Using Large Language Models for Cyber Threat News Prioritization

Abstract

Organizations adopting a 'collect-all' strategy for cyber threat intelligence (CTI) often face overwhelming volumes of cybersecurity news, leading to information overload and reduced operational efficiency. This paper investigates the use of Large Language Models (LLMs) to support cyber threat news prioritization in the context of the Thai banking sector. A dataset of 375 cybersecurity news articles was collected and labeled using LLM-based prompting at both coarse-grained (0-2) and fine-grained (1-10) relevance levels. Results show that LLMs can provide consistent relevance judgments when carefully prompted, but inconsistencies remain in borderline cases. To explain and validate LLM decisions, we conducted experiments with text-based classification using TF-IDF and Random Forest, achieving 72% accuracy, and keyword-based classification with logistic regression, which yielded lower accuracy but offered interpretability through risk-associated keywords. The findings suggest that while LLMs are useful for augmenting CTI workflows, they should be combined with traditional machine learning and human oversight to ensure reliability.