Large Language Models are Unreliable for Cyber Threat Intelligence

A recent study titled "Large Language Models are Unreliable for Cyber Threat Intelligence" critically examines the application of Large Language Models (LLMs) in automating Cyber Threat Intelligence (CTI) tasks. The research presents an evaluation methodology that assesses LLMs using zero-shot learning, few-shot learning, and fine-tuning approaches, focusing on their consistency and confidence levels. Experiments conducted with three state-of-the-art LLMs on a dataset of 350 CTI reports reveal that LLMs struggle with real-sized reports, exhibiting inconsistent performance and overconfidence. While few-shot learning and fine-tuning offer limited improvements, the findings raise concerns about relying on LLMs for CTI, especially in scenarios lacking labeled datasets where confidence is crucial.  

https://arxiv.org/abs/2503.23175

Comments

Post a Comment

Popular posts from this blog

Secure Vibe Coding Guide: Best Practices for Writing Secure Code

Ken Huang's "Secure Vibe Coding Guide" emphasizes the importance of integrating security into the software development lifecycle. The guide provides best practices for writing secure code, including input validation, proper authentication mechanisms, and secure data storage techniques. It also highlights the necessity of regular code reviews and staying updated with the latest security vulnerabilities and patches. By following these guidelines, developers can create applications that are resilient against common security threats and contribute to a safer digital environment.  https://kenhuangus.substack.com/p/secure-vibe-coding-guide
Read more

KEVIntel: Real-Time Intelligence on Exploited Vulnerabilities

KEVIntel is a dynamic platform providing up-to-date information on known exploited vulnerabilities (KEVs). It aggregates data from over 50 public sources, including CISA, and enriches each entry with metadata such as EPSS scores, online mentions, scanner inclusion, and exploitation status. The platform aims to serve as an early warning system, offering insights even before official publications. KEVIntel supports various formats like JSON, CSV, and RSS, facilitating integration into security operations. These entries include CVSS scores, exploit status, and links to proof-of-concept code, aiding organizations in prioritizing remediation efforts.   https://kevintel.com/
Read more

OWASP SAMM Skills Framework Enhances Software Security Roles

The OWASP SAMM Skills Framework, introduced on February 9, 2025, is a new initiative donated by Siemens to enhance software security practices within organizations. This framework assigns specific responsibilities to SAMM (Software Assurance Maturity Model) streams, clarifying which roles are involved in advancing each stream. It provides guidance on the necessary skills and training for each role, aligning SAMM-related activities with appropriate stakeholders and their required competencies. This alignment helps organizations identify the right personnel and visualize shared responsibilities, ensuring a structured approach to secure product development. Implementing the OWASP SAMM Skills Framework involves several key steps. Organizations must first map responsibilities to roles, ensuring each task is assigned to the right individual. Next, they must evaluate and align stakeholders with specific SAMM streams, validating that those assigned understand and accept their roles in advanci...
Read more