Threat Intelligence Report Summarization

Introduction

In an era where cyber threats are increasingly sophisticated and frequent, organizations must leverage advanced technologies to strengthen their defenses. A promising approach is the synthesis and summarization of threat intelligence using Large Language Model (LLM) agents. This article explores how these agents can analyze diverse data sources — such as static and dynamic analysis reports, antivirus data, MITRE techniques and general sample information — to create a comprehensive understanding of potential threats.

Background Information

Threat intelligence involves the collection and analysis of information regarding potential or current threats to an organization’s assets. It encompasses various data types, including indicators of compromise (IOCs), attacker behaviors, and capabilities. Traditionally, this process has relied on human analysts to synthesize information from multiple reports, which can be time-consuming and prone to errors. However, the integration of LLM agents offers a transformative solution.

Leveraging LLM Agents for Threat Intelligence

LLM agents can process vast amounts of data quickly and produce a fair synthesis and an accurate report summary. By employing specialized agents, tuned for different input data categories, we can achieve a more nuanced understanding of threats. For example, an LLM agent dedicated to static analysis can analyze key indicators from malware reports, while another focused on dynamic analysis can evaluate real-time behaviors of threats in a controlled environment.

LLM Model and Report Summarization

Input

The input data for the summarization is a complete Spectra Analyze report for the selected sample. The report includes comprehensive threat intelligence data for the sample, compiled from a multitude of antivirus scanners, dynamic analysis sandboxes, threat intelligence feeds, and Spectra Intelligence static analysis. JSON-formatted input easily exceeds several MB for even the simplest of malicious samples.

The goal is to quickly provide SOC Analysts with an accurate and useful summary of this data.

Model

Several base models available on AWS Bedrock were considered, namely:

• Amazon Titan text-lite-v1, and text-express-v1
• Meta llama3 70b-instruct-v1
• Anthropic Claude v2, v2.1, 3-haiku-v1, 3-sonnet-v1, instant-v1, and 3–5-sonnet-v1

After some testing we settled on using the 3-5-sonnet-v1. This model proved optimal for us in terms of context window size, response time, pricing, and most importantly — summarization quality.

LLM Agents

For the summary, we process several categories of input data. Each input category is processed using a dedicated LLM agent, with each agent’s prompt fine-tuned to processing just its data. This approach yields excellent results, even when using a stock LLM.

The input data, regardless of the source it was originally gathered from, is divided into several independent categories:

• Sample Indicators (IoC)
• Dynamic Analysis data
• MITRE Techniques
• Sample information
• Classification data

LLM Agents differ only by the system-prompt used, and the input data each selects from the Spectra Analyze input report. Analyses done by each of the agents are mostly independent of each other, due to the fact that each agent focuses on its dedicated aspect of the sample analysis. Thus, agents are run concurrently as soon as their input becomes available. The chart below visualizes the flow of the data and the summarization of the output at each stage of the processing.

Summarization data flow

LLM System-prompt Template

Each LLM Agent is given its own system-prompt, calibrated to the data it is processing. In essence, all of the prompts follow a similar template, such as this one:

Prompt:

As an expert cybersecurity threat analyst, create a clear and concise summary. Focus on the following points:

1. 1st Goal: definition
2. 2nd Goal: definition, etc.

Ensure that the summary is clear and easy to understand.

Sample Indicators Agent

The Sample Indicators Agent summarizes the capabilities of the sample based on the indicators of malicious behavior found in the ReversingLabs’ static binary analysis report, providing a clear picture of what to look for during threat detection.

system-prompt goals:

1. Common Capabilities: Identify and describe the capabilities that are typically found in malicious computer files.
2. Exploitation Methods: Explain how these capabilities can be exploited by attackers.

Dynamic Analysis Agent

Assessing and understanding the capabilities of a potential threat is crucial. The Dynamic Analysis Agent aggregates the data on actions and behaviors collected during dynamic analysis within sandbox environments such as Cape, Cuckoo, Fireeye, Joe, and ReversingLabs cloud.

system-prompt goals:

1. Dynamic Analysis Behavior: Describe the behaviors exhibited by a file during dynamic analysis.
2. Malware Actions: Highlight the specific actions that are commonly associated with malware.

MITRE Techniques Agent

The MITRE Agent summarizes the detected MITRE Techniques (the how), and groups them into appropriate MITRE Tactics (the why).

system-prompt goals:

1. MITRE ATT&CK Techniques: Identify and list the specific MITRE ATT&CK techniques observed in the given computer file.
2. MITRE ATT&CK Tactics: Group the identified Techniques into Tactics.
3. Technique Details: For each identified technique, provide a brief description.

Classification Components Agent

The Classification Components Agent summarizes the sample verdicts from multiple sources:

• Antivirus scanners
• Next generation antivirus software
• ReversingLabs’ static binary analysis
• Dynamic Analysis sandbox results

system-prompt goals:

1. File Classification Verdicts: Summarize the classification verdicts for a specific computer file as reported by various threat analysis tools.
2. Malicious vs. Non-Malicious: Clearly differentiate between the tools that classified the file as malicious and those that did not.

Sample Information Agent

• overall classification, i.e., ReversingLabs’ verdict
• classification reason
• classification result
• count of extracted files from the sample
• sample file type
• certificates validation results
• interesting network references, e.g., unique ipv4, ipv6, domain, url, and email address references

By synthesizing information from network traffic analysis, LLM agents can highlight unusual patterns that may indicate an ongoing attack or vulnerability.

system-prompt goals:

1. Classification: In one sentence, describe the file’s classification (e.g. malicious, benign).
2. File Details: In the next sentence, specify the file type and the number of extracted files (if applicable).
3. Certificate Validation: If provided, include the file’s certificate validation in the third sentence.
4. Network References: If available, list any interesting network references in the fourth sentence.

Behavior Summary Agent

The Behavior Summary agent describes the behavior of a sample, from the following inputs:

• sample indicators summary
• dynamic analysis summary
• MITRE summary

system-prompt goals:

1. Capabilities: Outline the capabilities of the file that indicate its potential threat level.
2. MITRE ATT&CK Techniques: Identify any relevant MITRE ATT&CK techniques associated with the file, referencing the MITRE ATT&CK matrix as needed.
3. Dynamic Analysis Signatures: Describe the dynamic analysis signatures that highlight the file’s operational impact.

Master Summary Agent

Ultimately, the Master Summary Agent assembles the report summary by aggregating inputs from:

• classification summary
• sample information summary
• behavior summary

By analyzing the classification and behavioral patterns, this LLM agent estimates the malicious potential of the analyzed sample, allowing organizations to proactively adjust their defenses. Below the LLM system-prompt is an example output from the Master Summary Agent, i.e., a report summary example for a complex threat sample that received mixed verdicts from different TI sources.

system-prompt goals:

1. Sample Behavior: Describe the behavior of the sample as observed in the analysis.
2. Threat Classification: Provide insights into the threat classification of the sample.
3. Sandbox Analysis: Summarize the results from the sandbox analysis, including any relevant findings.
4. Coherent narrative: Incorporate insights from static analysis, classification, informational content, and behavioral analysis to create a coherent narrative.
5. Target audience: target the summary towards cybersecurity professionals and executive decision-makers.

Example Report Summary

Example Report Summary

Executive Summary for sample 359dacbc078e8f62c62e94d02b60eabcedf8a33c

The analyzed sample, identified as AhnLab V3 Lite (Vdwrlyyzu.exe), presents a complex and contradictory threat profile that warrants further investigation. This 32-bit portable executable .NET application, exhibits characteristics that have triggered conflicting responses from various security analysis tools.

Key Findings:

1. Threat Classification: The sample received mixed verdicts across different analysis methods. While traditional antivirus scanners flagged it as malware (specifically Win32.Spyware.Negasteal), next-generation antivirus tools did not detect any threats. This discrepancy suggests the possibility of sophisticated evasion techniques or a potentially new, unrecognized threat.
2. Static Analysis: The file was classified as malicious during static analysis, corroborating the traditional antivirus findings. It possesses capabilities related to device configuration, monitoring, and cryptography.
3. Dynamic Analysis: Results from sandbox environments were inconsistent. Cuckoo Sandbox reported no malicious behavior, while FireEye Sandbox classified it as malicious. Joe Sandbox deemed it suspicious, indicating potential but not definitive malicious activity.
4. Behavioral Analysis: Surprisingly, no clear indicators of malicious behavior were observed across capabilities, MITRE ATT&CK techniques, or dynamic analysis signatures. This absence of suspicious attributes contradicts some of the earlier classifications and suggests a low threat level.
5. Additional Features: The file possesses a valid certificate and contains 27 extracted files, which may require further examination.

Conclusions and Recommendations:

The conflicting results across various analysis methods present a challenging scenario for threat assessment. While some indicators suggest malicious intent, others point towards benign behavior. This inconsistency could be due to sophisticated evasion techniques, a new and unrecognized threat, or potentially a false positive in some detection systems.

Given these contradictions, we recommend:

1. Conducting a thorough manual analysis to resolve the conflicting verdicts and determine the true nature of the file.
2. Performing additional dynamic analysis in a controlled environment to observe runtime behavior.
3. Investigating the context, origin, and intended purpose of the file within your environment.
4. Maintaining heightened monitoring of this file and any associated activities.
5. Updating threat intelligence and detection systems to account for this sample's unique characteristics.

This case underscores the importance of multi-layered analysis and the need for continuous refinement of threat detection methodologies in the face of evolving cyber threats.

Analysis

The use of LLM agents for threat intelligence synthesis presents several advantages. Firstly, it shows potential to enhance speed and accuracy of threat detection, allowing organizations to respond to incidents more rapidly. Secondly, it reduces the cognitive load on human analysts, enabling them to focus on strategic decision-making rather than data processing. However, organizations must also consider the challenges, such as being aware of and addressing potential biases and rare hallucinations in AI-generated insights.

LLM Hallucinations

Drama — Initially, the output was at times dramatic, e.g., reading a register became extensive file-system access and surveillance.

Bias — Another early challenge was the propensity to false-positive outputs. That is, often treating non-malicious inputs over-cautiously, e.g., there is no clear indication that the sample is malicious but caution is warranted as this may be a very sophisticated malware.

Fantasy — The few encountered hallucinations included non-existing indicators and capabilities, and an occasional digit appended or omitted from the long string of random digits comprising a cryptographic hash.

Enhancing Quality and Minimizing Hallucinations

Judicious selection of the pertinent input data, along with the careful system-prompt calibration, and the tuning of the temperature and top_k hyperparameters, have lowered both the drama and the bias substantially.

The proposed multi-agent architecture proved highly successful in avoiding fantasy. In a large body of manually reviewed reports, very few such errors have been observed. The key elements to that success were:

• having each agent highly focused on summarizing a specific aspect of the report,
• providing only the input relevant to that agent’s particular domain, and
• the strictly defined system-prompts.

Digit hallucinations were mitigated by instructing the agents not to print the hashes. Instead, they are added deterministically to the output.

Conclusion

The integration of LLM agents into the threat intelligence synthesis process represents a significant advancement in cybersecurity. By automating the analysis of diverse data sources, organizations can enhance their understanding of threats and improve their defensive strategies. As cyber threats continue to evolve, leveraging AI-driven solutions will be essential for maintaining robust security postures.

As large language models become more prevalent, we are moving away from the strictly deterministic, binary, black-and-white results. Occasional inaccuracies will become acceptable if the overall outcome is beneficial. Organizations should explore the implementation of LLM agents in their threat intelligence frameworks. By investing in these technologies, they can stay ahead of emerging threats and ensure a proactive approach to cybersecurity.

Thank you

Thank you for reading this article. I hope you found the information valuable. If you have any feedback or suggestions, please reach out.

Acknowledgments

A huge shout-out to the amazing engineering team working on the project, especially colleagues Mirano Tuk, Nikola Grubišić, and Tin Deranja. You guys have been the main driving force behind this project, from the initial concept to the final push to get it done.

Also, Igor Lasic — thank you for the vision, guidance, and trust in the team.

Working with all of you is a joy.