Network IOCs feed (TCF-0301)

This service provides a continuous feed of malicious URLs and domains identified by Spectra Intelligence.

When a URL is observed delivering malicious files, the feed also includes relevant metadata for the downloaded file, such as the SHA1 hash, SHA256, and the ReversingLabs threat name.

To reduce feed volume, when a domain is classified as malicious, individual malicious URLs on that domain are no longer sent as separate feed records. Instead, the feed emits a single record for the domain itself, identified by uri_type: domain. Classification applies to the host, so a record for evil.example.com replaces the URL records on evil.example.com only. It doesn't extend to the parent domain example.com, or to any other host under it.

If the domain later stops being classified as malicious, any URL on that domain that is newly classified as malicious afterward reappears in the feed as a url record. URLs that remained malicious the whole time are not automatically resent.

To check the current classification of a specific URL or domain, use TCA-0403 URL Report or TCA-0405 Domain Report.

The feed stores records for the last 365 days.

The feed returns a maximum of 1000 records per request, and provides the latest queried timestamp up to which the records are included. In your next request, increment that timestamp by 1 to fetch the next batch of records.

If the requested timestamp is not within the last 365 days, the service will respond with the status code 400 Bad Request.

Continuous Feed Query

View OpenAPI Specification

Request

GET /api/feed/malware_uri/v1/query/{time_format}/{time_value}[?format=xml|json|tsv]

• time_format
  • Format in which the time value will be specified. Supported values are: timestamp (number of seconds since 1970-01-01 00:00:00), utc (UTC date in the YYYY-MM-DDThh:mm:ss format). This will also determine which format will be used in the last_timestamp field of the response.
  • Required
• time_value
  • Accepts values in the format set by time_format
  • Required
• format
  • Specifies the format in which the resulting data will be returned. Supported values are: xml (default), json, tsv (Tab Separated Values, delimiter character \t 0x09)
  • Optional

To retrieve at most 1000 latest records, use the following query:

GET /api/feed/malware_uri/v1/query/latest[?format=xml|json|tsv]

Response

When no malicious file was downloaded from the URL, the record omits the samples field and contains only record_on, uri, threat_level, and uri_type.

{
  "rl": {
    "malware_uri_feed": {
      "entries": [
        {
          "record_on": "string",
          "uri": "string",
          "samples": [
            {
              "threat_name": "string",
              "sha1": "string",
              "sha256": "string"
            }
          ],
          "uri_type": "string"
        }
      ],
      "last_timestamp": "string",
      "time_range": {
        "to": "string",
        "from": "string"
      }
    }
  }
}

---

rl.malware_uri_feed.entries[]

• record_on
  • When the record was included in the feed. Format: utc (YYYY-MM-DDThh:mm:ss).
• samples
  • List of samples with optional threat names. For a record with uri_type: domain, the list covers files downloaded from any URL on that domain. Contains the following:
    • sha1: SHA1 hash of the sample, for example 9300addf4d28d3c1419d0080fbb415bb50e8a19c
    • sha256: SHA256 hash of the sample, for example 9f0da89c727b6efb3d2a3f845254bf89c626980a70a5817918d9c43ae65858c0
    • threat_name: optional threat name, for example Win32.Trojan.Dinwod
• uri
  • A URL or domain classified as malicious by ReversingLabs network classification algorithms. When uri_type is domain, this field contains a host, which can include subdomains: for example, evil.example.com. Such a record doesn't imply that the parent domain example.com is malicious.
• threat_level
  • Malware severity indicator expressed as an integer between 0 and 5. Values from 1 to 5 indicate threats from lowest to highest severity, 0 is reserved for known URIs. For unknown URIs, this value is omitted.
  • In real-world situations, threat level values are typically interpreted in the following way:
    • Threat Level 4, 5 - immediate response required (e.g., different types of Trojans, URI found on a blocklist or matches a known malware regex)
    • Threat Level 2, 3 - should be examined within 24 hours (e.g., first stage exploits, URLs with homoglyph variations)
    • Threat Level 1 - not urgent, but should be periodically reviewed (e.g. Adware / PUA, misleading subdomains).
• uri_type
  • Indicates whether the record is a url or a domain. Records with uri_type: domain are emitted when a domain itself is classified as malicious, replacing the individual malicious URL records on that domain (see the note above about volume reduction).

Examples

Fetching all new URIs with changes from 2023-01-01 00:00:00

/api/feed/malware_uri/v1/query/utc/2023-01-01T00:00:00

Fetching all new URIs with changes from 2023-01-01 00:00:00 using the timestamp format; response in JSON, XML, and TSV format

/api/feed/malware_uri/v1/query/timestamp/1672527600?format=json
/api/feed/malware_uri/v1/query/timestamp/1672527600?format=xml
/api/feed/malware_uri/v1/query/timestamp/1672527600?format=tsv

Fetching the first available changes

/api/feed/malware_uri/v1/query/timestamp/0

Fetching all latest changes in XML, JSON, and TSV format

/api/feed/malware_uri/v1/query/latest?format=json
/api/feed/malware_uri/v1/query/latest?format=xml
/api/feed/malware_uri/v1/query/latest?format=tsv

PULL Query

For a given point in time, this query returns a list of malicious URLs and domains that have been detected in the Spectra Intelligence system.

The starting point for this query is defined using the start query. If the user has not previously requested this query, nor has the START query been called, it will return records starting with the current timestamp. Every subsequent call will continue from the timestamp where the previous call ended. In case that the timestamp of the previous call is older than 365 days, the subsequent call will autocorrect this timestamp to the oldest available (i.e. current - 365 days), and corresponding records will be returned.

When using the pull query continuously, it is useful to implement a pause between API requests to allow more results to accumulate in the feed, increasing the number of results returned per API request and reducing the total number of API calls needed. However, when catching up from a distant past timestamp, minimizing the pause between requests will speed up the process. Once the timestamp from the latest API response matches the current time, both methods (with and without pause) yield a similar number of records per unit of time.

If you want to keep fetching the latest available records as they appear in the feed, you can use a short pause (e.g., 1 second) between requests. And if you need to catch up on a large backlog of data from the past, it is better to make continuous API calls without pausing.

Unless the limit parameter is specified, the feed returns up to 1000 records and any surplus records sharing the same timestamp. That ensures all the records with the same timestamp will be included in the recordset. The limit parameter must not be greater than 1000.

This endpoint is built to be queried by a single thread (single instance). Any concurrent requests will be blocked until the previous request is fulfilled.

View OpenAPI Specification

Request

GET /api/feed/cf/malware_uri/v1/query/pull[?format=xml|json][&limit=N]

• format
  • Parameter defining the format in which the resulting data will be returned. Supported values are: xml (default), json.
  • Optional
• limit
  • The number of records to return in the response. The maximum and default value is 1000. Note that the response may include more records than requested to ensure that all records with the same timestamp are returned.
  • Optional

Response

The response format is the same as in the feed-query.

Examples

Fetching all new URI changes since last pull in XML and JSON format

/api/feed/cf/malware_uri/v1/query/pull?format=json
/api/feed/cf/malware_uri/v1/query/pull?format=xml

Fetching a smaller batch of results per request

/api/feed/cf/malware_uri/v1/query/pull?limit=100
/api/feed/cf/malware_uri/v1/query/pull?format=json&limit=100

START Query

This query sets the starting timestamp for the previously described pull-query. It is used to reprocess an already pulled time span, obtain data from an older period that was not previously retrieved, or skip ahead to a more recent timestamp when older data is no longer relevant. Resetting the start time in these cases helps manage the data retrieval process effectively.

The starting timestamp must be within the last 365 days, otherwise the service will respond with the status code 400 Bad Request.

View OpenAPI Specification

Request

PUT /api/feed/cf/malware_uri/v1/query/start/[time_format]/[time_value]

• time_format
  • Format in which the time value will be specified. Supported values are: timestamp - number of seconds since 1970-01-01 00:00:00; utc - UTC date in the YYYY-MM-DDThh:mm:ss format
  • Required
• time_value
  • Accepts values in the format set by time_format
  • Required

Examples

Set the start time using UTC format

/api/feed/cf/malware_uri/v1/query/start/utc/2023-01-01T00:00:00

Set the start time using the timestamp format

/api/feed/cf/malware_uri/v1/query/start/timestamp/1672527600

Response

A successful query returns an HTTP 200 OK message with an empty response body.