Network IOCs feed (TCF-0301)

This service provides a continuous feed of malicious URLs identified by Spectra Intelligence.

When a URL is observed delivering malicious files, the feed also includes relevant metadata for the downloaded file, such as the SHA1 hash, SHA256, and the ReversingLabs threat name.

The feed stores records for the last 365 days.

The feed returns a maximum of 1000 records per request, and provides the latest queried timestamp up to which the records are included. In your next request, increment that timestamp by 1 to fetch the next batch of records.

If the requested timestamp is not within the last 365 days, the service will respond with the status code 400 Bad Request.

Continuous Feed Query

View OpenAPI Specification

Request

GET /api/feed/malware_uri/v1/query/{time_format}/{time_value}[?format=xml|json|tsv]

• time_format
  • Format in which the time value will be specified. Supported values are: timestamp (number of seconds since 1970-01-01 00:00:00), utc (UTC date in the YYYY-MM-DDThh:mm:ss format). This will also determine which format will be used in the last_timestamp field of the response.
  • Required
• time_value
  • Accepts values in the format set by time_format
  • Required
• format
  • Specifies the format in which the resulting data will be returned. Supported values are: xml (default), json, tsv (Tab Separated Values, delimiter character \t 0x09)
  • Optional

To retrieve at most 1000 latest records, use the following query:

GET /api/feed/malware_uri/v1/query/latest[?format=xml|json|tsv]

Response

Records returned from third party sources contain only the uri, threat_level and uri_type fields.

{
  "rl": {
    "malware_uri_feed": {
      "entries": [
        {
          "record_on": "string",
          "uri": "string",
          "samples": [
            {
              "threat_name": "string",
              "sha1": "string",
              "sha256": "string"
            }
          ],
          "uri_type": "string"
        }
      ],
      "last_timestamp": "string",
      "time_range": {
        "to": "string",
        "from": "string"
      }
    }
  }
}

---

rl.malware_uri_feed.entries[]

• record_on
  • When the record was included in the feed. Format: utc (YYYY-MM-DDThh:mm:ss).
• samples
  • List of samples with optional threat names. Contains the following:
    • sha1: SHA1 hash of the sample, for example 9300addf4d28d3c1419d0080fbb415bb50e8a19c
    • sha256: SHA256 hash of the sample, for example 9f0da89c727b6efb3d2a3f845254bf89c626980a70a5817918d9c43ae65858c0
    • threat_name: optional threat name, for example Win32.Trojan.Dinwod
• uri A URL that we downloaded a malicious file from, or a third party reputation provider found it malicious.
• threat_level
  • Malware severity indicator expressed as an integer between 0 and 5. Values from 1 to 5 indicate threats from lowest to highest severity, 0 is reserved for known URIs. For unknown URIs, this value is omitted.
  • In real-world situations, threat level values are typically interpreted in the following way:
    • Threat Level 4, 5 - immediate response required (e.g., different types of Trojans, URI found on a blocklist or matches a known malware regex)
    • Threat Level 2, 3 - should be examined within 24 hours (e.g., first stage exploits, URLs with homoglyph variations)
    • Threat Level 1 - not urgent, but should be periodically reviewed (e.g. Adware / PUA, misleading subdomains).
• uri_type
  • This field is included for the purpose of backward compatibility. Always equal to url.

Examples

Fetching all new URIs with changes from 2023-01-01 00:00:00

/api/feed/malware_uri/v1/query/utc/2023-01-01T00:00:00

Fetching all new URIs with changes from 2023-01-01 00:00:00 using the timestamp format; response in JSON, XML, and TSV format

/api/feed/malware_uri/v1/query/timestamp/1672527600?format=json
/api/feed/malware_uri/v1/query/timestamp/1672527600?format=xml
/api/feed/malware_uri/v1/query/timestamp/1672527600?format=tsv

Fetching the first available changes

/api/feed/malware_uri/v1/query/timestamp/0

Fetching all latest changes in XML, JSON, and TSV format

/api/feed/malware_uri/v1/query/latest?format=json
/api/feed/malware_uri/v1/query/latest?format=xml
/api/feed/malware_uri/v1/query/latest?format=tsv

PULL Query

For a given point in time, this query returns a list of malicious URIs that have been detected in the Spectra Intelligence system.

The starting point for this query is defined using the start query. If the user has not previously requested this query, nor has the START query been called, it will return records starting with the current timestamp. Every subsequent call will continue from the timestamp where the previous call ended. In case that the timestamp of the previous call is older than 365 days, the subsequent call will autocorrect this timestamp to the oldest available (i.e. current - 365 days), and corresponding records will be returned.

When using the pull query continuously, it is useful to implement a pause between API requests to allow more results to accumulate in the feed, increasing the number of results returned per API request and reducing the total number of API calls needed. However, when catching up from a distant past timestamp, minimizing the pause between requests will speed up the process. Once the timestamp from the latest API response matches the current time, both methods (with and without pause) yield a similar number of records per unit of time.

If you want to keep fetching the latest available records as they appear in the feed, you can use a short pause (e.g., 1 second) between requests. And if you need to catch up on a large backlog of data from the past, it is better to make continuous API calls without pausing.

Unless the limit parameter is specified, the feed returns up to 1000 records and any surplus records sharing the same timestamp. That ensures all the records with the same timestamp will be included in the recordset. The limit parameter must not be greater than 1000.

This endpoint is built to be queried by a single thread (single instance). Any concurrent requests will be blocked until the previous request is fulfilled.

View OpenAPI Specification

Request

GET /api/feed/cf/malware_uri/v1/query/pull[?format=xml|json][&limit=N]

• format
  • Parameter defining the format in which the resulting data will be returned. Supported values are: xml (default), json.
  • Optional
• limit
  • The number of records to return in the response. The maximum and default value is 1000. Note that the response may include more records than requested to ensure that all records with the same timestamp are returned.
  • Optional

Response

The response format is the same as in the feed-query.

Examples

Fetching all new URI changes since last pull in XML and JSON format

/api/feed/cf/malware_uri/v1/query/pull?format=json
/api/feed/cf/malware_uri/v1/query/pull?format=xml

Fetching a smaller batch of results per request

/api/feed/cf/malware_uri/v1/query/pull?limit=100
/api/feed/cf/malware_uri/v1/query/pull?format=json&limit=100

START Query

This query sets the starting timestamp for the previously described pull-query. It is used to reprocess an already pulled time span, obtain data from an older period that was not previously retrieved, or skip ahead to a more recent timestamp when older data is no longer relevant. Resetting the start time in these cases helps manage the data retrieval process effectively.

The starting timestamp must be within the last 365 days, otherwise the service will respond with the status code 400 Bad Request.

View OpenAPI Specification

Request

PUT /api/feed/cf/malware_uri/v1/query/start/[time_format]/[time_value]

• time_format
  • Format in which the time value will be specified. Supported values are: timestamp - number of seconds since 1970-01-01 00:00:00; utc - UTC date in the YYYY-MM-DDThh:mm:ss format
  • Required
• time_value
  • Accepts values in the format set by time_format
  • Required

Examples

Set the start time using UTC format

/api/feed/cf/malware_uri/v1/query/start/utc/2023-01-01T00:00:00

Set the start time using the timestamp format

/api/feed/cf/malware_uri/v1/query/start/timestamp/1672527600

Response

A successful query returns an HTTP 200 OK message with an empty response body.