Skip to main content

Reputation and metadata alerts (TCA-0206)

The Reputation and Metadata Changes service allows users to subscribe to up to one billion sample and URL hashes, tracking changes in their status over a period of time. These subscriptions, valid for a duration of 1 year, enable users to monitor any changes to the subscribed samples or URLs. When changes occur, they are delivered through the Data Change Feed API.

To subscribe to a list of samples, users can:

  • use the Spectra Intelligence Alert Subscription API
  • use the Spectra Intelligence file upload API to set the parameter <?subscribe=data_change> in their metadata upload requests

To unsubscribe from the Data Change Feed for previously subscribed samples or URLs, users can use the Alert Subscription API - Unsubscribe Query.

General Info about Requests/Responses​

  • All requests support the format parameter which allows changing the response format
  • Default response format is xml, except for bulk queries, where default format is the same as the POST format
  • The number of hashes in a bulk request must not be larger than one hundred (100)
  • POST requests must set in HTTP header field Content-Type: application/octet-stream

How to Use Alert Subscription API and Data Change Feed​

The subscription procedure consists of two general steps.

Step 1 - Subscribe to samples​

Users submit a list of sample or URL hashes they want to track in a bulk HTTP POST request to the Alert Subscription API. The POST request can be sent as XML or JSON.

Example: submitting a list of hashes as JSON

curl –X POST https://<server>/api/subscription/data_change/v1/bulk_query/subscribe/json -d '
{ "rl": {
"query": {
"hash_type":"sha1",
"hashes": [
"84dbb381485e6ee9d69fbad3c6f020be88230e9b",
"81648fd7871017eef87e0efa166e00e891a6ea9d",
"0761c95bd639466aff2d65c1c092e2cae40f900c",
"552ec95c8ffe3a0cb014eee508b87730461c8f26",
"d234189a25892325746699fe2ab39a40b2d14d6e"
]
}}}

'

Step 2 - Monitor changes to subscribed samples​

Users can poll the Data Change Feed using HTTP GET requests to check whether there have been any changes to the samples or URLs they have subscribed to.

The Data Change Feed returns the last_timestamp value that can be used in the next request up until 60 seconds from the current time. The value determines the time period for which the user wants to retrieve information about changes to subscribed samples.

For example:

  1. User sends a subscription request at timestamp T0.
  2. The user then issues a GET request at timestamp T1.
  3. The request contains the T0 timestamp as the last_timestamp value, and the API returns the changes that happened between T0 and T1 (if there have been any changes).

The response returned by the API contains SHA1 hashes of subscribed samples or URLs and sections that have changed in the requested period of time.

For every sample or URL with detected changes, the following sections are included:

SectionDescription
xrefAV scanning results have been updated.
sample_availableThe sample has become available for processing.
malware_presenceSample's Malware Presence classification, or any part of its metadata, has changed (including changes to the sample's Malicious/Unknown/Suspicious/Known status).
sample_became_shareableThe sample became shareable and is available for download.
dynamic_analysisDynamic analysis report has been completed and is available for querying.

Subscribe query​

This query is used for subscribing to a list of samples or URLs for which the changed sections (if there are any) will be delivered in the Data Change Feed. To subscribe to a list of samples or URLs, the user should submit the sample or URL hashes in a POST request. All hashes in a request should be of the same type. The maximum amount of hashes that can be submitted in one request is 100.

Request​

POST /api/subscription/data_change/v1/bulk_query/subscribe/{post_format}

Path parameters:

  • post_format
    • Accepts the following values: xml or json. It defines the POST body format and the response format.

Request body:

{
  "rl": {
    "query": {
      "hash_type": "string",
      "hashes": [
        "string"
      ]
    }
  }
}
  • hash_type can be sha1, md5 or sha256 for samples, and sha1 for URLs that were seen by the Spectra Intelligence system. Note: Samples or URLs that have not yet been seen can be subscribed only with SHA1 hash values.
  • hash values must be valid hashes defined by the hash_type parameter

Response​

{
  "rl": {
    "subscription_data_change": {
      "hash_type": "string",
      "subscribed": [
        {}
      ],
      "invalid_hashes": [
        "string"
      ],
      "unknown_hashes": [
        "string"
      ]
    }
  }
}
  • hash_type
    • The hash_type used in the request
  • subscribed
    • A list of hashes that were successfully subscribed
  • invalid_hashes
    • A list of invalid hashes
  • unknown_hashes
    • A list of unknown hashes (only if MD5 and SHA256 hash_type is used)

Unsubscribe query​

This query is used for unsubscribing from a list of samples or URLs that the user was previously subscribed to. Submitting a sample or URL hash in a POST request to this endpoint removes the associated sample or URL from the list of user's subscriptions.

The maximum amount of hashes that can be submitted in one request is 100. Changes for unsubscribed samples or URLs will no longer be delivered in the Data Change Feed.

Request​

POST /api/subscription/data_change/v1/bulk_query/unsubscribe/{post_format}

Path parameters:

  • post_format
    • Accepts the following values: xml or json. It defines the POST body format and the response format.

Request body:

{
  "rl": {
    "query": {
      "hash_type": "string",
      "hashes": [
        "string"
      ]
    }
  }
}

  • hash_type

    • The hash_type used in the request

  • unsubscribed

    • A list of hashes that were successfully unsubscribed

  • invalid_hashes

    • A list of invalid hashes

  • unknown_hashes

    • A list of unknown hashes (only if MD5 and SHA256 hash_type is used)

  • hashes must be valid hashes defined by hash_type parameter

  • hash_type can be sha1, md5 or sha256 for samples, and sha1 for URLs that were seen by the Spectra Intelligence system. Samples or URLs that have not yet been seen can be unsubscribed only with SHA1 hash values.

Examples​

Subscribe to a list of SHA1 hashes​

/api/subscription/data_change/v1/bulk_query/subscribe/json
{
  "rl": {
    "query": {
      "hash_type": "sha1",
      "hashes": [
        "84dbb381485e6ee9d69fbad3c6f020be88230e9b",
        "81648fd7871017eef87e0efa166e00e891a6ea9d",
        "0761c95bd639466aff2d65c1c092e2cae40f900c",
        "552ec95c8ffe3a0cb014eee508b87730461c8f26",
        "d234189a25892325746699fe2ab39a40b2d14d6e"
      ]
    }
  }
}

Unsubscribe from a list of SHA1 hashes​

/api/subscription/data_change/v1/bulk_query/unsubscribe/json
{
  "rl": {
    "query": {
      "hash_type": "sha1",
      "hashes": [
        "84dbb381485e6ee9d69fbad3c6f020be88230e9b",
        "81648fd7871017eef87e0efa166e00e891a6ea9d",
        "0761c95bd639466aff2d65c1c092e2cae40f900c",
        "552ec95c8ffe3a0cb014eee508b87730461c8f26",
        "d234189a25892325746699fe2ab39a40b2d14d6e"
      ]
    }
  }
}

Data change feed: PULL​

This query returns the next set of samples or URLs to which the user is subscribed. The starting point for this query is defined using the START query.

If the user has not previously requested this query or called the START query, it will return records starting with the current timestamp. Every subsequent call will continue from the timestamp where the previous call ended.

Unless the limit parameter is specified, the query returns a maximum of 1000 records and any surplus records sharing the same timestamp. This ensures that all the records with the same timestamp will be included in the recordset. The limit parameter must not be greater than 1000.

This endpoint is built to be queried by a single thread (single instance). Any concurrent requests will be blocked until the previous one is fulfilled.

Request​

GET /api/feed/data_change/v3/pull[?format=xml|json|tsv][&events=xref,sample_available,malware_presence,sample_became_shareable,dynamic_analysis][&limit=N]

Query parameters:

  • events
    • Accepts a list of sections that should be included in the response, delimited by commas. If omitted, includes all values. Otherwise, list one or more of the sections:
      • xref
      • sample_available
      • malware_presence
      • sample_became_shareable
      • dynamic_analysis
    • Optional
  • format
    • Specifies the response format. Supported values: xml (default), json, tsv (Tab Separated Values, delimiter character \t 0x09)
    • Optional
  • limit
    • Number of records to return in the response. The maximum value is 1000. Note that the response may include a little more than the requested number of records to ensure that all the records with the same timestamp are returned.
    • Optional

Response​

{
  "rl": {
    "data_change_feed": {
      "entries": [],
      "last_timestamp": 0,
      "time_range": {}
    }
  }
}

rl.data_change_feed.entries[]

{
  "record_on": "string",
  "sha1": "string",
  "updated_sections": [
    "string"
  ],
  "sha256": "string",
  "md5": "string"
}
  • sha1
    • SHA1 hash value of the sample or URL
  • md5
    • MD5 hash value of the sample, if available
  • sha256
    • SHA256 hash value of the sample, if available
  • updated_sections
    • See rl.data_change_feed.entries[].updated_sections

rl.data_change_feed.entries[].updated_sections

  • xref
    • Changed AV scanning results
  • malware_presence
    • Changed Malware Presence result
  • sample_became_shareable
    • Sample become shareable and is available for download
  • dynamic_analysis
    • Dynamic analysis report completed and available for querying

Data change feed: START​

This query sets the starting timestamp for the previously described PULL query.

Request​

PUT /api/feed/data_change/v3/start/{time_format}/{time_value}

Path parameters:

  • time_format
    • parameter accepts these options: timestamp, utc. timestamp is the number of seconds since 1970-01-01 00:00:00. utc is the date and time in format YYYY-MM-DDThh:mm:ss
    • Required
  • time_value
    • Accepts values defined by individual option set by time_format
    • Required

Continuous data change feed query​

This query returns a recordset with samples or URLs that the user is subscribed to from the requested timestamp onwards. The feed returns up to 1000 records and any surplus records sharing the same timestamp. The response also contains the latest timestamp up to which the events are included in the response.

To fetch the next recordset, use the the last_timestamp value from the response, increase it by 1 and submit it in a new request.

The response format is the same as with the PULL query.

Request​

GET /api/feed/data_change/v3/query/{time_format}/{time_value}[?format=xml|json|tsv][&events=xref,sample_available,malware_presence,sample_became_shareable,dynamic_analysis]
  • time_format
    • Accepts these options: timestamp, utc. timestamp is the number of seconds since 1970-01-01 00:00:00. utc is the date and time in format YYYY-MM-DDThh:mm:ss
    • Required
  • time_value
    • Accepts values defined by individual option set by time_format.
    • Required
  • events
    • Accepts a list of sections that should be included in the response, delimited by commas. If omitted, includes all values. Otherwise, list one or more of the sections: xref, sample_available, malware_presence, sample_became_shareable, dynamic_analysis
    • Optional
  • format
    • Specifies the response format. Supported values: xml (default), json, tsv (Tab Separated Values, delimiter character \t 0x09)
    • Optional

Examples​

Fetching records with the PULL query​

/api/feed/data_change/v3/pull?format=json

Fetching samples that changed their classification

/api/feed/data_change/v3/pull?events=malware_presence

Fetching sample_available and xref changes

/api/feed/data_change/v3/pull?events=sample_available,xref

Fetching sample_available and xref changes and setting the response format

/api/feed/data_change/v3/pull?events=sample_available,xref&format=json

Fetching dynamic_analysis changes

/api/feed/data_change/v3/pull?events=dynamic_analysis

Fetching dynamic_analysis changes and setting the response format

/api/feed/data_change/v3/pull?events=dynamic_analysis&format=json

Setting the starting timestamp for the PULL query​

Using the timestamp format

/api/feed/data_change/v3/start/timestamp/1430485200

Using the utc format

/api/feed/data_change/v3/start/utc/2015-05-01T13:00:00