Reputation and metadata alerts (TCA-0206)

The Reputation and Metadata Changes service allows users to subscribe to up to one billion sample and URL hashes, tracking changes in their status over a period of time. These subscriptions, valid for a duration of 1 year, enable users to monitor any changes to the subscribed samples or URLs. When changes occur, they are delivered through the Data Change Feed API.

To subscribe to a list of samples, users can:

use the Spectra Intelligence Alert Subscription API
use the Spectra Intelligence File Upload API and set the query parameter ?subscribe=data_change in their metadata upload requests

To unsubscribe from the Data Change Feed for previously subscribed samples or URLs, users can use the Alert Subscription API - Unsubscribe Query.

General Info about Requests/Responses

All requests support the format parameter which allows changing the response format
Default response format is xml, except for bulk queries, where default format is the same as the POST format
The number of hashes in a bulk request must not be larger than one hundred (100)

How to use the Alert Subscription API and Data Change Feed

The subscription procedure consists of two general steps.

Users submit a list of sample or URL hashes they want to track in a bulk HTTP POST request to the Alert Subscription API. The request body can be XML or JSON.

Example: submitting a list of hashes as JSON

curl --location 'https://data.reversinglabs.com/api/subscription/data_change/v1/bulk_query/subscribe/json?format=json' \
--header 'Authorization: Basic bXNjb3R0OnRoYXQncyB3aGF0IHNoZSBzYWlkCg==' \
--header 'Content-Type: application/json' \
--data '{
  "rl": {
    "query": {
      "hash_type": "sha1",
      "hashes": [
        "a7afddb68260a60f86c02a021efba7f216c2e7cf",
        "ca03064987d3c4465f91552ba8b6a883eecfd3e5",
        "b363713a938afcd3c74603827fab79e935b2b09b"
      ]
    }
  }
}'

Step 2 - Monitor changes to subscribed samples

Users can poll the Data Change Feed using HTTP GET requests to check whether there have been any changes to the samples or URLs they have subscribed to.

The Data Change Feed returns the last_timestamp value that can be used in the next request up until 60 seconds from the current time. The value determines the time period for which the user wants to retrieve information about changes to subscribed samples.

For example:

User sends a subscription request at timestamp T0.
The user then issues a GET request at timestamp T1.
The request contains the T0 timestamp as the last_timestamp value, and the API returns the changes that happened between T0 and T1 (if there have been any changes).

The response returned by the API contains SHA1 hashes of subscribed samples or URLs and sections that have changed in the requested period of time.

For every sample or URL with detected changes, the following sections are included:

xref: AV scanning results have been updated.
sample_available: The sample has become available for processing.
malware_presence: The sample's Malware Presence classification, or any part of its metadata, has changed (including changes to the sample's Malicious/Unknown/Suspicious/Known status).
sample_became_shareable: The sample became shareable and is available for download.
dynamic_analysis: Dynamic analysis report has been completed and is available for querying.

This query is used for subscribing to a list of samples or URLs for which the changed sections (if there are any) will be delivered in the Data Change Feed. To subscribe to a list of samples or URLs, the user should submit the sample or URL hashes in a POST request. All hashes in a request should be of the same type. The maximum amount of hashes that can be submitted in one request is 100.

Request

POST /api/subscription/data_change/v1/bulk_query/subscribe/{post_format}

Path parameters:

post_format
- Accepts the following values: xml or json. It defines the POST body format and the response format.

Query parameters:

format
- Accepts the following values: xml or json. Can be used to override the default response format.

Request body:

{
  "rl": {
    "query": {
      "hash_type": "string", // sha1, sha256, or md5
      "hashes": [
        "string"
      ]
    }
  }
}

hash_type can be sha1, md5 or sha256 for samples, and sha1 for URLs that were seen by the Spectra Intelligence system. Note: Samples or URLs that have not yet been seen can be subscribed only with SHA1 hash values.
hash values must be valid hashes defined by the hash_type parameter

Response

{
  "rl": {
    "subscription_data_change": {
      "hash_type": "string", // requested hash type
      "subscribed": [ 
        "string" // list of successful subscriptions
      ],
      "invalid_hashes": [
        "string"
      ],
      "unknown_hashes": [ 
        "string" // only if MD5 or SHA256 were used
      ]
    }
  }
}

Unsubscribe query

This query is used for unsubscribing from a list of samples or URLs that the user was previously subscribed to. Submitting a sample or URL hash in a POST request to this endpoint removes the associated sample or URL from the list of user's subscriptions.

The maximum amount of hashes that can be submitted in one request is 100. Changes for unsubscribed samples or URLs will no longer be delivered in the Data Change Feed.

Request

POST /api/subscription/data_change/v1/bulk_query/unsubscribe/{post_format}

The request format (body, parameters) is identical to the format in the subscribe query.

Response

{
  "rl": {
    "subscription_data_change": {
      "hash_type": "string", // requested hash type
      "unsubscribed": [ 
        "string" // list of successful unsubscriptions
      ],
      "invalid_hashes": [
        "string"
      ],
      "unknown_hashes": [ 
        "string" // only if MD5 or SHA256 were used
      ]
    }
  }
}

Examples

/api/subscription/data_change/v1/bulk_query/subscribe/json

{
  "rl": {
    "query": {
      "hash_type": "sha1",
      "hashes": [
        "84dbb381485e6ee9d69fbad3c6f020be88230e9b",
        "81648fd7871017eef87e0efa166e00e891a6ea9d",
        "0761c95bd639466aff2d65c1c092e2cae40f900c",
        "552ec95c8ffe3a0cb014eee508b87730461c8f26",
        "d234189a25892325746699fe2ab39a40b2d14d6e"
      ]
    }
  }
}

Unsubscribe from a list of SHA1 hashes

/api/subscription/data_change/v1/bulk_query/unsubscribe/json

{
  "rl": {
    "query": {
      "hash_type": "sha1",
      "hashes": [
        "84dbb381485e6ee9d69fbad3c6f020be88230e9b",
        "81648fd7871017eef87e0efa166e00e891a6ea9d",
        "0761c95bd639466aff2d65c1c092e2cae40f900c",
        "552ec95c8ffe3a0cb014eee508b87730461c8f26",
        "d234189a25892325746699fe2ab39a40b2d14d6e"
      ]
    }
  }
}

Data change feed: PULL

This query returns the next set of samples or URLs to which the user is subscribed. The starting point for this query is defined using the START query.

If the user has not previously requested this query or called the START query, it will return records starting with the current timestamp. Every subsequent call will continue from the timestamp where the previous call ended.

Unless the limit parameter is specified, the query returns a maximum of 1000 records and any surplus records sharing the same timestamp. This ensures that all the records with the same timestamp will be included in the set of records. The limit parameter must not be greater than 1000.

This endpoint supports single threaded requests. Any concurrent request will be blocked until the previous one is fulfilled.

Request

GET /api/feed/data_change/v3/pull

Query parameters:

events
- Accepts a list of sections that should be included in the response, delimited by commas. If omitted, includes all values. Otherwise, list one or more of the sections:
  - xref
  - sample_available
  - malware_presence
  - sample_became_shareable
  - dynamic_analysis
- Optional
format
- Specifies the response format. Supported values:
  - xml (default)
  - json
  - tsv(tab-separated values, delimiter character \t (0x09))
- Optional
limit
- Number of records to return in the response. The maximum value is 1000. Note that the response may include a little more than the requested number of records to ensure that all the records with the same timestamp are returned.
- Optional

Response

{
  "rl": {
    "data_change_feed": {
      "entries": [],
      "last_timestamp": 0,
      "time_range": {}
    }
  }
}

rl.data_change_feed.entries[]

{
  "record_on": "string",
  "sha1": "string",
  "updated_sections": [
    "string"
  ],
  "sha256": "string",
  "md5": "string"
}

sha1
- SHA1 hash value of the sample or URL
md5
- MD5 hash value of the sample, if available
sha256
- SHA256 hash value of the sample, if available
updated_sections
- See rl.data_change_feed.entries[].updated_sections

rl.data_change_feed.entries[].updated_sections

xref
- Changed AV scanning results
malware_presence
- Changed Malware Presence result
sample_became_shareable
- Sample become shareable and is available for download
dynamic_analysis
- Dynamic analysis report completed and available for querying

Data change feed: START

This query sets the starting timestamp for the previously described PULL query.

Request

PUT /api/feed/data_change/v3/start/{time_format}/{time_value}

Path parameters:

time_format accepts these options:
- timestamp: number of seconds since 1970-01-01 00:00:00
- utc: date and time in the format YYYY-MM-DDThh:mm:ss
time_value
- Accepts values defined by individual option set by time_format

Continuous data change feed query

This query returns a set of records (samples or URLs) that the user is subscribed to from the requested timestamp onwards. Essentially it combines the START and PULL queries into one.

The feed returns up to 1000 records and any surplus records sharing the same timestamp. The response also contains the latest timestamp up to which the events are included in the response.

To fetch the next set, use the last_timestamp value from the response, increase it by 1 and submit it in a new request.

The response format is the same as with the PULL query.

Request

GET /api/feed/data_change/v3/query/{time_format}/{time_value}

Path parameters:

time_format accepts these options:
- timestamp: number of seconds since 1970-01-01 00:00:00
- utc: date and time in the format YYYY-MM-DDThh:mm:ss
time_value
- Accepts values defined by individual option set by time_format

Query parameters:

events
- Accepts a list of sections that should be included in the response, delimited by commas. If omitted, includes all values. Otherwise, list one or more of the sections:
  - xref
  - sample_available
  - malware_presence
  - sample_became_shareable
  - dynamic_analysis
- Optional
format
- Specifies the response format. Supported values:
  - xml (default)
  - json
  - tsv(tab-separated values, delimiter character \t (0x09))
- Optional
limit
- Number of records to return in the response. The maximum value is 1000. Note that the response may include a little more than the requested number of records to ensure that all the records with the same timestamp are returned.
- Optional

Examples

Get samples you subscribed to, set response format to JSON: /api/feed/data_change/v3/pull?format=json
Get samples that changed their classification: /api/feed/data_change/v3/pull?events=malware_presence
Get dynamic analysis changes and set the response format: /api/feed/data_change/v3/pull?events=dynamic_analysis&format=json
Set the start time for the pull query, use UTC: /api/feed/data_change/v3/start/utc/2015-05-01T13:00:00

General Info about Requests/Responses​

How to use the Alert Subscription API and Data Change Feed​

Step 1 - Subscribe to samples​

Step 2 - Monitor changes to subscribed samples​

Subscribe query​

Request​

Response​

Unsubscribe query​

Request​

Response​

Examples​

Subscribe to a list of SHA1 hashes​

Unsubscribe from a list of SHA1 hashes​

Data change feed: PULL​

Request​

Response​

Data change feed: START​

Request​

Continuous data change feed query​

Request​

Examples​

General Info about Requests/Responses

How to use the Alert Subscription API and Data Change Feed

Step 1 - Subscribe to samples

Step 2 - Monitor changes to subscribed samples

Subscribe query

Request

Response

Unsubscribe query

Request

Response

Examples

Subscribe to a list of SHA1 hashes

Unsubscribe from a list of SHA1 hashes

Data change feed: PULL

Request

Response

Data change feed: START

Request

Continuous data change feed query

Request

Examples