Analyze URL (TCA-0404)

This service allows users to submit a URL for analysis. Essentially, the analysis is a crawling process that will start looking for files to download from the submitted URL. When downloaded, the files are sent for analysis to the ReversingLabs file processing pipeline.

Key functionalities of the URL analysis service:

• Service can download a maximum of 50 samples per analysis. Each of those samples can be up to 100 MB in size.
• Service supports redirects i.e. analysis will be performed on the final URL
• Files are downloaded only from the submitted URL, no recursion (crawl depth = 1) e.g. if a user submitted the following URL: http://www.example.com/freshcontent, only that URL will be crawled and http://www.example.com/freshcontent/newest will not
• Automatic re-analyses will be performed in a regular cadence to retrieve new malware versions/mutations deployed on the same URL and new malware files deployed on website/opendirectory

The report about the performed analysis and reputation information for the downloaded samples can be retrieved via the TCA-0403 URL Threat Intelligence service, along with a list of samples that were downloaded from the submitted URL.

Processing of all downloaded samples might take around 30 minutes from submission. When the processing is completed, the TCA-0403 URL Analysis Notification service will publish an alert, the analysis report will contain the latest data, and the downloaded samples will have updated classification and metadata.

Querying the TCA-0403 /report endpoint before the analysis is complete may return mixed data: the last_analysis object will be populated with new information, while the statistics object may still show old classification data. Within 10 minutes from submission, the /downloaded_files endpoint returns a list of downloaded samples, but their metadata and classifications will be populated with information obtained during previous URL analyses.

Privacy

This service can only access and analyze publicly reachable online resources. Consequently, all submitted URLs and downloaded files will be treated as public, and will be visible and accessible to all Spectra Intelligence users.

This API is rate limited to 2 requests per minute.

Analyze URL Query

This query requests an analysis of the submitted URL. Only one URL can be submitted per request. Submitting the same URL consecutively does not initiate new analyses until the first analysis has completed.

Request

POST /api/networking/url/v1/analyze/query/{format}

Path parameters:

• format
  • Defines the format of the request body (XML or JSON)
  • Required

Request body:

{
  "rl": {
    "query": {
      "url": "string",
      "response_format": "string"
    }
  }
}

• url
  • Specifies the URL for which the analysis will be started. Provide the full URL of a website including the protocol (https://www.example.org). Only http and https protocols are supported. If the protocol is missing from the submitted URL, http will be automatically prepended to the URL. Note that URL normalization is performed during request submission, so duplicate and empty elements in the URL may be automatically removed or converted.
  • Required
• response_format
  • An optional parameter that allows choosing the response format. Supported values are xml and json; the default is xml
  • Optional

Response

The response includes the submitted URL in a normalized format, information about the status of the analysis, and the analysis_id that can be further used to get a list of samples downloaded during that specific analysis.

{
  "rl": {
    "requested_url": "string",
    "status": "string",
    "analysis_id": "string"
  }
}

• requested_url
  • The submitted URL in normalized format.
• status
  • Indicates the status of the analysis job for the requested URL. Returns started when the analysis job has been successfully initiated. No other status indicators are currently supported.
• analysis_id
  • A 16 character string identifying the current analysis.

Examples

Starting a analysis for the URL https://www.myabandonware.com using JSON as the POST payload format, and requesting that the response is in JSON as well.

Request:

/api/networking/url/v1/analyze/query/json

{
  "rl": {
    "query": {
      "url": "https://www.myabandonware.com",
      "response_format": "json"
    }
  }
}

Response:

{
  "rl": {
    "requested_url": "https://myabandonware.com/",
    "status": "started",
    "analysis_id": "159134594237612a"
  }
}

Custom Response Status Codes

This API implements custom response status codes:

CODE	DESCRIPTION
403	The server understood the request, but is refusing to fulfill it. A possible cause may be that the URL that was sent to analysis is already being analyzed. In this case, the API will also return an "Analysis in progress" message.