YARA hunting (TCA-0303)

The ReversingLabs YARA Hunting service enables users to create custom YARA rules containing textual or binary patterns, and upload them to the service to obtain matches using the APIs described in this document. When a sample matches the pattern found in a YARA rule, it receives the classification defined by that rule.

YARA rules can be grouped into rulesets - collections of one or more rules that are defined inside one YARA code text. Rules inside a single ruleset are allowed to be interdependent; i.e., one rule can reference another (see the "Referencing other rules" section in the YARA documentation).

For users who need to create fine-grained, complex rulesets, YARA provides modules as a way of extending the default set of features. The ReversingLabs YARA Hunting Service supports the following YARA modules: PE, ELF, Dex, Macho, String, Math, Hash, Time, and Dotnet.

The Ruleset Administration API is used to administer one's collection of YARA rulesets. A RESTful interface provides operations to add, delete, and query individual rulesets.

The Matches Feed API is used to obtain generated matches.

YARA rulesets do not become active in the system immediately upon upload/creation. All user-submitted rulesets go through a validation phase where they are tested for correctness and efficacy.

Note that a ruleset can be declared invalid even if it is syntactically correct. One reason for validation failure could be that a ruleset is too broad - it matches a disproportionately large number of sample files and would generate an inordinate amount of matches.

The YARA Hunting service scans/matches files of up to 200 MB in size.

Ruleset Administration API

Create a ruleset

This query tries to create a new ruleset with content given inside the POST request. The ruleset name serves as a unique identifier for the ruleset. Uploaded YARA text is treated as a collection of YARA rules, and is subject to ruleset validation procedures.

Request

POST /api/yara/admin/v1/ruleset

Headers: Content-Type: application/json

Request body:

{
  "ruleset_name": "string",
  "text": "string",
  "sample_available": 0
}

• ruleset_name
  • String, containing the ruleset name, between 3 and 48 characters long and conforming to the following regular expression: ^[a-z,A-Z,0-9,_-]*$
• text
  • UTF-8 encoded string containing YARA rules. YARA rules should be stringified, for example using the json.dumps() method or other similar tool. To properly escape the string, add a backslash (\) before each quotation mark (") and turn each newline into an escaped character (\n).
• sample_available
  • If set to true, the Matches Feed API will return only samples that are available for download to the user.
  • If set to false, the Matches Feed API will return only samples that are not available for download to the user.
  • If the parameter is not set in the request, the Matches Feed API will return all samples, regardless of their download availability status.

Response

{
  "ruleset_name": "string",
  "ruleset_sha1": "string",
  "sample_available": 0
}

• ruleset_name
  • string specified by the user in the request
• ruleset_sha1
  • hex-encoded string representing the SHA1 digest of the submitted ruleset text
• sample_available
  • this field shows sample download availability status. If this parameter was specified in the POST request body by the user, filtering will be applied and the response will only contain samples that are either available (true) or not available (false) for download

Delete a ruleset

This query initiates the deletion procedure of an existing ruleset.

DELETE /api/yara/admin/v1/ruleset/{ruleset_name}

Path parameters:

• ruleset_name
  • name of the existing ruleset (specified in the creation request)

Get ruleset information

Request

GET /api/yara/admin/v1/ruleset/{ruleset_name}

Path parameters:

• ruleset_name
  • name of the existing ruleset (specified in the creation request)

Response

{
  "ruleset_name": "string",
  "valid": true,
  "approved": true
}

• ruleset_name

  • string specified by the user in the request

• valid - three possible values:

  • null - ruleset validation is still pending, and it hasn't yet been deployed
  • true - ruleset has undergone the validation procedure, has been declared valid and is either deployed or it will be deployed upon approval
  • false - ruleset has undergone the validation procedure, has been declared invalid and will not be deployed

• approved - three possible values:

  • null - ruleset approval is still pending, and it hasn't yet been deployed
  • true - ruleset has been reviewed and approved by an administrator, and is either deployed or it will be deployed in the near future
  • false - ruleset has been reviewed by the administrator, has been rejected and will not be deployed

Get ruleset text

This query fetches the ruleset content.

Request

GET /api/yara/admin/v1/ruleset/{ruleset_name}/text

Path parameters:

• ruleset_name
  • name of the existing ruleset (specified in the creation request)

Response

{
  "text": "string"
}

Get all rulesets

This query lists the user's ruleset collection.

Request

GET /api/yara/admin/v1/ruleset

Response

[
  {
    "ruleset_name": "string",
    "valid": true,
    "approved": true
  }
]

• ruleset_name
  • string, specified by the user in the creation request
• valid - three possible values:
  • null - ruleset validation is still pending, and it hasn't yet been deployed
  • true - ruleset has undergone the validation procedure, has been declared valid and is either deployed or it will be deployed upon approval
  • false - ruleset has undergone the validation procedure, has been declared invalid and will not be deployed
• approved - three possible values:
  • null - ruleset approval is still pending, and it hasn't yet been deployed
  • true - ruleset has been reviewed and approved by an administrator, and is either deployed or it will be deployed in the near future
  • false - ruleset has been reviewed by the administrator, has been rejected and will not be deployed

If the authenticated user does not have any uploaded rulesets, this query will return an empty list.

The HTTP response code "200 OK" means that all rulesets were successfully fetched, even if the user doesn't have any uploaded rulesets.

Matches Feed API

Fetch matches

This query returns a set of YARA ruleset matches in the specified time range for a particular user.

The feed stores records for the last 365 days.

Request

GET /api/feed/yara/v1/query/{time_format}/{time_value}[?format={response_format}]

Path parameters:

• time_format
  • Format in which the time value will be specified. Possible values are: timestamp - number of seconds since 1970-01-01 00:00:00; utc - UTC date in the YYYY-MM-DDThh:mm:ss format
  • Required
• time_value
  • Time range start
  • Required

Query parameters:

• format
  • Specifies the response format. Supported values: xml (default), json
  • Optional

The time range is defined as the time period from the time specified by the time_value parameter up to the time of the request being made.

If the requested timestamp is not within the last 365 days, the service will respond with the status code "400 Bad Request".

The latest time value is 10 seconds before the current time. Specifying a time value outside this period will yield a "400 Bad Request" status code.

The feed will return at most 1000 records, starting from the earliest one. However, if a single second contains more than 1000 matches, all of them will be returned in a single query.

The response data provides the latest timestamp until which events are included. To fetch the next recordset, the timestamp from the response field last_timestamp should be increased by 1 and used in the subsequent query as the time_value parameter. The maximum time span for a single request is limited to 24h. All time values are in UTC, independent of the input format.

Example Requests

Get all YARA matches from 2016-05-20 00:00:00 as timestamp:

/api/feed/yara/v1/query/timestamp/1463702400

Get all the YARA matches from 2016-05-20 00:00:00 UTC as date string:

/api/feed/yara/v1/query/utc/2016-05-20T00:00:00

Fetch all YARA matches from 2016-05-20 00:00:00 in the JSON format:

/api/feed/yara/v1/query/timestamp/1463702400?format=json

Fetch all YARA matches from 2016-05-20 00:00:00 in the XML format:

/api/feed/yara/v1/query/timestamp/1463702400?format=xml

Response

A query response will contain zero or more match entries that are found within the specified time period.

A single entry represents a match produced by a specific YARA ruleset against a particular sample file. The entry contains information about the sample file (SHA1, file size) as well as information on one or more rules from the ruleset that matched the sample file (rule name, tags and meta fields and their values).

Additional data propagated from YARA engine is a list of matching strings contained in matched_data objects. The matched_data object is a 3 dimensional vector with the form (int match_offset, base64_string string_identifier, base64_string matched_string).

The entry also contains information about sample download availability as a boolean value in the sample_available field.

If a sample was matched by several rulesets, each will produce its own entry.

JSON Response Format

{
  "rl": {
    "feed": {
      "entries": [],
      "last_timestamp": 0,
      "name": "string",
      "time_range": {}
    }
  }
}

---

rl.feed.entries[]

{
  "sha1": "string",
  "file_type": "string",
  "timestamp": 0,
  "ruleset_name": "string",
  "rule": [
    {
      "meta": [
        [
          "string",
          "string"
        ]
      ],
      "identifier": "string",
      "tag": [
        "string"
      ],
      "matched_data": [
        {
          "string_identifier": "string",
          "match_offset": 0,
          "matched_string": "string"
        }
      ]
    }
  ],
  "sample_available": true
}