Malware presence change events feed (TCF-0111)

The Malware Presence Change Events Feed provides a continuous list of new malicious samples, false positives (FPs) and malicious samples for which the ReversingLabs threat name changed.

Event types included in the feed:

• New malicious - event type representing files for which the classification changed to MALICIOUS (including false negatives - FNs)
• False positive (FP) - event type representing files for which the classification changed from MALICIOUS or SUSPICIOUS to KNOWN
• Threat name changes - event type representing malicious files for which the threat name changed

By including event_type information in the response, the service enables users to filter records by event and easily get newly discovered samples that ReversingLabs classified as malicious, or information about false positive samples - malicious or suspicious samples for which the classification changed into KNOWN after rescanning.

The service also provides information about threat name changes for malicious samples. The threat_name field is returned in the response for "New malicious" and "Threat name changes" event types.

Threat names follow the ReversingLabs Malware Naming Standard <naming> (platform-subplatform.malware_type.malware_family). The following are examples of threat names formatted according to the standard:

Win32.Ransomware.Teslacrypt Android.Trojan.Smsbot Script-JS.Downloader.Nemucod

By comparing the threat_name fields in the response for the "Threat name changes" event type, users can track malware type changes, malware family changes, as well as changes in the affected platform.

Similarly, based on the threat_name field information in the response for "New malicious" event type, users can compile a list of new malicious samples by specific platform, malware type, or malware family.

The feed stores records for the last 365 days.

ReversingLabs Malware Naming Standard

The ReversingLabs detection string consists of three main parts separated by dots. All parts of the string will always appear (all three parts are mandatory).

platform-subplatform.type.familyname

1. The first part of the string indicates the platform targeted by the malware.

   This string is always one of the strings listed in the Platform string table. If the platform is Archive, Audio, ByteCode, Document, Image or Script, then it has a subplatform string. Platform and subplatform strings are divided by a hyphen (-). The lists of available strings for Archive, Audio, ByteCode, Document, Image and Script subplatforms can be found in their respective tables.

2. The second part of the detection string describes the malware type. Strings that appear as malware type descriptions are listed in the Type string table.

3. The third and last part of the detection string represents the malware family name, i.e. the name given to a particular malware strain.

   Names "Agent", "Gen", "Heur", and other similar short generic names are not allowed. Names can't be shorter than three characters, and can't contain only numbers. Special characters (apart from -) must be avoided as well. The - character is only allowed in exploit (CVE/CAN) names (for example CVE-2012-0158).

Examples

If a trojan is designed for the Windows 32-bit platform and has the family name "Adams", its detection string will look like this:

Win32.Trojan.Adams

If some backdoor malware is a PHP script with the family name "Jones", the detection string will look like this:

Script-PHP.Backdoor.Jones

Some potentially unwanted application designed for Android that has the family name "Smith" will have the following detection string:

Android.PUA.Smith

Some examples of detections with invalid family names are:

Win32.Dropper.Agent
ByteCode-MSIL.Keylogger.Heur
Script-JS.Hacktool.Gen
Android.Backdoor.12345
Document-PDF.Exploit.KO
Android.Spyware.1a
Android.Spyware.Not-a-CVE
Win32.Trojan.Blue_Banana
Win32.Ransomware.Hydra:Crypt
Win32.Ransomware.HDD#Cryptor

Platform string

The platform string indicates the operating system that the malware is designed for. The following table contains the available strings and the operating systems for which they are used.

String	Short description
ABAP	SAP / R3 Advanced Business Application Programming environment
Android	Applications for Android OS
AOL	America Online environment
Archive	Archives. See Archive subplatforms for more information.
Audio	Audio. See Audio subplatforms for more information.
BeOS	Executable content for Be Inc. operating system
Boot	Boot, MBR
Binary	Binary native type
ByteCode	ByteCode, platform-independent. See ByteCode subplatforms for more information.
Blackberry	Applications for Blackberry OS
Console	Executables or applications for old consoles (e.g. Nintendo, Amiga, ...)
Document	Documents. See Document subplatforms for more information.
DOS	DOS, Windows 16 bit based OS
EPOC	Applications for EPOC mobile OS
Email	Emails. See Email subplatforms for more information.
Firmware	BIOS, Embedded devices (mp3 players, ...)
FreeBSD	Executable content for 32-bit and 64-bit FreeBSD platforms
Image	Images. See Image subplatforms for more information.
iOS	Applications for Apple iOS (iPod, iPhone, iPad…)
Linux	Executable content for 32 and 64-bit Linux operating systems
MacOS	Executable content for Apple Mac OS, OS X
Menuet	Executable content for Menuet OS
Novell	Executable content for Novell OS
OS2	Executable content for IBM OS/2
Package	Software packages. See Package subplatforms for more information.
Palm	Applications for Palm mobile OS
Script	Scripts. See Script subplatforms for more information.
Shortcut	Shortcuts
Solaris	Executable content for Solaris OS
SunOS	Executable content for SunOS platform
Symbian	Applications for Symbian OS
Text	Text native type
Unix	Executable content for the UNIX platform
Video	Videos
WebAssembly	Binary format for executable code in Web pages
Win32	Executable content for 32-bit Windows OS's
Win64	Executable content for 64-bit Windows OS's
WinCE	Executable content for Windows Embedded Compact OS
WinPhone	Applications for Windows Phone

Archive subplatforms

String	Short description
ACE	WinAce archives
AR	AR archives
ARJ	ARJ (Archived by Robert Jung) archives
BZIP2	Bzip2 archives
CAB	Microsoft Cabinet archives
GZIP	GNU Zip archives
ISO	ISO image files
JAR	JAR (Java ARchive) archives
LZH	LZH archives
RAR	RAR (Roshal Archive) archives
7ZIP	7-Zip archives
SZDD	Microsoft SZDD archives
TAR	Tar (tarball) archives
XAR	XAR (eXtensible ARchive) archives
ZIP	ZIP archives
ZOO	ZOO archives
Other Archive identification	All other valid Spectra Core identifications of Archive type

Audio subplatforms

String	Short description
WAV	Wave Audio File Format
Other Audio identification	All other valid Spectra Core identifications of Audio type

ByteCode subplatforms

String	Short description
JAVA	Java bytecode
MSIL	MSIL bytecode
SWF	Adobe Flash

Document subplatforms

String	Short description
Access	Microsoft Office Access
CHM	Compiled HTML
Cookie	Cookie files
Excel	Microsoft Office Excel
HTML	HTML documents
Multimedia	Multimedia containers that aren't covered by other platforms (e.g. ASF)
Office	File that affects multiple Office components
OLE	Microsoft Object Linking and Embedding
PDF	PDF documents
PowerPoint	Microsoft Office PowerPoint
Project	Microsoft Office Project
Publisher	Microsoft Office Publisher
RTF	RTF documents
Visio	Microsoft Office Visio
XML	XML and XML metafiles (ASX)
Word	Microsoft Office Word
Other Document identification	All other valid Spectra Core identifications of Document type

Email subplatforms

String	Short description
MIME	Multipurpose Internet Mail Extensions
MSG	Outlook MSG file format

Image subplatforms

String	Short description
ANI	File format used for animated mouse cursors on Microsoft Windows
BMP	Bitmap images
EMF	Enhanced Metafile images
EPS	Adobe Encapsulated PostScript images
GIF	Graphics Interchange Format
JPEG	JPEG images
OTF	OpenType Font
PNG	Portable Network Graphics
TIFF	Tagged Image File Format
TTF	Apple TrueType Font
WMF	Windows Metafile images
Other Image identification	All other valid Spectra Core identifications of Image type

Package subplatforms

String	Short description
NuGet	NuGet packages
DEB	Debian Linux DEB packages
RPM	Linux RPM packages
WindowStorePackage	Packages for distributing and installing Windows apps
Other Package identification	All other valid Spectra Core identifications of Package type

Script subplatforms

String	Short description
ActiveX	ActiveX scripts
AppleScript	AppleScript scripts
ASP	ASP scripts
AutoIt	AutoIt scripts (Windows)
AutoLISP	AutoCAD LISP scripts
BAT	Batch scripts
CGI	CGI scripts
CorelDraw	CorelDraw scripts
Ferite	Ferite scripts
INF	INF Script, Windows installer scripts
INI	INI configuration file
IRC	IRC, mIRC, pIRC/Pirch Script
JS	Javascript, JScript
KiXtart	KiXtart scripts
Logo	Logo scripts
Lua	Lua scripts
Macro	Macro (e.g. VBA, AmiPro macros, Lotus123 macros)
Makefile	Makefile configuration
Matlab	Matlab scripts
Perl	Perl scripts
PHP	PHP scripts
PowerShell	PowerShell scripts, Monad (MSH)
Python	Python scripts
Registry	Windows Registry scripts
Ruby	Ruby scripts
Shell	Shell scripts
Shockwave	Shockwave scripts
SQL	SQL scripts
SubtitleWorkshop	SubtitleWorkshop scripts
WinHelp	WinHelp Script
WScript	Windows Scripting Host related scripts (can be VBScript, JScript, …)
Other Script identification	All other valid Spectra Core identifications of Script type

Type string

This string is used to describe the general type of malware. The following table contains the available strings and describes what each malware type is capable of.

String	Description
Adware	Presents unwanted advertisements
Backdoor	Bypasses device security and allows remote access
Browser	Browser helper objects, toolbars, and malicious extensions
Certificate	Classification derived from certificate data
Coinminer	Uses system resources for cryptocurrency mining without the user's permission
Dialer	Applications used for war-dialing and calling premium numbers
Downloader	Downloads other malware or components
Dropper	Drops malicious artifacts including other malware
Exploit	Exploits for various vulnerabilities, CVE/CAN entries
Format	Malformations of the file format. Classification derived from graylisting, validators on unpackers
Hacktool	Software used in hacking attacks, that might also have a legitimate use
Hyperlink	Classifications derived from extracted URLs
Infostealer	Steals personal info, passwords, etc.
Keylogger	Records keystrokes
Malware	New and recently discovered malware not yet named by the research community
Network	Networking utilities, such as tools for DoS, DDoS, etc.
Packed	Packed applications (UPX, PECompact…)
Phishing	Email messages (or documents) created with the aim of misleading the victim by disguising itself as a trustworthy entity into opening malicious links, disclosing personal information or opening malicious files.
PUA	Potentially unwanted applications (hoax, joke, misleading...)
Ransomware	Malware which encrypts files and demands money for decryption
Rogue	Fraudulent AV installs and scareware
Rootkit	Provides undetectable administrator access to a computer or a mobile device
Spam	Other junk mail that does not unambiguously fall into the Phishing category, but contains unwanted or illegal content.
Spyware	Collects personal information and spies on users
Trojan	Allows remote access, hides in legit applications
Virus	Self-replicating file/disk/USB infectors
Worm	Self-propagating malware with exploit payloads

PULL with timestamp

This query returns samples with a newly calculated or changed malware presence classification and threat name, starting from the timestamp provided in the request. The feed will return at most 1000 records, or a little bit over 1000 if there are multiple records with the same timestamp.

To retrieve the next batch of records, the timestamp from the response field last_timestamp should be increased by 1 and used in the subsequent query as the value of the time_value parameter.

If the requested timestamp is not within the last 365 days, the service will respond with the status code 400 Bad Request.

GET /api/feed/mwp_change_events/v1/query/{time_format}/{time_value}/[?format=xml|json][&sample_available=false|true][&limit=N]

Request parameters

• time_format
  • Format in which the time value will be specified. Supported values are: timestamp - number of seconds since 1970-01-01 00:00:00; utc - UTC date in the YYYY-MM-DDThh:mm:ss format
  • Required
• time_value
  • Accepts values in the format set by time_format
  • Required
• format
  • Parameter defining the format in which the resulting data will be returned. Supported values are: xml (default), json, tsv
  • Optional
• sample_available
  • If this parameter is set to true in the request, filtering will be applied and the response will contain only samples that are present in the ReversingLabs storage and available for download. When set to false, the query will return all samples, regardless of their download availability status. The default is false, meaning that if the parameter is not provided in the request, filtering is not applied.
  • Optional
• limit
  • The number of records to return in the response. The maximum and default value is 1000. Note that the response may include more records than requested to ensure that all records with the same timestamp are returned.
  • Optional

Response format

The response contains a list of records matching the requested timestamp. Every record in the list includes the following information:

• SHA1, MD5, and SHA256 hashes associated with the sample,
• event type,
• sample classification,
• threat name,
• sample size,
• sample type,
• the timestamp of the update.

An empty response is returned if no records for the requested timestamp are available.

Response fields are described in the following table. Example responses can be found further in the document. Fields that do not appear in the response are considered empty. There will be no fields with empty or null values in the response.

• SHA1
  • SHA1 of the sample
• SHA256
  • SHA256 of the sample
• MD5
  • MD5 of the sample
• event_type
  • The type of event that triggered the record in the feed. Possible values: NEW_MALICIOUS; FP; THREAT_NAME_CHANGE
• classification
  • Current Malware Presence status designation. The status designation is calculated by a proprietary ReversingLabs algorithm that adapts and improves as new information about the file and the threat is discovered. The algorithm takes the following into account: Spectra Intelligence antivirus scan results, threat and trust factors, parent/child relationships, certificates, and other metadata-specific information.
• threat_name
  • Sample's detected threat name (platform-subplatform.malware_type.malware_family)
• sample_size
  • Sample size (in bytes)
• sample_type
  • Sample type
• record_on
  • Timestamp of the update, indicating the date and time when the sample record was updated in the feed.

Response Examples

{"rl": {
    "mwp_change_events_feed": {
        "time_range": {
            "from": "YYYY-MM-DDTHH:MM:SS",
            "to": "YYYY-MM-DDTHH:MM:SS"
        },
        "entries": [
            {
                "sha1": "21841b32c6165b27dddbd4d6eb3a672defe54271",
                "sha256": "2f6ed...55346",
                "md5": "3133c2231fcee5d6b0b4c988a5201da1",
                "event_type": "FP",
                "classification":"KNOWN",
                "sample_size":"636416",
                "sample_type":"PE/Exe",
                "record_on":"1470221633000"

            },
            {...},
                …
            ],
            "last_timestamp": "YYYY-MM-DDTHH:MM:SS_or_timestamp", -- the format is the same as the requested time format
        }
    }
}

{"rl": {
    "mwp_change_events_feed": {
        "time_range": {
            "from": "YYYY-MM-DDTHH:MM:SS",
            "to": "YYYY-MM-DDTHH:MM:SS"
        },
        "entries": [
            {
                "sha1": "21841b32c6165b27dddbd4d6eb3a672defe54271",
                "sha256": "2f6ed...55346",
                "md5": "3133c2231fcee5d6b0b4c988a5201da1",
                "event_type": "THREAT_NAME_CHANGE",
                "classification":"MALICIOUS",
                "threat_name":"Win32.Trojan.Nsis"
                "sample_size":"636416",
                "sample_type":"PE/Exe",
                "record_on":"1470221633000"

            },
            {...},
                …
            ],
            "last_timestamp": "YYYY-MM-DDTHH:MM:SS_or_timestamp", -- the format is the same as the requested time format
        }
    }
}

PULL without timestamp

For this query type, ReversingLabs is tracking the user's last query state, so the user doesn’t have to provide a timestamp in the request to retrieve the next batch of records.

PULL Query

This query returns a list of classification and threat name changes since a particular point in time. The starting point for this query is defined using the START query. If the user has not previously requested this query or called the START query, it will return records starting with the current timestamp. Every subsequent call will continue from the timestamp where the previous call ended. In case that the timestamp of the previous call is older than 365 days, the subsequent call will autocorrect this timestamp to the oldest available (i.e. current - 365 days), and corresponding records will be returned.

Unless the limit parameter is specified, the feed returns up to 1000 records and any surplus records sharing the same timestamp. This ensures that all the records with the same timestamp will be included in the recordset. The limit parameter must not be greater than 1000.

This endpoint is built to be queried by a single thread (single instance). Any concurrent requests will be blocked until the previous one is fulfilled.

Request parameters

• format
  • Parameter defining the format in which the resulting data will be returned. Supported values are: xml (default), json
  • Optional
• sample_available
  • If this parameter is set to true in the request, filtering will be applied and the response will contain only samples that are present in the ReversingLabs storage and available for download. When set to false, the query will return all samples, regardless of their download availability status. The default is false, meaning that if the parameter is not provided in the request, filtering is not applied.
  • Optional
• limit
  • The number of records to return in the response. The maximum and default value is 1000. Note that the response may include more records than requested to ensure that all the records with the same timestamp are returned.
  • Optional

Response format

The response format is the same as in the PULL with timestamp.

START Query

This query sets the starting timestamp for the previously described PULL query.

The starting timestamp must be within the last 365 days, otherwise the service will respond with the status code 400 Bad Request.

PUT /api/feed/mwp_change_events/v1/query/start/[time_format]/[time_value]

Request parameters

• time_format
  • Format in which the time value will be specified. Supported values are: timestamp - number of seconds since 1970-01-01 00:00:00; utc - UTC date in the YYYY-MM-DDThh:mm:ss format
  • Required
• time_value
  • Accepts values in the format set by time_format
  • Required

Response format

A successful query returns an HTTP 200 OK message with an empty response body.

Examples

Format query field

Retrieving all new detections from 2019-05-09T07:25:10

[GET]

/api/feed/mwp_change_events/v1/query/timestamp/1557386701
/api/feed/mwp_change_events/v1/query/utc/2019-05-09T07:25:10

Retrieving all new detections from 2019-05-09T07:25:10 with samples available in the storage

[GET]

/api/feed/mwp_change_events/v1/query/timestamp/1557386701?sample_available=true
/api/feed/mwp_change_events/v1/query/timestamp/1557386701?sample_available=true&format=json

Retrieving all new samples from 2019-05-09T07:25:10 in JSON and XML formats

[GET]

/api/feed/mwp_change_events/v1/query/timestamp/1557386701?format=json
/api/feed/mwp_change_events/v1/query/timestamp/1557386701?format=xml

Setting the initial timestamp to 2017-03-26 10:33:20

[PUT]

/api/feed/mwp_change_events/v1/query/start/timestamp/1557386701

Pulling records from the latest timestamp

[GET]

/api/feed/mwp_change_events/v1/query/pull