Connectors
The Connectors service allows automatically retrieving a large number of files from external sources and analyzing them on the appliance. Events for the Connectors service are logged as CEF messages and can be monitored if System Alerting is enabled on the appliance.
Connectors can only be configured by the appliance administrator(s), not by regular users.
To manage settings for each connector, access the Administration ‣ Connectors page. The sidebar on the left lists all currently supported types of connectors.
Select a connector in the sidebar to access its configuration dialog. If a connector is disabled or if it has not been previously configured on the appliance, the dialog contains only the Enable connector button. Click the button to start configuring the connector.
Network Connector - File Share
The Network File Share connector allows connecting up to 5 shared network resources to the appliance. When the network shares are connected and mounted to the appliance, it can automatically scan the network shares and submit files for analysis. After analyzing the files, the appliance can optionally sort the files into folders on the network share based on their classification status.
The Network File Share connector supports SMB and NFS file sharing protocols.
Currently, it is not possible to assign a custom name to each network share. The only way to distinguish between configured network shares is to look at their addresses. If there are 3 configured network shares, and the network share 2 is removed, the previous network share 3 will automatically “move up” in the list and become network share 2.
Configuring Network Shares
To add a new network share:
- make sure the connector is enabled
- expand the Shares section in the Network File Share Connector dialog and fill in the relevant fields.
| Field | Mandatory | Description |
|---|---|---|
| Address | Mandatory | Enter the address of the shared network resource that will be mounted to the appliance. The address must include the protocol (SMB or NFS). Leading slashes are not required for NFS shares (example: nfs:storage.example.lan). The address can point to the entire network drive, or to a specific folder (example: smb://storage.example.lan/samples/collection). When the input folder and/or sorting folders are configured, their paths are treated as relative to the address configured here. Note: If the address contains special characters, it may not be possible to mount the share to the appliance. The comma character cannot be used in the address for SMB shares. Some combinations of ? and # will result in errors when attempting to mount both the SMB and the NFS shares. |
| Username | Optional, SMB only | Enter the username for authenticating to the SMB network share (if required). Usernames and passwords for SMB authentication can only use a limited range of characters (ASCII-printable characters excluding the comma). |
| Password | Optional, SMB only | Enter the password for authenticating to the SMB network share (if required). Usernames and passwords for SMB authentication can only use a limited range of characters (ASCII-printable characters excluding the comma). |
| Input folder | Optional | Specify the path to the folder on the network share containing the files to be analyzed by Spectra Analyze. The folder must exist on the network share. The path specified here is relative to the root (address of the network file share). If the input folder is not configured, the root is treated as the input folder. |
Using Advanced Connector Options
Advanced options for the Network File Share connector refer to actions that the connector service can perform on the files after the Spectra Analyze appliance finishes analyzing them.
Specifically, the connector can be configured to automatically sort files into user-defined sorting folders on the network share. Files are sorted into folders based on the classification status they receive during analysis (malicious, suspicious, known, unknown).
Advanced options can be configured for every network share individually. This means that the sorting criteria, folder names and folder paths can be different on each configured network share.
The connector will operate and analyze files even if these advanced options are disabled. They only control the post-analysis activities.
| Field | Description |
|---|---|
| Delete source files | Selecting the checkbox will allow the connector to delete files on the network share after they have been processed. |
| Enable automatic file sorting | Selecting the checkbox will allow the connector to store analyzed files and sort them into folders on every configured network share based on their classification status. This checkbox toggles the availability of other options in the Advanced Options section. |
| Goodware folder | Specify the path to folder into which the connector will store files classified as “Known” (goodware). This field is mandatory when Enable automatic file sorting is selected. The path specified here is relative to the address of the network file share. If the folder doesn’t already exist on the network share, it will be automatically created after saving the configuration. |
| Malware folder | Specify the path to folder into which the connector will store files classified as “Malicious” (malware). This field is mandatory when Enable automatic file sorting is selected. The path specified here is relative to the address of the network file share. If the folder doesn’t already exist on the network share, it will be automatically created after saving the configuration. |
| Unknown folder | Specify the path to folder into which the connector will store files without classification (“Unknown” status). The path specified here is relative to the address of the network file share. If this field is left empty, unknown files will be stored either to the Goodware or to the Malware folder, depending on the “Allow unknown” setting. |
| Known threshold | Files classified as Goodware with the risk score value higher than the one configured here will be stored into the configured Malware folder. Goodware files with the risk score less than or equal to the value configured here will be stored into the configured Goodware folder. Supported values are 0 to 5. Default is 5 (saves all to the Goodware folder). This field is mandatory when Enable automatic file sorting is selected. |
| Allow unknown | When selected, files with the “Unknown” classification status are stored into the configured Goodware folder. If this checkbox is not selected, files with the “Unknown” status are either stored into the Unknown folder (if the Unknown folder is configured), or to the Malware folder (if the Unknown folder is not configured). |
| Allow suspicious | When selected, files classified as “Suspicious” will be stored into the configured Goodware folder. If this checkbox is not selected, files classified as “Suspicious” will be stored into the configured Malware folder. |
Starting the Connector
After providing the required information, click Test connection to verify that the appliance can access the configured network share. When the button is clicked, the appliance attempts to connect and mount the network share.
To remove all configured settings for the current network share, click Remove item.
To add another network share, click Add item. Up to 5 network shares can be added in this way. If there are already 5 network shares connected to the appliance, at least one must be removed by clicking Remove item before adding another.
When all network shares are configured successfully, click Start connector at the bottom of the page. This will initiate the connector service on the appliance. The service mounts configured network shares, automatically retrieves all files from them, and submits the files for analysis on the appliance.
The service will continually scan the network shares for new files (approximately every 5 minutes). If any of the existing files on the network share has changed since the last scan, it will be treated as a new file and analyzed again. The service supports a retry mechanism that attempts to retrieve each file up to 10 times. Failures are recorded in the system log.
All files retrieved from the network share(s) and analyzed on the appliance are accessible to Spectra Analyze users from the Submissions page. They are distinguished from other files by the unique username fileshare_connector. Additionally, each file retrieved via the connector has a set of User Tags automatically assigned to it. Those User Tags are based on the file metadata, and can contain information about the file source, the last modification time in the original location, file permissions, and more.
If advanced options are not enabled, the connector service will not perform any additional actions on the files retrieved from network shares after the Spectra Analyze appliance finishes analyzing them. The users can see the analysis results for each file on its Sample Details page.
Handling Rescanned and Renamed Files
The Network File Share connector supports several scenarios involving rescanned and renamed files. The connector has the ability to automatically rename files, which allows it to handle duplicates and files manually renamed by the user. The advanced file sorting options must be configured for the connector to be able to move files after they are analyzed.
| SCENARIO | RESULT |
|---|---|
| A new file is analyzed, but a file with the same filename already exists in the output folder. Their hashes are identical. | The original file remains in the output folder. The last modified timestamp value in the file metadata is updated for the original file. Its filename remains unchanged. The new file is removed after analysis. |
| A new file with the same filename as an old file is analyzed. Their hashes are identical. However, the old file no longer exists in the output folder, or the new file has been uploaded for the first time. | The new file is saved to the output folder. Its filename remains unchanged. |
| A new file is analyzed, but a file with the same filename already exists in the output folder. Their hashes are different. | The new file is renamed and saved to the output folder. The file renaming pattern is to add (#) after the original file name; for example: Name.extension would be saved as Name(1).extension, Name(2).extension, Name(3).extension, etc. |
| A file has been analyzed previously and moved into one output folder (A). Based on reanalysis, it should be moved to a different output folder (B). | The file is moved to a different output folder (B). Its filename remains unchanged. |
Pausing and Disabling the Connector
While the connector service is active, the Start connector button changes into Pause connector. Clicking this button temporarily halts the connector service, which in turn stops scanning the network shares for new files. The connector service records the last state and is able to resume scanning when Start connector is clicked again.
While the connector is running, it is possible to modify its configuration and save it by clicking Save changes without having to pause or disable the connector.
If the connector service is active during a scheduled or manually executed Purge action, the system will automatically stop the service before performing the Purge action, and start it after the Purge action is complete.
To disable the entire connector service on the appliance, click Disable connector at the bottom of the page. When the connector is disabled, it will not be possible to reconfigure, start, or pause it until the service is enabled again.
The current connector configuration will be preserved and restored when the service is enabled again. Likewise, all files retrieved from the network share(s) and analyzed by Spectra Analyze will remain on the appliance.
Email Connector - IMAP AbuseBox
The IMAP AbuseBox connector allows connecting to a Microsoft Exchange server and analyzing retrieved emails on the Spectra Analyze appliance.
Requirements
- IMAP must be enabled on the Exchange server.
- A new user account must be configured on the mail server and its credentials provided to the connector in the configuration dialog.
- A dedicated email folder must be created in the Exchange user account, and its name provided to the connector in the configuration dialog. All emails forwarded to that folder are collected by the connector and automatically sent to the appliance for analysis.
When the analysis is complete, emails with detected threats get classified as malicious. If the Automatic message filing option is enabled, malicious emails are moved to the specified Malware folder on the configured Exchange user account.
Emails with no detected malicious content do not get classified. They can optionally be moved to the specified Unknown folder on the configured Exchange user account.
To improve performance and minimize processing delays, each email sample gets analyzed and classified only once. When the Automatic message filing option is enabled, each email sample is moved only once, based on its first available classification.
Because of that, it is recommended to enable classification propagation and allow retrieving Spectra Intelligence classification information during sample analysis instead of after. Administrators can enable these two options in the Administration ‣ Configuration ‣ Spectra Detect Processing Settings dialog. This will improve classification of emails with malicious attachments.
Configuring the Exchange user account
To configure the connection with the Exchange user account:
- make sure the connector is enabled
- fill in the fields in the Exchange setup section of the IMAP Connector dialog.
| Field | Mandatory | Description |
|---|---|---|
| Server domain | Mandatory | Enter the domain or IP address of the Exchange server. The value should be FQDN, hostname or IP. This should not include the protocol (e.g., http). |
| Email folder | Mandatory | Enter the name of the email folder from which the email messages will be collected for analysis. This folder must belong to the same Exchange user account for which the credentials are configured in this section. The folder name is case-sensitive. |
| Connection Type | Mandatory | Supports IMAP (Basic Authentication) and Exchange (OAuth2) methods of authentication. Depending on the selection, the next section of the form will ask for different user credentials: Basic Authentication asks for a username and password, OAuth2 asks for a client ID, client secret and tenant ID. |
| Email address | Mandatory | Enter the primary email address of the configured Exchange user account. |
| Access Type | Mandatory | Delegate is used in environments where there’s a one-to-one relationship between users. Impersonation is used in environments where a single account needs to access many accounts. |
| Connect securely | Optional | If selected, the connector will not accept connections to Exchange servers with untrusted or expired certificates. |
After providing the required information, click Test connection to verify that the appliance can access the configured Exchange account.
Using Advanced Connector Options
Advanced options for the IMAP connector refer to actions that the connector service can perform on the emails after the Spectra Analyze appliance finishes analyzing them.
Specifically, the connector can be configured to automatically sort emails into user-defined sorting folders on the connected Exchange user account. Emails are sorted into folders based on the classification status they receive during analysis (malicious, suspicious, known, unknown).
Emails classified as malicious are sorted into the configured “Malware” folder.
By default, emails classified as suspicious are also sorted into the “Malware” folder. If the Allow suspicious option is selected, then they are sorted into the configured “Unknown” folder.
Emails without detected malicious content (classified as Goodware, or not classified at all = Unknown) are always sorted into the configured “Unknown” folder.
The connector will operate and analyze emails even if these advanced options are disabled. They only control the post-analysis activities.
| Field | Description |
|---|---|
| Enable automatic message filing | Selecting the checkbox will allow the connector to move analyzed emails and sort them into email folders in the configured Exchange email user account. This checkbox toggles the availability of other options in the Advanced Options section. |
| Malware folder | Specify the name of the email folder into which the connector will store emails classified as “Malicious” (malware). This folder will be created if it doesn’t exist. This field is mandatory when Enable automatic message filing is selected. |
| Unknown folder | Specify the name of the email folder into which the connector will store emails with no malicious content detected. This folder will be created if it doesn’t exist. This field is mandatory when Enable automatic message filing is selected. |
| Allow suspicious | When selected, emails classified as “Suspicious” will be moved to the configured Unknown folder. If this checkbox is not selected, files classified as “Suspicious” will be moved to the configured Malware folder. |
Starting the Connector
When the configuration is complete, click Start connector at the bottom of the page. This will initiate the connector service on the appliance. The service connects to the configured Exchange user account, automatically retrieves emails from it, and submits them for analysis on the appliance.
All emails retrieved from the Exchange server and analyzed on the appliance are accessible to Spectra Analyze users from the Submissions page. They are distinguished from other files by the unique username abusebox_connector. Additionally, each file retrieved via the connector has a set of User Tags automatically assigned to it. Those User Tags are based on the file metadata, and can contain information about the file source, the last modification time in the original location, file permissions, email subject, recipient and sender addresses, and more.
If advanced options are not enabled, the connector service will not perform any additional actions on the emails retrieved from the Exchange server after the Spectra Analyze appliance finishes analyzing them. The users can see the analysis results for each file on its Sample Details page.
Pausing and Disabling the Connector
While the connector service is active, the Start connector button changes into Pause connector. Clicking this button temporarily halts the connector service, which in turn stops scanning the mailbox for new emails. The connector service records the last state and is able to resume scanning when Start connector is clicked again.
While the connector is running, it is possible to modify its configuration and save it by clicking Save changes without having to pause or disable the connector.
If the connector service is active during a scheduled or manually executed Purge action, the system will automatically stop the service before performing the Purge action, and start it after the Purge action is complete.
To disable the entire connector service on the appliance, click Disable connector at the bottom of the page. When the connector is disabled, it will not be possible to reconfigure, start, or pause it until the service is enabled again.
The current connector configuration will be preserved and restored when the service is enabled again. Likewise, all files retrieved from the Exchange server and analyzed by Spectra Analyze will remain on the appliance.
Storage Connector - S3
The S3 connector allows connecting up to 5 S3 buckets to the appliance. When the buckets are connected and mounted to the appliance, it can automatically scan the buckets and submit files for analysis. The files can be placed into the root of each bucket, or into an optional folder in each of the buckets.
After analyzing the files, the appliance can optionally sort the files into folders on the S3 bucket based on their classification status.
Currently, it is not possible to assign a custom name to each S3 file storage input. The only way to distinguish between configured buckets is to look at their names. If there are 3 configured S3 file storage inputs, and input 2 is removed, the previous input 3 will automatically “move up” in the list and become input 2.
Configuring S3 Buckets
To add a new S3 bucket:
- make sure the connector is enabled
- expand the S3 File Storage Inputs section in the S3 dialog and fill in the relevant fields.
| Field | Mandatory | Description |
|---|---|---|
| AWS S3 Access Key ID | Mandatory | The Access Key ID for AWS S3 account authentication. In cases where the appliance is hosted by ReversingLabs and Role ARN is used, this value will be provided by ReversingLabs. |
| AWS S3 Secret Access Key | Mandatory | The Secret Access Key for AWS S3 account authentication. In cases where the appliance is hosted by ReversingLabs and Role ARN is used, this value will be provided by ReversingLabs. |
| AWS S3 region | Mandatory | Specify the correct AWS geographical region where the S3 bucket is located. This parameter is ignored for non-AWS setups. |
| Enable Role ARN | Optional | Enables or disables authentication using an external AWS role. This allows the customers to use the connector without forwarding their access keys between services. The IAM role which will be used to obtain temporary tokens has to be created for the connector in the AWS console. These temporary tokens allow ingesting files from S3 buckets without using the customer secret access key. If enabled, it will expose more configuration options below. |
| Role ARN | Mandatory and visible only if Role ARN is enabled | The role ARN created using the external role ID and an Amazon ID. In other words, the ARN which allows the appliance to obtain a temporary token, which then allows it to connect to S3 buckets without using the customer secret access key. |
| External ID | Mandatory and visible only if Role ARN is enabled | The external ID of the role that will be assumed. Usually, it’s an ID provided by the entity which uses (but doesn’t own) an S3 bucket. The owner of that bucket takes the external ID and creates an ARN with it. |
| Role session name | Mandatory and visible only if Role ARN is enabled | Name of the session visible in AWS logs. Can be any string. |
| ARN token duration | Mandatory and visible only if Role ARN is enabled | How long before the authentication token expires and is refreshed. The minimum value is 900 seconds. |
| AWS S3 bucket | Mandatory | Specify the name of an existing S3 bucket which contains the samples to process. The bucket name can be between 3 and 63 characters long, and can contain only lower-case characters, numbers, periods, and dashes. Each label in the bucket name must start with a lowercase letter or number. The bucket name cannot contain underscores, end with a dash, have consecutive periods, or use dashes adjacent to periods. The bucket name cannot be formatted as an IP address. |
| AWS S3 folder | Optional | The input folder inside the specified bucket which contains the samples to process. All other samples will be ignored. The folder name can be between 3 and 63 characters long, and can contain only lower-case characters, numbers, periods, and dashes. Each label in the folder name must start with a lowercase letter or number. The folder name cannot contain underscores, end with a dash, have consecutive periods, or use dashes adjacent to periods. The folder name cannot be formatted as an IP address. If the folder is not configured, the root of the bucket is treated as the input folder. |
| S3 endpoint URL | Optional | Enter a custom S3 endpoint URL. Specifying the protocol is optional. Leave empty if using standard AWS S3. |
| Server Side Encryption Type | Optional | Leave blank unless the bucket policy enforces SSE headers to be sent to S3. Valid options are either “AES256” or “aws:kms” |
| Connect securely | Optional | If selected, the connector will not accept connections to S3 buckets with untrusted or expired certificates. This setting only applies when a custom S3 endpoint is used. |
| Enable Selection Criteria Using Metadata | Optional | If selected, the connector only fetches files that have certain object metadata saved. This metadata contains information about the classification of the file. This means that the files must have been pre-processed and saved with their metadata, which is an option in Spectra Detect (saving files after processing together with such metadata). The supported file metadata are classification and threat name. |
Using Advanced Connector Options
Advanced options for the S3 connector refer to actions that the connector service can perform on the files after the Spectra Analyze appliance finishes analyzing them.
Specifically, the connector can be configured to automatically sort files into user-defined sorting folders on the S3 bucket. Files are sorted into folders based on the classification status they receive during analysis (malicious, suspicious, goodware, unknown).
Advanced options can be configured for every bucket individually. This means that the sorting criteria, folder names and folder paths can be different on each configured S3 bucket.
The connector will operate and analyze files even if these advanced options are disabled. They only control the post-analysis activities.
| Field | Description |
|---|---|
| Enable Same Hash Rescan | Selecting the checkbox will force the connector to rescan samples that share the same hash. |
| Delete source files | Selecting the checkbox will allow the connector to delete files on the S3 bucket after they have been processed. |
| Enable automatic file sorting | Selecting the checkbox will allow the connector to store analyzed files and sort them into folders on every configured S3 bucket based on their classification status. This checkbox toggles the availability of other options in the Advanced Options section. |
| Goodware folder | Specify the path to folder into which the connector will store files classified as Goodware. This field is mandatory when Enable automatic file sorting is selected. The path specified here is relative to the address of the S3 bucket. If the folder doesn’t already exist on the bucket, it will be automatically created after saving the configuration. |
| Malware folder | Specify the path to folder into which the connector will store files classified as “Malicious” (malware). This field is mandatory when Enable automatic file sorting is selected. The path specified here is relative to the address of the S3 bucket. If the folder doesn’t already exist on the bucket, it will be automatically created after saving the configuration. |
| Unknown folder | Specify the path to folder into which the connector will store files without classification (“Unknown” status). The path specified here is relative to the address of the S3 bucket. If the folder doesn’t already exist on the bucket, it will be automatically created after saving the configuration. |
| Suspicious folder | Specify the path to folder into which the connector will store files classified as “Suspicious”. The path specified here is relative to the address of the S3 bucket. If the folder doesn’t already exist on the bucket, it will be automatically created after saving the configuration. |
After providing the required information, click Test connection to verify that the appliance can access the configured S3 bucket. When the button is clicked, the appliance attempts to connect and mount the bucket.
To remove all configured settings for the current S3 bucket, click Remove item.
To add another S3 bucket, click Add item. Up to 5 S3 buckets can be added in this way. If there are already 5 S3 buckets connected to the appliance, at least one must be removed by clicking Remove item before adding a new one.
Starting the Connector
When all file inputs are configured successfully, click Start connector at the bottom of the page. This will initiate the connector service on the appliance. The service mounts configured S3 buckets, automatically retrieves all files from them, and submits the files for analysis on the appliance.
All files retrieved from the S3 buckets and analyzed on the appliance are accessible to Spectra Analyze users from the Submissions page. They are distinguished from other files by the unique username s3_connector. Additionally, each file retrieved via the connector has a set of User Tags automatically assigned to it. Those User Tags are based on the file metadata, and can contain information about the file source, the last modification time in the original location, file permissions, and more.
If advanced options are not enabled, the connector service will not perform any additional actions on the files retrieved from S3 buckets after the Spectra Analyze appliance finishes analyzing them. The users can see the analysis results for each file on its Sample Details page.
Pausing and Disabling the Connector
While the connector service is active, the Start connector button changes into Pause connector. Clicking this button temporarily halts the connector service, which in turn stops receiving and analyzing new files. The connector service records the last state and is able to resume scanning when Start connector is clicked again.
While the connector is running, it is possible to modify its configuration and save it by clicking Save changes without having to pause or disable the connector.
If the connector service is active during a scheduled or manually executed Purge action, the system will automatically stop the service before performing the Purge action, and start it after the Purge action is complete.
To disable the entire connector service on the appliance, click Disable connector at the bottom of the page. When the connector is disabled, it will not be possible to reconfigure, start, or pause it until the service is enabled again.
The current connector configuration will be preserved and restored when the service is enabled again. Likewise, all files retrieved from the network share(s) and analyzed by Spectra Analyze will remain on the appliance.
Email Connector - SMTP
The SMTP connector allows analyzing incoming email traffic on the appliance to protect users from malicious content. When enabled, the connector service collects emails (with attachments) and uploads them to the appliance for analysis. Each email message is saved as one file. If email uploading fails for any reason, the connector automatically retries to upload it to the appliance.
When the analysis is complete, each email message receives a classification status from the appliance. In this operating mode, the connector acts as an SMTP relay. Therefore, the connector should not be used as a front-end service for accepting raw email traffic, but only as a system inside an already established secure processing pipeline for SMTP email.
To allow the SMTP connector to inspect and collect email traffic, users must ensure that the SMTP traffic in their network is diverted to port 25/TCP prior to configuring the connector on the appliance.
Additional port configuration may be required on the appliance. Because it involves manually modifying configuration files, this action can cause the appliance to malfunction. Contact ReversingLabs Support for instructions and guidance.
Profiles
There are two profiles for this connector: Default and Strict. These two profiles correspond to different Postfix configuration files. In the Default case, you don’t enforce TLS traffic and you accept any SMTP client. This corresponds to the following Postfix configuration:
mynetworks = 0.0.0.0/0 [::]/0
smtpd_tls_security_level = may
smtp_tls_security_level = may
In the Strict profile, you do enforce TLS and you can also specify trusted SMTP clients (highlighted line 1 in the example below; see Postfix docs for the specific syntax). The relevant portion of the configuration looks like this in Strict mode:
mynetworks = 0.0.0.0/0 [::]/0
smtpd_tls_security_level = encrypt
smtp_tls_security_level = encrypt
smtpd_tls_mandatory_ciphers = high
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_mandatory_exclude_ciphers = aNULL, MD5
Starting the Connector
After the connector is enabled, click the Start connector button. This will initiate the connector service on the appliance.
Pausing and Disabling the Connector
While the connector service is active, the Start connector button changes into Pause connector. Clicking this button temporarily halts the connector service, which in turn stops receiving and analyzing new email traffic. The connector service records the last state and is able to resume scanning when Start connector is clicked again.
If the connector service is active during a scheduled or manually executed Purge action, the system will automatically stop the service before performing the Purge action, and start it after the Purge action is complete.
To disable the entire connector service on the appliance, click Disable connector at the bottom of the page. When the connector is disabled, it will not be possible to reconfigure, start, or pause it until the service is enabled again.
Use Case: Accept Email from Any Email Client or Server
To configure the service to accept email from any email client or server, follow these steps:
-
Enable the SMTP connector
-
Contact ReversingLabs Support with:
- Your intended email forwarding address range
- A formal request to enable email forwarding
-
After receiving your request, ReversingLabs will:
- Create an MX DNS record for your hosted service
- Configure receiver restrictions to accept email only from your domain
-
Once ReversingLabs has added the MX record to the DNS for your hosted service, create email forwarding rules in your email system to automatically forward emails to the hosted service for processing.
Storage Connector - Azure Data Lake
The Azure Data Lake connector allows connecting up to five Azure Data Lake Gen2 containers to the appliance. When the containers are connected and mounted to the appliance, it can automatically scan them and submit files for analysis. The files can be placed into the root of each container, or into an optional folder in each of the containers.
This connector is not compatible with containers that have the Blob Soft Delete feature enabled.
After analyzing the files, the appliance can optionally sort the files into folders on the container based on their classification status.
Currently, it is not possible to assign a custom name to each data lake input. The only way to distinguish between configured containers is to look at their names. If there are three configured data lake inputs, and input 2 is removed, the previous input 3 will automatically “move up” in the list and become input 2.
Configuring Azure Data Lake containers
To add a new Azure Data Lake container:
- make sure the connector is enabled
- expand the Azure Data Lake Inputs section in the Azure data lake dialog and fill in the relevant fields.
| Field | Mandatory | Description |
|---|---|---|
| Storage account name | Mandatory | The name of the storage account. |
| Storage access key | Mandatory | The access key used for Shared Key Authentication. This value should end in == |
| Container | Mandatory | Specify the name of an existing Azure Data Lake container which contains the samples to process. The value must start and end with a letter or number and must contain only letters, numbers, and the dash (-) character. Consecutive dashes are not permitted. All letters must be lowercase. The value must have between 3 and 63 characters. |
| Folder | Optional | The input folder inside the specified container which contains the samples to process. All other samples will be ignored. |
Using Advanced Connector Options
Advanced options for the Azure Data Lake connector refer to actions that the connector service can perform on the files after the appliance finishes analyzing them.
Specifically, the connector can be configured to automatically sort files into user-defined sorting folders on the Azure Data Lake container. Files are sorted into folders based on the classification status they receive during analysis (malicious, suspicious, goodware, unknown).
Advanced options can be configured for every container individually. This means that the sorting criteria, folder names and folder paths can be different on each configured Azure data lake container.
| Field | Description |
|---|---|
| Delete source files | Selecting the checkbox will allow the connector to delete source files on Azure Data Lake storage after they have been processed. |
| Enable automatic file sorting | Selecting the checkbox will allow the connector to store analyzed files and sort them into folders based on their classification. |
| Goodware folder | Specify the path to folder into which the connector will store files classified as Goodware. This field is mandatory when Enable automatic file sorting is selected. The path specified here is relative to the address of the Azure Data Lake container. If the folder doesn’t already exist on the container, it will be automatically created after saving the configuration. |
| Malware folder | Specify the path to folder into which the connector will store files classified as Malicious. This field is mandatory when Enable automatic file sorting is selected. The path specified here is relative to the address of the Azure Data Lake container. If the folder doesn’t already exist on the container, it will be automatically created after saving the configuration. |
| Unknown folder | Specify the path to folder into which the connector will store files without classification (“Unknown” status). The path specified here is relative to the address of the Azure Data Lake container. If the folder doesn’t already exist on the container, it will be automatically created after saving the configuration. |
| Suspicious folder | Specify the path to folder into which the connector will store files classified as “Suspicious”. The path specified here is relative to the address of the Azure Data Lake container. If the folder doesn’t already exist on the container, it will be automatically created after saving the configuration. |
After providing the required information, click Test connection to verify that the appliance can access the configured Azure Data Lake container. When the button is clicked, the appliance attempts to connect and mount the container.
To remove all configured settings for the current Azure Data Lake container, click Remove item.
To add another Azure Data Lake container, click Add item. Up to five Azure Data Lake containers can be added this way. If there are already five Azure Data Lake containers connected to the appliance, at least one must be removed by clicking Remove item before adding a new one.
Starting the Connector
When all file inputs are configured successfully, click Start connector at the bottom of the page. This will initiate the connector service on the appliance. The service mounts configured Azure Data Lake containers, automatically retrieves all files from them, and submits the files for analysis on the appliance.
All files retrieved from the Azure Data Lake containers and analyzed on the appliance are accessible to Spectra Analyze users from the Submissions page. They are distinguished from other files by the unique username azure-data-lake_connector. Additionally, each file retrieved via the connector has a set of User Tags automatically assigned to it. Those User Tags are based on the file metadata, and can contain information about the file source, the last modification time in the original location, file permissions, and more.
If advanced options are not enabled, the connector service will not perform any additional actions on the files retrieved from Azure Data Lake containers after the Spectra Analyze appliance finishes analyzing them. The users can see the analysis results for each file on its Sample Details page.
Pausing and Disabling the Connector
While the connector service is active, the Start connector button changes into Pause connector. Clicking this button temporarily halts the connector service, which in turn stops receiving and analyzing new files. The connector service records the last state and is able to resume scanning when Start connector is clicked again.
While the connector is running, it is possible to modify its configuration and save it by clicking Save changes without having to pause or disable the connector.
If the connector service is active during a scheduled or manually executed Purge action, the system will automatically stop the service before performing the Purge action, and start it after the Purge action is complete.
To disable the entire connector service on the appliance, click Disable connector at the bottom of the page. When the connector is disabled, it will not be possible to reconfigure, start, or pause it until the service is enabled again.
The current connector configuration will be preserved and restored when the service is enabled again. Likewise, all files retrieved from the network share(s) and analyzed by Spectra Analyze will remain on the appliance.
Global Configuration
In addition to every connector service having specific configuration settings, there is a Global Configuration section at the bottom of every connector page. These settings apply to all configured connectors.
| Field | Description |
|---|---|
| Save files that had encountered errors during processing | Original files that were not successfully uploaded will be saved to /data/connectors/connector-[CONNECTOR_NAME]/error_files/ |
| Max upload retries | Number of times the connector will attempt to upload the file to the processing appliance. Upon reaching the number of retries, it will be saved in the error_files/ destination or be discarded |
| Max upload timeout | Period (in seconds) between upload attempts of the sample being re-uploaded. |
| Upload algorithm | The algorithm used for managing delays between attempting to reupload the samples. In Exponential backoff, the delay is defined by multiplying the Max upload timeout parameter by 2, until reaching the maximum value of 5 minutes. Linear backoff will always use the Max upload timeout value for the timeout period between reuploads. |
| Max upload delay | In case the appliance is under high load, this parameter is used to delay any new upload to the appliance. The delay parameter will be multiplied by the internal factor determined by the load on the appliance. |
| Database cleanup period | Specifies the number of days for which the data will be preserved. |
| Database cleanup interval | Specifies the time (in seconds), in which the database cleanup will be performed. |