Spectra Detect Redundancy — Manager Clustering and Failover

important

This section is only available in OVA deployments.

Spectra appliances are designed for full redundancy. If one component fails, another automatically takes over using the same configuration settings.

This chapter describes how redundancy works for Spectra Detect Manager.

Creating Manager Redundancy

important

A proxy or Load balancer is REQUIRED for Spectra Detect Manager to function properly in Redundancy.

warning

Creating a cluster erases all existing data on the secondary SDM. As the secondary, use an SDM that has no data you need.

Use the following endpoint to identify the active/primary Spectra Detect Manager:

GET /api/v1/health-check/

This endpoint returns a 200 status code if the cluster node is primary and up. It returns a 503 status code if the cluster node is secondary or down. The endpoint does not provide a response body.

Before clustering, add the active/primary manager (the one you will use to initiate redundancy) to the proxy or load balancer. After redundancy setup completes, update the proxy to include both managers (this is described in step 7).

1. Configure Allowed Hosts

As admin, go to Administration > Spectra Detect Manager > General > Network Settings.

• Verify that the Application URL is set to the proxy IP or domain.

• In Allowed Hosts, ensure the following are listed:

  1. Current Detect Manager IP address and domain
  2. Proxy IP and domain
  3. Secondary IP and domain
  4. localhost and 127.0.0.1

  Enter one entry per line.

• Click Save on the primary manager.

• Repeat for the secondary manager, but only update the Allowed Hosts section. Don't change an existing Application URL on the secondary manager, because clustering replaces it. If the field is empty, for example on a new SDM, set it to the secondary's own https:// address so you can save.

Note: Changing the Application URL updates the configuration on all connected appliances (Hub, Workers, and Analyze).

warning

Both the primary and secondary manager must have each other's IP addresses added to Allowed Hosts before you initiate the cluster. If either node is missing the other's IP, the cluster setup may fail and leave both managers stuck in the secondary state, requiring manual recovery. This prerequisite applies to both GUI and API-driven setup workflows.

2. Create Redundant Cluster

Go to Administration > Redundancy. The Cluster Configuration tab says that the cluster isn't configured. Click Create Redundant Cluster.

The Redundant Cluster Setup Wizard opens, with the steps Establish Connection, Check Prerequisites, and Run the Configuration Process.

3. Establish Connection

Fill in the fields, then click Next.

Field	Description
Local IP Address	The IP address of this SDM, the primary. Select it from the list.
Secondary Spectra Detect Manager IP Address	The IP address of the secondary SDM.
Secondary Spectra Detect Manager URL	The URL of the secondary SDM, for example https://<secondary IP address>.
Secondary Spectra Detect Manager GUI User	A user on the secondary SDM. The user must be an admin.
Secondary Spectra Detect Manager GUI Password	The password of that user.
VPN Network	The network for the VPN that SDM automatically sets up between the two SDMs. The default is 100.127.255.248/29.
Failover Timeout in Seconds	How long the primary can be unavailable before the secondary takes over. Set a value between 3 and 600. The default is 15.
Disable TLS Certificate Sharing	By default, the SDMs in a cluster share the same TLS certificate. Select this option if each SDM needs its own TLS certificate. Users then see a different certificate after a failover or switchover.

4. Check Prerequisites

• The wizard runs the following checks:
  • Checking connectivity between appliances
  • Verifying that both appliances can be configured as a cluster
  • Verifying software version compatibility between appliances: both SDMs must run the same version.
• If a check fails, the message shows what's missing, for example an IP address that isn't in Allowed Hosts.
• When all checks pass, the message "All checks successfully passed. Click the "Next" button to proceed." appears.
• Click Next to continue.

5. Run the Configuration Process

• Click Start Configuration, then Confirm in the Start cluster configuration? dialog.

• While the configuration runs, the wizard shows Configuration in progress... with the percentage complete, and the RL Cluster log.

• Do not refresh or navigate away during setup; leaving the page interrupts the process and it cannot be reopened.

• Shortly after the configuration starts, the primary switches to the maintenance screen and stays unavailable until the configuration finishes. The secondary is unavailable as well.

• When setup completes, both managers reboot and return online automatically. Wait until both are reachable before proceeding.

• The maintenance screen doesn't update by itself. When the primary is reachable again, for example in a new browser tab, reload the wizard page to see the result.

  important

  When initiating redundancy on a Spectra Detect Manager with Central logging enabled and a large volume of stored sample data, the initial sync to the secondary manager may take longer than 30 minutes.

  If you refresh the screen after 30 minutes, you may see a "Configuration failed" status with a "Rollback Configuration" button, even though clustering is still in progress. Do not roll back while data is syncing.

  To verify sync progress, SSH to each SDM and run the top command. If the rsync process is using ~70% CPU or more, the sync is still in progress. Wait until the rsync process on both the primary and secondary is under ~10% CPU before refreshing the GUI.

• When the "Configuration finished successfully" message appears, click Finish.

6. Verify Status

• Open the Cluster Configuration tab to see which manager is primary and which is secondary.
• The secondary manager is read-only while redundancy is active.
• The Status tab provides redundancy health and logs.
• You can initiate a switchover, upgrade the cluster, or remove the cluster configuration here.
• Check the Manage and View the Status of Redundancy section for more details.

7. Update the Proxy or Load Balancer

• After the redundancy setup is complete, update the proxy or load balancer to include both manager IPs.
• Access the Spectra Detect GUI via the proxy or load balancer; it automatically directs users to the primary node.
• When you later remove redundancy, follow the guidance in the Post De-clustering Actions section below to keep only one manager active.

Manage and View the Status of Redundancy

Redundancy Status

Go to Administration > Redundancy. The page has the following tabs:

• Cluster Configuration shows the IP addresses of the primary and secondary SDM. It also has the Upgrade Cluster, Switchover, and Remove Cluster Configuration sections.
• Status shows the health of the cluster:
  • Cluster Status: the Cluster mode, for example OPERATIONAL, and the Overall status, for example OK.
  • Node Status: the Node mode, PRIMARY or SECONDARY, the IP, and the Version of each node.
  • A table that shows a status icon for the Cluster resource manager, Database, and VPN status of the primary and the secondary.
  • Logs: the RL Cluster, RL Daemon, and Corosync logs.

Managing Switchovers

1. Go to Administration > Redundancy > Cluster Configuration.
2. Click Manual Switchover to make the secondary manager primary.
3. In the Promote the Secondary appliance to Primary? dialog, click Confirm. The dialog notes that the switch can take some time, depending on the amount of data.
4. The system enters maintenance mode for several minutes.
5. When complete, the proxy or load balancer automatically directs traffic to the new primary.

Upgrading Redundant Managers

Use the Upgrade Cluster wizard to upgrade both SDMs in a redundant cluster. The wizard upgrades one node at a time: first the secondary, then a switchover that makes the upgraded secondary the new primary, then the new secondary. The primary keeps serving traffic while the secondary upgrades. While the secondary upgrades, automatic failover is off.

Known Issues

• After the wizard finishes, the cluster can stay in maintenance mode, and the secondary shows the maintenance page instead of the secondary node page.
• If you reload or close the wizard page after Upgrade complete. appears, the wizard no longer shows Finish, and the cluster stays in update mode with automatic failover off. Don't reload or close the page before you click Finish. If you did, send POST /api/v1/cluster/update/ with {"update_finished": true} to finish the upgrade.

Before each upgrade step, take a snapshot of the SDM you're about to upgrade, or back it up some other way.

To upgrade the cluster:

1. Go to Administration > Redundancy > Cluster Configuration and click Upgrade Cluster.
2. In the Upgrade Secondary step, choose the update package in one of the following ways:
   • Under Select Update File, click Choose File and select a .bin or .rlpm file.
   • Under Available Updates, click Download next to a version, wait for Download Complete, then select it. This list is only available when the appliance has access to Spectra Intelligence.
3. Click Start Upgrade, then Confirm. The secondary upgrades and restarts, and the RL Cluster log shows the progress. When Upgrade complete. appears, click Next.
4. In the Switchover step, click Initiate Switchover, then Confirm. The secondary becomes the primary, and your session is interrupted while the switchover runs. The page doesn't return to the wizard on its own. After the switchover finishes, reload the page, and the wizard resumes at the Upgrade New Secondary step. If you opened the wizard on the original primary's own address instead of the Application URL, open the new primary instead.
5. In the Upgrade New Secondary step, choose the update package again, then click Start Upgrade and Confirm.
6. When Upgrade complete. appears, click Finish to return to the cluster page.

To stop an upgrade that's in progress, click Cancel Upgrade, then Confirm. Partial changes on the secondary aren't rolled back automatically.

Removing Manager Redundancy

1. Go to Administration > Redundancy > Cluster Configuration.
2. Click Remove Cluster Configuration, then confirm. Data isn't wiped from the SDMs.
   • The system enters maintenance mode for a few minutes
   • Once complete, the primary manager retains its Application URL and Allowed Hosts configuration
   • The secondary manager's Application URL and Allowed Hosts are cleared, leaving it as a standalone SDM with no appliance connections
   • Review Post De-clustering Actions after this step to retire the unused manager

If the Secondary is unavailable during de-clustering

• You can still remove the cluster following the directions in Step 2 above. If the offline secondary manager was permanently removed or destroyed, no additional cleanup is required after Step 2.
• If you have access to the server, you can manually run the below command to clean up the de-clustered Spectra Detect Manager that was offline during the initial de-cluster action.

sudo /bin/cluster_manage destroy  

If the secondary comes back online after de-clustering, it must be manually de-clustered. Contact ReversingLabs Support for help.

Post De-clustering Actions

important

After de-clustering, shut down or remove one of the Spectra Detect Managers.

All connected appliances continue communicating through the proxy or load balancer, which remains configured to route to the active manager.

Remove the inactive or retired manager from the proxy or load balancer configuration to prevent connection attempts to a non-existent node.

If you plan to re-establish redundancy with a new secondary manager, repeat the steps in Creating Manager Redundancy.

warning

After de-clustering, confirm that the Application URL on the primary manager is correct before reconnecting any appliances. The primary retains its full configuration and continues to push the Application URL to all connected appliances (Hub, Workers, and Analyze). The secondary's Application URL and Allowed Hosts are automatically cleared during de-clustering, so it will no longer communicate with or affect connected appliances.

Resetting SDM Configuration

Using rlapp configure --reset

If SDM experiences deployment or redundancy issues that cannot be resolved through standard troubleshooting, use the rlapp configure --reset command to reset the base configuration.

danger

All SDM configuration and data will be lost. This is a destructive operation intended for recovery scenarios only. Contact ReversingLabs Support before using this command.

sudo rlapp configure --reset

Use this command in the following cases:

• After failed clustering or de-clustering operations
• When SDM configuration becomes corrupted
• To recover from deployment issues

This command resets the following:

• Databases: Postgres and Clickhouse
• Configuration files: clickhouse, postgres, rlapp, nginx, monit, snmp, rsyslog, etc.
• rlapp data directories: /data/rlapp-tmp, /data/rlapp-samples, /data/rlapp-backup, etc.