# Spectra Detect Redundancy — Manager Clustering and Failover

> Configure Spectra Detect Manager redundancy with active-passive clustering, automatic failover, and load balancer integration for high availability deployments.

Spectra appliances are designed for full redundancy. If one component fails, another automatically takes over using the same configuration settings. 

This chapter describes how redundancy works for **Spectra Detect Manager**.

## Creating Manager Redundancy

**Important: A proxy or Load balancer is REQUIRED for Spectra Detect Manager to function properly in Redundancy.**


Use the following endpoint to identify the active/primary Spectra Detect Manager:

`GET /api/cluster/check/`

Before clustering, add the active/primary manager (the one you will use to initiate redundancy) to the proxy or load balancer. After redundancy setup completes, update the proxy to include both managers (this is described in [step 7](#7-update-the-proxy-or-load-balancer)).

### 1. Configure Allowed Hosts

As admin, go to [Administration > Spectra Detect Manager > General > Network Settings](../Admin/ManagerSettings.md#general).

- Verify that the **Application URL** is set to the proxy IP or domain.  
- In **Allowed Hosts**, ensure the following are listed:  
   1. Current Detect Manager IP address and domain  
   2. Proxy IP and domain  
   3. Secondary IP and domain  
   4. localhost and 127.0.0.1
   ![Allowed Hosts](./images/redundancy_1.png)
- Click **Save** on the primary manager.  
- Repeat for the secondary manager, but **only update the Allowed Hosts section**. Do not modify the Application URL on the secondary manager. It will be automatically replaced during clustering.

**Note:** Changing the Application URL updates the configuration on all connected appliances (Hub, Workers, and Analyze).  

### 2. Create Redundant Cluster

Go to **Administration > Redundancy**, then click **Create redundant cluster**.

![Create Redundant Cluster](./images/redundancy_2.png)

### 3. Establish Connection  

![Establish Connection](./images/redundancy_3.png)

- Fill out the required fields.  
- A VPN is automatically established between redundant managers.  
- The user on the secondary manager must be an admin.  
- Set **Failover Timeout (seconds)** between 3 and 600; 30 is recommended.  
- Usually, the machines in a redundant cluster share the same TLS certificate. However, if you need individual TLS certificates for each machine, you can *Disable TLS Certificate Sharing*.
- Click **Next**.

### 4. Check Prerequisites  

![Check Prerequisites](./images/redundancy_4.png)

- The system validates if your environment supports redundancy.  
- When all checks pass, the message "All checks successfully passed" appears.  
- Click **Next** to continue.

### 5. Run the Configuration Process  

![Run the Configuration Process](./images/redundancy_5.png)

- Click **Start Configuration**, then confirm in the popup.
- **Do not refresh or navigate away** during setup; leaving the page interrupts the process and it cannot be reopened.
- The system automatically switches to the maintenance screen while redundancy is configured.
- When setup completes, both managers reboot and return online automatically. Wait until both are reachable before proceeding.

   **Important: When initiating redundancy on a Spectra Detect Manager with Central logging enabled and a large volume of stored sample data, the initial sync to the secondary manager may take longer than 30 minutes.**
If you refresh the screen after 30 minutes, you may see a "Configuration failed" status with a "Rollback Configuration" button, even though clustering is still in progress. Do not roll back while data is syncing.

   To verify sync progress, SSH to each SDM and run the `top` command. If the `rsync` process is using ~70% CPU or more, the sync is still in progress. Wait until the `rsync` process on both the primary and secondary is under ~10% CPU before refreshing the GUI.

- When the "Configuration finished successfully" message appears, click **Finish**.

![Finish the Configuration Process](./images/redundancy_6.png)

### 6. Verify Status

- Open the **Cluster Configuration** tab to see which manager is primary and which is secondary.
- The secondary manager is read-only while redundancy is active.
- The **Status** tab provides redundancy health and logs.
- You can initiate a switchover or remove the cluster configuration here.
- Check the [Manage and View the Status of Redundancy](#manage-and-view-the-status-of-redundancy) section for more details.

### 7. Update the Proxy or Load Balancer

- After the redundancy setup is complete, update the proxy or load balancer to include both manager IPs.
- Access the Spectra Detect GUI via the proxy or load balancer; it automatically directs users to the primary node.
- When you later remove redundancy, follow the guidance in the [Post De-clustering Actions](#post-de-clustering-actions) section below to keep only one manager active.

## Manage and View the Status of Redundancy

### Redundancy Status

Go to **Administration > Redundancy > Cluster Configuration**.  

![Cluster Configuration](./images/redundancy_7.png)

- The **Cluster Configuration** tab shows which manager is primary or secondary.  
- Click the **Status** tab for detailed health checks and logs.  
- Three log views are available: RL Cluster, RL Daemon, and Corosync.
   
![Cluster Status](./images/redundancy_8.png)

### Managing Switchovers

![Manual Switchover](./images/redundancy_9.png)

1. Go to **Administration > Redundancy > Cluster Configuration**.  
2. Click **Manual Switchover** to make the secondary manager primary.  
3. Confirm the action when prompted.  
![Manual Switchover](./images/redundancy_10.png)
4. The system enters maintenance mode for several minutes.  
5. When complete, the proxy or load balancer automatically directs traffic to the new primary.

## Removing Manager Redundancy

1. Go to **Administration > Redundancy > Cluster Configuration**.  
![Remove Cluster Configuration](./images/redundancy_11.png)
2. Click **Remove Cluster Configuration**, then confirm  
   - If the Application URL was set to the proxy IP or domain, no changes are sent to the connected appliances
   - The system enters maintenance mode for a few minutes
   - Once complete, you will have two independent Spectra Detect Managers. Review [Post De-clustering Actions](#post-de-clustering-actions) after this step to retire the unused manager.  

### If the Secondary is unavailable during de-clustering

- You can still remove the cluster following the directions in Step 2 above. If the offline secondary manager was permanently removed or destroyed, no additional cleanup is required after Step 2.
- If you have access to the server, you can manually run the below command to clean up the de-clustered Spectra Detect Manager that was offline during the initial de-cluster action.  

```bash
sudo /bin/cluster_manage destroy  
```

If the secondary comes back online after de-clustering, it must be manually de-clustered. Contact [ReversingLabs Support](mailto:support@reversinglabs.com) for help.

## Post De-clustering Actions

**Important: **After de-clustering, shut down or remove one of the Spectra Detect Managers.****
All connected appliances continue communicating through the proxy or load balancer, which remains configured to route to the active manager.

Remove the inactive or retired manager from the proxy or load balancer configuration to prevent connection attempts to a non-existent node.

If you plan to re-establish redundancy with a new secondary manager, repeat the steps in [Creating Manager Redundancy](#creating-manager-redundancy).
