Redundancy System
Spectra Analyze > Administration > Configuration & Update > Redundancy System
Spectra Analyze allows administrators to set up two Spectra Analyze instances into a redundant cluster. The cluster provides switchover capabilities to ensure no data is lost during system upgrades or primary appliance failures.
Only administrators can access and use the redundancy system feature.
To create or manage a redundant cluster, on the primary appliance, go to Administration > Configuration & Update > Redundancy System.
This chapter describes redundancy on the same subnet. Spectra Analyze automatically supports redundancy on the same network and takes care of all the details, such as setting up a virtual (shared) IP address.
However, to set up a redundant system across the internet, users need to set up their own load balancer, for example, HAProxy. This load balancer routes requests to the secondary Spectra Analyze in case the primary one fails. In that case, the procedure to set up a redundant system is the same as in the remainder of this chapter, with the exception that the Floating IP Address is ignored.
How redundancy works on Spectra Analyze
A redundant cluster connects two Spectra Analyze appliances: one primary (master) and one secondary (standby). Data synchronizes from primary to secondary at regular intervals, ensuring both contain identical data. The secondary appliance serves as a replica of the primary, and seamlessly takes over if the primary fails.
In a cluster, the primary appliance handles tasks such as file uploads and analysis. The secondary receives all new samples and database updates from the primary but doesn't analyze or process files. When the primary appliance fails, all processing transfers to the secondary appliance through automatic switchover. Administrators can also perform manual switchovers and promote the secondary to primary at any point.
The cluster is controlled from the primary appliance. The administrator decides which of the two appliances to use as the primary, and performs all cluster configuration and management actions from it.
During the cluster configuration process, the primary appliance is accessible and can be used, while the secondary appliance is inaccessible and cannot be used.
Once the cluster is configured, the floating IP address should be used to access the cluster instead of individual IP addresses of the appliances.
- All data on the secondary appliance is erased during configuration. You must confirm the warning before proceeding.
- System performance may be impacted during configuration due to background processes.
- If cluster configuration is canceled or interrupted, the secondary appliance may become non-functional depending on the stage at which the interruption occurred. Administrators may attempt resetting it using
tcbase resetfrom the appliance console. - After the cluster has been configured, the initial synchronization between the primary and secondary appliances can take up to several hours depending on data volume, increasing network traffic due to data replication.
Prerequisites for setting up a redundant cluster
- The administrator must have access to both Spectra Analyze appliances so that they can log into the secondary appliance during the configuration process.
- The Spectra Analyze appliances must be running the same software version.
- The system specifications of the two Spectra Analyze appliances should be as similar as possible; especially the RAM and the storage size. Considerable differences in system specifications can cause synchronization issues.
- The link between the Spectra Analyze appliances must be at least 1 Gbps. Using a lower bandwidth link in situations with databases over 60 GB can cause database synchronization issues.
- Both appliances must be able to connect to each other via SSH.
- Host names and IP addresses must be defined for both appliances, as well as the shared (floating) IP address:
- Go to Administration > Configuration & Update > Configuration > General > Allowed Hosts.
- Add the host name and IP address of the primary and secondary appliance.
- Add the floating IP address.
- Perform this on the other appliance as well.
Creating a redundant cluster
Go to Administration > Configuration & Update > Redundancy System. If no previous cluster configuration exists, only creation options appear on this page.
Click Create redundant cluster, and follow the wizard to configure the cluster.
Under STEP 1 - Establish connection, provide the following information:
- Floating IP address: IP address for cluster access. Must be in the Allowed hosts list on both appliances to allow them to communicate. For more information, see Prerequisites for setting up a redundant cluster.
- Local IP address: IP address of the current (primary) appliance.
- Secondary Spectra Analyze IP address: IP address of the secondary appliance.
- Secondary Spectra Analyze URL: host name of the secondary appliance in the format http(s)://XX.XX.X.XXX.
- Secondary Spectra Analyze user: administrator username on the secondary appliance.
- Secondary Spectra Analyze password: administrator password on the secondary appliance.
- Disable TLS Certificate Sharing: machines in a redundant cluster typically share the same TLS certificate. Select this option to use individual certificates on the appliances.
After providing all information, click Next.
Under STEP 2 - Check prerequisites, the primary appliance checks prerequisites. If successful, click Next to proceed.
Under STEP 3 - Run the configuration process, click Start configuration to initiate cluster configuration. A confirmation dialog warns about data removal from the secondary appliance. The configuration wizard displays background activities during configuration. In the RL Cluster tab, click Toggle autoscroll on/off to control log scrolling.
When configuration completes successfully, click Finish to open the redundancy system page with cluster status information.
Redundant cluster status
To check cluster status, navigate to the floating IP address defined during cluster configuration and go to Administration > Configuration & Update > Redundancy System. The Cluster configuration section shows configured IP addresses and provides options for manual switchover or cluster configuration removal.
Under Status, you can see detailed information about each node and cluster component, and browse system logs related to the redundancy functionality.
The status of the whole cluster is indicated by one of the following modes.
| Cluster Mode | Description |
|---|---|
| No Cluster | The cluster has not been defined. |
| Negotiate | The secondary appliance is not synchronized with the primary appliance. Failover is not yet possible. The cluster needs to decide which appliance to set as the primary. |
| Syncing | The secondary appliance is in the process of synchronizing with the primary appliance. Failover is currently not possible. |
| Maintenance | Primary and secondary appliances are being upgraded to new versions. |
| Operational | The cluster is operational, and the secondary appliance is synchronized with the primary one. Failover is now possible. |
The status of individual nodes (appliances) in the cluster is indicated by the following modes.
| Node Mode | Description |
|---|---|
| Single | This node is not joined to any cluster. |
| Secondary Not Synced | This node is configured as the secondary, but data synchronization with the primary node has not started yet. |
| Syncing | This node is configured as the secondary. Data synchronization with the primary node is in progress. |
| Secondary | This node is configured as the secondary and is now fully synchronized. |
| Primary | This node is configured as the primary. |
Users can also check which appliance in the cluster is acting as the primary using the Redundant Status API.
Manual switchover
Once operational, the redundant cluster requires minimal interaction beyond status checks. Automatic switchover occurs when the primary appliance fails.
However, users can manually perform a switchover at any time if necessary. To do this, from the Spectra Analyze interface on the primary appliance's IP address, go to Administration > Configuration & Update > Redundancy System > Cluster configuration. Click Manual switchover and confirm.
Performing a manual switchover from the floating IP address of the redundancy cluster may cause issues.
A switchover promotes the secondary to primary appliance while configuring required services in the background. The duration of the process depends on system specifications and data volume. Any existing RabbitMQ file processing queues are not replicated to the secondary appliance.
Upgrading a redundant cluster
The redundancy system feature allows Spectra Analyze administrators to upgrade the appliances in the cluster without affecting the cluster configuration. Upgrades should be performed from the primary appliance.
To upgrade both appliances in the cluster, go to Administration > Configuration & Update > System Update on the primary appliance. Upload and apply the upgrade package to the primary appliance. After that, the upgrade package is uploaded to the secondary appliance.
While the primary appliance is being upgraded, it enters maintenance mode and background services are shut down. This includes services related to the redundancy functionality, which means that the data replication from the primary to the secondary appliance is stopped during the upgrade.
The redundancy functionality remains unavailable until the secondary appliance finishes upgrading. However, once the upgrade of the primary appliance is done, users can normally access and work on the primary Spectra Analyze appliance while the secondary is being upgraded.
If the primary appliance is connected to the ReversingLabs Spectra Detect Manager, it is also possible to upgrade it from the Spectra Detect Manager interface. Starting the appliance upgrade from Spectra Detect Manager automatically upgrades the entire cluster, including the secondary appliance.
Removing cluster configuration
If the cluster functionality is no longer needed, administrators can remove the cluster configuration.
Removing the cluster functionality requires a full reinstallation of the secondary appliance. For more information, see the Setup and initial configuration.
To dismantle the cluster, from the Spectra Analyze interface on the cluster's floating IP address, go to Administration > Configuration & Update > Redundancy System > Cluster configuration. Click Remove cluster configuration and confirm. Note that the process of removing the cluster configuration may take some time.
Appliances are removed from the cluster and the configured floating IP is deactivated. All primary appliance data is preserved, and the appliance returns to single-node operation mode where it can continue to process files.
Troubleshooting
The cluster may experience split-brain during switchover, where both nodes in the cluster act as the primary.
You can detect this under Administration > Configuration & Update > Redundancy System > Status, where the Component status table shows database issues on the secondary appliance. Other related cluster issues appear here, such as database connection failures or standby mode on the primary appliance.
To further troubleshoot and mitigate cluster configuration issues, contact ReversingLabs Support.