Spectra Detect Performance Testing
This page presents the results of large-file capacity testing of the Spectra Detect Worker. The tests measured how large a single submission a Worker can process end to end, and what limits that size in practice.
The short answer is that the limit depends on what's inside the file, not on the file size alone. No product-imposed file size limit was found in the tested range. What decides whether a large submission completes is the working storage, the memory, and the processing time that its content needs.
Test Setup
All tests used a standalone Spectra Detect Worker with the default processing configuration. Two hardware configurations were used:
| Test system | vCPU | RAM | Working storage | Saves unpacked files to external storage |
|---|---|---|---|---|
| System A | 32 | 128 GB | 1.5 TB | Yes |
| System B | 32 | 192 GB | 5.8 TB | No (default) |
System A is a Worker Profile 3 class system. System B is the same server after a memory and storage upgrade.
Other conditions that apply to every run:
- Files were submitted through the standard REST API. No special endpoint or client was used.
- Files larger than 1 GB were handled by the Worker's dedicated large-file processing path.
- The Processing Timeout was left at its default of 8 hours per file.
- Each run was a single submission on an otherwise idle Worker.
How the Tests Were Run
All samples were real, publicly available data: open-weight AI model files and published training datasets. No synthetic filler was used, so the results reflect the kind of content a customer would submit.
Archives were packaged as uncompressed TAR files. The Worker ships with decompression-bomb protection, and a compressed archive would have mixed that protection into the file size measurement. An uncompressed TAR still presents the Worker with a single file of the stated size.
Packaging everything into one archive was only a way to measure the largest single submission. Normally, submit a model's or dataset's files individually, as their publishers distribute them (about 5 GB each); for compressed training data, this also keeps the memory use and processing time per submission lower.
A run counted as passed only when the whole pipeline completed:
- The upload was accepted.
- The content was fully extracted and analyzed.
- The analysis report was retrieved.
- The SHA-256 hash in the report matched the submitted file.
An accepted upload alone never counted as a pass. Sizes were increased step by step, the Worker was reset to a clean state between runs, and disk, memory, and processing time were recorded throughout.
Results
The tests covered three types of submissions, because they behave differently:
- Model archives: many model files in one archive. The Worker extracts the archive and analyzes every file in it.
- Single model files: one large model file. There's nothing to extract, so the Worker analyzes the file as it is.
- Compressed training-data archives: compressed text datasets in one archive. Their content expands during analysis.
Model Archives
Archives of open-weight large language model files in safetensors format. The largest archive based on a single model held a 671-billion-parameter model (163 files). The 999.9 GB and 1,604.2 GB archives were merged sets of files from several models.
| Archive size | Test system | Processing time | Result |
|---|---|---|---|
| 24.3 GB | A | 15 min | Passed |
| 48.6 GB | A | 27 min | Passed |
| 102 GB | A | 61 min | Passed |
| 205.7 GB | A | 100 min | Passed |
| 401.4 GB | A | 142 min | Passed |
| 23.8 GB | B | 12 min | Passed |
| 688.6 GB | B | 211 min | Passed |
| 999.9 GB | B | 305 min | Passed |
| 1,604.2 GB | B | — | Didn't complete (stopped at the 8-hour processing timeout) |
Model archives passed at every size up to 999.9 GB, with no memory pressure. The 1,604.2 GB archive uploaded normally but was stopped by the default 8-hour processing timeout before analysis finished, with resources nowhere near exhausted.
Single Model Files
Single model files in GGUF format. The 49.6 GB file is a genuine model file, exactly as published. Model publishers split anything larger into smaller parts, so the larger test files were assembled from real model data.
| File size | Test system | Processing time | Result |
|---|---|---|---|
| 49.6 GB | A | 7.4 min | Passed |
| 99.2 GB | A | 14 min | Passed |
| 198.1 GB | A | 27 min | Passed |
| 439.2 GB | B | 58 min | Passed |
| 713.3 GB | B | 95 min | Passed |
Single files are the cheapest content per GB at every size tested, with under 10 GB of memory used.
Compressed Training-Data Archives
Archives of web-crawl text datasets in Parquet format with Snappy compression, exactly as the datasets are published.
The memory results in this section were measured with Spectra Core 5.5 and earlier. Spectra Core 5.6.0 optimizes memory use for this content, so these memory figures don't apply to Workers running Spectra Core 5.6.0 or later.
| Archive size | Test system | Processing time | Result |
|---|---|---|---|
| 50 GB | A | 38 min | Passed |
| 75 GB | A | — | Didn't complete (memory) |
| 100.6 GB | A | — | Didn't complete (memory) |
| 203.2 GB | A | — | Didn't complete (working storage and memory) |
| 75.5 GB | B | 39 min | Passed |
| 100.6 GB | B | 51 min | Passed |
| 125.7 GB | B | 63 min | Passed |
| 203.2 GB | B | 103 min | Passed |
| 400.3 GB | B | 201 min | Passed |
| 601.9 GB | B | 305 min | Passed |
| 800.5 GB | B | 402 min | Passed |
| 1,001.9 GB | B | — | Didn't complete (stopped at the 8-hour processing timeout) |
For this content type, memory decides pass or fail. From about 75 GB upward, analysis reaches a peak of about 112–132 GB of RAM per archive, nearly independent of the archive size. Smaller archives peaked lower: 81 GB for the 50 GB archive and 67 GB for a 43.5 GB archive. On System A, with 128 GB of RAM, a 50 GB archive passed, while the 75, 100.6, and 203.2 GB archives were stopped as memory approached the system's total. On System B, with 192 GB of RAM, the same archives passed, and so did archives up to 800.5 GB.
A control test submitted the same data once as Snappy-compressed Parquet (43.5 GB) and once as uncompressed Parquet (74.4 GB). The memory peak followed the size of the submitted archive, not the size of the decompressed data, so the guidance below applies to the archive size regardless of Parquet compression.
What Governs Large Submissions
File size. No product-imposed file size limit was found in the tested range. Uploads were accepted at every size tested, up to 1.6 TB.
Working storage. An archive needs about twice its size in free working storage during analysis with the default configuration, for both model content and compressed training data. For example, a 700 GB model archive needs about 1.4 TB of free working storage, and a 1 TB archive about 2 TB.
When the Worker saves unpacked files to external storage, model archives need about three times their size. Compressed training data can expand to about seven times its size or more, and wasn't bounded in testing.
Memory. Model content, whether in an archive or as a single file, needs modest memory. In testing, available memory dropped by up to about 15 GB, on a 401.4 GB model archive. With Spectra Core 5.5 and earlier, compressed training data peaks at about 112–132 GB of RAM per archive from about 75 GB upward, nearly independent of archive size. Smaller archives peak lower. A 128 GB system supported 50 GB per archive, and a 192 GB system was validated to 800 GB per archive.
Processing time. Processing time is predictable:
| Content type | Approximate processing time |
|---|---|
| Single model files | About 8 seconds per GB |
| Model archives | About 30–38 seconds per GB up to about 200 GB, about 18 seconds per GB from about 700 GB |
| Compressed training data | About 30 seconds per GB |
At very large sizes, the default 8-hour per-file processing timeout becomes the governing limit. At the measured rates, it caps model archives at roughly 1.57 TB and compressed training data at roughly 950 GB. The timeout is configurable, with no maximum, so it can be raised for larger submissions.
Large files are processed one at a time by a single process. A large file that takes 10 hours to analyze holds every other large file in the queue for those 10 hours.
Expected Behavior With Very Large Files
- Fuzzy hashes are omitted for very large files. If SSDEEP or TLSH is enabled, the hash isn't calculated for files larger than about 700 MB (SSDEEP) or 3.9 GB (TLSH).
- YARA results can be partial for very large files inside an archive. YARA scanning of an individual file of about 12 GB or more inside an archive can reach its per-file scan timeout. The submission still completes, with partial YARA results for that file.
- Large files get exactly one processing attempt. Files on the large-file path aren't retried if processing fails. A failed file is counted in
file_processing_failurein the file processing statistics. Its report request still returns HTTP 200, with an empty report (tc_reportisnull), so an HTTP 200 alone doesn't mean the analysis succeeded. To try again, resubmit the file, for example after raising the processing timeout. - Retrieve reports promptly. Reports are removed after the Task age limit (default 90 minutes).
- Upload time comes on top of processing time. The processing times on this page cover analysis only. Uploads ran at about 180–200 MB/s in testing, so a 1 TB upload takes about 90 minutes.
- Reports for compressed training data are large. The 800.5 GB archive produced a 2.3 GB report, and a 578 MB report took 36 minutes to download. Reports for model archives stay under 2 MB.