Skip to main content

Benchmark Tool

The hlquery benchmark tool measures indexing and query performance under configurable load profiles.

Overview​

The benchmark binary provides multiple test modes:

  • Standard benchmark: Create collections and insert documents with configurable parameters
  • Search benchmark: Test search performance on existing data
  • Flood mode: Continuous stress testing with random operations
  • Detailed benchmark: Comprehensive testing of all API routes
  • Fake mode: Insert sample data for quick testing

Installation​

The benchmark tool is built alongside the main HLQuery server. After building HLQuery, you'll find the benchmark executable in the run/bin/ directory:

./run/bin/hlquery-benchmark --help

Basic Usage​

Standard Benchmark​

Run a standard benchmark with default settings:

./run/bin/hlquery-benchmark --url http://localhost:9200

Customize the benchmark:

./run/bin/hlquery-benchmark \
  --url http://localhost:9200 \
  --collections 50 \
  --documents 10000 \
  --threads 8 \
  --batch-size 200

Search Benchmark​

Test search performance on existing collections:

./run/bin/hlquery-benchmark --url http://localhost:9200 --search

Flood Mode (Continuous Stress Testing)​

New in recent update: Flood mode now runs continuously until stopped with Ctrl+C, randomly creating collections and documents.

./run/bin/hlquery-benchmark --url http://localhost:9200 --flood

Flood mode features:

  • Continuous operation: Runs until you press Ctrl+C (like --nofork mode)
  • Random operations: Each worker thread randomly performs:
    • 40% probability: Create a new collection
    • 50% probability: Insert random documents into existing collections
    • 10% probability: Perform random searches
  • Real-time statistics: Prints stats every 10 seconds showing:
    • Runtime
    • Collections created
    • Documents inserted
    • Searches performed
    • Errors encountered
  • Thread-safe: Uses mutexes and atomic variables for safe concurrent operations
  • Graceful shutdown: Handles SIGINT/SIGTERM signals for clean termination

Example output:

----------------------------------------------------------------
FLOOD MODE: Continuous Stress Testing
----------------------------------------------------------------

This will continuously:
  - Randomly create collections
  - Randomly insert documents into collections
  - Randomly perform searches

Press Ctrl+C to stop...

Flood benchmark running... Press Ctrl+C to stop.

[Stats] Runtime: 10s | Collections: 15 | Documents: 234 | Searches: 3 | Errors: 0
[Stats] Runtime: 20s | Collections: 28 | Documents: 567 | Searches: 8 | Errors: 0
...

Fake Mode​

Insert sample data quickly:

./run/bin/hlquery-benchmark --url http://localhost:9200 --fake

This creates 630 records across 19 collections:

  • 100 fictional people profiles with names, biographies, occupations, interests, and U.S. demo locations
  • 100 recognizable U.S. university catalog records with meaningful IDs, city/state, broad institution type, and synthetic search topics
  • 100 synthetic anomaly records with structured operational fields
  • 50 meaningful records in each of the saas, finance, fashion, and ecommerce collections
  • 10 curated records in each additional sample collection, including food, stocks, music, science, books, movies, art, travel, sports, history, technology, and math
  • shared is_synthetic, data_notice, embedding (float[]), and location_name fields; wholly synthetic collections also include demo location (geo_point) values

Fake-mode records are designed for public demonstrations. People, generated organizations, artworks, companies, and incidents are explicitly fictional. University names and city/state locations are real catalog references, while their search-topic annotations are synthetic. No university ranking, score, enrollment count, or invented campus coordinate is included. Market records contain no live prices or recommendations.

Example university catalog search:

curl "http://localhost:9200/collections/universities/documents/search?q=computer%20science&sort_by=catalog_order:asc&limit=5"

Example vector search against fake data:

curl -X POST http://localhost:9200/collections/technology/vector_search \
  -H "Content-Type: application/json" \
  -d '{"field_name":"embedding","vector":[0.1,0.2,0.3,0.4],"topk":5,"metric_type":"cosine"}'

Example geo search against fake data:

curl "http://localhost:9200/collections/food/documents/search?q=*&filter_by=_geo_radius(location,40.7306,-73.9352,5km)&sort_by=_geo_distance(location,40.7306,-73.9352):asc"

Detailed Benchmark​

Run comprehensive testing of all API routes:

./run/bin/hlquery-benchmark --url http://localhost:9200 --detailed

Dump Collections​

List all collections and their documents:

./run/bin/hlquery-benchmark --url http://localhost:9200 --dump

Command Line Options​

Options:
  --url URL          Server URL (default: http://localhost:9200)
  --auth TOKEN       Authentication token
  --collections N    Number of collections to create (default: 2, max: 1000 without --flood)
  --documents N      Total number of documents to insert (default: 50000 per collection, max: 1000000 without --flood)
  --threads N        Number of threads (default: 8)
  --batch-size N     Documents per bulk insert batch (default: 500)
  --advanced [FILE]  Output detailed JSON metrics (default: adv.json)
  --detailed [FILE]  Run comprehensive benchmark testing ALL routes
                     and functionalities (includes --advanced)
  --search           Run search benchmark on previously inserted data
  --dump             Dump all collections and their documents
  --fake             Insert public-safe synthetic demo collections from run/benchmark/*.json
  --flood            Flood server with continuous random data generation for stress testing
                     (runs until stopped with Ctrl+C, randomly creates collections and documents)
  --id ID            Run UUID/ID for correlation (default: auto-generated timestamp)
  --seed SEED        Seed for deterministic runs
  --verify           Perform full deterministic document read-back and SHA-256 verification
  --verify-after-restart Verify counts after server restart (requires manual restart)
  --verify-final-counts  Verify benchmark collection counts before reporting success
  --check-consistency    Check consistency of /status, /stats, /metrics, /doctotal at end
  --cleanup          Delete all benchmark-tagged collections at end
  --durability-config PATH  Describe server durability settings from this file
  --log-file FILE    Structured log file (JSON lines format)
  --verbose, -v      Show detailed progress information
  --help, -h         Show this help message

Examples​

Quick Performance Test​

# Start server
./run/hlquery start --nofork &

# Run benchmark
./run/bin/hlquery-benchmark --url http://localhost:9200 --collections 10 --documents 1000 --threads 4

Stress Testing​

# Run continuous flood test
./run/bin/hlquery-benchmark --url http://localhost:9200 --flood --threads 4 --verbose

# Press Ctrl+C when done

Advanced Metrics​

# Generate detailed JSON metrics
./run/bin/hlquery-benchmark --url http://localhost:9200 --advanced metrics.json --collections 20 --documents 5000

Understanding Results​

Standard Benchmark Output​

The standard report separates:

  • Ingest: End-to-end client generation, JSON serialization, HTTP transfer, parsing, and batched storage writes
  • Durability sync: Counter verification plus the server's FlushAndSync
  • Ingest rate and durable rate: Documents per second before and after the durability barrier
  • Document data and ingest bandwidth: Logical field bytes, so runs with different payload sizes are not compared using document count alone
  • Conditions: Client/server versions, pre-existing aggregate document and physical RocksDB/SSTable baselines, and the indexing policy

The standard ingest benchmark does not claim that the lazy search index is fully built. First-search index construction is explicitly excluded and shown as such in the report. Use --search to measure queries against existing data. The benchmark also reports WAL/fsync settings as unknown unless --durability-config PATH is supplied; it does not guess the daemon's settings from an unrelated local config file.

Flood Mode Statistics​

Flood mode prints periodic statistics:

  • Runtime: Total elapsed time
  • Collections: Number of collections created
  • Documents: Total documents inserted
  • Searches: Number of searches performed
  • Errors: Error count

Advanced JSON Output​

When using --advanced, you get detailed metrics in JSON format including:

  • Per-operation timings
  • Success/failure rates
  • Throughput metrics
  • Error details

Advanced Features​

Run ID and Correlation​

The benchmark automatically generates a unique run ID (timestamp-based) for each run. You can also provide your own with --id:

./run/bin/hlquery-benchmark --id my-test-run-001 --collections 50 --documents 5000

The run ID is stored in the advanced JSON output and can be used to correlate benchmark runs with server logs.

Persistence and Verification​

The benchmark automatically waits for all ingest workers and then invokes an explicit RocksDB SyncWAL durability barrier. A non-2xx durability response fails the run instead of publishing a durable throughput number. Add --verify for a full deterministic read-back of every benchmark document, including its fields, payload hash, aggregate SHA-256 checksum, and duplicate/missing checks. Verification time is reported separately and excluded from ingest throughput. Use --verify-final-counts for the lighter per-collection counter comparison.

Use --check-consistency to verify that /status, /stats, /metrics, and /doctotal all report consistent counts:

./run/bin/hlquery-benchmark --check-consistency --collections 10 --documents 1000

Verifying After Restart​

Run one benchmark with restart verification enabled:

./run/bin/hlquery-benchmark \
  --verify-after-restart \
  --advanced results.json \
  --collections 50 \
  --documents 5000

After ingest and durability sync, the tool pauses and asks you to restart the server. Restart that same server, wait until it is healthy, and then press Enter so the benchmark can compare the post-restart counts.

Latency Percentiles​

The advanced JSON output includes latency percentiles (P50, P90, P99) for batch insertions and other operations, useful for regression testing and performance analysis.

Monotonic Checks in Flood Mode​

Flood mode now includes monotonic checks to ensure counters never decrease, helping detect data loss or counting issues during stress testing.

Best Practices​

  1. Start with small tests: Begin with fewer collections and documents to verify connectivity
  2. Use an isolated server for publishable numbers: Existing data, active indexing, compaction, and other clients can materially change throughput
  3. Record the versions and baseline: Do not compare runs with different client/server builds or pre-existing datasets as if they were equal
  4. Use appropriate thread counts: Useful write parallelism is bounded by the number of collections because each collection serializes its counter update
  5. Sweep batch sizes around the 2000-document default: Bigger is not always faster and can cross storage or WAL batch limits
  6. Test incrementally: Gradually increase load to find performance limits
  7. Use flood mode for stress testing: Use it to find breaking points and test stability
  8. Check server health: Ensure the server is healthy before and after benchmarks
  9. Use --check-consistency: Verify endpoint consistency after benchmarks
  10. Save advanced metrics: Use --advanced to save detailed metrics for analysis
  11. Limit defaults: Default collections/documents are limited to prevent accidental huge runs (use --flood to bypass)
  12. Clean up after tests: Use --cleanup to remove benchmark collections when done

Troubleshooting​

Connection Errors​

If you see connection errors:

  • Verify the server is running: ./run/hlquery status
  • Check the URL is correct
  • Ensure firewall rules allow connections
  • Check authentication token if required

Performance Issues​

If benchmarks are slow:

  • Compare the client and server versions printed under Conditions
  • Check the reported server baseline for pre-existing data
  • Check server CPU, memory, disk, indexing, and compaction activity
  • Test on an isolated instance before treating a number as a regression
  • Sweep thread and batch counts; do not assume more is faster
  • Check network latency and verify the daemon's actual durability configuration

Interrupting a Benchmark​

One Ctrl+C gracefully cancels active benchmark requests and exits with status 130. A second Ctrl+C is only an emergency forced exit. If the first interrupt does not stop promptly, verify that run/bin/hlquery-benchmark is the freshly installed binary.

Count Mismatches After Restart​

If document/collection counts don't match after restart:

  • Check that flush/sync completed successfully (look for "✓ Flush/sync completed" in output)
  • Verify WAL and SSTables are persisted correctly
  • Use --check-consistency to identify which endpoints disagree
  • Check server logs for errors during benchmark

Data Loss Warnings​

If you see "ERROR: Server reports X documents but benchmark inserted Y":

  • This indicates potential data loss or counting issues
  • Check server logs for errors
  • Verify disk space is available
  • Check for server crashes during benchmark
  • Use --check-consistency to verify endpoint consistency

Modes and Caveats​

Persistence Expectations​

The benchmark automatically synchronizes its WAL durability boundary at the end, but note:

  • WAL (Write-Ahead Log) is synced to disk
  • Memtable flushing is not included in the default throughput barrier
  • --verify performs full application-level read-back and checksum validation
  • --verify-final-counts checks the final per-collection counts
  • Use --verify-after-restart to explicitly test persistence

Fake Collections​

Fake collections are created only by --fake. They are separate from standard bench_* collections and do not count toward the --collections parameter.

Fake collections include is_synthetic, data_notice, embedding, location, and location_name fields. Use embedding for vector search and location for geo radius, box, and distance-sort tests. Do not present synthetic ranks, profiles, market scenarios, or incidents as real-world information.

The 50-item category collections are available at /collections/saas, /collections/finance, /collections/fashion, and /collections/ecommerce.

The source fixtures live in run/benchmark/. Regenerate them deterministically after editing the fixture definitions:

./tools/generate-benchmark-fixtures.py

Default Limits​

To protect against accidental huge runs, default limits are applied:

  • Maximum 1000 collections (without --flood)
  • Maximum 1,000,000 documents (without --flood)
  • Use --flood to bypass these limits for stress testing

See Also​