Skip to main content

Benchmark Tool

The hlquery benchmark tool measures indexing and query performance under configurable load profiles.

Overview​

The benchmark binary provides multiple test modes:

  • Standard benchmark: Create collections and insert documents with configurable parameters
  • Search benchmark: Test search performance on existing data
  • Flood mode: Continuous stress testing with random operations
  • Detailed benchmark: Comprehensive testing of all API routes
  • Fake mode: Insert sample data for quick testing

Installation​

The benchmark tool is built alongside the main HLQuery server. After building HLQuery, you'll find the benchmark executable in the run/bin/ directory:

./run/bin/benchmark --help

Basic Usage​

Standard Benchmark​

Run a standard benchmark with default settings:

./run/bin/benchmark --url http://localhost:9200

Customize the benchmark:

./run/bin/benchmark \
  --url http://localhost:9200 \
  --collections 50 \
  --documents 10000 \
  --threads 8 \
  --batch-size 200

Search Benchmark​

Test search performance on existing collections:

./run/bin/benchmark --url http://localhost:9200 --search

Flood Mode (Continuous Stress Testing)​

New in recent update: Flood mode now runs continuously until stopped with Ctrl+C, randomly creating collections and documents.

./run/bin/benchmark --url http://localhost:9200 --flood

Flood mode features:

  • Continuous operation: Runs until you press Ctrl+C (like --nofork mode)
  • Random operations: Each worker thread randomly performs:
    • 40% probability: Create a new collection
    • 50% probability: Insert random documents into existing collections
    • 10% probability: Perform random searches
  • Real-time statistics: Prints stats every 10 seconds showing:
    • Runtime
    • Collections created
    • Documents inserted
    • Searches performed
    • Errors encountered
  • Thread-safe: Uses mutexes and atomic variables for safe concurrent operations
  • Graceful shutdown: Handles SIGINT/SIGTERM signals for clean termination

Example output:

----------------------------------------------------------------
FLOOD MODE: Continuous Stress Testing
----------------------------------------------------------------

This will continuously:
  - Randomly create collections
  - Randomly insert documents into collections
  - Randomly perform searches

Press Ctrl+C to stop...

Flood benchmark running... Press Ctrl+C to stop.

[Stats] Runtime: 10s | Collections: 15 | Documents: 234 | Searches: 3 | Errors: 0
[Stats] Runtime: 20s | Collections: 28 | Documents: 567 | Searches: 8 | Errors: 0
...

Fake Mode​

Insert sample data quickly:

./run/bin/benchmark --url http://localhost:9200 --fake

This creates:

  • 100 synthetic people profiles with first, middle, and last names plus short biographies
  • 100 ranked university profiles
  • 50 documents in each of the saas, finance, fashion, and ecommerce collections
  • 10 documents for each additional sample collection, including food, stocks, music, science, books, movies, art, travel, sports, history, technology, and math
  • shared embedding (float[]) and location (geo_point) fields on every fake collection, so vector and geo search can be tested immediately

Example vector search against fake data:

curl -X POST http://localhost:9200/collections/technology/vector_search \
  -H "Content-Type: application/json" \
  -d '{"field_name":"embedding","vector":[0.1,0.2,0.3,0.4],"topk":5,"metric_type":"cosine"}'

Example geo search against fake data:

curl "http://localhost:9200/collections/food/documents/search?q=*&filter_by=_geo_radius(location,40.7306,-73.9352,5km)&sort_by=_geo_distance(location,40.7306,-73.9352):asc"

Detailed Benchmark​

Run comprehensive testing of all API routes:

./run/bin/benchmark --url http://localhost:9200 --detailed

Dump Collections​

List all collections and their documents:

./run/bin/benchmark --url http://localhost:9200 --dump

Command Line Options​

Options:
  --url URL          Server URL (default: http://localhost:9200)
  --auth TOKEN       Authentication token
  --collections N    Number of collections to create (default: 2, max: 1000 without --flood)
  --documents N      Total number of documents to insert (default: 50000 per collection, max: 1000000 without --flood)
  --threads N        Number of threads (default: 8)
  --batch-size N     Documents per bulk insert batch (default: 50)
  --advanced [FILE]  Output detailed JSON metrics (default: adv.json)
  --detailed [FILE]  Run comprehensive benchmark testing ALL routes
                     and functionalities (includes --advanced)
  --search           Run search benchmark on previously inserted data
  --dump             Dump all collections and their documents
  --fake             Insert realistic sample collections, including 50-item SaaS, finance, fashion, and ecommerce collections
  --flood            Flood server with continuous random data generation for stress testing
                     (runs until stopped with Ctrl+C, randomly creates collections and documents)
  --id ID            Run UUID/ID for correlation (default: auto-generated timestamp)
  --seed SEED        Seed for deterministic runs
  --no-fake-collections  Disable fake helper collections (products/foods/cart/items)
  --verify-after-restart Verify counts after server restart (requires manual restart)
  --check-consistency    Check consistency of /status, /stats, /metrics, /doctotal at end
  --dry-run          Generate collections/docs in memory but don't send to server
  --cleanup          Delete all benchmark-tagged collections at end
  --log-file FILE    Structured log file (JSON lines format)
  --verbose, -v      Show detailed progress information
  --help, -h         Show this help message

Examples​

Quick Performance Test​

# Start server
./run/hlquery start --nofork &

# Run benchmark
./run/bin/benchmark --url http://localhost:9200 --collections 10 --documents 1000 --threads 4

Stress Testing​

# Run continuous flood test
./run/bin/benchmark --url http://localhost:9200 --flood --threads 4 --verbose

# Press Ctrl+C when done

Advanced Metrics​

# Generate detailed JSON metrics
./run/bin/benchmark --url http://localhost:9200 --advanced metrics.json --collections 20 --documents 5000

Understanding Results​

Standard Benchmark Output​

The benchmark provides:

  • Collection creation time: How long it took to create all collections
  • Document insertion time: Time to insert all documents
  • Throughput: Documents per second
  • Error count: Number of errors encountered

Flood Mode Statistics​

Flood mode prints periodic statistics:

  • Runtime: Total elapsed time
  • Collections: Number of collections created
  • Documents: Total documents inserted
  • Searches: Number of searches performed
  • Errors: Error count

Advanced JSON Output​

When using --advanced, you get detailed metrics in JSON format including:

  • Per-operation timings
  • Success/failure rates
  • Throughput metrics
  • Error details

Advanced Features​

Run ID and Correlation​

The benchmark automatically generates a unique run ID (timestamp-based) for each run. You can also provide your own with --id:

./run/bin/benchmark --id my-test-run-001 --collections 50 --documents 5000

The run ID is stored in the advanced JSON output and can be used to correlate benchmark runs with server logs.

Persistence and Verification​

The benchmark automatically:

  • Flushes all pending HTTP requests before exiting
  • Calls flush/sync endpoints to ensure data persistence
  • Records final collection/document counts from the server
  • Compares server-reported counts with benchmark-inserted counts

Use --check-consistency to verify that /status, /stats, /metrics, and /doctotal all report consistent counts:

./run/bin/benchmark --check-consistency --collections 10 --documents 1000

Verifying After Restart​

To verify that data persists correctly after a server restart:

  1. Run benchmark with --advanced to save final counts:
./run/bin/benchmark --advanced results.json --collections 50 --documents 5000

  1. Restart the server manually

  2. Run with --verify-after-restart to compare counts:

./run/bin/benchmark --verify-after-restart --advanced results.json

Latency Percentiles​

The advanced JSON output includes latency percentiles (P50, P90, P99) for batch insertions and other operations, useful for regression testing and performance analysis.

Monotonic Checks in Flood Mode​

Flood mode now includes monotonic checks to ensure counters never decrease, helping detect data loss or counting issues during stress testing.

Best Practices​

  1. Start with small tests: Begin with fewer collections and documents to verify connectivity
  2. Monitor server resources: Watch CPU, memory, and disk usage during benchmarks
  3. Use appropriate thread counts: Too many threads can overwhelm the server
  4. Test incrementally: Gradually increase load to find performance limits
  5. Use flood mode for stress testing: Perfect for finding breaking points and testing stability
  6. Check server health: Ensure the server is healthy before and after benchmarks
  7. Use --check-consistency: Verify endpoint consistency after benchmarks
  8. Save advanced metrics: Use --advanced to save detailed metrics for analysis
  9. Limit defaults: Default collections/documents are limited to prevent accidental huge runs (use --flood to bypass)
  10. Clean up after tests: Use --cleanup to remove benchmark collections when done

Troubleshooting​

Connection Errors​

If you see connection errors:

  • Verify the server is running: ./run/hlquery status
  • Check the URL is correct
  • Ensure firewall rules allow connections
  • Check authentication token if required

Performance Issues​

If benchmarks are slow:

  • Check server resource usage
  • Reduce thread count
  • Increase batch size for bulk operations
  • Check network latency
  • Verify server configuration

Flood Mode Not Stopping​

Flood mode should stop on Ctrl+C. If it doesn't:

  • Check for signal handling issues
  • Verify you're using the latest version
  • Try sending SIGTERM: kill -TERM <pid>

Count Mismatches After Restart​

If document/collection counts don't match after restart:

  • Check that flush/sync completed successfully (look for "✓ Flush/sync completed" in output)
  • Verify WAL and SSTables are persisted correctly
  • Use --check-consistency to identify which endpoints disagree
  • Check server logs for errors during benchmark

Data Loss Warnings​

If you see "ERROR: Server reports X documents but benchmark inserted Y":

  • This indicates potential data loss or counting issues
  • Check server logs for errors
  • Verify disk space is available
  • Check for server crashes during benchmark
  • Use --check-consistency to verify endpoint consistency

Modes and Caveats​

Persistence Expectations​

The benchmark automatically flushes and syncs data at the end, but note:

  • WAL (Write-Ahead Log) is synced to disk
  • SSTables are flushed from memtable
  • Final counts are recorded for verification
  • Use --verify-after-restart to explicitly test persistence

Fake Collections​

By default, the benchmark creates helper collections (products, foods, cart, items) unless --no-fake-collections is used. These collections:

  • Are separate from benchmark collections
  • Don't count toward --collections parameter
  • Can be excluded with --no-fake-collections to avoid confusion in doctotal/hlquery-cli cols results

Fake collections include embedding, location, and location_name fields. Use embedding for vector search and location for geo radius, box, and distance-sort tests.

The 50-item category collections are available at /collections/saas, /collections/finance, /collections/fashion, and /collections/ecommerce.

Default Limits​

To protect against accidental huge runs, default limits are applied:

  • Maximum 1000 collections (without --flood)
  • Maximum 1,000,000 documents (without --flood)
  • Use --flood to bypass these limits for stress testing

See Also​