Skip to main content

Server Services and Routines

This page explains the internal services, routines, and background processes that run inside the hlquery server.

Architecture Overview​

hlquery is built on an event-driven architecture with the following core components:

  • HTTP Server: Handles incoming HTTP requests
  • Socket Engine: Manages network I/O using epoll/poll
  • RocksDB Storage Engine: Provides persistent storage using RocksDB
  • Search API: Processes search queries and document operations
  • Thread Pool Manager: Manages worker threads for concurrent processing
  • Timer Manager: Handles periodic tasks and scheduled operations
  • Log Manager: Centralized logging system

Main Event Loop​

The server's main event loop (hlquery::Run()) orchestrates all system operations:

Loop Structure​

while (true) {
    // 1. Check shutdown flags
    if (force_exit || shutting_down) break;
    
    // 2. Periodic database flush (once per second)
    if (now != oldtime) {
        Database->Flush();
    }
    
    // 3. Network event processing
    SocketEngine::DispatchEvents();
    
    // 4. Pending write processing
    SocketEngine::DispatchTrialWrites();
    
    // 5. Deferred action processing
    ActionList::ProcessActions();
    
    // 6. Lazy operations
    DaemonHandler::ProcessLazyOperations();
    
    // 7. Timer processing
    Timers->Tick();
    
    // 8. Socket engine optimization
    DaemonHandler::ProcessSocketEngineOptimization();
}

Core Routines​

1. Network Event Processing​

Function: SocketEngine::DispatchEvents()

Processes all pending I/O events from the socket engine (epoll/poll). This handles:

  • Incoming HTTP requests
  • Connection management
  • Network I/O operations
  • Client disconnections

Frequency: Every event loop iteration

Performance: Uses adaptive timeout based on pending work:

  • Non-blocking (timeout=0) when pending work exists
  • Blocking (timeout=-1) when idle for maximum efficiency

2. Pending Write Processing​

Function: SocketEngine::DispatchTrialWrites()

Attempts immediate writes for sockets that are ready, reducing latency for response delivery.

Frequency: Every event loop iteration

Purpose: Optimize response delivery by writing data as soon as sockets are writable

3. Deferred Action Processing​

Function: ActionList::ProcessActions()

Executes queued operations that were deferred to avoid blocking the event loop. This includes:

  • Expensive database operations
  • Background indexing tasks
  • Metadata updates

Frequency: Every event loop iteration

Purpose: Prevent blocking operations from affecting request latency

4. Lazy Operations​

Function: DaemonHandler::ProcessLazyOperations()

Handles expensive operations that can be processed incrementally during idle periods:

  • Background compaction
  • Index optimization
  • Cache warming
  • Statistics collection

Frequency: Every event loop iteration (with internal throttling)

Purpose: Utilize idle CPU cycles for maintenance tasks

5. Timer Processing​

Function: Timers->Tick()

Executes scheduled periodic tasks:

  • Database flushes
  • Health checks
  • Statistics collection
  • Cache cleanup
  • Connection timeout handling

Frequency: Every event loop iteration (timers fire based on their schedule)

Purpose: Handle time-based operations and scheduled maintenance

6. Socket Engine Optimization​

Function: DaemonHandler::ProcessSocketEngineOptimization()

Adjusts system behavior based on load:

  • Adaptive sleep intervals
  • Throughput optimization
  • Connection pool sizing
  • I/O buffer tuning

Frequency: Every event loop iteration

Purpose: Optimize performance based on current system load

7. Database Flush​

Function: Database->Flush()

Periodically flushes in-memory data to disk for durability.

Frequency: Once per second (when system time changes)

Purpose: Ensure data durability and prevent data loss

HTTP Server Services​

Request Processing Pipeline​

  1. Connection Acceptance: New connections are accepted by the HTTP server
  2. Request Parsing: HTTP requests are parsed into HttpRequest structures
  3. Route Resolution: Routes are resolved using ResolveHttpRoute() function
  4. Authentication: API key validation (if enabled)
  5. Handler Execution: Appropriate handler function is called
  6. Response Generation: HttpResponse is generated
  7. Response Sending: Response is sent back to client

Route Handlers​

All route handlers are implemented in the SearchAPI class:

Collection Management​

  • HandleCreateCollection() - Create new collections
  • HandleDeleteCollection() - Delete collections
  • HandleListCollections() - List all collections
  • HandleGetCollection() - Get collection details
  • HandleUpdateCollection() - Update collection schema

Document Management​

  • HandleAddDocument() - Add documents
  • HandleUpdateDocument() - Update documents
  • HandleDeleteDocument() - Delete documents
  • HandleGetDocument() - Retrieve documents
  • HandleListDocuments() - List documents in collection
  • HandleBulkImportDocuments() - Bulk import documents
  • HandleDeleteDocumentsByFilter() - Delete documents by filter

Search Operations​

  • HandleSearch() - Full-text search
  • HandleVectorSearch() - Vector similarity search
  • HandleMultiSearch() - Multi-collection search
  • HandleFacetCounts() - Faceted search
  • HandleExportDocuments() - Document export
  • HandleAutocomplete() - Autocomplete suggestions (planned)
  • HandleSpellcheck() - Spell checking (planned)
  • HandleAnalyticsClick() - Click analytics (planned)

Configuration Management​

  • HandleListSynonyms() - List synonyms in collection
  • HandleListAllSynonyms() - List all synonyms (global endpoint)
  • HandleCreateOrUpdateSynonym() - Manage synonyms
  • HandleGetSynonym() - Get synonym details
  • HandleDeleteSynonym() - Delete synonyms
  • HandleListStopwords() - List stopwords in collection
  • HandleListAllStopwords() - List all stopwords (global endpoint)
  • HandleCreateStopword() - Add stopwords
  • HandleDeleteStopword() - Delete stopwords
  • HandleListOverrides() - List search overrides
  • HandleCreateOrUpdateOverride() - Manage overrides
  • HandleGetOverride() - Get override details
  • HandleDeleteOverride() - Delete overrides
  • HandleListAliases() - List aliases
  • HandleCreateOrUpdateAlias() - Manage aliases
  • HandleGetAlias() - Get alias details
  • HandleDeleteAlias() - Delete aliases

System Endpoints​

  • HandleHealth() - Health check
  • HandleStatus() - Detailed status
  • HandleStats() - Server statistics
  • HandleMetrics() - JSON metrics (currently minimal)
  • HandleConnections() - Connection count
  • HandleRocksDB() - RocksDB engine stats
  • HandleDocTotal() - Document counts
  • HandleUpdateCounters() - Update counters

Thread Pool Services​

Thread Pool Manager​

Class: hlquery_threadpool::ThreadPoolManager

Manages multiple specialized thread pools:

HTTP Thread Pool​

  • Handles HTTP request processing
  • Processes search queries
  • Manages document operations

Database Thread Pool​

  • Handles database I/O operations
  • Processes RocksDB compaction

Search Thread Pool​

  • Processes search queries
  • Handles indexing operations
  • Manages vector operations

Tuning tip: If searches feel slow even when CPU is idle, confirm the target collection is fully indexed, store RocksDB on fast NVMe/SSD with a generous block cache, and raise search_pool_threads (30+ threads) inside the <performance …> block in run/conf/hlquery.conf. This prioritizes the search pool and gives worker threads enough room to service bursts without waiting on HTTP/write pools.

Thread Pool Lifecycle​

  1. Initialization: Thread pools are created after fork (if daemon mode)
  2. Worker Threads: Each pool spawns worker threads
  3. Task Queue: Tasks are queued and distributed to workers
  4. Shutdown: Graceful shutdown waits for tasks to complete

RocksDB database Services​

DBManager​

Class: DBManager

Provides persistent key-value storage using RocksDB's RocksDB architecture:

Core Operations​

  • Put: Write key-value pairs
  • Get: Read values by key
  • Delete: Remove keys
  • Flush: Persist in-memory data to disk

Background Services​

  • Compaction: Merges SSTables to optimize storage
  • Block Cache: Caches frequently accessed data blocks
  • Index Cache: Caches index and filter blocks
  • Write-Ahead Log (WAL): Ensures durability

Storage Components​

Memtable​

  • In-memory write buffer
  • Sorted by key
  • Flushed to disk when full

SSTables (Sorted String Tables)​

  • Immutable on-disk data files
  • Organized in levels
  • Merged during compaction

Block Cache​

  • LRU cache for data blocks
  • Reduces disk I/O
  • Configurable size

Search Services​

SearchAPI​

Class: SearchAPI

Singleton service that handles all search-related operations:

Search Processing​

  1. Query Parsing: Parses search parameters
  2. Collection Lookup: Validates collection exists
  3. Index Search: Searches inverted index
  4. Result Ranking: Scores and ranks results
  5. Filtering: Applies filters
  6. Sorting: Sorts results
  7. Pagination: Paginates results
  8. Highlighting: Highlights matching terms
  • Embedding generation
  • Vector similarity calculation
  • Hybrid search (lexical + vector)

HybridStorageManager​

Class: hlquery_storage::HybridStorageManager

Manages document storage and indexing:

Services​

  • Document Storage: Stores documents in collections
  • Index Management: Maintains inverted indexes
  • Metadata Management: Tracks collection metadata
  • Counter Management: Maintains document counts

Logging Services​

LogManager​

Class: LogManager

Centralized logging system with multiple log targets:

Log Targets​

  • Console output
  • File logging
  • Syslog integration
  • Custom handlers

Log Levels​

  • Critical: System-critical errors
  • Error: Error conditions
  • Warning: Warning messages
  • Normal: Normal operations
  • Debug: Debug information

Timer Services​

TimerManager​

Class: TimerManager

Manages periodic and scheduled tasks:

Timer Types​

  • One-shot: Fire once after delay
  • Periodic: Fire at regular intervals
  • Scheduled: Fire at specific times

Common Timers​

  • Database flush timer (1 second)
  • Health check timer (30 seconds)
  • Statistics collection timer (60 seconds)
  • Cache cleanup timer (5 minutes)

Initialization Sequence​

The server initializes in the following order:

  1. Configuration Loading: Load and parse configuration files
  2. Log Manager: Initialize logging system
  3. Timer Manager: Create timer system
  4. Database Engine: Initialize RocksDB database
  5. Search API: Initialize search services
  6. User Auth: Initialize authentication (if enabled)
  7. Thread Pools: Initialize thread pools (after fork)
  8. HTTP Server: Start HTTP server
  9. Metadata Scan: Scan and load collection metadata
  10. Ready State: Mark server as ready to accept requests

Shutdown Sequence​

Graceful shutdown follows this sequence:

  1. Shutdown Signal: Receive shutdown signal
  2. Stop Accepting: Stop accepting new connections
  3. Complete Requests: Wait for in-flight requests to complete
  4. Flush Database: Flush all pending writes
  5. Close Connections: Close all client connections
  6. Shutdown Thread Pools: Stop thread pools gracefully
  7. Cleanup Resources: Release all resources
  8. Exit: Terminate process

Performance Optimizations​

Adaptive Behavior​

The server implements adaptive behavior patterns:

  • Adaptive Timeouts: Socket timeouts adjust based on load
  • Lazy Processing: Expensive operations deferred to idle periods
  • Connection Pooling: Reuses connections for efficiency
  • Batch Processing: Groups operations for efficiency
  • Cache Warming: Preloads frequently accessed data

Resource Management​

  • Memory Pools: Reduces allocation overhead
  • Lock-Free Structures: Minimizes contention
  • Zero-Copy I/O: Reduces data copying
  • Efficient Data Structures: Optimized for common operations

Monitoring and Observability​

Metrics Collection​

The server collects various metrics:

  • Request counts and latencies
  • Cache hit/miss rates
  • Connection counts
  • Memory usage
  • Disk I/O statistics
  • Search performance

Health Checks​

Multiple health check mechanisms:

  • /health endpoint for basic health
  • /status endpoint for detailed status
  • Internal health monitoring
  • Automatic recovery from transient failures

Error Handling​

Error Recovery​

The server implements robust error handling:

  • Graceful Degradation: Continues operating with reduced functionality
  • Automatic Retry: Retries failed operations
  • Circuit Breakers: Prevents cascading failures
  • Resource Limits: Prevents resource exhaustion

Logging and Diagnostics​

All errors are logged with context:

  • Error type and message
  • Stack traces (in debug mode)
  • Request context
  • System state

Research & Future Work​

  • Daemon Resiliency & Observability: We are instrumenting the daemon with structured health telemetry, per-stage query timings, and automatic restart safeguards so background deployments recover faster.
  • TTC Instrumentation: Time-to-Consistency (TTC) metrics will tie RocksDB compaction scheduling to write visibility, giving operators visibility into how soon a write becomes search-visible while keeping throughput sane.
  • Hybrid Filtering & Tiered Storage: Future work will add selectivity-aware filtering planners and cold-tier offload for vector shards, which will require coordination between the search services documented here and the new dispatcher/routing layers.

For implementation notes and experiments, see etc/research/2026-02-26.md in the repo.