Skip to main content

hlog

hlog is the lightweight log ingestion worker that ships with hlquery. It tails one or more files, turns each line into a structured event, and can push those events into an hlquery collection such as logs.

Unlike the main daemon, hlog is not a search server. It is a small pipeline runner with a simple shape:

input_file -> event mapping -> filters -> outputs

The worker is configured with etc/hlog/run/conf/hlog.conf, while the main hlquery daemon still uses run/conf/hlquery.conf.

What hlog Is Good For​

  • Ingesting hlquery.log, queries.log, or app logs into a searchable collection
  • Building a searchable operational log stream without adding a separate log stack
  • Enriching raw file lines with fields such as host, tags, pipeline name, and timestamps
  • Replaying failed inserts from a local failure buffer

Build and Run​

Build from the etc/hlog directory:

cd etc/hlog
./configure
make

Common wrapper commands:

./run/hlog start
./run/hlog start --nofork
./run/hlog status
./run/hlog stop
./run/hlog restart
./run/hlog test

The normal entrypoint is etc/hlog/run/hlog. It handles pidfiles, backgrounding, status checks, and stop/restart logic. The compiled binary lives at etc/hlog/build/bin/hlog.

Typical Workflow​

Example: ingest the main daemon log into a logs collection.

  1. Start hlquery.
  2. Create a collection for logs.
  3. Point hlog at the daemon log file.
  4. Start hlog.
  5. Query the ingested log lines with search or SQL.

Example collection:

curl -X POST http://localhost:9200/collections \
  -H "Content-Type: application/json" \
  -d '{
    "name": "logs",
    "fields": [
      {"name": "message", "type": "string"},
      {"name": "source_file", "type": "string"},
      {"name": "source_path", "type": "string"},
      {"name": "pipeline", "type": "string"},
      {"name": "source_kind", "type": "string"},
      {"name": "observed_at", "type": "string"},
      {"name": "tags", "type": "string"}
    ]
  }'

Start hlog against the daemon log:

cd etc/hlog
./run/hlog start --file ../../run/logs/hlquery.log --mode refresh --interval 1000

Check the data:

curl "http://localhost:9200/collections/logs/documents/search?q=error&limit=10"

Or with SQL:

curl "http://localhost:9200/sql?sql=SELECT%20COUNT(*)%20AS%20total_logs%20FROM%20logs%3B"
curl "http://localhost:9200/sql?sql=SELECT%20source_file%2C%20COUNT(*)%20AS%20total_docs%20FROM%20logs%20GROUP%20BY%20source_file%20ORDER%20BY%20total_docs%20DESC%3B"

Configuration File​

The pipeline is described in etc/hlog/run/conf/hlog.conf.

Example:

<hlog>

<input_file path="tests/file.txt"
            start_position="end"
            method="inotify">

<event message_field="message"
       path_field="source_path"
       file_field="source_file"
       date_field="observed_at"
       date_format="%Y-%m-%dT%H:%M:%S"
       tags_value="log_line"
       include_date="true">

<filter_add_field field="pipeline" value="hlog">

<filter_add_field field="source_kind" value="file">

<output_stdout enabled="true">

<output_hlquery enabled="true"
                endpoint="http://127.0.0.1:9200"
                collection="logs"
                auth_method="bearer"
                auth_token=""
                timeout="5">

If auth_token is empty, hlog sends requests without authentication headers. When your hlquery endpoint requires auth, set auth_token and choose auth_method="bearer" or auth_method="api-key".

Pipeline Stages​

  • input_file: tail a file
  • event: rename or enable generated event fields
  • filter_add_field: add constant fields to every event
  • filter_drop_if_contains: drop lines when a field contains a substring
  • output_stdout: print structured events to stdout
  • output_hlquery: send events into a collection

Watch Methods​

  • inotify: use Linux kernel file notifications
  • refresh: rescan the file every refresh_ms
  • auto: use inotify when available, otherwise fallback to refresh

Start Position​

  • start_position="beginning": read the whole file from the start
  • start_position="end": only ingest newly appended lines

Use beginning when replaying old logs. Use end when attaching to a live file and you only want new lines.

Event Fields​

Default emitted fields include:

  • id
  • message
  • path
  • file
  • host
  • ingested_at
  • tags

The <event> block lets you rename those fields or disable some of them.

Example:

<event message_field="line"
       date_field="created_at"
       date_format="%Y.%m.%d"
       include_host="false">

Minimal event payload:

<event message_field="raw"
       include_date="false"
       include_tags="false">

Configure-Time Overrides​

etc/hlog/configure can rewrite values in the generated hlog.conf, which is useful for fast local setup.

Examples:

./configure --input=../../run/logs/hlquery.log
./configure --input=../../run/logs/queries.log --collection=query_logs
./configure --endpoint=http://127.0.0.1:9200 --collection=logs --method=refresh
./configure --auth-method=api-key --auth-token=secret-token

Useful flags:

  • --input=/path/to/file
  • --method=inotify|refresh|auto
  • --collection=logs
  • --endpoint=http://127.0.0.1:9200
  • --auth-method=bearer|api-key
  • --auth-token=FOO
  • --environment=dev
  • --host=my-host
  • --tags=my-tag
  • --include-date=true|false
  • --include-tags=true|false
  • --timeout=5

Run ./configure --help inside etc/hlog for the full list.

Failure Buffer and Replay​

If output_hlquery cannot insert an event, hlog appends the raw line and timestamp to etc/hlog/run/data/hlog_failed.log.

That gives you a simple replay path after the server is healthy again:

cd etc/hlog
./configure --input=../../run/data/hlog_failed.log
./run/hlog start --nofork

When replaying, use start_position="beginning" so every saved line is emitted again. After a successful replay, truncate or remove the failure log to avoid duplicate inserts.

Querying Ingested Logs​

Once lines are in a collection, you can use normal search, filters, and SQL.

Examples:

curl "http://localhost:9200/collections/logs/documents/search?q=timeout&limit=20"
curl "http://localhost:9200/sql?sql=SELECT%20COUNT(*)%20AS%20total_logs%20FROM%20logs%3B"
curl "http://localhost:9200/sql?sql=SELECT%20source_kind%2C%20COUNT(*)%20AS%20total_docs%20FROM%20logs%20GROUP%20BY%20source_kind%3B"

Inside talk:

localhost:9200> use logs
localhost:9200|logs> search error 10 0 observed_at:desc
localhost:9200|logs> sql: SELECT COUNT(*) AS total_logs FROM logs;
localhost:9200|logs> sql: SELECT source_file, COUNT(*) AS total_docs FROM logs GROUP BY source_file ORDER BY total_docs DESC;

Operational Notes​

  • hlog can fall back to file log targets declared in hlquery.conf when no explicit input_file stages are configured.
  • refresh mode is simpler and portable; inotify is usually cheaper on Linux for active files.
  • If a log file is rotated externally, prefer validating behavior in your environment before relying on unattended ingestion.
  • For analytics-style log queries, SQL grouped aggregates are usually easier to consume than raw hit output.

For implementation details and the standalone worker layout, see etc/hlog/README.md.