tekmerdb-ingest turns a document into individual claims and inserts each one into a running engine over HTTP. It handles PDF, DOCX, TXT, MD, URLs, and Wikipedia articles.
tekmerdb-ingest report.pdf --source "IEA Report 2024"
Two things are required: the input (a file path, URL, or Wikipedia article title) and --source/-s, the name attached to every claim it inserts.
tekmerdb-ingest is a standalone binary. It only talks to the engine over HTTP, so it doesn't need to be installed or run on the same machine as tekmerdb — run it from your laptop, a CI runner, wherever the source document already is, and point it at the engine's address:
tekmerdb-ingest report.pdf --source "IEA Report 2024" --engine http://203.0.113.10:3000
--engine / -e — engine base URL, default http://localhost:3000--domain / -d — overrides the domain on every claim; otherwise queried from the engine's /statusengine_host = "127.0.0.1", which only accepts connections from its own machine — a remote tekmerdb-ingest run will fail to connect. To ingest from another machine, the engine must first be opened up by setting engine_host = "0.0.0.0" in tekmerdb-server.conf and restarted (see Configure → Network binding). Only do this behind a firewall — the engine's PATCH endpoint has no authentication.
The type is auto-detected from the input (file extension, URL, or bare title). Force it explicitly with --type if detection guesses wrong. Valid values: pdf, docx, txt, md, url, wiki.
tekmerdb-ingest report.pdf --source "IEA Report 2024"
Reads the PDF's embedded text layer directly (no OCR) and reconstructs true reading order from glyph positions, including detecting and correctly ordering two-column layouts. A scanned or image-only PDF with no text layer will yield no claims.
tekmerdb-ingest https://example.com/report --source "Example Report" --type url
Fetches the page's raw HTML and pulls text only from content tags — <p>, headings, <li>, <blockquote>, <article> — discarding nav/scripts/markup. It does not execute JavaScript, so content that a site renders client-side after page load won't be captured; server-rendered pages and most articles/blogs work well.
tekmerdb-ingest "Grid inertia" --source Wikipedia --type wiki
A bare title with no path separators or file extension is auto-detected as a Wikipedia article and fetched directly by name.
tekmerdb-ingest notes.txt --source "Field notes" --type txt
Plain text extraction — DOCX paragraphs, or the raw file contents for TXT/MD.
tekmerdb-ingest report.pdf --source "IEA Report 2024" \ --chunk-mode sentence --min-len 40 --max-len 600
--chunk-mode — sentence (default) or paragraph--min-len / --max-len — claim length bounds in characters, default 40–600--confidence — base confidence assigned before scoring, default 0.7| Mode | Produces | Pros | Cons |
|---|---|---|---|
sentence |
One claim per sentence | Each claim is a single, atomic statement — the right granularity for the engine to score and compare. Boundary detection is tuned against real-world quirks (abbreviations, bulleted lists, PDF line breaks that don't align with sentence ends). | More claims per document, so more inserts overall. |
paragraph |
One claim per paragraph | Fewer, larger claims — fewer inserts for the same document. | Bundles multiple, possibly unrelated statements into a single claim, which muddies what's actually being scored. Paragraphs are split naively on blank lines with no sentence awareness, so a paragraph over --max-len gets cut at a plain word-count boundary — it can end mid-sentence. |
Use sentence mode. It's the default, it's the mode the boundary-detection edge cases were built for, and paragraph mode's only real advantage — fewer HTTP requests — isn't worth trading away claim quality for.
tekmerdb-ingest report.pdf --source "IEA Report 2024" --dry-run
Parses and splits the document into claims without inserting anything or requiring a reachable engine — use it to check chunking before committing a real run.
tekmerdb-ingest report.pdf --source "IEA Report 2024" \ --concurrency 8 --log ingest-run.ndjson
--concurrency — concurrent HTTP inserts, default 4--log — writes an NDJSON record of every claim and its insert result to the given file