Two maps
do all the
work.

Two maps do the work. OffsetIndex takes a document id and gives back a segment file, a byte offset, and a byte length. InvertedIndex takes a term and gives back the list of documents containing it, with how often each one uses it.

Both maps live in memory. Neither is the source of truth. The segment files on disk are, and both maps can be rebuilt from them. Everything else in Shardly is the machinery that keeps the maps honest when the power goes out.

A read is one fs.readSync call at a known offset. A search walks posting lists and does not open a segment file until it already knows which ten documents to return. If you understand those two sentences, the rest of this page is detail.

The length stops at the closing brace

Records are separated by newlines, so the file is valid NDJSON and you can read it with tail. But length counts only the JSON itself. That way the buffer handed to JSON.parse holds exactly one object and nothing else, with no trimming step that could go wrong on the last record in a file.

Many readers, one writer

Storage keeps a read-only file descriptor per segment in this.readFds, so a hot read never pays for open. It keeps exactly one write descriptor, on the newest segment, in append mode. That asymmetry removes the need for any lock: writes only ever touch the end of one file, and reads only ever touch bytes that are already final.

An id carries no order, and that is a real cost

Every id comes from randomUUID(). Random ids spread evenly and never collide, which is what you want. They also mean two documents written a second apart sort next to each other by pure accident. There is no way to ask for "every document after this one" without a full pass, and liveDocIds() returns insertion order because that is what a JavaScript Map happens to preserve, not because anything guarantees it. A monotonic id would buy range scans and cursor pagination. Shardly does not need them, so it does not have them.

Why 64 MiB, and not any other number

SEGMENT_MAX_BYTES caps each file at 64 MiB. Nothing in the read path cares how large a segment is, because reads seek directly to an offset. The cap is not a performance knob.

It exists so compaction has units to work with, and so a corrupted file loses a bounded amount of data instead of everything. Pick 4 GiB and one bad sector takes the corpus. Pick 1 MiB and you drown in file descriptors.

The file is the authority on its own size

openCurrentSegment() sets currentSegmentSize from fs.statSync(p).size, never from the snapshot. This looks like a detail and is not.

Recovery can restore a write the snapshot never saw. If the next append trusted the snapshot, it would place a new record on top of a live one and silently destroy it. Asking the filesystem costs one syscall at startup and removes the entire class of bug.

Two records per write looks wasteful. It is not.

Ask what a single record would tell you. A lone "I intend to write 231 bytes at offset 4096" leaves recovery unable to separate a finished write from a torn one. A lone "I wrote 231 bytes at offset 4096" is a lie if the process dies before the segment fsync returns, because the log entry is durable and the data is not.

You need the pair. The pending record bounds where the damage can be. The committed record proves the damage is not there. Neither one is enough alone.

What Shardly actually promises

write() returns to the caller only after step 3. An acknowledged write survives any crash, full stop. A write still in flight might survive and might not, and both outcomes are correct, because nobody was told it succeeded.

That is the whole contract. It is narrower than "nothing is ever lost", and it is the only kind of promise a single machine can actually keep.

Why a tombstone is committed from birth

Delete records are written as committed straight away, with no pending phase. A tombstone is a flag in memory, and the WAL record is the only durable part of it. There is no second file to keep in step, so there is nothing to be half-done.

Where the honesty is

The four checks confirm structure. They cannot confirm meaning. A record that is byte-complete, parses as JSON, and carries the right id can still have a flipped bit inside a string value, and every check will pass it. Detecting that needs a checksum per record, and Shardly does not have one.

Adding a CRC to the record format is the single most valuable change left in this file. I am saying so here rather than letting you discover it.

A torn tail should not condemn the file

readAll() swallows parse errors on individual lines. That is narrow and intentional. The last line of the log is the one most likely to be torn by the exact crash being recovered from, and one bad tail should not make the whole log unreadable.

The fiddly part is the arithmetic, not the protocol

writeBatch() builds the whole NDJSON blob in memory first, assigning ids and computing offsets as it goes, rotating segments where the 64 MiB cap demands. Every offset is measured against this.currentSegmentSize plus Buffer.byteLength(chunk, "utf8"), which is the committed size plus the bytes buffered but not yet flushed.

It has to be Buffer.byteLength and not String.length. One multi-byte character puts every later offset in the batch off by the difference. That bug is invisible on ASCII test data and corrupts every read on real text, which is the worst combination a bug can have.

New segments are numbered past the current one

They start at currentSegmentIndex + 1, so a fresh file can never collide with one that live offsets point at. The old files are readable through the entire operation. Nothing is unlinked until the new snapshot is durable.

Leaving the orphan is a decision, not an oversight

A crash before step 3 leaves a partial new segment that nothing references. No startup sweep removes it, and that is a choice rather than an omission.

A sweep that unlinks unreferenced segments is exactly the code that deletes real data the day a snapshot goes missing for an unrelated reason. Leaving the orphan costs disk. The next compaction reclaims it anyway, because it reuses the same number and opens with "w". Even if a normal rotation reaches that number first, it opens with "a" and sizes itself from the file, so new records land after the dead bytes and every offset stays correct.

Wasted space, never corruption. I will take that trade every time.

Renaming a file is atomic, and that is not the same as durable

writeJsonAtomic() writes to ${target}.tmp, calls fsync on it, renames it over the target, and then calls fsync on the parent directory. That last step is easy to skip and it matters here more than usual.

A rename is atomic with respect to readers, but the directory entry is not durable until the directory itself is flushed. snapshot() truncates the write-ahead log immediately afterward. Without the directory fsync, a power failure in that gap loses the snapshot and the log that would have rebuilt it. Two durable-looking writes, zero durable state.

Why the two snapshots are not written atomically together

The offset snapshot and the inverted-index snapshot are written in sequence. A crash between them leaves a fresh offset file next to a stale index file.

The version check catches that on the next start and rebuilds. Making the pair atomic would need a two-phase write across two files for no gain, because the inverted index is derived data. Rebuilding it is always safe and never wrong. Spending complexity to protect something you can always regenerate is how storage layers get hard to read.

The rule

SNAPSHOT_EVERY_N_WRITES is 500. Anything other than an exact version match rebuilds the index from the segments. Equal counts no longer pass for equal state.

The character class is ASCII-only

/[^a-z0-9\s]+/g runs after toLowerCase(), so every character outside a-z0-9 becomes a separator. tokenize("Zürich café") gives ["z", "rich", "caf"]. tokenize("東京 検索") gives an empty array.

Non-English text is not ranked poorly. It is unindexable. For the bundled English corpus this never shows up, which is exactly why it is worth writing down.

The stemmer over-stems and under-stems

Six suffix rules, not Porter. stem("cares") returns "car", colliding with the actual word car, so a query for cars matches documents about caring. That is over-stemming.

Meanwhile stem("running") gives "runn" and stem("runs") gives "run", so two forms of one verb never meet. That is under-stemming, and it is a plain defect rather than a trade-off. Porter handles it with a doubled-consonant rule.

Why keep it anyway

It is a dozen lines you can read in full and reason about, and swapping in Porter is a contained change to one function that already has tests around it.

The reason to write the failures down is simpler. A stemmer nobody has measured is a stemmer everyone trusts too much.

Symmetry saves it

Query text and document text go through the identical function. A collision costs precision on both sides evenly instead of breaking recall in one direction, which is the difference between a blunt tool and a broken one.

Removing a document means removing it from every posting list it appears in. The obvious way is to walk the whole index and filter each list. It is correct, it passes every unit test, and it costs O(vocabulary) per delete while allocating a replacement array for every term in the corpus, whether or not the document touched it.

docTerms maps a document id to the distinct terms it contributed. The loop becomes O(terms in this document). The gap widens as the vocabulary grows, which is the direction real corpora go.

docTerms is not in the snapshot format. load() rebuilds it by walking the postings once, which costs one pass over data already being parsed and keeps old snapshots readable. Deriving beats storing when the derivation is cheaper than the format change.

The memory cost is one array of string references per document. The strings are the same objects already used as keys in index, so this is pointers, not text.

The test that pins the middle setting

A ranker that punishes length is as broken as one that rewards it. tests/rank.test.ts covers both ends: a short document beats an over-long one at equal term frequency, and a genuinely more relevant long document still wins when it earns it. The second test is the one that catches an over-eager b.

Why every hit carries a breakdown

Each result includes a TermScore[] with the term, its frequency in that document, its IDF, and its contribution, sorted by contribution. A test asserts the contributions sum to the total.

That is what lets an interface answer "why did this rank here" instead of showing a number nobody can check. A relevance score you cannot decompose is a relevance score you cannot debug.

The version that was replaced

The first implementation looked up each winner's term frequency by searching that term's posting list, once per winner. On a term appearing in 20,000 documents with a limit of 10, it re-scanned 20,000 postings ten times to recover ten numbers.

It now walks each term's postings once, filling a Map<docId, Map<term, tf>> keyed by a Set of winner ids. The work drops to O(terms × postings) no matter what the limit is.

total counts matches, not results

total reports every document matching at least one query term, which is scores.size, not the size of the returned page. Reporting the page size would make every search look like it found exactly ten things, and pagination would have nothing to paginate.

When the heap steps aside

If limit is Infinity or larger than the candidate count, selectTopN() sorts everything instead. A heap cannot beat a sort when you want the whole list, and pretending otherwise is how you end up with a slower "optimized" path.

What the comparison is worth

Both sides compute BM25 over the same documents with the same k1 and b, and topHitsMatch verifies the fast path returns the same answer as the slow one. A speedup with a different answer is a bug report, not a benchmark.

Where the number is soft

speedup divides by Math.max(indexed.tookMs, 1e-6). On a small corpus where the indexed path finishes in microseconds, that ratio is measuring timer resolution as much as anything. Treat the ordering as real and the exact multiplier as approximate.

These are local measurements on one machine, not a performance guarantee. Run npm run seed and npm run bench on your own hardware for numbers that mean something to you.

Detecting binary before decoding it

Files with an unrecognized extension get checked first. isProbablyBinary() reads the first 8 KiB and returns true on any NUL byte. Otherwise it flags files where more than 30% of bytes are C0 control characters outside tab, newline, and carriage return.

A NUL byte is conclusive. The 30% threshold is a heuristic, tuned to let odd-but-real text through while rejecting compiled output. Extensions on the TEXT_EXTENSIONS list skip the check entirely, because a .ts file is text even if it opens with something strange.

A PDF with no text is reported, not indexed

PDFs go through unpdf. If extraction yields nothing, the file is returned as skipped with a reason rather than stored as an empty document.

A scanned page that silently produces zero tokens is precisely the thing you want to see in the response. Indexing it as an empty document hides the problem and pollutes avgdl at the same time.

Malformed NDJSON lines are skipped, not fatal

One bad line in a 200,000-line dump should not reject the dump. Same reasoning as the torn WAL tail.

Politeness is a feature of the GitHub adapter

Concurrency runs through mapLimit(), an eighteen-line worker pool. Promise.all over every file would open hundreds of sockets and trip the rate limiter on the first repository. Caps on repositories, files per repository, total files, and file size are parameters with defaults, so the HTTP route holds a fixed policy while tests pick their own.

Progress is streamed, not awaited

POST /corpus/wikipedia/index streams NDJSON progress events as it works. A request that returns after 2,386 documents tells the user nothing for the whole duration. The streaming version is about twenty lines.

previewDoc() had to learn to recurse

GET /documents and GET /search trim string values to 280 characters and set _truncated. The first version only walked top-level keys.

A nested {content: {body: "…"}} from a deep GitHub index shipped whole, so responses ran to megabytes while _truncated was absent, which is the worst possible combination: wrong data and a flag saying it is fine. It now recurses through objects and arrays. GET /documents/:id still returns the full document, because that is the detail view.

parseLimit() clamps rather than trying

Unparseable or non-positive values fall back to the default, and everything is capped. A client asking for limit=100000000 gets the cap, not an attempt and a heap crash.

buildServer() returns the engine too

Tests need the handle. Production needs the onClose hook that snapshots and closes descriptors. Both SIGINT and SIGTERM route through app.close(), so a container stop is a clean shutdown rather than a recovery on next boot.

The structures are the same, which is the whole point

An inverted index is an inverted index whether the postings are JavaScript objects or variable-byte integers in a mapped file. BM25 is BM25 whether you compute it in TypeScript or in a JIT that has been tuned for twenty years. Segments are immutable and merged in both systems, and both put a durable log in front of the data.

Every row in that table is an engineering answer to a scale Shardly does not operate at. Skip lists matter when a posting list has ten million entries. A finite state transducer for the term dictionary matters when the vocabulary does not fit in memory. Positions matter the moment somebody types a phrase in quotes.

What you get for reading this one instead

You can read all of Shardly in an afternoon and hold it in your head afterwards. Every constant is in one file. Every fsync is on a line you can point at. When a search returns the wrong document, you can decompose the score term by term and find out why.

That is not a substitute for Lucene and is not trying to be. It is the thing you build once, so that Lucene stops being magic.