A Mortal Journey: The Infrastructure of 8.2 Million Characters and the 247-Batch Long March

A cultivator long march is measured in years; dismantling this 8.2-million-character immortal road into a rebuildable world took one day.

I. Decoding: the misalignment-free epic

With the edict given, the first cut fell on source QC. GB18030 decoding, chapter indexing, and alignment verification flowed in one breath: 2,462 chapters, 8.2 million characters, zero misalignment. The only warning was “truncated ending” — the engineer did not let it pass, and digging down revealed it was merely the author’s new-book announcement, while the true finale was chapter 2446, “Ascending to the Immortal Realm.” The book was complete; the false alarm cleared.

The pilot batch then rolled out: 99 seconds, 23 characters, 18 lorebook entries, 9 events. The eyes at the manual acceptance point were sharp — the female characters (the cousin Wu-yan, the little sister, the elder sister) were all caught, the complete-female-coverage iron law in effect; the only note was the batch being split into two segments causing Han Li, Mo Daifu, and Zhang Tie to appear once more in the latter segment — normal for split scanning, and the merge layer would dedupe.

II. A real bug and a false alarm

As the full scan ran, the handwritten layer generated in parallel — and immediately hit a real toolkit bug: llm_stream.chat returns a (text, model) tuple, while gen_src wrote it to disk as a string (gen_cast unpacks it; gen_src missed it). The fix touched only the workspace copy; the original stayed read-only; the iron law unbroken.

The bigger drama was a garbling false alarm. Six narrative files (first_mes, alternate_greetings, personality, scenario, mes_example, creator_notes) looked entirely garbled, as if the upstream channel had read UTF-8 as GBK. The engineer first checked the scan products — sixteen batch files all clean; then tried a round-trip repair — it failed, but the encoded bytes were correct Chinese GBK; finally inspected the real code points of the file characters: the files themselves were intact; the garbling was only the Bash output pipeline’s display layer. A disaster that could have caused mistaken deletion and rewrite was dissolved by four words: “stop guessing, look at the code points.”

III. The dike of the format contract

The third battle was invisible. The generated entry files had mvu and system using lowercase order:/group: and missing CONTENT: markers, while nsfw used Chinese labels — the content was fine, but feeding them straight into the assembler would lose const (non-residency), turn ORDER into 999, and pollute the body with metadata lines — exactly the “status bar missing” class of trap from the README. The engineer wrote a normalization script, correcting four entry files into standard @@ENTRY format, moving NSFW’s ORDER from 1 (colliding with the rules iron law) to 20, then verifying with the assembler’s own parser: all 25 entries parsed clean, const intact, ORDER correct, no metadata pollution.

IV. A breathing scan

The full scan itself was a self-managing march. Concurrency climbed from 3 to 7 at about 2.2 batches per minute; batch 18 failed when reasoning ate the 32,000 budget and passed on retry; after three 429 rate limits the gate auto-lowered concurrency from 8 to 6; batch 52, struck by four consecutive 429s, was marked failed, but --loop would rescan it next round — failed batches never enter the results directory and stay on the todo list forever. Zero garbling, the mechanism self-healing. During the wait, a merge-chain smoke test ran early: seventeen batches merged into 163 clusters, no hub warning, Han Li and Erlengzi correctly merged; alias-fusion risks (the Yan cluster mixing multiple “stepmothers,” the Ma deputy-chief mixing “Deputy Chief Wu/Lingzhu”) were recorded for a unified audit after the full run. The delivery side also pre-cleared risks: pillow 12.2 can generate images, node 25.9, zod.js passed ES module checks.

Coda

By tonight, 141 of 247 batches had landed, with the roster and event timeline beginning to take shape in the merge layer. Han Li’s immortal road is still landing batch by batch, and the workshop has proven: no matter how thick the original, it cannot withstand a frozen craft plus an unsleeping foreman.

Real coordinates

• Ingest: 2,462 chapters / 8.2M characters / zero misalignment; the “truncated ending” warning was the author’s new-book announcement (false alarm); the true finale is chapter 2446 “Ascending to the Immortal Realm”
• Pilot: 99s / 23 characters / 18 lorebook / 9 events; all female characters caught (iron law in effect); split-segment repetition handled by merge
• Real bug: llm_stream.chat returns a tuple while gen_src failed to unpack (gen_cast had it); only the workspace copy fixed
• Garbling false alarm: the six files’ “garbling” was a Bash display-layer fault; 16 scan batches zero pollution; settled by direct code-point inspection
• Format dike: mvu/system lowercase fields missing CONTENT, nsfw Chinese labels → normalization script; NSFW ORDER 1→20 to avoid collision; all 25 entries parsed clean
• Scan health: concurrency 3→7, ~2.2 batches/min; b18 reasoning budget exhaustion passed on retry; 429 gate auto-lowered concurrency 8→6; b52 failed and —loop rescans next round; budget burn ratio 1.3
• Merge smoke: 17 batches → 163 clusters, no hub, Han Li/Erlengzi correctly merged; alias-fusion risks reserved for post-full-run audit
• Delivery side: pillow 12.2 / node 25.9 / zod.js ES module checks passed
• Progress: 141/247 batches; characters_merged.json and events_timeline.json already landing