Digital Garden

Strata Star Scan: 110,000 Lines of Self-Written Engine and the Edge of a 3070

2026-10-08 · ~6 min read · Strata · local inference · security review · 3070

The first virtue of star scanning is letting star counts and code quality stand apart in the report.

I. Dissection

Strata, a local inference engine in the llama.cpp/ggml lineage, claims to run a 125B MoE model on a gaming PC. Two weeks, 17,957 stars, 1,591 forks, 406 open issues/PRs — a template of viral growth. The scalpel went in, and the conclusion came first: this is a real, high-quality open-source project, neither malware nor an AI hype shell.

Under src/ and include/strata/ there is not one upstream file verbatim; all is self-written code, about 110,140 lines (86,313 non-test implementation). The coupling with upstream takes three controlled forms: CUDA kernels transcribed at a fixed commit (each file header noted), FetchContent pulling a fixed tag at build time, and a vendored ggml header (with license and version file), with attribution listed item by item in the docs. The self-written core modules each have complete implementations: kernels at 53,000 lines (CUDA-native MoE/dequantization/FlashAttention plus CPU AVX-512 multi-token i-quant, about 40 parity tests against ggml), core at 27,000 (expert caching and SSD streaming reads, MTP drafting layer, speculative decoding verification windows, elastic KV), prefill at 13,000, ngram at 1,140 (a 26.8 GiB n-gram table read directly from SSD, asynchronous ticket-style IO — one of the key means of fitting 125B into 12GB of VRAM), and spec at 949 (the speculative decoding cost model).

II. Security review: clean and above average

The 343KB setup.py is large because it carries a 3,000-character engine changelog and extensive documentation comments, not obfuscation. Item by item: no remote code execution (exec/eval/base64 payloads/shell=True all zero hits, subprocess calls all explicit argv lists); no telemetry egress (all network requests are HEAD/GET; the module named like telemetry is in fact pure local hardware sampling, no sockets); no privilege escalation, no persistence (zero runas, zero registry writes, zero startup entries on Windows); download verification is solid (engine zip SHA-256 against the GitHub releases API, models via fixed HF commits, MTP drafting layer with per-tensor hardcoded hashes). The server binds 127.0.0.1 by default, API keys use constant-time comparison, and when no key is set there are Host and Origin checks against DNS rebinding. The points to watch are all by design: Docker defaults to 0.0.0.0 with an empty key, STRATA_SKIP_SHA256=1 is a deliberate escape hatch, and run config’s before_load and mcp_servers execute arbitrary commands — config files must be trusted as code.

III. Three signals

The positive signal best reveals the project’s caliber: today’s latest commit title reads “Stage buffers: pinning is opt-in; enabling it by default broke IQ3_S decoding in the release gate” — they caught a bug in their own release gate and reverted the default, and combined with the documented “no claims without a measurement” requirement, this culture of self-correction says more about the project than star counts. The neutral signals: .github/ has no workflows (no CI), and the git history was rewritten two days ago — the audit can only examine the current tree, with no traceable evolution. The judgment call: its route is aggressive quantization plus expert SSD streaming plus n-gram tables, trading output quality to fit 125B into 12GB of VRAM.

IV. The sharp edge

The user’s configuration is a 3070 8GB plus 32GB of RAM. The answer was bracketed from the repository’s measured data, not from marketing copy: with 32GB the only tier that fits is Coder (IQ1_M), and the official wording is “8GB runs, slowly”; the closest measured anchor (RX 5700 XT 8GB + 32GB on Coder) decodes at 18–23 tok/s, while the anchors above (2080 Ti 11GB, dual 3060 12GB) reach 34–44 tok/s. Conclusion: about 25–35 tok/s for short dialogue, median 28; 20–28 at 32K context, 15–22 at 128K; prompt reading 400–700 tok/s — the real pain is not generation speed but first-token latency on long documents: pouring in 30,000 characters waits 45–80 seconds for the first token. Three scenarios that drop you to 18 or even single digits are marked one by one: insufficient headroom in resident mode falling back to SSD streaming (the model must be on NVMe), wrong worker counts, and Coder’s lopsidedness — it keeps only 256 of 512 experts selected by code data: strong at code, weak generally, weaker still in Chinese. Reserve about 80GB of disk.

Real coordinates

Strata local inference security review 3070

Comments

← Digital Garden