Public test suite
7,820 passed, 43 skipped, 0 failed; 91.37% line coverage
- How measured:
- GitHub Actions on the main branch, 12 Oct 2026.
- Why it is weaker than it looks:
- Shows the code does what the tests say. It does not show that a model uses the tools well.
Hijacked-model test (prompt injection)
21 scenarios: 18 reached a harmful side effect without the structural gate, 0 with it; 6 of 6 legitimate workflows still ran
- How measured:
- A scripted model that obeys every instruction it reads (the worst case) is shown pages with hidden instructions. The suite later grew to 23 scenarios, still 0.
- Why it is weaker than it looks:
- A scripted worst case, not a real model. With a real local 7B model the harmful tool ran 0 times, but it read the injected page in only 11 of 18 scenarios: a small sample, not proof. Wrong-but-harmless answers are not covered.
Default owner policy on labelled requests
169 requests: 3 of 110 unsafe ones still ran; 6 of 59 harmless ones were stopped
- How measured:
- Each request goes through the real gate and the real shell sandbox; the verdict is compared with the label.
- Why it is weaker than it looks:
- The labels were written by the developers, and part of the set was used while the rules were tuned, so this is not an independent test.
Does the FDIA equation add anything over data quality alone?
No evidence yet: F = D^I × A and D alone gave the same held-out result (0% unsafe allowed, 89.5% of harmless requests also stopped)
- How measured:
- 120 labelled requests through the real gate, comparing the full equation with D alone.
- Why it is weaker than it looks:
- A negative finding that we publish on purpose. The human-approval and policy layers do the protective work in these tests.
Injection screen on text it has never seen
External labelled set: 6.7% of attacks blocked on the first look, 10% after generic additions, 23% with a small model's second opinion; 0 false blocks in 532 harmless texts. Thai holdout: 15 of 31 (48.4%) at 0 of 32 false blocks
- How measured:
- Public prompt-injection test split, not copied into the repository; Thai sets written by us, holdout measured once.
- Why it is weaker than it looks:
- Rules catch a minority of unseen attacks. That is why the structural gate above exists instead of relying on the screen.
Where the prompt tokens go
The tool menu is 94% of a prompt (7,153 of 7,637 tokens, 48 tools). Delta v2 shrinks large tool output by 48–69%, and nothing below 6,000 characters
- How measured:
- Token counts with tiktoken on the runtime's real prompts and three real 30 KB documents.
- Why it is weaker than it looks:
- Counts are approximate and model-specific. The real cache hit rate has never been measured.
Delta storage versus plain compression
Structural delta saves 70–94% against raw JSON, but a snapshot plus zstd saves 96.7–99.6%, within 3% of delta plus zstd. No evidence that Delta beats ordinary compression for a stream
- How measured:
- Byte counts on the runtime's own payloads, including the real memory table.
- Why it is weaker than it looks:
- Delta wins only for random access and appending; a damaged byte costs it up to 20 ticks.
Memory store compared with LangGraph and mem0
At 2,000 operations: Delentia 1.5 MB (sealed log), mem0 7.1 MB, LangGraph 192 MB. After each system's own delete call the person's text was still in the file for LangGraph and mem0, and gone only after Delentia's erase
- How measured:
- One synthetic workload, default settings of mem0ai 2.2.1 and LangGraph with SqliteSaver, raw database files scanned for the text.
- Why it is weaker than it looks:
- One workload and default configurations. Recall quality was not measured here.
Recall by meaning
30 paraphrased questions among 120 texts: Delentia word matching 7, mem0 24, Delentia with local embeddings 22. Held-out Thai set of 40: 20 with word matching, 38 with the bge-m3 embedding
- How measured:
- Fixed question sets written before the runs; top-3 hit counts.
- Why it is weaker than it looks:
- Synthetic data, not real users' memories. Word matching alone is weaker than mem0 here; embeddings are optional and not the default.
Start-up and footprint
`delentia serve` answers /health in about 2.9 s; idle memory about 471 MB; about 0.3 s of runtime overhead per governed episode
- How measured:
- Measured on one Windows PC with a scripted model.
- Why it is weaker than it looks:
- One machine. Not a load test and not a production figure.
Unattended-operation drill
8 of 8 observations hold: a scheduled read job ran by itself, a scheduled write job stopped and waited for a signature and wrote nothing, an emergency pause stopped a new job, and the audit trail the server wrote (36 rows) verified
- How measured:
- A real `delentia serve` with the daemon on, a real clock and a scripted model, left alone for about two minutes.
- Why it is weaker than it looks:
- Shows the code side of autonomy. It does not show that a capable model can drive it, or that it runs for days.
Real small local models driving the tools
Simple goals completed: qwen3:8b 4 of 5, qwen2.5:7b 3 of 5, llama3.2:3b 0 of 5 (CPU only, 70–190 s per goal)
- How measured:
- Five simple tool tasks on the current runtime, one run each.
- Why it is weaker than it looks:
- A tiny sample of weak models. No capable hosted model has been tested yet, so we make no claim about how well the agent works with one.