Comparative Evaluation
The Agent Reliability Matrix
Three paradigms of autonomous AI development: unguided baseline agents, passive prompt rules, and TypeSafe System One reflex hooks.
Technical Capability Matrix
| Capability / Invariant | Baseline Agent | Passive Prompt Pack | jev-superpowers |
|---|---|---|---|
| Dependency Verification | Invented or hallucinated crates | Instructed not to hallucinate (ignored under load) | jev-scout registry gate (0% hallucinations) |
| Destructive Command Safety | Executes rm -rf, drop table blindly | Subject to jailbreaks / prompt injection | jev-guard physical hook blocks execution |
| Local FOSS & Offline Mode | Cloud-only or heavy Ollama chains (2-5s) | Requires continuous cloud token egress | Dual backend: TypeSafe Cloud or local Laya (421M open weights, 33ms) |
| Stop-Hook Completion Gate | Declares done with failing tests | "Double check tests pass" text prompt | limpet binary halts exit code on failure |
| Build / Triage Latency | Dumps 2000 lines into context window | Dumps 2000 lines into context window | jev-axi triage isolates root cause in 92ms |
| Pre-Commit Diff Checking | Commits broken code silently | Advises clean commits in instructions | git-jev evaluates staged diff before git commits |
| Context Token Consumption | Fast context window exhaustion | Bloated system prompt (4k-8k tokens) | Dynamic skills loaded on demand (<1k tokens) |
Failure Analysis
Why Soft Prompts Fail at Scale
In experiments across 100 multi-turn coding sessions, agents guided solely by Markdown system prompts experienced instruction drift by turn 16.
Turn 1: System prompt: "Never use deprecated API v1" -> Compliance: 100%
Turn 8: User requests urgent database fix -> Compliance: 92%
Turn 16: Context buffer exceeds 28k tokens -> Compliance: 61%
Turn 24: Agent uses deprecated API v1 -> Compliance: 0% (Rule erased from active attention)
With jev-superpowers:
Turn 24: Agent attempts deprecated API call
PreToolUse Hook: triggers jev-guard -> Execution physically rejected (Exit 1).
Compliance remains 100% indefinitely.