Public routing evaluation lab
Test thresholds. Measure trade-offs.
Explore an unverified candidate dataset, adjust the threshold, and inspect which simulated decisions pass or need review.
- Engine
- TypeSafe Jev System 1
- Latency
- < 15ms deterministic
- Outcomes
- 5 bounded categories
- Policy
- Fail-closed on uncertainty
- Starting gate
- 0.80 confidence score
First benchmark
One task. Five bounded outcomes.
JevLab measures a routing suggestion. It does not execute a tool, approve an action, or create a human ticket.
- code_search
- test_runner
- docs_lookup
- code_review
- none
The Policy Dilemma
Why Confidence Thresholds Matter in Agent Routing
TypeSafe Jev (jev-1.13.0) is designed as a high-speed System One decision engine for agentic workflows. Instead of token-by-token generative reasoning, it evaluates unstructured input against declared schemas in milliseconds, returning both a typed routing suggestion and an empirical confidence score.
Under-Gating Hazard (Threshold Too Low)
Adopting suggestions with a low threshold (e.g., 0.50) forces the router to guess on unclear or multi-intent prompts. This leads to silent misroutes—dispatching tests when the user wanted search, or triggering incorrect capabilities.
Over-Gating Bottleneck (Threshold Too High)
Setting the threshold too cautiously (e.g., 0.95+) escalates 90% of routine tasks back to expensive frontier LLMs or human queues. This erodes the 200x speed and 400x cost advantages that made you deploy Jev in the first place.
The JevLab Empirical Solution
JevLab eliminates guesswork. You can test thresholds interactively on real double-blind benchmarks, visualize the exact trade-off between coverage and error rate, and export production-hardened TypeScript code.
Three-Step Calibration
How JevLab Calibrates Your Routing Strategy
From offline evaluation to production deployment in your TypeScript codebase without wasting API credits.
Explore Double-Blind Evidence
Inspect benchmark samples blind-labeled by two independent humans and adjudicated with pinned criteria. Challenge cases highlight ambiguous and adversarial prompts.
Tune the Threshold Locally
Move the confidence slider from 0.00 to 1.00. The browser immediately recalculates the 5×5 confusion matrix, precision/recall, and auto-adopt vs handoff rates locally.
Export Deterministic TypeScript
Download a complete, runnable TypeScript routing module with pinned model versions, runtime schema validation, and fail-closed error handling.
Capability Boundaries
Understanding the Five Bounded Outcomes
JevLab evaluates a five-choice classification router. Each choice represents a strict capability boundary with zero tool execution side-effects:
code_search
Finds symbol definitions, callers, references, or config files. Never modifies files.
test_runner
Runs existing test suites and inspects exit codes. Never writes new test logic.
docs_lookup
Reads official docs, setup guides, and README files. Never executes commands.
code_review
Reviews diffs for bugs, maintainability, or security issues. Never commits changes.
none
Confidently rejects requests that fall outside the four supported capabilities.
Measurement Formulas
Empirical Policy Metrics at a Glance
Every metric reported in JevLab is based on raw API responses and fixed mathematical definitions:
Auto-Adopt Rate
The proportion of total tasks where confidence meets or exceeds threshold t. Measures how much workload your agent autonomously handles.
Auto-Adopted Error Rate
The percentage of misclassifications among auto-adopted decisions. The primary risk metric to minimize before going to production.
Handoff Rate
The proportion of total tasks safely escalated to your fallback queue or human operators due to insufficient confidence.
Architectural Benchmarks
Comparative Research & Decision Engine Guides
Explore independent engineering analyses evaluating non-autoregressive decision models, open-source ModernBERT encoders, and hosted routing primitives:
Laya vs TypeSafe Jev Benchmark
An 8-dimension technical evaluation contrasting Convai's open-source ModernBERT-large (32.8ms local P50, 0.081 ECE) against TypeSafe's hosted primitive. Analyzes latency, cross-lingual reach, and broad-domain decay across Banking77 and CLINC150.
Production Confidence Thresholding
An operational manual explaining why raw confidence values cannot be treated as accuracy rates. Covers empirical tuning techniques, the three-band action policy, and production fail-closed patterns for agent routing pipelines.
Build Your Own Jev
An engineering blueprint for self-hosting System 1 decision primitives in hours using Qwen-2.5, ModernBERT, and SGLang. Detailed matrix comparing Laya, openjev-sglang, Drex, AnyJev, and VON with 30-line Python code.
Does Jev Hallucinate?
Deconstruct the "zero hallucination" marketing claim. Understand the crucial distinction between schema determinism and semantic truth, prompt injection vectors in decision nodes, and defense-in-depth guardrails.
OpenRouter and Access
Jev Router, Classification, and API Access
Three explainers for the OpenRouter listings. Each one separates Jev Router from the Jev 1.13 decision model, and labels vendor prices and signup status with the source and the date they were checked.
Jev Router
What typesafe/jev-router selects, how the listed price differs from a flat token card, and when a chat router is the wrong tool for a typed decision.
Jev on OpenRouter
How to call structured classification through the Decisions API, which listing to pin, and how that call differs from chat completions.
Jev API Access
TypeSafe direct versus an OpenRouter key, where signup status stands, and the rate limits TypeSafe publishes with a warning that they still move.
Frequently Asked Questions
Common Questions About JevLab & Routing Policies
What is a Jev confidence threshold?
In JevLab, the confidence threshold (0.00 to 1.00) is the operating cutoff applied to TypeSafe Jev's output score. Suggestions with confidence at or above the threshold are automatically adopted, while suggestions below the threshold hand off to your existing escalation workflow.
What is the difference between none and handoff in JevLab?
none is an explicit, bounded classification outcome indicating that a task is outside the router's five defined capabilities. In contrast, handoff is a policy decision meaning the model's confidence was below your operating cutoff (or the response failed), so the task is returned to the caller for human or fallback review.
Is JevLab affiliated with TypeSafe?
No. JevLab is an independent, unofficial evaluation project and policy lab. It is not affiliated with, endorsed by, or maintained by TypeSafe AI.
Does JevLab execute agent tools?
No. JevLab evaluates routing suggestions and simulates policy outcomes only. It never executes code search, runs tests, reads documentation, inspects pull requests, or performs any tool actions.
How should I read coverage vs error rate when raising the threshold?
Raising the confidence threshold reduces the auto-adopt rate (coverage) and increases the handoff rate, while typically decreasing the error rate among auto-adopted decisions. JevLab visualizes this trade-off so you can select a threshold based on your application's risk tolerance.