Methodology
Measure the policy, not a promise.
JevLab evaluates a five-label routing suggestion and the threshold policy applied to it. It does not measure tool execution, authorize actions, or claim that an interactive threshold is production-ready.
Current evidence
Sources, review, and splitting
A published sample declares whether it is human-authored, AI-authored synthetic, or sanitized real-world material. Private source paths, reviewer identities, and unreleased splits are never copied into the public artifact.
- Two people independently label each task while the proposed answer, model output, and the other reviewer's answer are hidden.
- Initial agreement is recorded before discussion. A designated adjudication record fixes the final label or moves an irreducibly ambiguous task into the challenge track.
- Semantic groups are frozen into validation or held-out test; related rewrites cannot span both. The starter candidates are never used as an independent test set.
- Validation supports threshold exploration. A test threshold must be frozen from validation before the held-out test run and cannot be adjusted on its report page.
Threshold policy
A response routes only when it is valid and its raw confidence is greater than or equal to the selected threshold. A lower confidence or any failed response hands control back to the caller. The 0.80 starting value is an interaction default, not a recommendation.
Metrics and denominators
Let N be all samples, V valid responses, A valid responses at or above threshold, and C correct responses within A. Failed responses stay in N and never count as valid predictions.
| Metric | Formula |
|---|---|
| Auto-adopt rate | A / N |
| Accepted error rate | (A − C) / A; N/A when A = 0 |
| Handoff rate | (N − A) / N |
| Failure rate | (N − V) / N |
| Valid accuracy | Correct valid responses / V |
| Overall correct | Correct responses / N |
The 5×5 matrix and per-label precision/recall include valid responses only. Fixed confidence buckets are descriptive diagnostics, not a calibration guarantee. Every displayed ratio includes its denominator.
Challenge cases
Ambiguous, multi-intent, or insufficient-context tasks are reported separately. Their labels and reasons are not sent to the model and do not influence route or handoff. Unlabeled cases never enter accuracy; confident forced choices remain visible as a limitation.
Measurement record
Runs pin the TypeSafe Direct provider, model jev-1.13.0, SDK, catalog, prompt, dataset, split, retry policy, timestamps, raw probabilities, confidence, attempts, duration, usage when returned, and normalized failures. A report is published only after an approval binds the exact source hashes. Existing public versions cannot be overwritten.
Latency is request timing, not downstream task time. Cost is not shown unless provider usage and a verified rate are both available; JevLab does not infer downstream savings.
Limitations
- The benchmark covers clear English routing tasks in five labels.
- A high-confidence suggestion can still be wrong or unsuitable, especially for ambiguous and multi-step tasks.
- Preview results describe only their exact reviewed dataset, provider, model, prompt, and run—not production reliability.
- Custom single runs have no human label and are never included in published accuracy.
Frequently asked questions
What is a Jev confidence threshold?
In JevLab, the confidence threshold (0.00 to 1.00) is the operating cutoff applied to TypeSafe Jev's output score. Suggestions with confidence at or above the threshold are automatically adopted, while suggestions below the threshold hand off to your existing escalation workflow.
What is the difference between none and handoff in JevLab?
none is an explicit, bounded classification outcome indicating that a task is outside the router's five defined capabilities. In contrast, handoff is a policy decision meaning the model's confidence was below your operating cutoff (or the response failed), so the task is returned to the caller for human or fallback review.
Is JevLab affiliated with TypeSafe?
No. JevLab is an independent, unofficial evaluation project and policy lab. It is not affiliated with, endorsed by, or maintained by TypeSafe AI.
Does JevLab execute agent tools?
No. JevLab evaluates routing suggestions and simulates policy outcomes only. It never executes code search, runs tests, reads documentation, inspects pull requests, or performs any tool actions.
How should I read coverage vs error rate when raising the threshold?
Raising the confidence threshold reduces the auto-adopt rate (coverage) and increases the handoff rate, while typically decreasing the error rate among auto-adopted decisions. JevLab visualizes this trade-off so you can select a threshold based on your application's risk tolerance.