JevLab

Scope

One routing cutoff. That is the whole measurement.

JevLab measures how a confidence cutoff trades auto-adopt, accepted error, and handoff. The task is one clear English request, classified into five labels. The lab simulates that policy.

Inside the measurement

  • The labels are code_search, test_runner, docs_lookup, code_review, none.
  • A valid response at or above the cutoff becomes a route suggestion. The slider starts at 0.80, which is an interaction starting point.
  • A lower score, or any failed response, becomes a handoff. Handoff returns the task to the caller. This site does not open a ticket.
  • The TypeScript export uses that same comparison. The direct adapter is pinned to jev-1.13.0 when a live call is configured.
  • English is the language of the benchmark. The interface can accept another language when live evaluation is enabled. Those runs are not part of a validated result.

Outside the measurement

  • Counting, arithmetic, and date math.
  • Searching a game, or simulating one step ahead in a puzzle. Those tasks need a benchmark built for that state. The labels here are work routes.
  • Executing code search, tests, documentation lookup, or review. A suggestion is data. The named tool does not run.
  • Approving a destructive tool call. A gate that may deny or ask, and must not approve, is a different case.
  • Choosing a cheaper or a larger model. Cost routing is not this case.
  • A directory of use cases, or a live playground that runs failure modes on demand.
  • A recommended production threshold. The lab does not mark one.

A confident wrong label

A high score can still name the wrong label. An ambiguous, multi-step, or under-specified task can receive a confident suggestion. The lab asks for one clear task. It does not detect ambiguity, and challenge labels are never sent to the model.

When a reviewed challenge set is published, it stays out of the main accuracy totals. Confident forced choices stay visible. How to read the confidence value separates ranking, gating, and a close call.

Current evidence

No reviewed benchmark is published. The lab fixture is an unverified candidate set with hand-authored simulations. No agreement rate or model accuracy is claimed. The status is kept on the methodology page.

JevLab is an independent project. It is not affiliated with TypeSafe.