Benchmark Analysis
Laya vs Jev Benchmark: How to Read the Conflicting Numbers
The public benchmarks comparing Convai's open-source Laya (ModernBERT-large) and TypeSafe's hosted Jev (System 1 primitive) appear completely contradictory at first glance. One report claims Laya beats Jev by 7.8x on speed and 0.766 vs 0.727 on accuracy; another shows Laya collapsing below random chance. Here is the factual breakdown of what those numbers actually measure, why they conflict, and what matters for production agent routing.
Why the numbers conflict: Laya's headline score of 0.766 accuracy comes from a specialized checkpoint (laya-typed-decisions) fine-tuned directly on the benchmark training split. In contrast, Laya's zero-shot base models score only 0.362 (English) and 0.342 (multilingual), which is below the 0.461 majority baseline. TypeSafe Jev achieves 0.727 zero-shot accuracy without task-specific tuning, and achieves higher soft distribution accuracy (0.580 vs 0.471).
Latency boundary: Laya runs at ~32.8 ms on a local T4 GPU (pure inference, zero WAN latency). Jev runs at 236–276 ms p50 as a hosted cloud API (includes public network round-trip).
1. The Core Discrepancy: Fine-Tuned vs Zero-Shot Generalist
To understand the benchmark battle between Laya and Jev, you must first separate the model checkpoints. The open-source Laya project maintains three distinct artifacts:
laya: Base English ModernBERT-large encoder (421M parameters).laya-multilingual: Multilingual mmBERT-base encoder (322M parameters).laya-typed-decisions: ModernBERT-large fine-tuned explicitly on the 2,000-sample typed decisions training distribution.
When publications claim "Laya outperforms Jev (0.766 vs 0.727)", they are evaluating the specialized checkpoint on its home turf. When third-party suites like JevBench v1.2 report Laya falling behind, they tested the base zero-shot checkpoint on broader intent tasks.
| Metric / Feature | TypeSafe Jev (1.13.0) | Laya (Fine-Tuned) | Laya (Base Zero-Shot) | Evaluation Context |
|---|---|---|---|---|
| Top-Label Accuracy | 0.727 (zero-shot) | 0.766 | 0.362 (below baseline) | 2,000 typed decision test set (baseline = 0.461) |
| Soft (Distribution) Accuracy | 0.580 | 0.471 | 0.312 | Evaluates full probability spread vs reference |
| Reported ECE (Calibration) | 0.144 (raw) / 0.246 (workflow) | 0.081 (after temp refit) | 0.213 (raw uncalibrated) | Lower is better; Laya requires post-hoc temperature refit |
| Inference Latency | 236 – 276 ms (hosted API) | 32.8 ms (local T4 GPU) | 32.8 ms (local T4 GPU) | Jev includes network WAN overhead; Laya is local process |
| Batched Throughput | Parallel multi-question API | 7.2 ms / query (50-batch) | 7.2 ms / query (50-batch) | Local batching saturates GPU memory efficiently |
| Choice Set Scaling (>20 opts) | Up to 255 options (~0.870) | Degrades on wide sets (~0.425) | Degrades on wide sets | Banking77 evaluation (Laya token budget constraints) |
| Cost & Licensing | $0.042 / 1M input (output free) | Apache-2.0 ($0 software) | Apache-2.0 ($0 software) | Laya requires self-hosted GPU infrastructure |
| Data Privacy Boundary | Cloud API egress | 100% Air-Gapped / On-Prem | 100% Air-Gapped / On-Prem | Zero data leaves private VPC when hosting Laya |
2. Programmable Zero-Shot vs Narrow Enums: The Founder Debate
When TypeSafe announced Jev, Replit founder Amjad Masad publicly questioned the premise on X:
"This is cool, but if your output domain is known in advance, why not just train a model to produce logprobs over enums?"
TypeSafe co-founder Diogo Almeida (co-inventor of ChatGPT) responded with the core design philosophy of Jev:
"Zero-shot + general == programmable. I would assume any extreme scale narrow task could then be fine-tuned for, but we'll see - I suspect putting it all in a shared cognitive core has big maintainability/generalization benefits."
This exchange explains the fundamental difference between the two approaches:
- Laya's philosophy favors fine-tuning a small, fast 421M encoder on your exact enum schema. If your routing labels never change, fine-tuning gives you exceptional 33ms local execution at 0.766 accuracy.
- Jev's philosophy favors an elastic zero-shot cognitive layer. You can change criteria, add new labels, or introduce ad-hoc questions in plain English without retraining or managing ML training pipelines.
3. Latency Boundaries: Local GPU vs Hosted Cloud API
Developer and researcher Matt Mastracci ran independent live evaluations comparing Jev against a local diffusion-based patch (DiffusionGemma-as-Jev on vLLM):
"DiffusionGemma and Jev make a similar number of mistakes, are both reasonable about their confidence when they do, but DiffusionGemma wins a bit on a PII test. I'd call it a tie... Is Jev faster than DiffusionGemma? No (API vs DGX Spark)."
When evaluating speed, remember that network topology dominates API calls:
- A hosted call to Jev incurs ~100–180ms of transatlantic/cross-country fiber latency, TLS handshake overhead, and gateway queueing before model computation begins.
- A local Laya deployment on an in-cluster GPU instance (e.g. AWS g4dn.xlarge with T4) eliminates internet traversal entirely, delivering true ~33ms p50 latency.
4. Why Top-Label Accuracy is Incomplete: Soft Accuracy and ECE
In automated agent routing, how the model fails is just as important as how often it is right. As Dan Shipper (CEO of Every) observed during early access:
"It doesn't produce words as output, it produces probabilities. So it can efficiently act as a judge in cases where you'd need a Fable-level model—but in our testing was 25x faster and 600x lower priced."
Comparing probabilities reveals why Jev maintains an architectural edge in safety-critical workflows:
- Soft Accuracy (0.580 Jev vs 0.471 Laya): When evaluating cross-entropy against true human ambiguity distributions, Jev captures second-place probabilities much more accurately. Laya tends to collapse all probability mass onto its top prediction.
- Expected Calibration Error (ECE): Laya reports an exceptional 0.081 ECE, but only after applyingtemperature fitting on held-out validation data. Out-of-the-box raw Laya has an ECE of 0.213, compared to Jev's published 0.144 baseline.
5. The Modern Agent Stack: Separating Thinking from Deciding
Community analysis by Simplifying AI summarized the emerging consensus across both commercial and open-source builders:
"Let the LLM research, plan, and write. Let Jev route, score, approve, or escalate. Let code execute the decision. That split changes the entire agent stack."
Meanwhile, open-source engineer Harsha Gundala demonstrated how fast this space moves by releasingQwen-2.5-1B-RLCD on Hugging Face shortly after Jev launched:
"They were building in stealth for 2 years, I was building in stealth for 2 hours... on Hugging Face now!"
6. Why Raw Benchmarks Don't Tell You Your Production Error Rate
In production, you never deploy a raw accuracy number like "0.766" or "0.727". Instead, you apply aconfidence threshold gate.
If your gate is set at 0.90, you only auto-adopt decisions where the model is highly certain. The remaining requests fail closed to a human reviewer or fallback LLM. The true question for your architecture is:"At what threshold do I achieve <1% accepted error while keeping auto-adopt volume above 70%?"
Interactive Policy Testing
Measure Auto-Adopt vs Handoff Trade-offs in JevLab
Don't guess confidence cutoffs from vendor whitepapers. Use JevLab's interactive slider to test real confusion matrices across 5 core agent routing outcomes (code_search, test_runner, docs_lookup, code_review, none) and export validated TypeScript code.
7. Verified Primary Sources & Public Discussion
Every citation and number on this page is traceable to verified primary disclosures, open-source repositories, and authenticated X threads:
- Diogo Almeida (TypeSafe Founder): Official Jev announcement, RLCD architecture, and $0.042/1M pricing.
- Doom Real-Time Intelligence Demo: ~10 decision calls/sec at ~$7/hour operating cost.
- Dan Shipper (Every CEO): First-hand production testing, "probabilities not words" paradigm.
- Matt Mastracci: Independent live evals of Jev vs DiffusionGemma on DGX Spark.
- Micha\u0142 Chromiak Technical Analysis: Rigorous cross-dataset evaluation of typed decision models in agentic AI.
- Convai Innovations Laya Repository: ModernBERT-large checkpoints, benchmark replication code, and license details.
- TypeSafe Official GA Release: Public general availability with no waitlist.
Frequently Asked Questions
Is Laya more accurate than TypeSafe Jev?
Only when using the specialized laya-typed-decisions checkpoint (0.766 vs 0.727). Laya's base zero-shot models score 0.362, falling below the 0.461 majority baseline. Jev provides reliable zero-shot accuracy across unfamiliar schemas without fine-tuning.
Can Laya run in an air-gapped VPC?
Yes. Laya is distributed with Apache-2.0 open weights on Hugging Face. You can deploy it in a private Kubernetes cluster or air-gapped GPU server with zero internet connectivity. Jev is exclusively available as a cloud-hosted API.
Why is Jev's latency higher than Laya?
Laya measures local GPU inference time (~33ms on Tesla T4). Jev measures total round-trip latency over the public internet (236–276ms p50), which includes WAN transmission, TLS handshakes, and gateway routing.
How does choice count affect performance?
Laya allocates a fixed token budget across candidate options. Beyond ~20 choices, accuracy degrades (e.g., Banking77 drops to 0.425). Jev supports up to 255 choice options and maintains ~0.870 accuracy on the same task.
Which model should I choose for agent routing?
Choose Laya if you have fixed routing schemas, in-house GPUs, strict air-gap compliance requirements, and sub-50ms latency SLAs. Choose Jev if you need zero-shot flexibility, wide option sets, calibrated soft distributions, and want to avoid maintaining GPU infrastructure.