A 4-billion-parameter open-source decision model finishes ahead of 103 open-weight systems, including some built on base models nearly eight times its size, on a benchmark that weighs intelligence, calibration, speed and cost equally.
MOUNTAIN VIEW, Calif.--(BUSINESS WIRE)--#ai--H2O.ai, the trusted Sovereign AI Platform for predictive, generative and agentic AI, with observability built in, today announced that H2O-Lightning-4B, its open-weights decision model, ranked first among 104 open-weight systems on the JevBench Composite Score leaderboard as of October 7. The model scores 72.5 out of 100, above the 71.5 posted by Jev, the benchmark's closed API reference system, at an estimated $0.021 per 1,000 decisions.
Much of the AI work inside banks, insurers, health systems, telcos and government agencies is a decision about a record: route this ticket, flag this clause, score this claim. TypeSafe AI built Jev around that idea. Jev is a model that answers a typed question with a calibrated probability instead of generated text. Each decision needs the right answer, a probability a risk team can set a threshold on, a latency that fits a live workflow, and a cost that stays predictable at millions of calls a month. JevBench scores all four on an equal-weight harmonic mean, so a weak axis drags down the whole score. An open 4-billion-parameter model scoring above a closed reference system is the shift regulated buyers have been waiting for. Owning the model no longer means giving up quality.
"Better decisions don't need bigger models. Classification is one of the most common enterprise use cases, and models like H2O-Lightning deliver frontier-model quality and lightning-fast decisions at about 100 times lower cost. Leading AI companies, including Microsoft, are now building decision models of their own. This is not a flash in the pan," said Sri Ambati, Founder and CEO of H2O.ai.
How it works
H2O-Lightning-4B is a fine-tune of Qwen3.5-4B. It answers typed decision questions about a record such as a document, ticket, policy or conversation, and returns a probability for every option. It supports three question types: choice (one of a set of named options), yes/no (the probability that a statement is true) and score (an ordinal level on a stated scale, with its probability distribution).
Each decision is one forward pass and one output token, so cost is driven by input tokens alone. The model runs on unmodified vLLM 0.30.0 with a small standard-library shim in front.
The results
H2O-Lightning-4B competes in a crowded, fast-moving field. TypeSafe AI set the bar with Jev. Independent developers and enterprise teams have since put more than 100 open-weight systems on the JevBench board, and its maintainers score every entry the same way. The category is now drawing the largest players: Microsoft last week introduced Microsoft-Decision-1, a 9-billion-parameter decision model evaluated on its own internal benchmark suite. About one point separates the top of the JevBench board, and the order will keep changing.
On JevBench v1.6.1, H2O-Lightning-4B scores 90 on calibration, 92.6 on speed, 60 on intelligence and 60 on cost. Its measured median latency is 29 milliseconds per decision. Against the second-ranked open-weight system, an adapter on a 31-billion-parameter model, it matches calibration, runs faster and costs less than half as much per 1,000 decisions (estimated $0.021 versus $0.045). Against the Jev reference, it runs at roughly one-third the latency and two-thirds the cost.
Open, and yours to own
H2O-Lightning-4B is released under the Apache 2.0 license and joins H2O.ai's open-source work, including H2O-3, h2oGPT, LLM Studio and Danube. Organizations own the model, their data and the results, and can run it on-premises, air-gapped, in a private cloud or in the cloud. The H2O Sovereign AI Platform works on the same principle: H2O AI Super Agent™ orchestrates predictive, generative and agentic AI in one system, with observability built in, and routes every job to the right model instead of sending everything to one costly frontier model.
"Calibration is the score we watch most. When H2O-Lightning-4B says 80 percent, it should be right about 80 percent of the time, so a team can automate the confident calls and send the rest to a person. We kept the serving path plain on purpose: stock vLLM, one forward pass, one token," said Jonathan C. McKinney, Chief of Technology at H2O.ai.
H2O-Lightning-4B is available now at huggingface.co/h2oai/h2o-lightning-4b.
Read the technical blog: https://h2o.ai/blog/2026/h2o-lightning-jevbench/
See full demo video: https://www.youtube.com/watch?v=2Qp04Wu0A14
Microsoft-Decision-1 announcement: https://commandline.microsoft.com/microsoft-decision-1-model-foundry/
About H2O.ai
H2O.ai created the industry's trusted Sovereign AI Platform because organizations should own their AI: their agents, models, data, and results, run anywhere from cloud to air-gapped. H2O Super Agent orchestrates predictive, generative, and agentic AI in one system, with observability and governance, empowering regulated and public sector organizations to get value from AI. Moving from pilot to production takes expertise, from top-ranked data scientists, including Kaggle Grandmasters, to forward-deployed engineers embedded with customer teams.
Built on a decade of open source, H2O.ai is trusted by 20,000+ organizations, half the Fortune 500. AT&T, Commonwealth Bank of Australia, Wells Fargo, Bank of America, Workday, Progressive Insurance, and NIH rely on H2O.ai to cut cost and risk and own their AI future.
Through its AI for Good program, H2O.ai applies technology and investment to non-profits and academia solving hard problems in health, education, and the environment. Learn more at h2o.ai.
Contacts
Media Contact
Bruna Smith
bruna.smith@h2o.ai
