How accurate and well-calibrated is JEV compared with LLMs on political science tasks?

JEV returns probabilities over answer options, not generated text

JEV is a commercial model that TypeSafe markets as a "System One" model, borrowing the label from the fast, intuitive mode of thinking. Where a generative LLM answers by writing tokens one at a time and the researcher reads a label or number out of that text, JEV takes a piece of text (the "state") plus one or more questions with a fixed answer set, and returns decisions with a probability distribution over the supplied options. It offers three output types: a choice from a user-provided category list, an item on a worded scale, and the probability of a yes answer to a binary question.

That design matters for how uncertainty is read. JEV returns a probability for every option the researcher supplies, rounded to two decimal places, regardless of how many options there are. LLM token probabilities are defined over possible next tokens instead, so a researcher has to map tokens to answer labels, discard probability mass on tokens that are not labels, and renormalise. Closed APIs also truncate what they report: for GPT-6 Luna, OpenAI returns at most the five most probable tokens at each position, so on a nine-point scale some levels receive no probability at all, and it returns none when the model reasons before answering.

The paper notes that TypeSafe has not published JEV's architecture, so the claim that the model reads the state once and evaluates questions in parallel cannot be checked directly. TypeSafe attributes its calibration to a post-training method it calls reinforcement learning for calibrated decisions, which it says optimises for "epistemically honest probabilities." The authors point out that calibration depends on the data as well as the model, so probabilities calibrated on one set of texts need not be calibrated on another.

Seven published replications, scored against human coding

The study re-runs seven published applications that evaluated LLMs for annotation, scaling, or data generation, keeping each study's human benchmark as the reference. Five are reading tasks, where the model is given a text, and two are recall tasks, where the model is given only a name and must draw on embedded knowledge: an ideological scaling of European political parties, and a reproduction of V-Dem's expert codes for 53 indicators in 2023.

JEV was called via OpenRouter (and TypeSafe's own endpoint for V-Dem), GPT-6 Luna was called directly from OpenAI servers, and Qwen3.8-27B was called via a third-party inference provider at 8-bit precision. The authors retained each published task and prompt wording where the interface permitted, removing reply-format instructions that JEV cannot follow.

The headline comparison is against the human reference, not against the other models. Against GPT-6 Luna and Qwen, "better" or "worse" means the paired 95% interval excludes zero, while "slightly better" or "slightly worse" means JEV's estimate is higher or lower but the interval includes zero. The authors also flag that the published models were trained after some of the original studies were released, so comparisons with those published models may favour the newer models.

JEV matches or nearly matches both LLMs on accuracy

Across the seven applications, JEV performs on par with its contemporary commercial and open-weight alternatives when human coding is the benchmark. In the V-Dem recall task, JEV has a small but clear lead over every model: its mean correlation with V-Dem is higher than GPT-6 Luna's by 0.05 [0.04, 0.07], GPT-4o's by 0.05 [0.03, 0.06], Qwen3.8-27B's by 0.09 [0.07, 0.11], and Llama-3.1 70B's by 0.18 [0.16, 0.20].

Elsewhere the picture is mixed rather than dominant. In the party manifesto and legislative speech replication, JEV's correlation with the benchmark is 0.97 on UK manifesto economic positions, 0.92 on social positions, and 0.81 on EU speech subsidies, rising to 0.86 when sentences are relevance-weighted. GPT-6 Luna scores 0.98, 0.95, and 0.95 on the same three, and Qwen3.8-27B scores 0.97, 0.94, and 0.90. The paired gaps show JEV below GPT-6 Luna on all three measures and below Qwen on the subsidies measure, with the economic and social gaps small.

The authors' summary is that JEV annotates and scales text in these political science contexts about as well as a mid-tier commercial LLM and, in many instances, as well as open-weight models that can be run locally. The V-Dem lead is the clearest single accuracy result in JEV's favour; the manifesto and speech results are the clearest cases where it trails.

No cost advantage over GPT-6 Luna at batch prices

TypeSafe charges only for input tokens and makes output free, and advertises large cost and speed advantages on the reasoning that output tokens cost more than input tokens and reasoning models can write hundreds or thousands of them before answering. The authors test that claim on the primary task of each application.

For classification with short labels, most of the bill is input. In these tasks GPT-6 Luna wrote about four tokens per answer, while JEV counts the question and every answer option as input and billed 1.2 to 3.3 times as many input tokens per decision as Luna. The result is that JEV is not cheaper than GPT-6 Luna at OpenAI's batch prices: Luna costs less on six of the seven tasks and about the same (3% more) on the seventh. Against Luna's standard prices, which apply when answers are needed at once rather than within a day, JEV is cheaper on six of the seven tasks.

Qwen was more expensive than both GPT-6 Luna and JEV when run through an external provider's API, at 2.35 to 5.41 times JEV's cost per decision, and batch processing was not available for that model. The authors note, however, that given the model's small size, many scholars can run it on a local commercial machine or on university supercomputer infrastructure at no cost to the researcher. JEV is faster than both alternatives: sending one request at a time, it returned each decision faster, which the authors describe as a clear speed advantage.

JEV's probabilities beat GPT-6 Luna's token probabilities

Probabilities serve two distinct uses in this literature, and the paper keeps them apart. First, they rank texts by how likely each answer is, so a researcher can screen a corpus for a rare category or route the least certain answers to a human or a more capable model; this requires the probabilities to rank answers well, which AUROC measures. Second, a researcher can weight each answer by its probability to build a continuous scale, or read the probabilities as rates; this also requires calibration, which expected calibration error (ECE) measures.

The distribution of GPT-6 Luna's probabilities is the starting point. Across all party comparisons in the replication of Di Leo et al. (N = 347,801), 74% of Luna's probabilities are below 0.01 or above 0.99, against 18% for JEV and 16% for Qwen. JEV returns more mid-range probabilities, which the authors say is what you would expect when two parties are close ideologically; Qwen does the same, so near-certainty is not a property of LLMs as such.

When each question is asked once, GPT-6 Luna has higher calibration error than JEV on all eight reading tasks with a categorical human label, which the authors read as suggesting TypeSafe's calibration claim may be warranted against Luna. The AUROC picture is similar: Luna's AUROC is lower than JEV's on seven of the eight tasks and level on the other. The authors are careful that the probability distribution difference is indicative but not conclusive evidence of poor calibration in GPT-6 Luna, and they report the ECE and AUROC results directly.

The calibration edge does not hold against Qwen3.8-27B

The comparison with the open-weight model is where the calibration story becomes task-dependent. Qwen3.8-27B's calibration differs from JEV's by type of task: its calibration error is lower on the four pairwise tasks, higher on three of the four labelling tasks, and higher on both attack-type arms and both recall tasks. JEV's advantage therefore holds relative to Luna's token probabilities but not consistently relative to Qwen's, and where JEV does show better calibration than Qwen, the authors note the differences are not dramatic.

The AUROC results follow the same pattern. Luna's AUROC is lower than Qwen's on all eight tasks. On the attack-type task, where Luna writes its probabilities in the reply, Luna's AUROC is above Qwen's in both arms and above JEV's only when given the codebook. JEV and Qwen are not consistently different from each other.

Two conditions change the calibration comparison itself. The gap against Luna disappears on the open-ended survey pairs when probabilities are averaged over the two presentation orders, and it reverses on the attack-type task, where Luna writes its probabilities in its reply. So the "better calibrated than GPT-6 Luna" result is specific to the one-call-per-question setting with categorical human labels, not a general property of the model.

Choose JEV for speed and parseable probabilities, not for cost

The paper's own conclusion is that JEV is a capable model for text scaling and annotation in political science contexts, but that there are no major gains in cost or capabilities that make it a clear choice for most applications. The main advantages lie in its speed and the ease of parsing the underlying probabilities. The authors state that unless researchers have a need for speed, JEV's only obvious advantage is ease of parsing the underlying choice probabilities.

The task-dependent split is the practical takeaway. Where a researcher needs a probability for every supplied option without mapping tokens to labels or renormalising, and where the task is one of the eight reading tasks with a categorical human label asked once per question, JEV's probabilities were better calibrated than GPT-6 Luna's. Where the comparison is against Qwen3.8-27B, or where probabilities are averaged over presentation orders, or where the model writes its probabilities in the reply, that edge is not consistent.

The authors also stress that their analysis still requires researchers to validate JEV as they would other LLMs. Because TypeSafe has not published the architecture, the training data, or the reward, the mechanism behind any calibration advantage cannot be checked directly, and calibration depends on the data as well as the model. The decision threshold the paper supports is therefore narrow: JEV is a reasonable choice when speed and directly parseable option probabilities matter, and it is not a clear choice on cost or on accuracy against the models tested here.

Sources used in this answer