Can a Turkish decision model score options without depending on their order?

Order invariance comes from scoring options blind to each other

ufakzeka-karar is an open Turkish decision model with 182,494,466 parameters, built on the lab's ufakzeka-1-base. Given a Turkish text and a question of a fixed answer type, it returns a temperature-scaled probability for every option plus an expected error that acts as a "not sure" signal, without generating text and in one CPU forward pass for up to ten options.

The mechanism behind the title question is architectural. A question is one sequence: a prefix holding the text, "Soru:" and the question, then each option as its own segment. The prefix attends only to itself, and each option attends causally to the prefix and itself, never to another option; every option starts at the position after the prefix. Each option's hidden states are mean-pooled and scored by one linear layer, and a softmax runs over the options. Because an option's logit does not depend on its pass-mates, the model's order sensitivity is 0.000 by construction, and the repository's permutation test on the real weights demands bit-for-bit equality.

This matters because option order is a known failure mode. On HakemBench's permutation probe, order sensitivity among the other measured models runs from 0.039 (Jev 1.13) to 0.590 (Laya). The paper's own comparison is sharper: a sequential head that reads options one after another, trained with shuffled options, was about as accurate but changed 2.3 to 2.8 percent of its answers when only the option order changed. The paper claims only the smaller result, that the order-invariant head is as accurate and never depends on option order, because the sequential head's four-point loss in the second development round did not reproduce in Run 1.

Cross-entropy beat REINFORCE by 10.2 macro F1 points

The training objective is cross-entropy against each row's target distribution, soft where labellers disagreed, plus on score questions the ranked probability score, so mass two levels off costs more than one. Both are strictly proper. Rows are weighted to each task's natural answer mix, and no reinforcement learning is used.

The ablation table gives the size of that choice. REINFORCE, with eight samples a question and the target's mass on the sampled answer as reward, lost 10.2 points (0.102) of macro F1 to cross-entropy in Run 1, with a 95 percent interval of −13.5 to −6.2, and lost 12.7 points in the second round. Every calibration measure was worse. The paper's explanation is structural: REINFORCE's expected reward is linear in the probabilities, so its optimum puts all mass on one answer, while cross-entropy's minimiser is the target.

The same table shows what did not help. Down-weighting easy rows was rejected at −2.2 points. Continued pretraining for 300M tokens cost 1.26 points. MoganBert-TR with the same head and data lost 0.79 points on decision tasks and 3.37 points on held-out questions. The backbone had been fixed as the lab's own before the encoder comparisons, so those are reported, not acted on.

It ranks 7th of 16, and three tracks are flagged

On the open set of HakemBench v1.0, which holds 2,346 items and 4,275 questions in seven tracks, the released model ranks 7th of 16 rows with a composite of 0.660 (95 percent interval 0.642 to 0.677). The composite is the equally weighted geometric mean of three axes averaged over tracks: decision quality (macro F1), calibration (Brier score relative to the uniform guess) and selective automation (AUGRC relative to a model wrong on every question).

Three hosted chat models (Gemini 3.8 Flash, GPT-5.6 Sol, GLM 5.3) and the decision API Jev 1.13 lead clearly at 0.825 to 0.888, and Kev 4B and Kev 9B also score higher. The intervals of ranks 5 to 9 overlap the model's own, so that band is not settled. On decision quality alone it is also behind Qwen3.5-4B (0.741) and DeepSeek V4 Pro (0.761), and on calibration (0.482) it is close to Kev 4B (0.503).

The ranking carries a disclosure that changes how it should be read. The released model is the last of three runs scored on HakemBench, and its numbers are not blind. The second run's new training data was aimed at the first run's errors on the full test set in guardrails, moderation and customer support, and the released run was trained after the second run's guardrail results on the full test set were read, under a protocol fixed in writing before any of its data, code or runs. Its guardrail, moderation and customer support numbers carry the flag "shaped by reading the test results". With every model scored on the other four tracks only, its composite is 0.678, 6th of 16.

Temperature scaling helped development but hurt held-out questions

The probabilities are the product here: a caller acts on answers above a threshold and sends the rest to a person, which works only as well as the probabilities are calibrated. Temperature scaling was fitted by cross-entropy on a fit part of the validation rows, and the form with the lowest cross-entropy on a selection part was chosen, preferring a simpler form within 0.001. For the released run that gave one global T = 1.2614.

The result is a split. Temperature scaling lowers calibration error (smooth ECE) on the development set but raises it on held-out support questions, from 0.027 to 0.045 for the first scored run, which never trained on them. The released model later trained on them, so its 0.036 to 0.064 is not an unseen-question test. The paper's likely reason is that a temperature fitted on mildly overconfident validation data overcorrects on questions at a different confidence level. The selection rule was fixed before the held-out result was seen, so T = 1.2614 ships as chosen; T = 1 would have kept its held-out calibration error at 0.036. On the benchmark the model stays somewhat overconfident, with a post-hoc temperature of 1.1733.

Calibration also varies by track. The guardrail track has the highest calibration error, a smooth ECE of 0.158 against 0.025 to 0.084 on the other tracks. Accuracy runs from 0.569 [0.534, 0.605] on customer support (1,340 questions, a flagged track) and 0.640 on fact-check triage to 0.952 on moderation.

The not-sure signal cannot be relied on to catch guardrail errors

The model returns an expected error that serves as a "not sure" signal, built as an isotonic map fitted on the selection part, the error being one minus the target's mass on the model's answer. Reading confidence as one minus the not-sure value, a threshold of 0.9 lets through 0.292 of the 4,275 open questions (1,248), and 0.062 of those answers are wrong.

At 0.99 the error is higher, 0.085 on 213 answers. Of those 213, 148 are guardrail answers, and 18 of those are wrong (0.122, Wilson interval 0.078 to 0.184); no other track has an error there. The paper's conclusion is direct: the signal cannot be relied on to catch guardrail errors. On customer support almost no answer reaches 0.9 (6 of 1,340). No other interval was computed for these figures.

The guardrail track is also where the model's own behaviour is weakest in absolute terms. It catches 194 of 217 attacks and passes 128 of 201 harmless look-alikes. The paper notes that every training attack was a template or a short translated prompt, and that the only guardrail texts in a real user's register were Run 2's harmless ones, a likely register shortcut.

Support routing falls below a baseline that reads no meaning

The weakest places by track are the support choice and score questions (accuracy 0.490 and 0.481) and the triage priority scale (0.500, macro F1 0.268). A surface-cue baseline, which answers from cues such as length, digits or a question mark rather than from meaning, beats the model over the whole open set on the support choice (0.516 against 0.490) and score questions (0.507 against 0.481), partly in-sample because its cues were fitted on part of the open set.

On the 84 support questions of the human check, answered by hand without seeing the labels, the model's accuracy is 0.476 [0.369, 0.583], below the baseline's 0.619 [0.512, 0.714]. The three leading hosted chat models and Jev 1.13 score 0.738 to 0.786, and DeepSeek V4 Pro 0.583.

Two scope conditions sit beside this result. The support questions are a flagged track, and Run 2's new support questions are the four held-out questions, with the held-out file sharing 887 of its 937 texts with the validation pool, so support is no longer a generalisation test. Their labels also come from the AI model family that settled the benchmark's support gold under the same rubric, so a support score there is partly agreement with the labeller.

World knowledge is weak and the model reads Turkish only

On the knowledge exam MMLU-Pro-TR the released model is slightly below chance: 0.098 [0.092, 0.103] against 0.111. The paper states plainly that world knowledge is weak. The model reads Turkish only, and the end of a text beyond 448 tokens is cut.

The external Turkish test sets show a mixed picture, and the paper is careful about which numbers are supervised. MASSIVE, MiDe22 and OffensEval numbers are supervised, not zero-shot, because their train splits are in the training data. MMLU-Pro-TR, XCOPA and X-FACT touch no training text, though all of X-FACT's Turkish claims come from one of FACTurk's sources. From Run 1 to Run 3 accuracy fell a little on the first four sets and rose on XCOPA and X-FACT; the intervals overlap on every set, and no paired test was run.

Two further boundaries belong here. The 181 court items and the 28 guardrail items taken from public prompt-injection sets are the test splits of datasets whose train splits are in the model's training data, so on those questions the model is supervised in-distribution while the other models answer zero-shot; its legal routing score (macro F1 0.754) and the guardrail counts should be read with that. Most training labels come from LLMs or rules, and no prompt-injection or relevance label was checked by a person.

One CPU thread, 102.5 ms a question, and a robustness split

The model answers a typed question in a median of 102.5 ms on one CPU thread. On an Apple M1 with one PyTorch thread and 13 sample questions, it took a median of 102.5 ms a question (69.8 to 194.5 ms) with peak resident memory of 986,038,272 bytes in 32-bit floating point. A CPU model can run on the caller's own machine, so the text need not leave it.

The robustness probe splits in two. Half of the model's robustness value (0.941 [0.917, 0.966], against Jev 1.13's 0.943) is 1.0 by construction because order sensitivity is 0.000. On the measured half, paraphrase agreement, it is 0.883 [0.833, 0.932] against Jev's 0.926 [0.889, 0.963], with overlapping intervals. The English Brier gap (Turkish minus English Brier score; negative means worse in English) is −0.303 [−0.377, −0.225]; the model was not trained for English.

Quantisation did not survive the acceptance rule. No int8 file stayed within the required 0.5 points of fp32 on macro F1, even with the 24 feed-forward down-projection matrices in fp32; dynamic quantisation lost 1.42 points, per channel 1.61, and down projections in fp32 0.67. The loss came from near-tie answers, so the release is fp32. Weights and code are under Apache-2.0.

Trust the order invariance, flag the three shaped tracks

The paper's most defensible claim is the one the title asks about. Order invariance is a property of the construction, verified bit-for-bit on the real weights, and it is the one result the paper explicitly narrows to what reproduced: the order-invariant head is as accurate and never depends on option order. The 10.2-point macro F1 gap over REINFORCE is the second result that survives its own ablation protocol, with the compared choices, measures and decision rule fixed in writing before each ablation ran.

The benchmark standing needs a different reading. The composite of 0.660 (7th of 16) and the four-track composite of 0.678 (6th of 16) are not blind numbers: every figure of the released model comes after readings of earlier runs' test results, and guardrail, moderation and customer support carry the flag "shaped by reading the test results". The paper also notes that the board's intervals resample items, but items a model wrote in one generation request are correlated, so they are likely too narrow.

For a reader deciding whether to use this model, the paper's own variables give the split. If the requirement is a Turkish typed-decision model whose answer cannot change when options are reordered, and that runs on one CPU thread in about 102.5 ms, the construction delivers that. If the requirement is support routing, the model is below a surface-cue baseline on the 84 human-checked support questions (0.476 against 0.619). If the requirement is a reliable "not sure" signal on guardrails, the signal misses those errors: at a 0.99 threshold, 148 of 213 answers are guardrail answers and 18 of those are wrong. The order-invariance claim is the one to build on; the flagged tracks are the ones to re-measure before trusting.

Sources used in this answer