Jais 3 vs Fanar 2: The Two Vendors Do Not Agree on What Fanar 2 Scores

J​ais 3 and Fanar 2 are the two Gulf flagship Arabic model families, and the most useful thing anyone can tell you about comparing them is that the vendors' own numbers for Fanar-2-27B-Instruct do not match. QCRI's model card reports ArabicMMLU at 74.67% and Belebele at 86.81% for its 27B instruct model. The J​ais-3-32B card, from the Institute of Foundation Models and Inception42, runs the same opponent on its own reference table and puts Fanar-2-27B at 66.3 on Arabic MMLU and 61.8 on Belebele. That is an eight-point gap on one benchmark and twenty-five points on another, for one model, and neither number is a fabrication — they are the output of different harnesses, different prompt formats, and in IFM's case an audited sample set. It is also the single most instructive thing on this page, because it means that a comparison of J​ais 3 and Fanar 2 built out of published numbers is a comparison of two measurement practices as much as two models.

The two cards, plainly

• J​ais-3-32B — 32B dense decoder-only from IFM, up to 524,288 tokens of context, Apache 2.0, weights published 29 September 2026. Ships with a reasoning parser (k2_horizon), a tool-call parser, and three selectable tool-call formats. Serves at tensor parallelism 2 in bfloat16.

• Fanar-2-27B-Instruct — 27B dense from QCRI at Hamad Bin Khalifa University, 32,768 tokens of context, Apache 2.0, released March 2026. Continually pretrained from google/gemma-3-27b-pt on roughly 166B Arabic and English tokens across three recipes and then merged, followed by a five-stage post-training pipeline.

• Scale range — J​ais 3 is a family, six base checkpoints from 0.9B to 375B including two mixture-of-experts models. Fanar 2's language line is essentially one instruct model at 27B, with siblings in other modalities rather than smaller text checkpoints.

• Licence — Apache 2.0 both sides, ungated both sides. Neither family makes you sign anything, which is worth stating because it is not the norm for the region.

• Base architecture — J​ais 3 is trained by IFM with a custom architecture (k2_horizon, custom remote code required). Fanar 2 is a continual-pretrain of G​oogle's G​emma 3 27B, which makes it a different kind of artifact: a well-executed adaptation rather than a from-scratch training run.

Where the numbers diverge, and why that is not a scandal

Take the benchmark names one at a time, because the label is doing less work than it appears to.

• ArabicMMLU versus Arabic MMLU* — Fanar's card says 74.67% for the benchmark it calls ArabicMMLU. IFM's table says 66.3 for the column it calls Arabic MMLU*, with the star meaning IFM excluded questions with ambiguous prompts, invalid options or indefensible gold labels. Two different sample sets, one of them cleaned, and no obligation on either lab to report the other's.

• Belebele, 86.81 versus 61.8 — a much larger gap, and the harness explanation stretches further here. Fanar's figure is dialectal Arabic; IFM's column reads the same benchmark name. Twenty-five points is not a rounding difference and the honest conclusion is that at least one of these runs is not measuring what the other one is.

• IF-Eval, 82.97 versus 49.8 — Fanar reports 82.97 on IF-Eval in its own improvement table. IFM's reference column puts Fanar-2-27B at 49.8 on Arabic IFEval. Same capability, different language and different prompt set.

• What both agree on shape, if not level — Fanar 2 is a large step over Fanar 1. QCRI reports +7.32 on ArabicMMLU, +10.68 on GSM8K and +8.27 on IF-Eval versus its own 9B predecessor. That claim is internally coherent and nobody disputes it.

The rule that falls out of this is not "one lab is cheating". It is that for Arabic benchmarks, a number without a harness attached is not a number. IFM at least marks which of its own figures it audited and publishes the excluded samples in a separate collection, which is more than most labs do — but it is still IFM auditing IFM's evaluation of a competitor.

Context and deployment are where the comparison is clean

These are the rows not subject to harness arguments, because they are architectural facts rather than measurements.

• Context — 524,288 tokens for J​ais-3-32B against 32,768 for Fanar-2-27B-Instruct. Sixteen times the window, and Fanar's card documents 54K long-context instructions used to reach 16K during training, which puts the trained long-context behaviour well inside the nominal 32K.

• Serving footprint — J​ais-3-32B at TP=2, bfloat16, vLLM ≥ 0.30.0 or SGLang ≥ 0.5.20. Fanar-2-27B is tested with transformers 4.57.6 and is a standard G​emma 3 27B shape, so it slots into any stack that already runs G​emma.

• Tooling shipped with the model — J​ais 3 includes the reasoning parser, the tool-call parser and a choice of json, xml or xml_typed tool formats as first-class serving flags. Fanar 2 supports generic tool calling plus ten built-in Fanar tools, but the integration surface is the vendor's own API rather than a set of framework flags.

• Third-party evaluation — neither has an Artificial Analysis entry, neither has an arena score, and neither has an independent replication. Both families' numbers are vendor-reported and the article says so before it quotes them, every time.

What Fanar 2 has that J​ais 3 does not

Three things, and they are not small.

• Native Arabic reasoning traces. Fanar 2 was trained on roughly 250,000 Arabic reasoning examples and generates its chain of thought in Arabic rather than translating from an English trace. J​ais 3 exposes a reasoning parser and reports strong Arabic agentic numbers, but its card does not make the same claim about the language the thinking happens in.

• Hallucination mitigation as a designed behaviour. Fanar 2 is trained to abstain — knowledge probing during training, five-step structured verification traces, calibrated "I don't know" responses — and it wraps spontaneous Quranic verse references in validation markers so a downstream system can check them. That last feature is unusual and has no counterpart in the J​ais 3 card.

• A product around the model. QCRI ships a chat interface, iOS and Android apps, and a documented API with an access-request flow, plus sibling models for image generation, image understanding and poetry. J​ais 3 is a release of weights with framework recipes; the speech model sits alongside it, and the rest is a research artifact.

Those are capability differences, not benchmark differences, and a fair reading is that Fanar 2 is more finished as a product while J​ais 3 is more ambitious as a model.

What the agentic columns actually say, with the asterisk attached

IFM's own table is the only published head-to-head between these two models and it is not flattering to Fanar: Terminal Bench 2.1 at 63.7 against 1.1, GDPVal at 75.5 against 0.0, MCPMark 66.1 against 12.1, SWE-Bench Verified 72.4 against 0.4, Arabic file processing 71.8 against 0.0.

Read those as what they are. They are IFM's measurements of a competitor's model, run by IFM, on IFM's harness, and published on IFM's model card. A zero on GDPVal and a 0.4 on SWE-Bench Verified for a 27B instruct model is plausible — these are agentic benchmarks that punish models without tool-call training data at scale, and Fanar 2's agentic work is oriented around its own ten tools rather than general software tasks. But a lab that reports its own model winning every row of a table it built is telling you about its evaluation, not about the world.

The defensible statement is narrower: on IFM's published harness, J​ais-3-32B outperforms Fanar-2-27B-Instruct on every agentic benchmark reported, and Fanar's card does not report agentic benchmarks at all, so nothing contradicts it and nothing corroborates it either. The published evidence is one-sided because only one side published.

Which one to pick

Long context, agentic work, or a need to pick a checkpoint size: J​ais 3, and it is not a close call on context. A 512K window against 32K decides document processing, repository-scale work and long agent trajectories by itself.

Arabic reasoning you intend to read, abstention behaviour you can rely on, Quranic-content handling, or a vendor with an app and a support path: Fanar 2. QCRI's post-training pipeline is genuinely elaborate — 4M SFT instructions, 280K DPO pairs, five stages, a deliberate capability-rebalancing pass to stop long-context adaptation from degrading short-form tasks — and that shows up in dimensions no leaderboard column captures.

If the decision is hard because both are plausible for the same workload, the thing that would settle it is your own prompts, run against both, scored by your own criteria. That is the experiment nobody has published for these two models, and it is worth more than every number quoted above.

Running it is less work than it used to be. OrcaRouter gives you one key across 200-plus models with 0% markup, provider list price passed through, so a bake-off between two Arabic checkpoints is a routing change rather than two integrations — and item-by-item scoring is exactly the case the routing DSL is built for, because it lets you send the same prompt to both models in one composed call instead of writing a comparison harness. Neither family is on our catalogue right now; the model list is where to check what is.

What would settle this

One independent run of both models on a shared harness, published with the prompts. That is what is missing, and it is missing for every mainstream Arabic model, not just these two — the whole field is running on self-reported numbers, which is why an eight-point disagreement about Fanar 2's ArabicMMLU can sit in public for seven months without anyone resolving it. Until someone does, treat the J​ais 3 table as IFM's argument and the Fanar 2 table as QCRI's, and treat the architectural facts — 512K against 32K, seven checkpoints against one, parser flags against a hosted API — as the part you can actually plan around.

What the numbers cannot tell you is whether any of it has gone wrong in the field: the documented AI incidents and agent failures around models in this class are collected in the OrcaRouter AI Incident Archive, where each entry carries its own sources, severity, confidence grade and disputed flag.