Jais 3 vs SILMA 9B: A Model Challenge Between Two Different Eras of Arabic AI

Open SILMA AI's website and the flagship it leads with is SILMA TTS v2.0, a text-to-speech model quoting 170 ms time-to-first-token for voice agents. Scroll to the footer and there is still a link to an Arabic LLM leaderboard, and the model this matchup names — SILMA-9B-Instruct-v1.0, a 9B bilingual Arabic and English instruct model from the silma-ai/ organisation — is still on the Hub. It was published on 17 August 2024. Against it sits Jais 3, the family from the Institute of Foundation Models and Inception42 whose weights landed on 29 September 2026 and whose serving recipes were stamped 5 October. That is a twenty-five-month gap and a change of industry category, and it makes this the strangest page in the set: not a matchup between two rivals, but between a checkpoint from the era when Arabic LLMs were scored by log-likelihood on translated benchmarks and a 2026 family that ships reasoning parsers and 512K context windows. The comparison is still worth making, because teams do still run SILMA 9B, and the question of what it would take to replace it is a real one.

What each side actually is

• Jais 3 is a family, not a model — six base checkpoints, Jais-3-0.9B, Jais-3-3.7B, Jais-3-7B, Jais-3-32B, Jais-3-MoVA-36B-A4B and Jais-3-375B-A23B, plus -Uno/ variants, all Apache 2.0 and all ungated. The dense models build on a custom k2_horizon/ architecture requiring trust-remote-code.

• SILMA 9B is one checkpoint. silma-ai/SILMA-9B-Instruct-v1.0/ is a Gemma 2 derivative — gemma2/ / Gemma2ForCausalLM/ — with five safetensors shards, a conversational pipeline tag, and Arabic and English declared as its languages.

• Licence is the sharpest practical difference. Jais 3 is Apache 2.0 with no gate. SILMA 9B carries the Gemma licence and an acknowledgement gate: the card's extra_gated_button_content/ field means you accept terms before the weights are yours, and the Gemma licence carries use restrictions that Apache 2.0 does not.

• The family that replaced it is small. SILMA AI's later releases are SILMA-Kashif-2B-Instruct-v1.0 from January 2025 and a pair of embedding models from October 2024. There is no SILMA 9B successor, no 27B, no MoE variant. The organisation's newest artifact by a wide margin is silma-tts/ from March 2026.

• Context — up to 524,288 tokens across most of the Jais 3 line, 128K on the 0.9B. SILMA's card publishes no context length figure at all.

The benchmark problem, stated before any number is quoted

SILMA 9B's published results come from a different measurement culture than Jais 3's, and the difference is not a matter of degree.

• SILMA's numbers are log-likelihood scores on the Open Arabic LLM Leaderboard harness — loglikelihood_acc_norm/ against Arabic MMLU (52.55), AlGhafa (71.85), ARC Challenge Arabic (78.19), ACVA (78.89), Arabic EXAMS (51.4), plus a raft of translated benchmarks. Its own headline is an Arabic Broad Benchmark score of 6.15 out of 10.

• Jais 3's numbers are generated-answer accuracy on an audited sample set. IFM reports Arabic MMLU at 89.3 for the 375B, 88.6 for the 32B, 85.2 for the 7B and 83.3 for the 3.7B, with the star marking benchmarks it cleaned of ambiguous questions, invalid options and indefensible gold labels — and it publishes the excluded samples.

• These are not the same measurement. Multiple-choice log-likelihood scoring and generated-answer accuracy on a cleaned set will not agree, and the gap between them is not a measure of model quality. Putting 85.2 next to 52.55 and calling it a 33-point win would be the single most misleading sentence available on this page, and it is the sentence most comparison articles would write.

• Neither has an independent score. No Artificial Analysis entry for either. No arena result. Every figure in this section is vendor-reported by whichever lab published it, and the Jais 3 figures additionally carry IFM's own footnote that the reasoning-effort setting used was high/ for every run.

The size-matched comparison is the one that means something

If SILMA 9B is what you have, the honest Jais 3 replacement candidate is Jais-3-7B or Jais-3-3.7B, not the flagship. Both are dense, both run at tensor parallelism 1 in bfloat16 in the vLLM recipe, and both are ungated.

• Footprint — Jais-3-7B at TP=1 on a single GPU against SILMA 9B's five-shard Gemma 2 9B. Comparable hosting cost, so the upgrade is a quality and capability question rather than a hardware one.

• Context — 524,288 tokens for Jais-3-7B against no published figure for SILMA 9B. This is the row where a workload gets re-architected rather than ported.

• Edge and quantised builds — GGUF for llama.cpp and Ollama on both the 3.7B and the 7B, and MLX 4-bit, 8-bit and bf16 for Apple silicon. SILMA 9B has GGUF conversions, but they are third-party uploads from 2024 and the vendor published none.

• Arabic-specific surfaces — IFM publishes a full Arabic benchmark suite per checkpoint covering dialects, diacritisation, parsing, IslamicQA, Arab culture and translation. SILMA's card reports through the leaderboard's aggregate columns and its own broad-benchmark score, with far less per-capability detail.

• Licence — Apache 2.0 against the Gemma licence plus a gate. For a commercial Arabic pipeline, that difference alone can decide the migration.

What SILMA 9B still has going for it

Two things, and neither is a benchmark.

• It is a Gemma. A gemma2/ checkpoint runs on every stack that already runs Gemma 2 — no --trust-remote-code/, no custom parser flags, no architecture-specific model implementation. Jais 3 needs trust-remote-code, a k2_horizon/ reasoning parser, a k2_horizon/ tool-call parser, and language-dependent sampling: temperature 1.0 for English and 0.7 for Arabic in thinking mode against a 0.2 default, with at least 32,768 output tokens recommended. Porting is real work.

• It is stable, and there is a leaderboard around it. SILMA AI maintains an Arabic LLM Broad Leaderboard space, and a two-year-old checkpoint has two years of production stories attached to it. That is not nothing when the alternative is a model published three weeks ago with no third-party evaluation, no technical report — IFM lists it as coming soon — and no published price.

The migration decision, and how to make it reversible

Replace SILMA 9B if your workload is long-context documents, agentic tool use, or commercially sensitive enough that the Gemma licence is a problem. The context row alone justifies it: a model with no published context figure against one that goes to half a million tokens is not a close call for document work.

Keep it if the pipeline is settled, the Gemma licence has already been cleared, and everything runs on stock transformers — because the honest counterweight to Jais 3's benchmark table is that the table is IFM's, the model is three weeks old, and there is no independent number behind any of it. Upgrading a working Arabic production path to a three-week-old checkpoint on the strength of vendor figures is the mistake this comparison exists to prevent.

The way to get both is to run them side by side first, and the reason that used to be expensive is that it meant two integrations and a comparison harness. OrcaRouter collapses that: one API across 200-plus models with 0% markup, provider list price passed through, so a bake-off is a routing change rather than a project — and the routing DSL is the right tool for the specific job of sending identical Arabic prompts to two models and scoring the answers, because it composes them into one call and returns both. Automatic failover is what makes the next step safe, letting a newly published model take a slice of real traffic while your settled model absorbs the rest and catches the requests the new one drops. Neither model is on our catalogue right now; the model list is where to look before designing around a self-host.

What to watch

Four things, in order of how much they would change this page. A SILMA 9B successor, which would turn a category mismatch back into a real matchup. Any independent evaluation of a Jais 3 checkpoint, which would make the whole benchmark section unnecessary. IFM's technical report, currently listed as coming soon. And a published price for Jais 3, of which there is none anywhere. Until at least one of those lands, the defensible reading of this comparison is architectural — licence, gating, context, tooling, size ladder — and the scores should be read as two labs measuring two different things in two different years.