Microsoft Mage-VL: A Codec-Native 4B Video Model, Shipped Without an Announcement

There is no Microsoft blog post about Mage-VL. No Azure newsroom entry, no Foundry catalog listing, no launch thread, nothing on the product channels where Microsoft normally introduces a model. What exists instead is a Hugging Face repository — microsoft/Mage-VL, six commits, 10.8 GB of weights, Apache-2.0 — a GitHub folder of inference scripts, a project page maintained by something calling itself the Microsoft Mage Team, and a 23-author arXiv report. Read together, those artifacts describe a 4B-scale vision-language model whose central idea is genuinely unusual: instead of decoding video into evenly spaced frames and pushing a dense grid of patches through a web-pretrained encoder, Mage-VL reads the compressed bitstream itself, using a from-scratch encoder called Mage-ViT to keep only the patches the codec spent bits on. Microsoft reports that this cuts visual tokens by more than 75% and delivers up to a 3.5x wall-clock speedup while matching Qwen3-VL-4B on static images and beating Microsoft's own 15B Phi-4-Reasoning-Vision on video.

That last sentence is the part to hold at arm's length. Every performance figure in this article traces back to Microsoft's own paper, model card, or project page. Ten days after the weights appeared, no independent party has reproduced any of it, no third-party leaderboard carries the model, and — as the Hugging Face page states plainly — it "isn't deployed by any Inference Provider," so there is not even a hosted endpoint someone could have benchmarked casually. What follows separates what the repository proves from what Microsoft merely claims, because on a release with no announcement those are very different categories.

What actually exists, ten days in

The verifiable surface of this release is small and worth enumerating precisely.

• Weights, dated July 26, 2026. Two safetensors shards of 4.97 GB and 4.52 GB, plus a separate 1.07 GB file named streammind_gate.safetensors. Hugging Face's own reader reports 5B parameters at BF16 — 4B in the language decoder, the rest split between the visual encoder and that gate.

• A technical report, submitted July 27, 2026 (arXiv 2607.24904), one version, 23 authors, titled "Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model."

• Runnable code, not just weights. The repo ships modeling_mage_vl.py, processing_mage_vl.py, two video processors including a dedicated codec_video_processing_mage_vl.py, and streammind_gate.py — about 175 KB of custom Python. The auto_map in config.json routes six Transformers classes into those files, which is why the repo carries the custom_code tag.

• Two licenses, not one. Mage-VL is Apache-2.0; the standalone Mage-ViT encoder is published separately under MIT.

• A working demo you don't have to install. Microsoft runs microsoft/mage-vl-demo as a Hugging Face Space on ZeroGPU, and two community Spaces already use the model.

• Early community traction, forming faster than the vendor's own communications. 268 likes, 435,784 downloads recorded for the last month, nine community quantizations and two finetunes in the model tree. Download counters include automated and mirror pulls, so treat the raw number as a signal of attention rather than of deployment.

• A sibling. Mage-Flow, a text-to-image and instruction-editing model built to the same fixed 4B budget, landed four days earlier on July 22. The GitHub repo presents Mage as "a family of lightweight, research-friendly multimodal models," which is the closest thing to a positioning statement anyone has published.

Set against that, the list of things that do not exist is just as informative. There is no Microsoft blog post or press release. There is no listing in Azure AI Foundry, which means no enterprise support path, no SLA, no managed endpoint. No inference provider serves it. There is no vLLM or SGLang support: a community request to add Mage-VL to SGLang was filed on July 28 as issue #32646 and, as of this writing, remains open with no linked pull request and no maintainer response. And there is no independent evaluation of any kind — the model is absent from the neutral leaderboards where a claim like "beats a 15B model on video" would normally get tested.

The one idea: read the codec, not the frames

Almost every video-capable VLM in production does the same thing. It decodes the video into RGB frames, samples them uniformly — one every second, or 32 across the clip, or whatever the budget allows — and passes each sampled frame through a vision transformer as a dense grid of patches. Every patch of every sampled frame becomes tokens. A static background wall costs exactly as many tokens as the person walking in front of it, and it costs them again in the next frame, and the next.

That is an enormous amount of redundant computation, and modern video codecs already solved the underlying problem decades ago. H.264 and HEVC do not store every frame; they store occasional anchor (I) frames in full and then describe the frames between them as motion vectors plus residuals — "this block moved here, and here is what changed." The interesting parts of a video are, almost by construction, the parts the encoder spent bits on.

Mage-ViT exploits that directly. Operating at 16x16 patch granularity, it keeps every patch of the anchor frames and, for predicted frames, retains only patches flagged as salient by the codec's own motion vectors and residual energy — the regions carrying real information, motion, or a scene change — while discarding the low-redundancy and fully redundant ones. Microsoft puts the resulting reduction at over 75% of visual tokens, with spatio-temporal context preserved because the anchor frames still carry the full scene. The design is codec-agnostic: the traditional path accepts H.264 or HEVC, and a neural path accepts DCVC-RT.

The elegance is that the motion estimation was already done. Every compressed video on the internet arrives with a map of where the action is, computed by the encoder and paid for by whoever uploaded it. A conventional pipeline throws that map away the instant it decodes to RGB, then spends GPU time rediscovering the same information. Mage-VL simply declines to discard it. Whether or not the benchmark numbers hold up, that observation is the durable contribution here — and it is the reason this release is worth reading even if you never download the weights.

The cleanest thing about the experiment

Buried in the setup is a design decision that makes the results far more interpretable than a typical model launch, and almost nobody covering this release has pointed it out: the language model is held fixed.

Mage-VL's decoder is Qwen3-4B-Instruct-2507, unmodified. The comparison baseline, Qwen3-VL-4B, uses the same 4B Qwen3 backbone with a conventional web-pretrained visual encoder. So when Mage-VL improves on Qwen3-VL-4B, the delta is attributable to the encoder and the codec-native tokenization, not to a bigger or better-trained language model. That is a controlled ablation dressed up as a product comparison, and it is the strongest methodological feature of the release.

It also cuts the other way, and honesty requires saying so. A same-backbone comparison is the fairest test of the encoder idea and simultaneously the framing most likely to flatter it — Microsoft chose the baseline that isolates its own contribution. The Phi-4 comparisons do not have that property: Phi-4-Reasoning-Vision-15B and Phi-4-MM-5.6B are different backbones, different training recipes, different post-training. "Beats our 15B model on video" is a real result but a much looser one, and it is also a comparison against Microsoft's own older work, which is the easiest kind to win.

The training scale is the other place the paper makes a genuinely surprising claim. Mage-ViT was pretrained from scratch on roughly 560M unlabeled images and 100M unlabeled video frames — a large corpus in absolute terms, but well short of the billions of curated image-text pairs behind the encoders it is competing with. The paper's first stated finding is that a strong VLM encoder does not require web-scale supervised data. If that holds up under independent scrutiny, it matters considerably more than any single benchmark row.

The numbers, and whose numbers they are

What follows is Microsoft-reported throughout, on Microsoft's own evaluation harness, against baselines Microsoft selected. Nothing here has been reproduced by a third party. Read it as a hypothesis with unusually specific error bars, not as a scoreboard.

• Video-MME — Mage-VL-4B 64.0 vs Qwen3-VL-4B 59.7 vs Phi-4-Reasoning-Vision-15B 55.3

• NExT-QA — 83.1 vs 79.8 vs 69.0

• LongVideoBench — 61.3 vs 57.7 vs 51.2

• VideoEval-Pro — 45.2 vs 20.7 for Phi-4

• Timelens-QVHighlight (temporal grounding) — 57.4 vs 34.9 vs 11.6

• Ref-DAVIS17 (referring tracking) — 25.83 vs 7.48 vs 2.15

• DocVQA-val — 95.14 vs 94.69 vs 92.79 (Phi-4-MM-5.6B)

• OCRBench — 81.80 vs 81.60 vs 81.70

• ChartQA — 84.88 vs 83.96 vs 83.40

• MMStar — 67.32 vs 62.04 vs 59.63

• RealWorldQA — 70.46 vs 70.85 vs 70.72, one of the rows Mage-VL loses

• MMBench-EN-dev — 84.02 vs 83.25, with Phi-4-Reasoning-Vision-15B ahead of both at 84.19

• CV-Bench-3D / CV-Bench-2D — 94.75 vs 92.30, and 82.13 vs 81.00

• EmbSpatial — 82.67 vs 77.50

• OVO-Bench (streaming) — 64.00 overall, described as state of the art among streaming architectures; the real-time visual perception subset averages 79.84% against 72.8% for Qwen3-VL-4B, at 1 fps

• Mage-ViT as a standalone encoder — above 86.3% on ImageNet at a 676-token budget, above 96.1% on Food-101

Three readings of that table are worth more than the table itself.

On images, "parity" is the honest word. DocVQA by 0.45, OCRBench by 0.20, ChartQA by 0.92, MMBench by 0.77 — these are within the range where a different prompt template or decoding seed could reverse the ordering, and RealWorldQA actually goes to Qwen3-VL-4B. Microsoft says as much, framing image performance as parity rather than a win, and that framing is correct. If your workload is document and image question answering, this release gives you no reason to move.

On video and temporal grounding, the gaps are large and consistent. Timelens-QVHighlight nearly doubles the baseline; Video-MME, NExT-QA and LongVideoBench all move by 3.6 to 4.3 points in the same direction with the backbone held fixed. Consistency across benchmarks that stress different things is the pattern you would expect if the encoder change is real rather than a tuning artifact.

Two rows should not be quoted without context. Ref-DAVIS17 at 25.83 against 7.48 looks like a 3.5x demolition, and the paper's headline spatial deltas include +11.0 on VSI-Bench and +53.1 on CrossPoint. When a baseline scores near the floor on a task, the delta mostly measures which model was trained to understand the task format — not which model is more capable. The same caution applies to the streaming results in absolute terms: on SoccerNet, Mage-VL's reported figures are 55.54 TimVal, 83.14 ROC-AUC and an F1 of 16.35. An F1 of 16.35 is a state-of-the-art number in a young evaluation, not a solved problem. Proactive streaming perception is early, and the leader's absolute score says so.

The gate: a model that decides when to speak

The second architectural idea is the one with the clearest product implications, and it explains that mysterious 1.07 GB file.

Mage-VL splits streaming into two processes, framed in the paper as System 1 and System 2. System 1 is a lightweight "cognition gate" that watches each rolling window of codec features and estimates the probability that something worth speaking about has just finished happening. Below a threshold, it stays silent and the expensive part of the model never runs. Above it, the full decoder is invoked to produce a response. The demo configuration uses 30-second causal windows at 1 fps, the CLI exposes the threshold directly as --gate_threshold, and the streaming entry point processes video segment by segment (inference_streaming.py --video_backend codec --segment_sec 8). Only the gate is trained in the final stage, on 3.35M streaming samples.

Two things about that are worth noticing. First, the gate is not a small classification head bolted on top: 1.07 GB of BF16 weights is roughly half a billion parameters, a real model in its own right, shipped as a separate checkpoint. Second, the filename is streammind_gate.safetensors — the naming suggests this component descends from earlier streaming-perception work rather than being invented for this paper, though the repo itself does not spell that lineage out.

Why it matters commercially: for always-on video, the dominant cost is not per-call latency, it is call frequency. A camera feed running 24/7 through a conventional VLM at 1 fps means 86,400 forward passes a day whether or not anything happened. A gate that stays quiet during the 99% of footage where nothing happens changes the shape of that bill, not just its size. Whether Microsoft's gate is accurate enough to trust with that decision is exactly the thing nobody outside the lab has tested.

Can you actually run it today?

Yes, if you have a GPU and patience. The friction is real and mostly lives in the video pipeline rather than the model.

Memory. Microsoft does not publish a VRAM requirement. From the weight index: 9.49 GB of shards plus the 1.07 GB gate is about 10.6 GB of BF16 parameters, so a 16 GB card is a realistic floor for image work and 24 GB or more is the sane target once you add KV cache for long video or a streaming window. That is arithmetic from file sizes, not a vendor spec — measure before you provision.

Custom code is mandatory. The auto_map points every Transformers entry point at the repo's own modules, so trust_remote_code is required. You are executing Microsoft's Python, not just loading tensors. There is no vLLM or SGLang path yet, which means no paged attention, no continuous batching, no production serving stack — a significant gap if you were hoping to put this behind an endpoint.

The codec path needs system tooling. FFmpeg and ffprobe must be on your PATH. The traditional codec backend depends on a codec-video-prep package that supplies a cv-preinfer step; the neural path needs DCVC-RT; the plain frames backend needs Decord. The requirements also pull in flash-attn and mamba-ssm, which compile CUDA extensions — install a PyTorch build matching your toolkit first or budget an afternoon for the build.

What the config tells you that the card does not. Maximum position embeddings are 262,144, so the decoder inherits Qwen3-4B's long context. The vision side runs at 448-pixel input with 16x16 patches, a 24-layer 1024-hidden encoder, 2x2 spatial merging, one token per second of video and a four-frame window. Training reached 384 frames of temporal length in stage three. A 262K ceiling is not the same as 262K of validated behavior, and 384 frames is the length the model was actually taught to handle.

Known rough edges. An open discussion on the repo, filed August 4 and still unanswered, reports token misalignment when images and videos are passed in the same request. Day-ten software behaves like day-ten software. If you want a look without any of this, the Microsoft-run Space on ZeroGPU is the zero-install option.

The license line is less simple than "Apache-2.0"

The model card says Apache-2.0. The Mage-ViT encoder says MIT. Both are about as permissive as open weights get. But the family repository states that "these models are released for research purposes only," with emphasis on responsible-AI review and human oversight — and that sentence sits uneasily beside an Apache-2.0 grant, which does not restrict commercial use. Add the dependencies: DCVC-RT and the codec preparation tooling carry their own terms, independent of the model's.

For a hobby project this is noise. For anything shipping to customers, it is the kind of ambiguity that should go to counsel before it goes to production, and the sort of question worth asking on the repo itself — where, notably, there is currently no Microsoft representative answering.

Should you build on this?

The decision splits cleanly along one line: whether your problem is a stream or a request.

If you are doing always-on perception — a camera feed, a live broadcast, a robot's field of view, a meeting that runs for an hour — Mage-VL is pointed exactly at you, and the economics of self-hosting are on your side. Per-token API pricing scales with frames, which is a brutal model for continuous video; a 4B model on your own hardware with a gate that stays silent through uneventful footage is a fundamentally different cost curve. The catch is that you are also signing up to be the first person outside Microsoft to find out whether the gate's judgment is any good.

If your problem is request-shaped — a user uploads a document, a PDF, a screenshot, a short clip and expects an answer — the case is much weaker. On exactly those tasks Mage-VL is at parity with a model you would still have to host yourself, and hosted multimodal endpoints are one API call away with no GPU, no ffmpeg build, and no trust_remote_code. On OrcaRouter, Gemini 3.6 Flash runs $1.50 per million input tokens and $7.50 per million output, which is the provider's list price passed straight through — we take 0% markup, so when a vendor cuts prices the cut is live on our side the same day rather than after a pricing review. One key reaches 200-plus models with automatic failover if a provider degrades, which is the practical reason to keep a hosted endpoint as your default and reserve self-hosting for the workloads that genuinely need it.

To be explicit, because the distinction matters: we do not host Mage-VL, and neither does anyone else. Hugging Face's own model page says no inference provider has deployed it. Today, running it means running it yourself.

What would change this read

Four things, in rough order of how much they would matter.

An independent evaluation is the big one. Every number above is a claim, and the claim most in need of testing is not a benchmark score but the 3.5x speedup, which was measured against uniform frame sampling on NExT-QA without published hardware, resolution, or frame-count details. Codec preprocessing moves real work onto the CPU and into ffmpeg; a wall-clock win measured end to end on someone else's rig is the only version of that number worth planning around.

Second, serving support. A merged vLLM or SGLang implementation would turn this from a research checkpoint into something you can put behind a load balancer. The SGLang issue is open and unclaimed; that is the thread to watch.

Third, a listing in Azure AI Foundry, which would signal that Microsoft intends this as a product rather than a paper. Nothing in the current release suggests that is imminent.

Fourth, and most oddly: whether Microsoft ever says anything. A 23-author technical report, a maintained project page, a hosted demo Space, and a sibling generative model four days earlier do not describe a leak or an accident — they describe a deliberate research publication that skipped the product megaphone entirely. The community filled the silence anyway, with nine quantizations and two finetunes inside ten days.

For most teams the right move is to read the paper, not download the weights. The codec-native idea is the takeaway, and it is portable: if reusing motion vectors the encoder already computed really buys a 75% token reduction at parity accuracy, that technique will show up in models with launch posts, provider support, and reproduced benchmarks. If you run continuous video today, the calculus is different — clone the repo, run your own clips through both backends, and measure the speedup yourself, because right now you would be the first.

Questions worth actually asking

Is Mage-VL just Qwen3-VL with a Microsoft label on it?

No, though the confusion is understandable. The language decoder is Qwen3-4B-Instruct-2507, used as-is — Microsoft did not train a new LLM. Everything else is new: Mage-ViT was pretrained from scratch, the codec-native tokenization has no counterpart in Qwen3-VL, and the streaming gate is an additional half-billion-parameter model. Reusing an open backbone and swapping the visual front end is a legitimate and increasingly common research strategy, and here it is also what makes the head-to-head interpretable. If you have compliance requirements around model provenance, note that the lineage runs through Alibaba's Qwen3 weights and check both licenses.

Does "3.5x faster" mean 3.5x cheaper to serve?

Not reliably. The figure is a wall-clock speedup on NExT-QA against uniform frame sampling, and Microsoft frames it as "up to." Two things dilute it in practice. Codec-native inference needs a preparation pass — ffmpeg, ffprobe, and the cv-preinfer step, or a DCVC-RT re-encode for the neural path — which consumes CPU time that a naive frames pipeline does not, and which does not appear in a GPU-side measurement. And the gain comes from token reduction, so it scales with how redundant your footage is: a mostly static security camera should do better than the quoted figure, while fast-cut edited video where nearly every patch changes should do worse. Measure it on your own clips.

Do I need special video files to use the codec path?

Mostly no, and this is the pleasant surprise. Ordinary MP4 files are already H.264 or HEVC, which is exactly what the traditional codec backend consumes — the motion vectors it needs are sitting in the file you already have. What you need to add is the tooling: FFmpeg and ffprobe on your PATH, plus the codec preparation package. The neural backend is the exception; DCVC-RT expects video encoded with that codec, so you would be re-encoding. And the plain frames backend remains available as a fallback that behaves like any other VLM, which is also the honest way to A/B the codec claim yourself.

Can I use it commercially?

The license says Apache-2.0, which permits commercial use, modification, and redistribution. The repository also says the models are "released for research purposes only." Those two statements point in different directions, and the gap has not been clarified by anyone at Microsoft — which, on a release with no announcement, no product listing, and no vendor presence in the repo discussions, is unsurprising. If money depends on the answer, get a lawyer to read both documents and the dependency licenses rather than trusting the license badge alone.