PAN 2 vs Cari4D: Watching an Interaction, and Predicting One

Cari4D knows what a hand did to an object. PAN 2 can show you what a hand would do to an object if you asked it to. Those are opposite directions of travel along the same axis, and until you say which direction your problem runs in, a comparison between them is a coin toss dressed as analysis. Cari4D is NVIDIA's category-agnostic 4D reconstruction line — the CoCoNet model that watches a monocular RGB video of one person handling one rigid object and recovers their metric-scale poses frame by frame, with a commercially licensed 231-million-parameter checkpoint on Hugging Face since 30 August 2026. PAN 2 is IFM's 29-billion-parameter open-weight world model, released as an Apache-2.0 checkpoint on 7 October 2026, that goes the other way: hand it a frame and a described action and it renders up to twenty seconds of 704×1248 video predicting what the action does. One reads. One writes. The reason they are worth putting on the same page is that the artefact connecting them — a structured description of who did what to which object — is precisely what each of them produces or consumes.

Two pipelines that look alike and share nothing

Cari4D is not one network responding to video. It is a chain of foundation models with a learned correction stage in the middle: UniDepth for metric depth, NLF and GENMO for human pose, Hunyuan3D for object meshes from a single image, FoundationPose for tracking, VolumetricSMPL for a signed-distance body model. CoCoNet — the part NVIDIA releases — renders the current human and object estimate into RGB, depth and masks, compares that render against the real observation through a DINOv2 ViT-B/14 RGB encoder and a ViT-S/14 six-channel encoder, runs two spatial-temporal attention blocks and regresses per-frame corrections through MLP heads.

PAN 2's chain runs the other way and has a language model in it. A conceptual reasoner — any instruction-following model — reads the observation and the requested actions and writes the next world state as a structured, time-grounded caption: events in order, actions bound to actors, camera motion. A frozen Qwen3.5-9B text encoder maps that caption into conditioning features. A 29B multi-modal diffusion transformer predicts the video latents the caption leaves open, conditioned on it and on the VAE latent of the starting frame, and a Wan 2.2 decoder renders them.

So the similarity that makes people pair them is real — both are about humans, hands, objects and the physics of contact — and the machinery underneath is not related at all.

• Direction — Cari4D recovers parameters from pixels; PAN 2 generates pixels from parameters.

• Output — Cari4D returns per-frame human pose and shape on the MHR rig, per-frame object pose, and two hand-contact logits; PAN 2 returns an MP4.

• Input — Cari4D takes one monocular RGB video; PAN 2 takes an optional first frame plus a structured action caption.

• Size — Cari4D CoCoNet is 231M parameters, trained to step 200,000; PAN 2's diffusion backbone is 29B, with a separate frozen 9B text encoder and a Wan 2.2 VAE.

• Metric scale — Cari4D's core claim is metric reconstruction from a single camera with no depth sensor; PAN 2 has no metric claim at all, and its output is a prediction rather than a measurement.

• Evaluation — Cari4D's numbers come from a CVPR 2026 paper plus a vendor card that admits the commercial model has no separate test set; PAN 2's come from IFM's own harness with no third-party reproduction.

The licence difference is the practical one

Both models are downloadable, which already puts them in a small club. What you may do with them differs more than their parameter counts suggest.

PAN 2's own code and weights are Apache-2.0, ungated, sixty files and 86.17 gigabytes, with third-party components — Wan 2.2, Qwen3.5-9B, Diffusers — under their own terms and not redistributed by MBZUAI. You download it, you run it, you ship what you build, subject to the usual Apache obligations. The `pan2-guardrail` repository ships separately with a request screener and a frame classifier; IFM reports that it cut harmful text-to-video requests producing unsafe output from 33.9% to 1.5% while stopping 11.5% of benign text-to-video requests and 3.3% of benign image-to-video requests, which is the kind of number you want before deploying anything that generates realistic footage.

Cari4D's position is more fragmented, and this trips up procurement teams. The research repository, nvidia/CARI4D, carries the NVIDIA License with "for research and development only" on the card. The commercial checkpoint, nvidia/cari4d_commercial, is under the NVIDIA Open Model Agreement and permits commercial use — but it is gated, you accept terms and share contact information before the files appear, and the released CoCoNet does not run alone. The wider pipeline still pulls NLF torchscript weights, FoundationPose weights, SMPL-H assets and Hunyuan3D meshes, each with its own licence. The commercial licence covers the part NVIDIA is releasing and nothing around it.

There is also a rig migration to plan for. The research line regresses SMPL pose and shape; the commercial checkpoint outputs Native Momentum Human Rig parameters. If you wrote integration code against the research tensors, the commercial card's output shape is not a drop-in.

Where the two compose, and why that is the interesting part

Here is the join that neither vendor advertises and that anyone building physical-AI data pipelines will recognise immediately.

PAN 2's interface is a structured caption: events in order, actions bound to actors, camera motion, inside a budget of 1,480 tokens soft and 1,536 hard. Producing that caption by hand is the single most tedious part of using the model, and the repository ships a prompt-rewriter skill precisely because free text does not survive the interface. Now consider what Cari4D emits from an ordinary video: per-frame human pose, per-frame object pose, and two hand-contact logits — a metric, dense, temporally consistent record of exactly the thing a world-state caption is supposed to describe.

That is a caption generator. Reconstruction gives you the ground truth of an interaction in parameter space; a language model turns those parameters into the structured description PAN 2 conditions on; the simulator then renders variations of that interaction. Loop it and you have a data engine: record one real manipulation, reconstruct it with Cari4D, caption it, and generate the counterfactuals — different object, different grip, different surface — that you could never afford to film. The same loop runs in reverse for evaluation, comparing a simulated clip against a reconstructed real one on the same rig.

Neither half of that loop is turnkey. Cari4D's commercial card states it was trained on 2,126 internal FORM-HOI sequences and that there is no separate testing dataset — evaluation is a qualitative demo set — and the internal "Daniel" corpus is not distributed. Its headline accuracy figures, 38% lower reconstruction error than prior art in distribution and 36% on unseen datasets, come from the CVPR paper and have not been independently reproduced. PAN 2, for its part, admits that about two in three simulated plans contain at least one step that is not carried out, that text actions specify dexterous contact only coarsely, and that how faithfully it simulates plans which should fail is unmeasured.

Read those two admissions together and the composition is honest rather than promotional: a reconstructed interaction is a measurement with a known training distribution, and a simulated one is a prediction with a known step-drop rate. Neither is ground truth about the world. Both are more useful than either alone.

Compute: a workstation versus a node

Cari4D runs on a single GPU and has been validated on Ampere-class hardware according to its card; the deployed model is 231M parameters and the surrounding foundation models are the bulk of the memory. It is the sort of pipeline a research team runs on a workstation overnight against a folder of clips.

PAN 2 does not fit that description. The vLLM recipe's headline is thirteen seconds of 704×1248 video in roughly 119 seconds on four B300 GPUs, with classifier-free guidance parallelised across two ranks and Ulysses sequence parallelism across two; the documented default is eight H200s in BF16 with VAE patch parallelism across four tiles. Fifty sampler steps with guidance on every step means a hundred forward passes through a 29B transformer per clip, which is why the parallelism is not optional. If you are generating counterfactuals in a loop, the loop's cost is the product of clips and that per-clip bill, and it dwarfs the reconstruction side by orders of magnitude.

That asymmetry is the practical planning fact of any hybrid. Reconstruction is cheap enough to run on everything you have; simulation is expensive enough that you run it only where a counterfactual changes a decision.

The planner slot, again

Both pipelines have a language model in them, and only one of those slots is a commodity you can route.

PAN 2's conceptual reasoner is explicitly interchangeable — IFM evaluated four and ranked them by full-plan completion: Claude Opus 5.5, GPT-6 Astra at medium effort, Kimi K3, Qwen3.8-Flash. All four are in the OrcaRouter catalogue, so the caption stage can run through a single endpoint with automatic failover and each provider's list price passed through at 0% markup, and you can benchmark planners against your own captions rather than trusting a ranking measured on IFM's tasks. Cari4D's correction stage has no such slot: CoCoNet is a fixed 231M network, and swapping it means retraining, not rerouting. OrcaRouter does not host PAN 2 or Cari4D and cannot route either — the renderer and the reconstructor both stay on your hardware.

The honest framing for the router here is narrow: one stage of a two-stage pipeline is a metered API call and the rest is not. That is still worth having, because the caption stage is where prompt sensitivity bites and where trying a second model costs a request rather than a training run.

Which one your problem needs

If you have video and need parameters — motion capture for animation, contact labelling for a dataset, metric pose for robot imitation learning, a QC pass over recorded manipulation — Cari4D is the tool and PAN 2 has nothing to offer. Start with the research checkpoint to evaluate, budget for the rig change and the third-party licences before you commit to the commercial one, and treat the accuracy figures as peer-reviewed vendor claims rather than reproduced ones.

If you have parameters or a description and need to see what happens — plan evaluation, counterfactual data generation, physical-reasoning research, an agent that previews its actions — PAN 2 is the model, and the bill is a multi-GPU node per clip. Read its limitations before you design around it: a step is dropped in most plans, long clips drift in appearance because each is simulated from at most one starting frame, and text specifies dexterous contact only coarsely.

If you have both problems, the answer is the loop, and the loop's weakest link is the caption contract. Cari4D gives you the measurement; a language model turns it into the structured description; PAN 2 renders the alternative. Three components, three licences, one research question — which is roughly what building on 2026's world models looks like, and why the comparison between any two of them is less useful than a diagram of how they connect.