RWKV-7 (Goose): Inside the Pull Request That Would Finally Load It in Transformers

The most-downloaded RWKV model on Hugging Face last month was not RWKV-7 (Goose), the architecture the project has shipped since March 2025, and it was not RWKV7-G1, the reasoning series trained on top of it. It was RWKV/rwkv-4-169m-pile: a 169-million-parameter RWKV-4 checkpoint from May 2023, with 8,824 downloads in the 30 days to 5 August 2026, against 215 for the 1.5B RWKV-7 Goose World-3 release and 316 for the 2.9B. The 2023 model is not better. It is the one that loads with a plain from_pretrained call and nothing else installed.

On 4 August 2026, a contributor named Hakureirm opened pull request #47780 against huggingface/transformers, titled "Add RWKV-7 (Goose)". As of 5 August it is open, unreviewed and unmerged, and it is the second attempt — an earlier proposal, #46984, was declined. Nothing has landed, and this piece should not be read as a release announcement. But the diff is public, and it addresses the single most boring, most consequential thing standing between the longest-running attention-free LLM lineage and the stacks most teams actually use: a loader.

What follows separates three kinds of claim, because coverage of RWKV usually runs them together. There is what is verifiably in the pull request and on the Hub, which you can check yourself. There is what the RWKV project reports about its own models, on its own evaluations. And there is the large set of questions nobody has answered publicly, which is where most of the interesting risk lives.

What is actually in the pull request

The mechanical facts, read from the PR page and the GitHub API on 5 August 2026:

• Scope — three commits, 12 files changed, 4,423 lines added and zero deleted, from the branch add-rwkv7-upstream into huggingface:main. Labelled "New model". No assignee, no milestone.

• What it adds — RWKV-7 as two public classes, Rwkv7Model and Rwkv7ForCausalLM, alongside an Rwkv7Cache built on the library's LinearAttentionLayer. The WKV state dtype is configurable independently of the model dtype, which matters because the recurrent state is where numerical drift accumulates in this family.

• How it runs — portable PyTorch with no third-party runtime dependency. Prefill uses a chunk-parallel form of the recurrence; decode runs a sequential single-token path. Parameter names follow the upstream RWKV reference implementation rather than being renamed to look transformer-ish.

• Test posture — beyond the standard model mixins, an integration test that matches BlinkDL's own runtime token-for-token, plus a NumPy reference implementation that shares no code with the modeling file. The repo's CI recap bot reports the latest run as success: 16 jobs, 179,151 tests, zero failures, 16 hours 9 minutes of compute.

• Where it stands — review has been requested from ArthurZucker and Rocketknight1; the page states that at least one approving review is required to merge, and neither has been given. GitHub's API still described the merge state as "unstable" when we read it, and a maintainer-facing bot has asked for the slow-test suites (auto and rwkv7) to be run before merge. In other words: green on the fast CI, not yet blessed by a human.

The most interesting part of the description is the concession. The earlier attempt, #46984, was declined because the published RWKV-7 checkpoints did not follow Transformers conventions, and the author's own framing is that "that objection was correct." The problem, laid out in the PR, is that RWKV-7 weights on the Hub come in two unusable-by-the-library shapes:

• The PTH repositories — raw .pth files, no safetensors. A library implementation cannot load them, and PyTorch pickle files are exactly what a security review at a large company will refuse.

• The HF repositories — these do ship model.safetensors, but each also ships a modeling_rwkv7.py and an auto_map, so loading one requires trust_remote_code. That is remote code execution as a condition of inference, which is why so many enterprise checklists stop there.

So the PR does two jobs at once. It adds a modeling file, and it points at a fresh set of conversions made directly from the canonical BlinkDL .pth releases that follow the standard layout — safetensors only, no pickle, no remote code, a normal config.json carrying architectures and model_type — spanning 0.1B to 7.2B. The smallest, Hakureirm/rwkv7-168m-pile-hf, has all 399 of its tensors verified bit-identical against the source .pth rather than spot-checked. That checkpoint is a Pile model, so its tokenizer is the ordinary GPT-NeoX-20B fast tokenizer rather than the RWKV World vocabulary — the same reason the library's existing RWKV page documents a Pile checkpoint too.

Why a checkpoint from 2023 out-downloads the current architecture

Transformers is at version 5.14.1, released 16 July 2026. Search its documentation for RWKV and you get exactly one model page, describing "the RWKV model (version 4)", contributed years ago, with RWKV/rwkv-4-169m-pile as the example and a default vocabulary of 50,277 tokens. There are no pages for RWKV-5, RWKV-6 or RWKV-7. Three architecture generations have shipped since the library's RWKV support was written, and none of them are in it.

The download numbers show what the ecosystem did about that: it routed around the library. Sorted by downloads in the last 30 days, the top RWKV-7 repositories are BlinkDL's raw .pth releases (rwkv7-g1 at 8,288, rwkv-7-world at 4,489) and a thick layer of community GGUF quantizations of the 13.3B G1 — several separate uploaders each moving one to three thousand downloads a month. The official transformers-format mirrors sit two orders of magnitude below the raw weights. The flash-linear-attention mirror of the 2.9B G1 manages 1,843.

That pattern has a simple explanation. llama.cpp merged RWKV v7 support on 17 March 2025 — a GGML_OP_RWKV_WKV7 kernel with CPU, CUDA, SYCL, Vulkan and Metal backends — about a day after the paper went up. If you wanted to run RWKV-7 on your own machine, the fast path was GGUF, and it has been for sixteen months. The path that did not exist was the one every fine-tuning script, every PEFT adapter, every evaluation harness and every internal serving wrapper assumes: AutoModelForCausalLM.from_pretrained, no flags.

What RWKV-7 (Goose) actually is

RWKV-7 is a recurrent neural network, not a transformer with a cheaper attention kernel, and the naming confuses people constantly. It carries a fixed-size state forward through the sequence instead of a growing cache of past keys and values. Concretely, against a standard attention model:

• Memory as context grows — a transformer's KV cache grows linearly with the number of tokens in flight; RWKV-7 keeps a state whose size is set by the architecture, not by the conversation. That is the whole efficiency argument.

• Cost per token — attention costs more per token as the context lengthens; RWKV-7's per-token inference cost is constant, which is why it keeps showing up in edge and always-on-stream proposals.

• Training shape — unlike a classic RNN, the recurrence is parallelizable over a chunk, so pretraining does not degenerate into a sequential crawl. That is what the PR's chunk-parallel prefill path implements.

• Context ceiling — there is no cache to blow out, so the project markets effectively unbounded context. What a fixed state cannot do is hold unbounded detail, which is a real limitation rather than a footnote.

The architectural claim in the paper, published 18 March 2025 by Bo Peng, Yu Zhang, Songlin Yang and Ruichong Zhang under the RWKV Project at the LF AI & Data Foundation, is a generalized delta rule with vector-valued gating, in-context learning rates and a relaxed value-replacement rule, plus a simplified MLP (the gating matrix removed, hidden dimension widened to compensate). The theoretical result attached to it is the more provocative half: RWKV-7 can perform state tracking and recognize all regular languages while remaining parallelizable in training, which the authors argue exceeds what transformers can do under standard complexity conjectures.

Everything is Apache 2.0. The family you can download runs 0.1B (12 layers, width 768), 0.4B (24 / 1024), 1.5B (24 / 2048), 2.9B (32 / 2560), 7.2B (32 / 4096) and 13.3B (61 layers, width 4096), all with a 65,536-token World vocabulary and head size 64. The base World series was trained on a 3.1-trillion-token multilingual corpus; the G1 "GooseOne" series continues that training on World v3.5, an expanded 5.16-trillion-token mix with more novels, web text, mathematics, code and reasoning data. G1 checkpoints add a think-tag reasoning mode, JSON function calling, and fill-in-the-middle from G1c onward. Naming is genuinely awkward: G0 means less than one epoch, G1 means more than one, and the suffix letters mark data revisions, with later letters carrying better data.

The numbers, and whose numbers they are

Here is the honest state of the evidence. The headline benchmark claim — that the 2.9B model set a new 3B state of the art on multilingual tasks and matched the English-language 3B state of the art on dramatically fewer training tokens — is the paper's own, published in March 2025 and evaluated against that period's 3B models. It went through OpenReview, which is more scrutiny than a vendor blog post gets, and it is still a self-reported result on a fifteen-month-old comparison set.

The project's other public measurement, UncheatableEval, is more interesting than a leaderboard row and gets almost no coverage. Instead of scoring multiple-choice benchmarks that leak into training sets, it measures compression rate on data that did not exist when the model was trained: new arXiv papers, fresh GitHub repositories, recent news. That design makes contamination much harder, and RWKV reports being competitive with same-size transformers on it. It is still an evaluation the project runs on itself.

What does not exist, as far as we can find, is an independent third-party index score for any RWKV-7 checkpoint — no neutral aggregator has put the 7.2B or 13.3B through the harness it runs on frontier models. So comparisons you may see against models like DeepSeek V4 Flash or Qwen3.8-Max are category errors twice over: nobody has run RWKV-7 on the same evaluation, and a 13.3B dense RNN is not competing for the same work as a frontier system. The defensible claim is narrower and more useful: at 1.5B to 13.3B, on constant memory, in twelve-ish languages, with permissive weights.

Running RWKV-7 today, and what that costs you

The Hub page for RWKV/RWKV7-Goose-World3-1.5B-HF is a good snapshot of the current friction. It is a 1.52B-parameter BF16 model with the RWKV World tokenizer, Apache 2.0, tagged custom_code, listing English, Chinese, Japanese, Korean, French, Arabic, Spanish and Portuguese. Its instructions tell you to install flash-linear-attention and a recent transformers before loading. And in the sidebar, where a hosted model would show providers, it says plainly: this model isn't deployed by any Inference Provider.

So your options today are all self-service:

• flash-linear-attention plus trust_remote_code — closest to normal Hub usage, but you are executing repository code and pulling in Triton kernels, which constrains you on hardware and on anything with a security review.

• GGUF via llama.cpp — the best-supported route in practice, including for the 13.3B, and the one the download numbers say people actually take. Great for local inference, not a training or fine-tuning path.

• The project's own runtime — the rwkv pip package and the reference repository, closest to canonical, furthest from the tooling your team already has.

None of those is an API you can call, and it is worth being direct about the implication: RWKV-7 is not on OrcaRouter, because it is not a hosted endpoint anywhere we can find. If you want RWKV-7, you run RWKV-7. What we can say honestly is where that leaves the rest of the stack. Evaluating an unproven architecture is only cheap if your production path does not depend on the outcome, and the cheapest way to keep that true is to not have a per-vendor integration for anything else — one OpenAI-compatible key across 200-plus models, provider list price passed straight through at 0% markup, and automatic failover when a provider degrades. Then a self-hosted RWKV-7 experiment on the two workloads where constant memory actually pays — a long-running stream, an on-device assistant, a summarizer that never stops — is an experiment, not a migration. That is the shape most teams should want here: a routed default, and a state-based model earning its way in on a specific job.

What could still stop this landing

Take the precedent seriously: a proposal to add this exact architecture was already declined once, on grounds the author accepts were valid. The new one is better argued and better tested, and it is still a community PR against a repository that is deliberately conservative about admitting architectures it will then maintain forever.

The specific open questions we would want answered before calling this done:

• Review, not CI — the automated suites are green; two maintainers have been asked and neither has approved. Transformers requires one approving review, and the slow tests have not been run.

• The dependency-free path's speed — pure PyTorch with a sequential single-token decode is portable, and portable is the entire point, but the PR does not publish throughput against the Triton kernels in flash-linear-attention. If native decode is materially slower, the library becomes the compatibility path while serious serving stays elsewhere.

• Which checkpoints arrive — the convention-compliant conversions cover 0.1B to 7.2B. The 13.3B G1, which is the one people actually want, is not in that set, and the documented example is a Pile model with a GPT-NeoX tokenizer rather than a World-vocabulary chat model. A merged loader with no flagship checkpoint behind it changes less than it looks like it does.

• The moving target — the project is not standing still. BlinkDL's G1 repository was updated the same week this PR opened, community quantizations have moved on to later data revisions than G1c, and RWKV-8 "Heron" has been previewed publicly with a suffix-automaton mechanism called ROSA. Heron is unreleased and unbenchmarked; we mention it only because a library integration that arrives late in a generation's life has a short shelf life.

Four questions the spec sheet does not answer

Can I use RWKV-7 in Transformers right now, or not?

Both, annoyingly, and the distinction is the whole story. You can load an RWKV-7 checkpoint through the transformers API today, if you install flash-linear-attention and pass trust_remote_code so the repository's own modeling file runs. What you cannot do is load it from the library itself, which is what makes it work by default in the tools built on top of the library — training and alignment scripts, adapters, evaluation harnesses, export paths — and what makes it pass a policy that forbids remote code. That second thing is what #47780 is for.

Does merging make the model better?

Not by a single point on any benchmark. It changes distribution, not quality — and for an architecture whose problem has never been quality, distribution is the binding constraint. The comparison is the one at the top of this piece: a 169M model from 2023 out-downloading current-generation weights forty to one, entirely on the strength of loading without flags.

If there is no KV cache, do I get unlimited context for free?

You get unlimited context length without the memory blow-up, which is not the same as unlimited recall. A fixed-size state has a fixed information capacity; feed it a million tokens and it cannot hold a million tokens' worth of retrievable detail. Attention with a full cache can, at a cost that grows the whole way. Treat RWKV-7's long-context story as "streams forever cheaply, compresses as it goes," and test the specific retrieval you need rather than trusting the word infinite.

Is it worth caring about at 13.3B when frontier models are hundreds of billions?

It depends entirely on whether constant memory is worth anything to you. If you are calling a hosted API and paying per token, almost certainly not — the frontier is far ahead on capability and you are not paying for the KV cache directly. If you are shipping inference onto hardware you do not control, or running a persistent stream where a growing cache is the thing that eventually kills the process, an architecture whose memory footprint does not move is a different kind of answer to a different question. Those are the workloads where a 2.9B RWKV-7 has been quietly competitive, and they are the ones where a native loader would matter most.

What we would watch next

Four concrete signals, in rough order of how much they would change our read. An approving review from ArthurZucker or Rocketknight1, which turns this from a hopeful diff into a scheduled feature. A convention-compliant conversion of the 13.3B G1 with the World tokenizer, which is what makes the loader worth having. Published throughput numbers for the dependency-free decode path against the Triton kernels, which decides whether native support is a serving option or a compatibility shim. And any sign of RWKV-8, which would tell you whether this integration arrives at the start of a generation or the end of one.

Until at least the first of those, the correct summary is the unglamorous one: RWKV-7 (Goose) is real, permissively licensed, downloadable up to 13.3B, and still not natively loadable in the library that most of the ecosystem is built on. A pull request opened on 4 August 2026 proposes to fix that. It has not been merged, and pull requests to add architectures to transformers do get closed.