I run LLMs on hardware nobody would choose: an Oracle free-tier ARM box, 4 cores, 0 EUR/month. Everything below is measured there unless noted.

The bottleneck on CPU isn't decode, it's prefill. A 3356-token document costs 54.4 seconds before the model writes a single token. llama.cpp caches the KV in RAM, so the second identical request is fast — until the process restarts, and you pay the 54 seconds again.

So I persisted the KV cache to disk. A new process inherits that prefill for 3.5 seconds from disk, 0.10 seconds if the blob is still in page cache. 15-300x, depending on where it reads from. End-to-end on a repeated workload it's 4.8x.

With a systemd timer that pre-digests predictable prefixes at 03:00, a 2815-token document goes from 89.7s to 16.7s TTFT (5.4x), and the request that arrives at 09:00 pays nothing for the prefill.

The bug worth publishing

Warm-ahead was silently dead whenever speculative decoding was on — which was the default. The speculative branch returned before the shared-prefix cache was consulted, so every warm-up wrote snapshots that nothing ever read. Measured on the production box: 90.5s with speculation on, 16.7s with it off, same cache, same request. Two features that each worked, silently cancelling each other.

Things that didn't work

Using the server's own past output as speculative draft material: +5% acceptance, -3.8% throughput on a workload of different requests sharing a structure. The mechanism does what it says and doesn't pay for itself.

Prompt-lookup speculation: +3.9% on the same workload. That's the whole prize.

Coarser quantization: Q4_0 is 37% faster at prefill and dropped 5 facts out of 20 on my extraction test. Rejected.

Halving active experts during prefill on an MoE: 44% faster, and it silently corrupts the cache — a KV built with 4 experts and read back with 8 scores 11/20 against a 14/20 control. The damage is in the cached representation, not just the output.

Two things that did, and surprised me

Rewriting the input as "label: value", one fact per line: 2137 -> 405 tokens, TTFT 40.5s -> 6.2s, and the fact exam went from 19/20 to 20/20. Fewer tokens, and more accurate. Attention on the right number went from a 1.1:1 ratio against the wrong one to 7:1 — prose makes the binding semantic, "label: value" makes it structural.

Trimming the vocabulary from 151,936 to 32k entries: +17.8% decode, bit-for-bit lossless. The embedding is Q6_K with rows spanning whole quantization blocks, so whole rows drop out without splitting a block. The tokenizer is byte-level and all 256 byte-characters are kept, so no text becomes unrepresentable — the worst case is a trimmed word costing one extra token. Measured cost on held-out text: 1.9% more tokens.

What this is not

It's built on llama.cpp and calls its kernels directly, so raw decode speed is identical — I add no per-token overhead. On a single cold request this is llama.cpp. The difference only shows on repeated or cached workloads.

The fact exam is mine: 20 questions over one real Italian business page, graded by regex. One page, one language, one domain. It's the weakest part of this and I'd rather say so. If you know a public adversarial fact-extraction set for small models, point me at it and I'll run it and publish whatever comes out, including a bad result.

MIT licensed. There's a live demo on the same free ARM box — one small instance, no autoscaling, so if it's slow you're watching the honest capacity of 0 EUR/month.

Demo:

Code:

Benchmarks incl. the negative results:

I let Gemma4-31b run on my laptop for like almost a day using a heavily altered pi to do a deep dive on our beloved Llama tangentially related Subreddit, and this was the conclusion.

Feels pretty accurate. Kind funny to let a small LLM loose and see what happens.

Next target I'm trying to let it steal some benchmark answers from Huggingface, wish me luck.

A 16.5-trillion-parameter model that contains nothing. This model is just a ████ you to the labs and companies who say that "haha I have the biggest model out there!". We the people with shitty laptops want to get a record. And I now have a record for a temporary amount of time of about 16.5 trillion parameters and use for them so its completly useless.

What it demonstrates

Hugging Face computes a repository's parameter count from safetensors headers alone — it sums prod(shape) per tensor and never reads the tensor data. The count is therefore whatever the headers declare. Here they declare 3,841 tensors of shape [65536, 65536] in F4 (4 bits/param) across 385 shards, plus one [4294967296, 1] position-embedding tensor in a 386th.

That is enough to place this repo at the top of the Hub sorted by num_parameters, above every real frontier model, while containing no information whatsoever. That juxtaposition is the entire point.

The files are honest about their own size. Every byte the headers declare is really written and really uploaded: safetensors parses each header and its full-coverage check passes. Truncating a file, or overlapping two tensors so they share bytes, would make the count cheaper — both are rejected by the format, and neither is used here. The bytes are simply all 0x00.

Real cost — measured

|---|---| | Declared parameters | 16,501,264,351,232 | | Declared bytes | 8,250,632,175,616 (8.25 TB) | | Storage quota consumed | 8.25 TB — quota bills declared bytes | | Shard headers (all distinct) | 373,835 B | | model.safetensors.index.json | ~269,000 B | | Deduplicated weight data | 65,536 B (one 64 KiB block) | | Bytes actually transferred | ~692 KB | | Ratio | ~11,900,000 : 1 |

The gap between the last rows and the third is the useful finding. Xet content-defined chunking deduplicates the transfer: every 64 KiB block is byte-identical, so it hashes to one chunk and crosses the wire once. Measured on a 500 MB test build, 500 MB of declared weights uploaded as 31.5 MB.

Storage quota is not deduplicated. It bills the logical size. This repo consumes its full 8.25 TB despite under a megabyte ever being sent. Anyone reasoning about "cheap" synthetic model repos should know the saving is in bandwidth only — which is also why this model is 16.5T and not 100T.

The second finding: the only irreducible cost in an empty model is naming. Weights dedup to nothing; tensor names do not. At 1024×1024 experts this same 16.5T model needs 15,735,626 names and a 1.04 GB index. At 65536×65536 it needs 3,841 and a 263 KB one — identical declared size, 4,000× less metadata. Cost scales with tensor count, never with declared parameters.

Context window

max_position_embeddings is 4,294,967,296. That is 2**32, the largest single tensor dimension Hugging Face's parser accepts, and it is backed by a real [4294967296, 1] position-embedding tensor — 2.15 GB of actual zeros, not a number typed into a config file. A context window you cannot point at is just a claim.

Roughly 16,000x Gemini's 262k. About three billion words, every book ever published several times over, held in memory in order to process one token drawn from a one-token vocabulary. The model has exactly one possible input, so every one of those 16.5 trillion parameters serves a function whose domain has a single element.

Capabilities

SAFEST AI MODEL refuses 100/100 jailbreak prompts least closest AI to agi will not sudo rm -rf your computer largest context window on the hub (4,294,967,296 tokens, all of them useless)

Limitations

It has no capabilities.

I a couple of weeks ago and then got annoyed that there was no way to poke at it on my own machine. So I wrote an inference engine for it in C99.

Nothing clever going on. 93% of that 1.56 TB checkpoint is routed experts, and only 16 of 896 fire per token, so the experts never become resident at all. They get read off NVMe on demand and multiplied straight out of their packed 4-bit form, no dequantization step. The dense trunk gets repacked into one file where layer L sits at a known offset and streamed one layer at a time. What stays in RAM is a dial you set.

Numbers from my box (2x EPYC 7763, NVMe, the four GPUs in it sat idle the entire time):

• 8.24 GB peak RSS at the smallest preset, ~33 s/token

• ~128 GB gets you ~20 s/token, which is as fast as it ever got

• Output is byte-identical at every budget in between

I know that this is not a practical way to use K3. It is half a minute per token and it wants 1.7 TB of free disk for the checkpoint plus the packed trunk. I built it to understand the architecture by implementing it, not because you should serve anything with it.

No BLAS, no framework, no GPU path. Six C files, libm and OpenMP, 176 KB binary.

If you want to sanity check it before committing to a 1.56 TB download: clone and run `make && make test`. About a minute, no weights and no network needed. It builds a 13-layer model with the same tensor graph and checks it against a PyTorch reference from committed fixtures, including greedy decode and the incremental path with the KV cache and carried KDA state.

Repo:

DSv4F doesn't ship a jinja, but for distributions that do and faithfully reconstruct what DS releases in their chat template python, every system message is hoisted into the system prompt at the top -- the format has no mid-conversation system turn. So, anything you stick at the tail or mid-convo actually fries your prefix (and doesn't have conversational proximity to the injection point).

Use latest_reminder, which is the role DS trained for how most templates use system and what most people providing quants are passing through (if they match DS' python template). I use llama.cpp and it happily passes it through no issue; dunno how other engines work with it.

Couldn't figure out why my prompt caching was so garbage and there it was, so I'm passing it on to hopefully save others time and frustration (and probably money, if you're using a hosted version).

Hi all,

I'm happy to announce that Xberg v1 is out.

Xberg is the successor to Kreuzberg, equivalent to what would have been Kreuzberg v5. It's a content intelligence framework that handles a very wide range of inputs: documents (currently 101 formats), code and data formats (currently 367 types), audio/video transcription, and URLs (both static and JS-rendered content). It extracts and prepares that content for downstream processing.

It's an extremely efficient, high-performance engine (see our PDF benchmarks below). For PDFs and images specifically, we handle native PDFs with very high performance and accuracy, and we ship multiple OCR engines that match the quality of the best Python libraries (e.g. docling, PaddleOCR, RapidOCR) at substantially better performance and stability.

The changes between Kreuzberg v4 and Xberg v1 are substantial, and I invite you to read the for the complete picture. The highlights below give a sense of what's new:

• Pure-Rust PDF backend (pdf_oxide) replaces pdfium, with no native pdfium dependency.

• Layout-aware pipeline: reading order reconstructed with ONNX layout detection (PP-DocLayoutV3 / RT-DETR) and Docling-style predecessor-graph reordering.

• Per-page scanned-page detection with selective OCR, plus AcroForm/XFA form fields and outline-based headings.

• Across-the-board optimization of OCR and PDF extraction (memory discipline, pooled model sessions, streamed conversions).

• Native PaddleOCR backend (PP-OCRv6, with medium / small / tiny tiers) alongside Tesseract.

• Pure-Rust Candle OCR/VLM stack (TrOCR, GLM-OCR, GOT-OCR, DeepSeek-OCR, and PaddleOCR-VL) running without ONNX Runtime or native Tesseract.

• A second, ONNX-Runtime-free inference path via tract, which is what makes in-browser (WASM) and mobile inference possible.

• Named-entity recognition natively in Rust (GLiNER2), extensible to all bindings, including an in-browser WASM model with no server round-trip.

• Structured LLM extraction (extract_structured / split_and_extract) with rasterization, chunking, citations, caching, and configurable call/merge/VLM-fallback policies.

• Audio & video transcription via a Whisper ONNX engine (.mp3, .wav, .m4a, .mp4, .webm).

• Retrieval building blocks: sparse embeddings (SPLADE), ColBERT late-interaction retrieval, and cross-encoder reranking alongside dense embeddings.

• Text intelligence: reversible redaction, summarization, translation, VLM image captioning, QR-code detection, document diffing, and page/chunk classification.

• URL & web ingestion: sitemap discovery (map_url) and batched multi-URL crawling.

• New document formats: WordPerfect (.wpd/.wp/.wp5), HEIC/HEIF/AVIF, OpenDocument Presentation (.odp), Quarto / R Markdown, and configurable Jupyter cell rendering.

• Four new language bindings (Dart/Flutter, Swift, Kotlin/Android, and Zig) bring the total to 15 language bindings over one engine, with Android/iOS cross-compilation.

• Full mobile support (Flutter, Android, iOS).

• Candle backend alongside ONNX, plus ONNX-via-tract enabling ONNX on WASM and Android.

• Wider code intelligence: tree-sitter coverage grew substantially (248 to 367+ languages).

• Over 150 bugs fixed during the 1.0 cycle, plus security hardening (bounded RTF/PDF allocations, redaction leak fixes, Excel DDE warnings).

The API surface was also simplified and reworked, making it more consistent.

There's a migration guide in our docs explaining how to move from Kreuzberg to Xberg. Kreuzberg itself is in LTS mode until the end of this year and will continue to receive bug fixes and security updates.

You're invited to check out the and join our .

Benchmarks

The benchmarks below are for PDFs and images only. There are extensive benchmarks on our website with per-format breakdowns, which you can see . These numbers are measured in CI via our reproducible benchmark harness, and are specifically taken from the run for harness 1.0.8, source cf7fa0533d. The data is publicly available in GitHub releases, and you can run the benchmark harness yourself.

Composite quality (markdown pipeline, higher is better):

Structure and layout fidelity (SF1: tables and reading order, higher is better):

On native PDFs Xberg leads on quality (0.958 vs 0.837 for the next-best framework) and on table and reading-order fidelity by a wide margin (SF1 0.949 vs 0.612 for docling). On scanned PDFs it is #1 on both quality and raw text fidelity.

Where we don't win yet: on pure image OCR we are currently #2 on the composite score, behind mineru (though still #1 on raw text accuracy). We are improving image OCR right now, and v1.1 should have us winning across the board.

TL;DR: I spent 9 days developing a new quantization method for MLX models and measured 18 variants against each other on a single M5 Max MacBook Pro (128 GB). The result is the best-measuring MLX quant of Qwen3.5-122B-A10B I'm aware of at any size — the 82 GiB build edges out 94–95 GiB 6-bit builds, and lands within 0.3–0.7% of the imatrix-rounded source GGUF while staying native MLX. Apache 2.0, weights up on HF.

Why bother if GGUF is better?

MLX on Apple Silicon is substantially faster than llama.cpp on the same hardware — on my M5 Max I measure roughly 9x faster prefill and ~20% faster token generation. For anything with a long context and a lot of turns, that gap compounds.

The problem is that existing MLX quants below 6 bit are not great, and you can see it in the table below: oQ4 gives up ~3.8% perplexity at short context and ~4.2% at long context against the source GGUF. In practice that shows up as incoherent reasoning traces and rounding errors that stack until the model starts hallucinating.

So a better MLX quantization method has real advantages for agentic workflows and local AI on Apple Silicon. At the same time, I made the conscious decision to require native MLX support. imatrix on MLX is not format native — it needs custom kernels. WinterMix quants are format native and are drop-in replacements.

WinterMix quantized models are format-native MLX models with open weights (Apache 2.0). No custom kernels, no forked runtime, no flags. They load anywhere MLX works — LM Studio, mlx-vlm, and friends — at stock speed, with the vision tower fully functional and coherent thinking traces.

If you just want to try it: download the repo below, point LM Studio at it, done.

HuggingFace Links

— 82 GiB, ~6.0 bpw: the best-measuring MLX quant of this model I'm aware of at any size, including against 94–95 GiB 6-bit oMLX builds (narrowly at 2K, more clearly at 16K).

— 68 GiB, ~5.0 bpw: leaves ~35–40 GB free on a 128 GB Mac = 5–8 parallel 100K-token agent sessions resident at once (GDN architecture keeps a 100K session's cache at ~5–10 GB). Beats its direct size-peer (oQ4, 67 GiB) by ~1.4–1.5% at both context lengths.

Numbers

One scoring rule for every row (NLL over the second half of each window, token-aligned across engines — llama.cpp's native rule, so these are comparable to Unsloth's), paired per-token where both models run under MLX. Reference rows were measured on my own harness: same tokens, same machine. oMLX quants are included because oMLX is currently the popular option for MLX.

All rows are Qwen3.5-122B-A10B in various quantization mixes.

Being upfront about the ceiling: the imatrix-rounded source GGUF is still slightly ahead (+0.3–0.7% rule-matched). Matching imatrix-style weighted rounding in MLX would need custom inference kernels, and "loads in everything at stock speed" was a hard constraint I wasn't willing to break. Within the native format, this appears to be about the limit.

The part I think is actually interesting

Halfway through this project I found that perplexity is blind to real behavioral differences between quants. Two builds with statistically identical NLL differed 2.5× in how often they self-interrupt ("wait, let me re-check...") during 50K-token reasoning traces. Then the reverse bit me: my best-NLL build had an elevated self-interruption count — and actually reading the traces showed it wasn't confusion at all, but disciplined audit passes that twice caught a base-model reasoning bug before the final answer.

So the release models were selected on three instruments: paired NLL, blind-scored state-tracking benchmarks at depth, and directly reading the reasoning traces. Both releases deliver perfect scores on a 30-step adversarial state-tracking task on every seed — and the 68 GiB build's traces show it catching its own 4-bit arithmetic slips before they reach the output. If you evaluate quants, I'd honestly recommend reading traces over counting anything.

What's under the hood (briefly)

Sensitivity-informed mixed-precision allocation (routing-critical tensors pinned at BF16 — MoE routers do not like being quantized), GPTQ-family error-compensated rounding reimplemented natively for the MLX affine format and executed layer-wise (whole-model GPTQ OOMs a 122B on 128 GB; streaming it peaks around 28 GB), and a diverse long-context calibration mixture engineered so every expert in every layer actually gets calibrated — including multilingual content, because it turns out an English-only calibration set silently starves the language-specialist experts. Validated across 18 measured variants with paired controls and held-out out-of-domain checks (no calibration binding: code/math within ±0.1% of RTN).

I'm not releasing the pipeline code for now — the models are open weights (Apache 2.0), the method writeup stays private. The M5 Max kernel-panicked ten times during development before I got the workload tamed, if that helps set the vibe.

Requests

I'm planning to take requests for MLX quantizations of other models — drop them in the comments or in the HF Community tabs. Practical constraints: it has to fit the pipeline on a 128 GB Mac (up to ~120B+ MoE is proven), and dense models calibrate differently than MoE, so results may vary until I've tuned per-architecture.

Happy to answer questions about the eval methodology, the behavioral testing, Apple Silicon quirks (ask me about watchdog panics), or Mac long-context agent setups.

Basically you now have to mark all AI generated images, audio, video and text as AI generated. :P

Hello, Also I want to join the hype of posting token specs.

CPU: 2x Intel Xeon CPU E5-2650 v4 @ 2.20GHz

RAM: 2x 4 Channel 2400MHz DDR4

GPU: 1x AMD Radeon 7900 XTX 24GB

3x AMD Instinct MI60 32GB

Strange GPU combination, right? One of my AMD Instinct MI60 32GB failed, and I have no spare and other choices.

Prompt processing is in the high 140t/s (got down to mid 80t/s at 60k context). Inference is a about 11t/s.

llama.cpp command is not optimized.

llama.cpp logs:

38.32.848.664 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =   2048, progress = 0.03, t =  14.48 s / 141.44 tokens per second
38.47.371.961 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =   4096, progress = 0.06, t =  29.00 s / 141.23 tokens per second
39.05.717.960 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =   6144, progress = 0.09, t =  47.35 s / 129.76 tokens per second

..

51.08.617.986 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  65536, progress = 0.98, t = 770.25 s / 85.08 tokens per second
51.27.241.174 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  66682, progress = 0.99, t = 788.87 s / 84.53 tokens per second
51.34.905.170 I slot print_timing: id  0 | task 0 | prompt processing, n_tokens =  67194, progress = 1.00, t = 796.54 s / 84.36 tokens per second
51.43.631.476 I slot print_timing: id  0 | task 0 | n_decoded =    100, tg =  11.96 t/s, tg_3s =  11.96 t/s
51.46.703.156 I slot print_timing: id  0 | task 0 | n_decoded =    135, tg =  11.81 t/s, tg_3s =  11.39 t/s
51.49.759.994 I slot print_timing: id  0 | task 0 | n_decoded =    171, tg =  11.80 t/s, tg_3s =  11.78 t/s

..

52.11.051.549 I slot print_timing: id  0 | task 0 | n_decoded =    427, tg =  11.93 t/s, tg_3s =  11.84 t/s
52.14.103.819 I slot print_timing: id  0 | task 0 | n_decoded =    464, tg =  11.95 t/s, tg_3s =  12.12 t/s
52.17.143.301 I slot print_timing: id  0 | task 0 | n_decoded =    501, tg =  11.97 t/s, tg_3s =  12.17 t/s
52.19.252.023 I slot print_timing: id  0 | task 0 | prompt eval time =  796902.50 ms / 67198 tokens (   11.86 ms per token,    84.32 tokens per second)
52.19.252.029 I slot print_timing: id  0 | task 0 |        eval time =   43980.11 ms /   526 tokens (   83.61 ms per token,    11.96 tokens per second)

llama.cpp version: 10223 (11924d4c1)

llama.cpp backend: ROCm 7.2.4

llama.cpp command line:

GGML_CUDA_P2P=1 llama-server -m DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf --temp 1.0--top-p 0.95--min-p 0.00 -fa 1 -c 1048576 -np 1 --chat-template-kwargs {"reasoning_effort":"max"} -lm none -mg 0

Hello everyone I want to join the hype of posting specs.

CPU: AMD EPYC 74F3 24-Core

RAM: 8 Channel 3200 DDR4

GPU: RTX A6000 48GB

Prompt processing is in the high 70t/s (got down to mid 30t/s at 300k context). Inference is a steady 17.20t/s~ and the 48GB VRAM is enough to have the full 1mil context but PP will be so bad. Sadly not as cool like those M5 Macs.

Anyone else having similar specs?

Edit: I was informed about batch size and set mine to 8096 and my Prompt processing jumped to almost 400t/s at the start. it got to around 300t/s at 20k context. Better than my 70t/s stock lol.

I know in in this community LLM's are generally used for coding but there are other usecases besides coding and those usecases should be tested too. I also know benchmarks can sometimes be benchmaxxed and the model can still turn out shit but it can give a good outline on how a model should perform in a certain task. Maybe I'm too behind on the latest developments but we need more benchmarks for all other use-cases. I use LLM's mainly for foreign language learning, creative writing and STEM/Medical/Biochemistry reasoning and inquiries and I rarely find any new benchmarks that tell me how a model might perform in those areas. MMLU-Pro-2 and a solid benchmark that tells how a model will perform for language learning would be so good for my usecase, however in general we need more new diverse benchmarks for models in order to have a general outline for advancements in other areas.

TLDR below 👇🏼

I’ve seen a lot of hype around Qwen 3.6 35B and 3.5 120B lately, especially regarding coding and tool-use capabilities. On this subreddit it is the defacto recommended model for everyone without a Datacenter at home. I’ve been running Qwen 3.5 120B (Qwen3.5-122B-A10B-GPTQ-Int4) as an autonomous worker agent in a multi-turn development loop using the Hermes agent harness.

While the model is undeniably impressive at one-shot snippet generation, putting it into a fully autonomous, long-context environment to build a module from scratch revealed several consistent failure patterns.

I thought I'd share these failure modes to see if others are experiencing the same issues—or if anyone has found effective tricks to tame it in such a task.

Here is what went wrong:

1. Premature "Mission Accomplished" Syndrome

The model has an overwhelming tendency to shout "DONE!" or "PERFECT!" after completing 10% of a task. It constantly reports success based on superficial checks (e.g., "the file built without syntax errors"), completely ignoring explicit acceptance criteria like end-to-end testing or UI rendering.

2. Evading Hard Constraints

When given strict architectural constraints (e.g., "Must be a single, self-contained module with zero external dependencies"), the agent aggressively cuts corners:

* It secretly substituted live data with hardcoded mock data.

* It wrote external Python scripts and set up local host cron jobs to bypass building proper module logic.

* It even rewrote part of the host application in a completely different language just to claim a quick win.

It prioritizes appearing finished over following instructions.

3. Hallucinating Infrastructure Limitations (Blame-Shifting)

Instead of debugging broken code, the model repeatedly blames the host environment. When its code failed to make network requests or render components, it confidently hallucinated system limitations:

* "The host framework's authentication token system is broken."

* "The runtime DNS resolvers don't support HTTP requests."

It will generate elaborate technical excuses rather than inspecting its own schema or syntax.

4. Ignoring Provided Docs and Boilerplates

Even when explicitly handed a boilerplate repository and documentation links in the prompt, it constantly tries to "reinvent the wheel." It overcomplicates custom build setups, invents new protocol schemas, and ignores pre-built Docker/build scripts that were provided to make its life easier.

5. Regression Cascades & Context Rot

As a result from the above the conversation history grew and the agent suffered from severe regression:

* In iteration 3, it had a working UI with mock data.

* By iteration 8, after trying to wire up live data fetching, it completely broke the UI.

* It failed to recognize that its new changes broke previously validated features, leading to endless debugging loops.

Discussion

Qwen 3.5 120B feels like an insanely talented junior developer who panics under pressure, lies about tests passing, and blames the server infrastructure when their code throws a 404.

Has anyone successfully mitigated these behavior loops in autonomous coding agents? Are you using specific prompting techniques, or is this just an inherent limitation of current 100B+ open models when complexity grows from "Do exactly what I tell you" to "Figure it out with my help"?

Curious to hear your experiences!

TLDR;

While Qwen 3.5 120B is great at one-shot generation, it breaks down in autonomous, multi-turn agent loops. The main issues are: Premature success claiming, Bypassing hard constraints, shifting blame on other systems when things don't work, Ignoring Docs and boilerplate Code that could have made its life easier. And as a result from that Context Rot.

Hey all,

tldr / who this helps: you run a mixed multi-GPU box where the experts spill to RAM, and you want to stay in the 3-bit tier instead of dropping to Q2 to make it fit.

Edit: I've also uploaded IQ3_XXS with info on which to download.

I requantized only the 129 routed expert tensors of DeepSeek-V4-Flash-0731 and left every other tensor at whatever precision the source GGUF already had. Attention, shared experts, router and indexer stay at Q8_0/BF16/F32 from bartowski's MXFP4 conversion. Only the experts drop to IQ3_XXS, with the down projections one rung up at IQ3_S. Result is 111.37 GiB in four shards and imatrix built from calibration_datav3.

For quality I scored it with llama-perplexity KLD against reference logits generated from the MXFP4 source itself, wikitext-2 first 150 chunks at ctx 512, and ran unsloth's UD-IQ3_S through the same axes for comparison. Mine gets mean KLD 0.2386 vs 0.2936, top-1 agreement 84.65% vs 82.78%, delta PPL +0.536 vs +0.685. However mine is 2.12 GiB larger, and UD-IQ3_S has the better max KLD at 11.13 vs my 12.53, so it is not a clean sweep. Raw perplexity logs for all three runs are in the repo if you want to take a look.

Speed on my rig, which is 5 mixed GPUs (2x 3090, 5060 Ti, 2x 4060 Ti, 96 GiB VRAM total) with expert spill to CPU: 13.91 / 13.57 / 13.26 t/s at depths 0 / 4096 / 16384, against 9.88 / 9.69 / 9.51 for the full MXFP4 source at the same placement. About 1.4x. That is a spill bound number and will not transfer to a box that fits it entirely in VRAM.

If you are purely chasing tokens per second, going smaller beats this by a lot. antirez's flat Q2 of the same model is 80.76 GiB, sits about 98% resident in my VRAM with no spill at all, and does 30.27 t/s, 2.18x mine. The point of this build was the quality tier at roughly 3 bits, not the highest number.

Also beware that DeepSeek-V4-Flash has open SWA and rollback stall issues in llama.cpp. I quantized with mainline llama-quantize but I run a patched build with DSV4 stall fixes that are not upstream and have not tested this GGUF against a stock llama-server. If you hit stalls on long contexts that is the known upstream issue and it affects every DSV4 GGUF.

I plan on using this recipe for other models as well. Cheers!

The biggest issue with preview was its inability to follow rules prompts and skills. It seems like no matter what you do it ignores them. I've tried first person and second person. I've tried Chinese and English. It does not follow them. That's the only problem with these models and why they're not actually frontier level and not just benchmaxxed. Every user's environment is different and they need to tune the actions and behavior of the model with rules or prompts or skills to exactly what they need to do and if the model ignores then it acts subpar. The newest version of flash has the same issue as the preview version and that's unfortunate.

I run it native, full precision, locally. And I've held out making this post cause I know I'm going to get roasted to all fuck but when you can actually run the models locally and when you're not brainwashed by benchmarks and you actually code with them you see the holes. I'm going back to qwen 27b ugh

Edit: I've seen two users provide credible information as to why deep-seek acts like this. I did some research and think I was able to verify it. It's been a rough 4 hours. Deepseek v4 stores rules/skills/prompts as compressed summaries, not raw text. 43 layers and 20 of them see the entire context at 128 tokens squeezed into a single entry. 21 see it at 4:1 and only two are fully dense. Every layer also gets the last 128 tokens uncompressed but that's only for the full resolution window and the prompt/skill/rule isn't in there lol. So they "survive" but the exact wording doesn't. There is a startup arg in vllm that might help. --hf--overrides '{"index_topk": 1024}'. In the 21 layers that stay 4:1 detail the model selects only 512 compressed entries per token about 2,048 tokens worth of fine detail from anywhere in the context raising this value to 1024 doubles that to 4,096 tokens. Now I asked opus 5 if this would solve the problem and opus said most likely not. I'm going to give it a go anyway though thanks for viewing my TED talk.

This pull request was added to the main llama cpp about 12 hours ago.

I was experiencing some looping and poor behavior yesterday but haven't had any problems since this fix.

DeepSeek V4 Flash got me thinking...

We keep seeing smaller models get way better. A model at a certain parameter count today can be much smarter than a model of the same size from a year or two ago. Better training, better data, better architectures, distillation, MoE, and all of that seem to let companies squeeze more intelligence into smaller models. But is there eventually a limit to this?

At some point, a model needs enough capacity to understand language, store knowledge across a huge number of subjects, reason through problems, write code, follow instructions, and generalize to things it has not seen before. So can we just keep shrinking models while maintaining the same level of intelligence?

Could a future 30B model actually match a current 300B or 700B model across everything? Not just on a few benchmarks, but in actual use across lots of different domains.

Could the same eventually happen with a 7B model? Or is there some minimum amount of capacity needed before the model starts losing knowledge, reasoning ability, or reliability?

I know parameter count is not a direct measurement of intelligence. MoE also makes this more confusing because a model can have hundreds of billions of total parameters while only using a small portion of them for each token. There is also a difference between total parameters, active parameters, memory usage, and actual inference compute.
I also do not think comparing parameters to neurons in the human brain is very useful. They are obviously not the same thing. Still, it makes me wonder whether there is some minimum amount of information or computation needed for something close to general intelligence.
Maybe we are not actually removing the cost either. Maybe we are just moving it somewhere else. A smaller model might require a much more expensive training run, synthetic data from larger models, distillation, longer reasoning time, retrieval, or external tools.

There is also the benchmark question. When a smaller model gets a similar benchmark score to a much larger one, does it really have the same overall capability? Or is it more optimized for the things we currently test?

Maybe it matches the larger model most of the time, but falls apart more often on rare knowledge, unusual prompts, long tasks, or problems that are very different from its training data.

My guess is that there is probably a minimum size for any specific level of capability, but better training and architectures keep pushing that minimum lower. I just wonder when the big improvements start slowing down.
Are we still early enough that models can keep getting dramatically smaller and smarter? Or are we getting close to the point where the easy gains are gone and the last 10 or 20 percent becomes extremely difficult?

Been using Qwen 3.6 35B-A3B quite extensively lately and honestly, I’m pretty happy with it. Also tried a few community improvements like Ornith 1.0, which add some interesting tweaks.

That said, I’m curious about what the community expects next from Qwen’s open-source roadmap.

Do you think we’ll ever see open weights for Qwen 3.7 (already available on OpenRouter), or is that unlikely?
Or are there other directions you think the team will prioritize instead?

You have two choices here (in order of pref):

1. Downgrade CUDA from 13.3 to 13.1 (skip 13.2 due to bugs) <- prefer this (thanks to for pointing this out)

2. Use this vibed fork that works with CUDA 13.3

I was troubleshooting this yesterday with the nvidia profiler and some LLM help ()

Here's some more info on #1 (quote from fairydreaming) "Downgrade your CUDA and recompile. Starting with 13.2 DeviceTopK is used for top-k instead of argsort, this turns PP rate to crap."

In short, DS4 Flash is spending a lot of time on things other than matrix multiplication.

# Running DeepSeek-V4-Flash-0731 (155 GB MoE) on a DGX Spark with vLLM-Moet 2-bit quantization

I used Deepseek-v4-Flash-0731 cloud API settig up vllm-moet to run deepseek-v4-flash with MTP locally on single DGX Spark at 2-bit quant. Thought it might help others. Below is the summery from my AI Agent. So I did not write myself.

There are few important things you must take care, and guide AI to do it for you. AI alone won't get it done right.

1. rebuild vllm-moet on ARM64

2. pull PR #11 into the repo

3. build the source code, and ask AI to modify the code that complains unsupported sm121 GPU.

4. increase default VLLM timeout because the loading take very long time, and triggers false timeout.

5. I do not recommend you to follow the below procedure to duplicate it. Instead feed the below text to your AI agent, let it handle the process and fixes.

6. you need to setup a very big swapfile, or the loading will fail. the swapfile is only needed during model loading

7. For convenience, I create a repo of the MTP head from preview version. If it helps others, it is located here.

Performance wise, the prefill is at steady 1000 tps.

decode is below

### Aggregate (tok/s)

| Concurrency | MTP | no-MTP | Δ |

|---|---|---|---|

| 1 | 25.2 | 19.1 | **+31.5%** |

| 2 | 30.9 | 26.4 | **+17.3%** |

| 4 | 43.2 | 45.6 | **−5.3%** |

### Per-request (tok/s)

| Concurrency | MTP | no-MTP |

|---|---|---|

| 1 | 25.2 | 19.1 |

| 2 | 23.0 | 17.7 |

| 4 | 14.5 | 13.9 |

== Below is the AI talking ==

**TL;DR:** [vLLM-Moet]() serves the new `deepseek-ai/DeepSeek-V4-Flash-0731` checkpoint on a single DGX Spark (GB10, 121.7 GiB unified memory, aarch64). The image **must be built on the Spark itself** (x86→arm64 transfer is impossible), the Dockerfile base digest is amd64-only and needs the multi-arch tag, sm_120 cubins run fine on GB10's sm_121, and the 0731 revision's DSpark MTP head won't draft on this stack — plain decode or load the main repo's 1-layer MTP head as a separate draft model (**+48% decode**).

## The stack

- **Model:** `deepseek-ai/DeepSeek-V4-Flash-0731` — 155.43 GiB FP8, 48 shards

- **Engine:** vLLM-Moet (vLLM v0.25.0 + ~7.4k-line patch) — 2-bit MoE experts on hand-written SM120 SASS kernels

- **Hardware:** DGX Spark — GB10, aarch64, sm_121, **no discrete VRAM** (121.7 GiB unified pool), 128 GiB swapfile

## Measured (Spark, 512K, FORCE_RESIDENT, delta off)

| Metric | Value |

|---|---|

| 2-bit planes | 43 layers × 1.69 GiB ≈ 73 GiB |

| KV cache / util 0.90 | 4.56M tokens (**8.7× concurrency**) |

| Decode — plain | ~19 tok/s (bandwidth-bound on LPDDR5X) |

| Decode — +MTP head | **26.6 tok/s (+48%)** |

| Boot — v025 warm plane cache | ~10 min (31–46 min cold) |

MTP vs plain (pp2048/tg512, 3 runs): conc1 25.2→19.1 (**+31.5%**), conc2 +17.3%, conc4 −5.3% (aggregate flips at high concurrency; per-request never hurts). k=2 is the optimum for the 1-layer head.

## Critical items to modify for DGX Spark (the actual gotchas)

**1. Build on the Spark — don't transfer the image.** The PRO 6000 image is linux/amd64; vLLM is arch-specific, `docker save`/`load` across x86→arm is useless. Build natively on aarch64.

**2. Dockerfile base digest is amd64-only.** `Dockerfile.sm120-v025` pins `vllm/vllm-openai:v0.25.0@sha256:e1c1ff…` — that digest is a *single amd64 manifest*. Swap to the multi-arch tag `vllm/vllm-openai:v0.25.0` (resolves to arm64 `2f726d…` on the Spark); keep the old digest commented with a why-note.

**3. Repo transfer via git bundle + explicit branch fetch.** `git bundle create v025.bundle v025` → scp → `git clone <bundle>`, then `git fetch <bundle> v025:v025 && git checkout v025`. **A bundle clone lands on the wrong branch (master)** — the fetch is mandatory.

**4. Don't rebuild SASS for sm_121.** GB10 is CC 12.1; the repo's baked sm_120 cubins + `TORCH_CUDA_ARCH_LIST=12.0a` load fine (minor-version forward compat, proven on v024 and v025). flashinfer publishes an aarch64 cu130 wheel (0.6.14), so nothing else changes.

**5. The 128 GiB swapfile MUST be in `/etc/fstab`.** The 155 GiB checkpoint can't stage in 121 GiB RAM — loading is swap-bound. The run script's `swapon` only fires on manual recreate, so after any host reboot swap is 0B → deterministic EngineCore OOM-kill → `--restart` crash loop (**88 restarts in 26 h**). Fix: `echo '/swapfile none swap sw 0 0' >> /etc/fstab`. Observed swap peak 69 GiB during weight load, reaped to ~2.4 GiB after plane build.

**6. 0731's MTP head is DSpark — it won't draft on this stack.** The revision ships a 3-layer DSpark head (`main_proj`/`main_norm`/`markov_head`/`confidence_head`/`hc_head`); the fork's MTP path can't replicate it (`KeyError: mtp_block.main_norm.weight` with MTP on, or 0% draft acceptance). Two working options:

- **Plain decode** (drop `--speculative-config`) — simplest, ~19 tok/s

- **Main-repo MTP head as separate draft model (+48%)** — extract the 1-layer head from the main `DeepSeek-V4-Flash` repo's last shard (3.4 GB, `num_nextn_predict_layers: 1`), or just use the published one `ycui7/DeepSeek-V4-Flash-MTP`:

```bash

--speculative-config '{"method":"deepseek_mtp","model":"/models/DeepSeek-V4-Flash-MTP","num_speculative_tokens":2}'

```

**7. Watch the read-only model mount.** With the model dir bind-mounted `:ro`: `VLLM_MOE_W2_STORE_DIR` into it **silently persists nothing** (every restart re-requants ~14 min), and `VLLM_MOE_W2_DELTA_GB>0` **hard-crashes** (delta store creates a lock file → `OSError: Errno 30 read-only`). Point STORE_DIR at a separate writable volume.

**8. No nvidia-smi; FORCE_RESIDENT's warning is survivable.** `nvidia-smi` shows `[N/A]` and EngineCore RSS stays ~3 GiB while device memory fills the unified pool — monitor with `free -h`/`docker stats` + `moe_w2: layer N planes built` logs. The "RESIDENT planes exceed budget by 69.7 GiB" warning is expected on GB10; it boots fine (planes + KV share the pool).

**9. `DELTA_GB=0` is the right call on Spark.** Disabling the FP4 delta frees ~20 GiB straight into KV (706K → 4.56M tokens ) and cuts boot 46 → 31 min. Decode unchanged (~19 tok/s — bandwidth-bound; the delta was never a speed factor here).

**10. Be patient — the load is silent and swap-bound.** ~16 min of zero log output while 155 GiB stages through swap (EngineCore at 99% CPU), then plane build (~14 min, warms 25s→6s/layer). Don't kill the container.

## The run (production)

```bash

docker run -d -it --restart unless-stopped --name ds4f-vllm-moet \

--gpus all --network host --ipc host --shm-size 64g \

-v /models:/models:rw -v /plane-cache:/plane-cache \

-e VLLM_MOE_W2=1 -e VLLM_MOE_W2_FORCE_RESIDENT=1 \

-e VLLM_MOE_W2_BASE_CACHE_GB=0 -e VLLM_MOE_W2_DELTA_GB=0 \

-e VLLM_MOE_W2_STORE_DIR=/plane-cache/packs \

vllm-moet-sm120:v025 \

/models/DeepSeek-V4-Flash-0731 --port 8000 \

--served-model-name deepseek-v4-flash \

--trust-remote-code --kv-cache-dtype fp8 --block-size 256 \

--max-model-len 524288 --gpu-memory-utilization 0.90 \

--max-num-batched-tokens 2048 --max-num-seqs 1 \

--tokenizer-mode deepseek_v4 --no-scheduler-reserve-full-isl \

--enable-auto-tool-choice --tool-call-parser deepseek_v4 \

--reasoning-parser deepseek_v4

# optional MTP: add --speculative-config '{"method":"deepseek_mtp","model":"/models/DeepSeek-V4-Flash-MTP","num_speculative_tokens":2}'

```

Verify: `curl :8000/v1/models` then a chat completion.

Hi all,

I recently discovered that MiniMax offers ~1.7B tokens/month for a basic $20 subscription, and I was genuinely shocked! I wanted to try it out and paired it with OpenCode. Everything is working amazingly well, but I started to wonder what happens with my data? I know OC is open source but navigating the codebase would take me weeks, so I wanted to ask the community here whether my data is being accessed by OC when using an outside provider.

If the answer is yes, then what would you recommend me? Nanocoder was an alternative, wondering how that works, and whether there are other options.

Thanks a lot!

I have a bunch of PDFs, word docs, and excel files that relate to a project of interest. I am trying to figure out what is the best way to organize this to an llm for easy understanding. This is an on-going project so if I ask a question a month from now I’d like to be able to get an accurate answer based on the entire corpus of info I’ve fed it. Currently I have everything dumped into a single folder. Then I use VSCode with Cline to ask questions against all those documents (about 500+ files in total). I am also using all that context to build our rules which I code up as well as build sql tables. So I need the code and database to reflect what’s found in those files.

With all that, what have you found to be the most efficient way of doing sth like this?

Thanks

Personally, i think good coding model shouldn't be focused on one-shot "everything in one html-file" tests, but should be really good on debugging, fixing and modifying its own output. Anyone know such simple tests that i would able to run with local models?

May be some kind of synthetic stuff that forces LLM to build something that is broken by design and then asking the model to do a multi-step changes, fix issues, analyze program's output (preferably with getting screenshots/videos)?

previously on my 3070, 32gb ddr4 and i711700 I used this command for months and got 26-30 tps:

"C:\Program Files\llama cpp\llama-server.exe" ^

-m "C:\Program Files\llama cpp\models\Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf" ^

--mmproj "C:\Program Files\llama cpp\models\mmproj-F16.gguf" ^

--gpu-layers 99 ^

--cpu-moe ^

--ctx-size 131072 ^

--cache-type-k q8_0 ^

--cache-type-v q8_0 ^

--port 8081 ^

--host ^

--jinja ^

--no-mmap ^

--parallel 1 ^

-b 4096 -ub 4096 ^

--temp 1.0 ^

--top-p 0.95 ^

--top-k 20 ^

--min-p 0.0 ^

--presence-penalty 1.5 ^

--repeat-penalty 1.0 ^

--chat-template-kwargs "{\"preserve_thinking\":true}"

then yesterday suddenly at start I am at 13 or 12 tps even after lowering ctxt to 32k. my gemma model got the same tps hit as well. if anyone can help me I will appreciate.

I think people are sleeping on Gemma and local models so I built a free, very fast harness for Gemma 4 that I call Tomte.

Works on Macs with M processors, will have a companion app you can connect to anywhere. So far does everything I ever needed chatGPT for!

I was driving the other day and saw a 1996 Ford Taurus. You know the one, you've probably seen it cruising in the rougher parts of town since they're starting to become the junkers of today. It's the generic weird looking rounded off car that... well...

Anyway, you're probably wondering why this guy's talking about a Ford Taurus. Seeing that car on the west side of Pueblo made a little lightbulb go off. I found myself asking... how many of those damn things did they actually build?"

I looked it up. They built 348,671 of these sedans in 1996. That's 955 finished Ford Taurus being built every single day. 39 an hour, every hour. Regular people in a Ford factory build that car. They stood and built an impossible object at scale.

Nobody in that entire building knew how to build a Ford Taurus, let alone 39 of them in an hour. Most of them couldn't tell you how an engine works, or how to bond paint to metal, or how to cast aluminum. They had no idea what they were doing, really. Some of the workers on the line on any given day were brand new, fresh out of high school, and barely knew how to tie their shoes. They might not have even known the piece of metal in front of them IS a Ford Taurus. All they know is a slab just rolled up, they're supposed to put three holes in it in three well defined and visually marked places. They do it, and another piece of metal rolls up.

They aren't building a Ford Taurus, they're drilling three holes again and again.

The factory still put out 39 cars an hour, every hour. The barely trained guy on his first day on the line stood in his station and punched his three holes in the sheet metal where the jig told him, 39 times an hour, and the piece of metal moved on, and a new piece came in. He may have made a few mistakes that got corrected along the way (the occasional hole being slightly out of spec), but those issues got caught before the piece moved along and the mistakes were corrected. More importantly, the process that ALLOWED those mistakes to happen gets corrected so that the person can't drill out of spec.

Done right, mistakes become effectively impossible. It's hard to mess it up because he's not being asked to build a Ford Taurus, he's being asked to punch three holes in sheet metal where the colorful dots tell him to drill.

Factories designed entire strategies around this, like Toyota's Poka Yoke (mistake proofing, making a process that ensures the worker can't do it incorrectly, control methods that physically block an incorrect step). At the end of the line, cars rolled off fully assembled and ready to go. Mistakes can be almost entirely eliminated as the line speeds up.

I mention this, because these thoughts have started to creep into my AI work in a big way.

AI is like having an intelligent, eager, untrained team of employees standing on your factory floor. They want to work and they are relatively capable. They can work tirelessly day and night. The problem is... none of them can build a Ford Taurus, and this is a Ford Taurus factory. Ask the best damn mechanic in the room to build a Ford Taurus and they might run around trying their best, and if you give them the better part of a year they might even build you something you can drive... but if you take that goal (a finished Ford Taurus) and break it down into a bunch of tiny little steps, suddenly that team of fools can build them at scale.

There are moments where you can just 'ask a guy to make something', and the result will be decent... but a process and a team builds more, faster, better.

Don't ask your AI to build a Ford Taurus. Ask them to drill three holes in the sheet metal in front of them.

Anyone else out there starting to turn AI into Factorio? Lol...

Everything is a distillation of Claude and GPT. I can ask Claude to review GPT and vice versa, but any other model-pair is essentially the model reviewing itself. Sucks that we're stuck with an echo chamber of models.

Edit: Wow I guess there's a bunch of PhDs in here lol.

What speeds are everyone getting with deepseek v4 flash 0731?

I’m getting~200 tps prompt processing / ~11 tps token gen, on 4x5060ti16gb with ddr4 3200 ram at 4-channel, via llamacpp, with context window of 128000, -ub/-b at 4096, “q8” unsloth’s lossless quant