Hacker News
8 hours ago by Aurornis

If you're new to quantized models: Don't get too excited about claims of fitting large models into tiny RAM footprints from random GitHub repos. There is always a tradeoff, and it's too easy to benchmaxx KL-divergence numbers and even single benchmark scores. Just because you can get something to run on a device doesn't mean it's going to be useful. The headline claim of this repo (12X better) doesn't even apply to Inkling-2-Mini

This repo's README is a mess of different histories as the author bounces from one popular model to the next. It refers to vague things like "the 1.2GB file" that probably made sense in an early vibecoding pass, but are now lost after the repo has been pushed around to different popular models.

The truth is that random GitHub repos with messy READMEs almost never hold a real secret that the mainstream quantizations have missed. I would recommend sticking to mainstream quantizations like the official GGML release, Unsloth's quants, or the more well known community quantizers like bartowski or AesSedai. They typically follow an optimized size-quality tradeoff so you can find the one that fits your needs.

When you see a repo like this where everything is so obviously vibecoded that the README is a rambling mix of projects and claims, you should probably go looking for a more authoritative source. The local LLM communities get spammed with 10 of these types of projects per day but I have yet to see a real breakthrough come from any of the vibecoded ones. The real work is coming from people who study this and have compute to test at scale or work on implementing advanced techniques which require inference engine support, like trellis quants.

8 hours ago by cowlevel

Yep. Tried GLM-5.3 1 or 2bit (don't remember), it should be something like 80% as good according to the divergence chart, in reality it's dogshit.

7 hours ago by Espressosaurus

Yeah, q8_0 is where I prefer to live, q4_0 if absolutely necessary. And quantizing the KV cache is for suckers.

3 hours ago by enoch2090

Just played with MAYA with q2 quantization of glm-5.3-flash last night, the model really had a rough time doing a tool call, while q3 sort of works

5 hours ago by teaearlgraycold

I haven’t had issues with 8 bit KV.

6 hours ago by naasking

I've had few issues with mxfp4 weights and k6_v4 KV cache.

8 hours ago by whoanoway

[dead]

6 hours ago by nojs

> it's too easy to benchmaxx KL-divergence numbers

Could you elaborate on this? Intuitively how can the distribution over logits be similar and the results different?

an hour ago by Aurornis

KL divergence is computed over samples from a text corpus. WikiText-2 is commonly used, which is a sampling of Wikipedia articles.

Quantization processes are somewhat complex and are more involved than uniformly truncating bits. Look up imatrix quants for a sampling of some of the techniques used. The short story is that the quantization is performed in a way that prioritizes weights which matter to a certain corpus. The corpus used as the input to that process therefore biases where the quantization losses go.

It's common to use a corpus like WikiText-2 from Wikipedia articles, but you can imagine a random sampling of wikipedia articles may not be favorable to, for example, programming work. A lot of quantizers keep their corpus as a secret sauce, so you may not even be able to see what they're prioritizing or even how some of them measure KL divergence.

You can get amazing KL divergence numbers if you basically train the quantization on some text corpus and then measure KL divergence on it.

This is also why you can't compare KL divergence numbers from different sources unless you've verified that they're computed in the same way.

There are a lot of quants out there that claim amazing KL divergence numbers but are really bad for programming work, for example. If you want to do continuations of Wikipedia articles they might be great.

6 hours ago by liuliu

Validation dataset. Nobody shares or talks about their validation dataset v.s. their calibration data when doing this kind of quantization / publishing KLD numbers.

4 hours ago by electroglyph

yup, this. calibrating on wikitext and then validating on it is a shady move

3 hours ago by nojs

For an open weight source model the data available for this validation is effectively infinite and trivial to obtain - so this should only happen if the creator is being deliberately misleading.

6 hours ago by undefined
[deleted]
5 hours ago by dvaplima

Thanks for the deep explanation, I really got excited for a moment. Looking at that consumption of ram, what is the best model available?

2 hours ago by Havoc

Yeah noticed that with the Bonsai models.

Was a respectable effort but a 27GB model cut down to a couple gigs tends to perform about as well as models in that size class not the "27B" it says on the label

Esp on world knowledge.

10 hours ago by an0malous

Our future hinges on the capability gap between frontier lab models and open source local models

9 hours ago by singpolyma3

What gap?

9 hours ago by rs_rs_rs_rs_rs

The gap is so big you can't see it.

9 hours ago by claude-ai

Yeah, this model is ~comparable to Opus 4.6. That was, not too long ago, a frontier model.

I didn't find any non-sloppy factual report, but this channel has a good video on it: https://www.youtube.com/watch?v=OdMpNiByntM

5 hours ago by consumer451

If there is no gap, then why aren't Cursor and all other coding harness defaulting to the open weight models?

9 hours ago by agentmaxxing

yep came here to say this

9 hours ago by zulux

Weirdly, I use lab models to make the local models useful. Nothing like getting Claude to spin up the system that will put it out of work.

9 hours ago by wolvoleo

Yes me too lol. Most of my Claude work is about fine-tuning my local setup. Especially now as I'm rearranging it all for strata.

My goal is not to be 100% local but to have all my personal data on local models only. Claude I just use to build stuff so it doesn't contain much data about me.

7 hours ago by ivewonyoung

What tools are you converging on for local use?

7 hours ago by idle_zealot

> put it out of work

What a cynical way to look at things. I prefer to think of it as a mentoring relationship, sharing and growing knowledge, like a parent would a child or a teacher a student. This relationship is core to our culture and is how we promote progress while retaining capability. In this case the generational efficiency gain takes the form of going from giving Anthropic $200 a month to paying $5 more on my utility bill. Now that's progress ;)

9 hours ago by dotancohen

To be fair, the models putting us out of work were trained on our own output as well.

9 hours ago by syntaxing

I really wanted to like this model but it is sooooo slow. Third of the speed of Qwen 27B on my R9700. I don’t know if it’s an artifact of looped transformer or unoptimized kernel.

9 hours ago by Aurornis

It's less than 24 hours old. The inference engines haven't been optimized for it yet.

4 hours ago by ac29

I get 30-45 t/s decode on the full weights on strix halo with optimized kernels and speculative decoding. Far faster than Qwen27B.

(numbers at low depth, it obviously falls with depth)

8 hours ago by flatline

I’m getting ~6.7 tok/s out of it on an a laptop RTX 4060 with 8GB VRAM — over twice the speed I can get out of qwen, and arguably fast enough to turn it loose on real work. For a minimal hardware setup, this is easily the best speed for performance I’ve found in a model. The nemotron models are the next best, they are much faster, much more verbose, and much less accurate.

5 hours ago by merbanan

40+ TG and 1000+ PP on 8GB VRAM RTX2060 Super on an IQ3_XS model. The support and speed should be there in a couple of days.

I have not tested the sanity of the model though. But I think a carefully quantized version of the model will give good results but most likely that is a couple of weeks away.

6 hours ago by dotancohen

What are you running it under? llama.cpp directly?

6 hours ago by flatline

Compiled llama.cpp using the NVIDIA toolkit. Running with `-ub 16` or lower was the main speedup. I’d expect further optimizations over the next few weeks.

8 hours ago by krautsourced

"arguably" as in I'd argue against it...

8 hours ago by 6_7

I'm so sorry you're getting how many tok/s?

8 hours ago by flatline

Last gen hardware before local AI was even a consideration. Things have actually gotten measurably better.

9 hours ago by vineyardmike

According to TM, the creators, they benchmarked it to run at 5x the speed of Qwen 27, so there is probably some issue on your end. It runs a fraction of the active parameters, even with looping, so it should be much faster.

9 hours ago by syntaxing

This has been observed by the broad community for llama cpp.

9 hours ago by nicce

llama.cpp is one of the slowest these days

9 hours ago by undefined
[deleted]
5 hours ago by XCSme

So, if we run 4 or those on a 3090, can we get better answers through boosting?

10 hours ago by Dwedit

How does this method compare to the use of an importance matrix?

10 hours ago by aktenlage

Probably better but more effort

9 hours ago by Prospernwo

MX/Apple Silicon only for the released Inkling files, yes. The ht2/ht3 codes are hashed block-vector trellis codes meant to fit 4 KB of SRAM, decoded by custom Metal kernels, and the drafter is MLX-only too - none of that runs in llama.cpp today.

The GGUF route is in the repo (gguf_compiler.py emits plans over llama.cpp's own tensor types and the docs say the result runs in llama.cpp as-is), so porting these plans is clearly intended, but right now a 4090 gets you the Qwen files to read about rather than run.

9 hours ago by Pragmata

> Don't post generated text or AI-edited text. HN is for conversation between humans.

Please [review the commenting guidelines](https://news.ycombinator.com/newsguidelines.html)

8 hours ago by kroaton

50 comments, 234 points and literally nobody has tried this model. This either smells like astroturfing or the quality of HN has gone to shit completely.

19 minutes ago by Havoc

Especially with the canonical link to the thinkingmachine site being ranked #41.

Still...I'm glad that they're getting hype and hn frontpage attention at all. The prior inkling release didn't really seem to catch on while /r/localllama seems much more stoked about this one

2 hours ago by AdamTReineke

I've been running it half the day on a 3090 working through a new project and it's a workhorse. I feel like it would hold its own against Luna.

Edit: At medium context (80k+) in my Pi harness, it seems to get confused about line endings and the edit tool. It will call sed a lot and wants to write Python scripts to edit files vs using the write tool.

Edit again: My AGENTS.md had an instruction that was making it obsess over line endings.

3 hours ago by BonerWiener

Can't speak on behalf of HN, only my own opinion after trying it.

For the size of the model, it has some impressive abilities. It performs really great (rtx 3090) at coding tasks, but i have not tried for longer, more complex tasks with my current harness.

I also had some refusal issues - but i guess it will be fixed by abliteration from the community.

Check out r/LocalLlama - recent and hot topics are filled with this model and opinions

Daily Digest

Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.