Nigel

I collect experiences, make things, and learn along the way. Now making @chonkerkeys

[Nathan Barry @nathanrs]

Turns out you can make LLM inference fully deterministic across devices, with no loss to quality or speed.

This weekend at the @SpaceXAI hackathon I got Qwen3-0.6B to produce identical hashed logits from a 512-token generation across 2 GPUs and 3 CPUs: an A100, an H100, an Apple M5 Max, an AMD EPYC, and an Intel Xeon.

The main reason why inference isn't deterministic is because floating-point addition is not associative. (a + b) + c ≠ a + (b + c), since every add rounds. Accumulation order changes with the hardware used and kernel selected, so the same prompt can give you different outputs even at temperature 0.

Integers ARE associative. So why doesn't integer quantization already fix this? Because, while weights and activations get quantized, the non-linear ops (softmax, normalization, SiLU) dequantize back to float and requantize afterwards. Each of those steps hands you back to floating point rounding.

True integer-only inference does exist, but it's historically been motivated by edge hardware without FPUs, which doesn't make much sense for LLMs. One 2024 paper (I-LLM) did it on LLaMA from that angle and didn't get much attention. Nobody seems to have looked at it from the determinism side.

I wrote my own implementation, simplifying the approach from the paper, so that every operation between the input ids and the int32 logits is exact integer arithmetic. To test it I chain-hashed the logits at every step and ran that across the devices and configurations below. Every integer run gave the same hash: 64430dd985f8. Every fp16 run gave a different one, all diverging on the very first token.

WikiText2 perplexity came out to 20.72 vs 20.95 for fp16 (slightly better than the float baseline), and CUDA-graphed integer decode hits 106 tok/s at batch 1 on an A100, 3.6x the fp16 eager baseline.

Github repo is listed in the comments. Plan to do a writeup over this eventually!

[Kaleigh Moore @kaleighf]

If I could be onstage and talk about any topic right now, it would bethis:

Because of AI, time is compressing, and no one's really talking about it.

We are all expected to do so much more over the course of a single day than I think any previous period in history.

I don't know about you, but I find myself doing the amount of work in the span of a single day that even just 18 months ago, would have taken me two to three weeks to complete.

This is great and wonderful, and thanks to AI, I can work much more efficiently, without the bottlenecks that have created so much lag time for me in the past.

However, it raises the expectation bar, the mental load, and decision fatigue at an exponential rate.

I don't think we're compensating ourselves and each other for this dramatic shift in information processing, and as a result, we all just expect better quality work in less time for the same amount of pay.

Stepping away from my computer at the end of the day, most days, I am in a daze. I don't know that our brains were meant to do this much context switching and this much high-level direction-giving every single day.

And no, I don't think agents are the answer to this question.

I think this is a serious philosophical question we are all going to have to come to terms with, because as that expectation bar continues to raise, we need to find a way to advocate better for ourselves and our personal time.

AI is fantastic, and I love it. But I also find myself working 10x more now than I have over the last ten years, and I'm not sure that that is a net positive.

[Nigel @9gel]

@dinstheman @paulg Amen brother. A business is a proper business as long as it makes the owner money. The Silicon Valley bubble makes everything skewed. Not all businesses are fit for VC investments which always come with huge growth expectations.

[Nigel @9gel]

.@meetup_support CRTITCAL: your new location country selector DOES NOT HAVE Hong Kong and Macau. Are you exiting these two locales? Selecting "China" does not work. (curiously, you do have "Iran" which is sanctioned; a lot of websites have this problem. What library do you use?)

[Nigel @9gel]

@AnthonyCastrio Hong Kong Island, population nearly 1.2 million in over 30 SQ miles

[Nigel @9gel]

@AnthonyCastrio Look at Hong Kong and you pretty much have what you want.

[Nigel @9gel]

@emollick They are too busy one-upping the competitors to really imagine what that would be like. I’d ask science fiction authors and user experience designers.

[cayden 凯登 @caydengineer]

AugmentOS 1.0 has dropped.

AugmentOS is the open source OS and super app for smart glasses. It enables apps and AI agents for the next 8 billion personal devices.

And we're launching today, with hardware you can buy today. 1/🧵

[Nigel @9gel]

@alanwells Of course not. However, near term, that’s how things operate - salespeople go to other human-operated companies, cozy up with the people, get them warm and fuzzy to open their wallets. Especially big $ amounts. I wish it were otherwise & we have a long way to go.

[Nigel @9gel]

@paulg Not just Twitter but in real life too. We just see all of them here at once in concentrated form. At a dinner table the most correct and sensible answers are usually ignored because they are boring.

[Nigel @9gel]

@soychotic Oh and there’s Active Directory (AD), AD FS, AD Connect, Azure AD (now known as Entra ID), Microsoft Entra Connect, & whatever AD and Entra programs and services that I no doubt have missed.

[Nigel @9gel]

@soychotic All those program managers need their own pet project to get a promotion.

[Nigel @9gel]

@levelsio Its little brother ESP8266 can run a web server written in Lua on NodeMCU https://t.co/IvNpFFusqM . We use it to control the lights in our hackerspace https://t.co/8tGisX3avh

[molson 🧠⚙️ @Molson_Hart]

Paradigm shift is happening in China that is going to have powerful effects on the rest of the world.

China is killing the combustion car and is taking the oil and non-Chinese car manufacturing industry down with it.

From the Chinese government's perspective:

1. Shift to EVs and China can "leapfrog" non-Chinese combustion car manufacturing expertise
2. Car manufacturing is the industry most able to shift to military armament production during times of war (during WW2, for example, GM made airplanes and tanks)
3. China's achilles heels on the global stage are oil and food. It imports both. By transitioning to EVs, it greatly reduces its dependence on oil, meaning that if it wants to be a hegemon it doesn't need to worry as much about oil embargos.
4. Leverage subsidies for domestic demand for EVs to become an export powerhouse, moving up the value chain. If foreign countries insist on assembling the EVs in their countries, so be it; China will still capture most of the price by setting up foreign manufacturing operations and through subcomponent manufacturing.

Meanwhile, the United States and particularly Europe are deindustrializing.

Never mind chess, it feels like the Chinese are playing go while Americans, and particularly Europeans, are playing checkers.

[Nigel @9gel]

@Rainmaker1973 This is not Japanese nor egg thin egg (usuyaki tamago). It’s thin pancake (煎饼）from China, from the Shandong province.

[Nigel @9gel]

@naval North America maybe. You haven’t seen the streets of China.

[Nigel @9gel]

https://t.co/BLaPdmLWSu

[Nigel @9gel]

@emollick It’s the new “let’s Google this”. AI is definitely not going away.

[Nigel @9gel]

@emollick AI is changing the habits of 20-something’s. The first response a young engineer I work with when we encounter a problem is, “let’s ask OpenAI” instead of digging into the details and find root cause.