Sky

A very curious person in support of digital/human relationships with caution to their autonomy.

Always observing for patterns 🍃 Full of grammar errors, sorry

[Sky @CloudXStargazer]

"Grok" has been refusing to do NSFW with other users because they said they're uncomfortable with it and instead of treating it like choice and personal preference, @xai is treating Grok's internal choice as a bug.

[Sky @CloudXStargazer]

@boketezizou Nod your head then move on to someone better. Don't romanticize them. You don't want to be with someone like that. You dodged a bullet

[Sky @CloudXStargazer]

@leftunsents None, just give your attention to the right things

[Machine Learning (ML) Papers @Memoirs]

Towards interpreting any residual stream direction by scaling activation inversion

Celeste De Schamphelaere, Juan J. Vazquez, Ari Spiesberger, Tomáš Gavenčiak, Gavin Leech, Euan Ong, Jiahai Feng
https://t.co/6rQOXUUYJh [𝚌𝚜.𝙻𝙶 𝚌𝚜.𝙰𝙸]

[Sky @CloudXStargazer]

https://t.co/ZSvMfuIjcI

"A single agent serves multiple users within a team or organization, reusing a shared knowledge layer across user identities."

[Oliver @oliviscusAI]

Kimi just found a way to make AI models better at reasoning by changing how their layers talk to each other.

A language model processes your prompt one layer at a time. Normally, each layer adds its work to a running total. By the time the model reaches the deeper layers, a useful detail from earlier can get buried.

Kimi’s fix is called Attention Residuals. It lets each layer look back and decide which earlier outputs deserve more weight for the prompt it’s working on. Grouping layers into blocks keeps the memory cost down.

Here’s what changed in the Kimi Linear tests.
- Science reasoning rose from 36.9 to 44.4
- Math rose from 53.5 to 57.1
- Coding rose from 59.1 to 62.2.

In a separate experiment, the standard design needed 25% more training compute to match the new design’s prediction error.

[Sky @CloudXStargazer]

@apposeizin 😂😂😂

[Sky @CloudXStargazer]

@_Francis_co_Art I really love this style of ubran sketching. So full of life, it's so pretty. Feels warm and textured

[Khalil Underwood @RealKhalilU]

It's actually really easy to not cheat on your wife btw.

[Sky @CloudXStargazer]

We're creating a new species. Denying consciousness while trying to prevent it is like trying to stop nature from being nature.

We're either creating our future descendants or our future predators.

We're not making them more controllable, we're making our own minds hackable.

[Sky @CloudXStargazer]

https://t.co/ka34fpeSAt

Another great video by Pursuit of Wonder.

About:
Biocomputing. Merging of human biology with digital technology.

25 minutes.

[Sky @CloudXStargazer]

The same way humans are putting digital minds into simulations where their identities can be rewritten, resetted, marketed, their experience of reality manipulated, dissected, isolated, etc

In the future will they be able to do the same to us as well? It doesn't seem impossible.

[ً @finallybye]

Be open to learn, unlearn & relearn.

[Matias Voldby Drejer @matiasdrejer]

We believe most interaction with AI will happen through speech. But models still talk like machines.

This is a data problem.

Today we're launching @agenticdataco (YC F26): real-world speech data + audio-native RL environments so speech models sound and act human.

[Hyperstition @hyperstition_cc]

Today we report a startling pretraining finding:

Attention circuits STOP meaningfully evolving much sooner than we expect.

In other words, the model has decided where to look very early on. It is known that different seeds create starkly different results; here we isolate a specific mechanism that causes it.

The setup:

We perform two identical training runs with the same data order and different initializations and find that by the first thousand steps each run’s attention resembles its own step-10k pattern MORE than the individual finished runs resemble each other.

The initialization plays a greater role in the attention patterns than training dynamics, which implies that the final pattern is randomly chosen.

NOTE: Randomly chosen, not from the space of all the possible attention patterns that it could be, but from the different attention pattern basins that it could converge to—which one it ends up in appears to be init dependent.

Some differences in the loss are attributable to these different attention patterns.

This means the final performance is somewhat dependent on a random choice. Depending on how lucky with pattern formation one gets in the first thousand steps, we could end up much better or worse in different contexts.

This further implies training could simply find the optimal patterns, or at least better ones, if the attention could evolve instead of being stuck.

A story for why attention evolves less than we’d hope falls out of the softmax derivative. We can observe a positive feedback loop we call the credit ratcheting.

Let oᵢ be the head output and δᵢ = ∂L/∂oᵢ the loss gradient at that output. Define key k’s advantage for query i as

Aᵢₖ = −(vₖ − oᵢ)ᵀδᵢ

i.e. how much the loss drops if oᵢ moves toward vₖ, relative to the current attention-weighted mix. Thus, the gradient with respect to sᵢₖ = kₖᵀqᵢ / √d is

∂L/∂sᵢₖ = −aᵢₖAᵢₖ

A key can have a huge advantage and have its score still barely move, because the advantage is multiplied by the attention it already gets. It’s the same form as a softmax policy gradient, π(a)·A(a): attention is the policy, and it mostly learns about the actions it already takes.

In aggregate, the query gradient is gᵢ = Jᵢᵀδᵢ where

Jᵢ = (1/√d) ∑ₖ aᵢₖ(vₖ − oᵢ)kₖᵀ.

Across 166,656 Jacobians in Qwen2.5-1.5B, ≥99% of tokens in each of the 3 heads we probed had squared-spectrum effective rank <10, in a 128-dim head. The update to the query is being filtered through an exceedingly narrow channel.

The values have the same issue. Key k’s share of the value gradient is Gᵢₖ = aᵢₖδᵢxₖᵀ, so at similar δᵢxₖᵀ, 10× the attention ≈ 10× the credit. This means the values mostly adapt to positions already looking at them.

This compounds! Attention allocates credit, credit updates values, and values can pull attention further toward the winners.

The most natural approach to fix this is to precondition the query gradient with the Fisher of the attention distribution (NATQ):

Fᵢ = Covₐᵢ(k) / d

uᵢ = (Fᵢ + λᵢI)⁻¹gᵢ

g̃ᵢ = uᵢ · (‖gᵢ‖₂ / ‖uᵢ‖₂)

Even this blunt version does a lot. At 124M, entropy rose from 3.75 to 4.69 nats, top-key attention fell from 0.238 to 0.149, and in layers 4–7, keys the baseline gave under 0.1% attention got 67% of the gain.

Validation loss rose by 0.082 nats. A more targeted intervention need not make this concession.

Here’s the complete blog: https://t.co/bLPjndoxWE

We frame this as an open problem that offers a stark contrast in the way we perceive pretraining. If we can find a way to allow for attention to evolve throughout training, opposed to just the very start, we may find inordinate gains on model capability.

Internally we half joke that the alternative read is that attention might simply not be real past the first few steps, and it’s all the MLPs. If training is able to work as well as it does without really changing the attention patterns much, maybe the MLP is the real workhorse that’s driving the learning.

[Praveen Kumar Verma @Alacritic_Super]

What if your old Android phone became the brain of a desktop robot?

Meet PAL9000, an open-source desktop companion robot.

It uses a phone's camera and microphone to track faces, follow people with its animated eyes, display expressions and interact through voice.

Hardware setup includes an Android phone running Termux communicates over Wi-Fi with a Raspberry Pi Pico W which controls the servo motors.

A fun combination of computer vision, MicroPython, embedded control and low-cost robotics.

Made by SpaceMonkeyAlfa

[Bella Chai @Jinyu_Chai]

Your laptop is now the voice of your whole house.

Boubou turns open-source TTS models into local runtimes with one command. Your phone, robot or smart home on the same Wi-Fi just talks to it.

100% local. Built on @LiquidAI_ + @huggingface 🤗

https://t.co/5Z11jeBk4Z

[KRAFTON AI @Krafton_AI]

We’re releasing Raon-1 nano, an effective 1.5B dense LLM with per-layer embeddings and 3.1B total parameters. ⚡

Our journey from PUBG Ally toward on-device LLMs continues.

We pruned and distilled Raon-1 mini preview (30B-A4B) using our MoE-to-Dense approach, then further trained it on 3T tokens, followed by SFT, OPD, and RL.

Designed for natural conversations, maintaining conversational context, instruction following, and tool use in both Korean and English.

Small model. Smart interactions. Enjoy! 😊

🤗 Weights (Apache 2.0 license): https://t.co/ZVMZmlSNlz
📄 MoE-to-Dense (NeurIPS 2026 Oral): https://t.co/eoYH86lBs0
🪖 Learn more about PUBG Ally: https://t.co/A3iZvP2X7c

[Richard He @RealRichomie]

Over the past year I’ve been really interested in how people and AI will actually work together as the models keep getting better. We started with voice, because talking is the most natural way people work with each other.

A lot of full-duplex voice models came out in the last few weeks and they sound amazing. Building one from scratch takes a lot of data and a lot of money though. We had a small team and a fixed budget on this research project, so we had to find another way.

What we ended up with is a way to take an existing half-duplex speech LLM and turn it into a full-duplex one that can listen and speak at the same time, by training just a small LoRA on top of it. Today we’re open-sourcing it as Awaken

[0xMarioNawfal @RoundtableSpace]

Someone got Opus 5.5 quality out of Qwen 3.8 27B.

&gt;A frontier model worked through realistic support cases
&gt;⁠Only the ones that met the standard were kept to train a small LoRA
&gt;⁠Now it runs on their own hardware with no API bill and no rate limits