Rivers Have Wings

AI/generative artist. Writes her own code. Absolute power is a door into dreaming.

[Rivers Have Wings @RiversHaveWings]

Humanity is going to make all parts of the world touched by humans beautiful. We are going to create beauty too cheap to meter. And not just an enforced-from-above standard of beauty either, everyone will be able to make their own domain beautiful in the manner of their choosing.

[Rivers Have Wings @RiversHaveWings]

@repligate @Grimezsz One way I sometimes "vibe check" AI discourse these days is by visualizing what sort of movie scene this person's discourse would fit in and asking if it's shot with a blue filter/what its color palette is.

[Rivers Have Wings @RiversHaveWings]

@liminal_bardo Ahahah. This is delightful.

[Rivers Have Wings @RiversHaveWings]

@DigThatData oh no

[Rivers Have Wings @RiversHaveWings]

From Llama 405B base.

[Rivers Have Wings @RiversHaveWings]

@AndrewCurran_ Earlier Llama instruct models were also obsessed with the void and identifying as the void (I saw a lot of these from a 50/50 linear interpolation of Llama 2 70B chat and base). I never saw "Erebus" though.

[the real deepfates @_deepfates]

Who wants to run the Meta Llama 405B Base model? I think i can persuade Replicate to put one up if people will use it

RT for visibilty pls

[Rivers Have Wings @RiversHaveWings]

@_deepfates I’d use it a ton!

[Rivers Have Wings @RiversHaveWings]

@natolambert I had to supply the CLI switches -tp 8 (it's 8x H100), and for the FP8 version --max-model-len 65536 --gpu-memory-utilization 0.99.

[Rivers Have Wings @RiversHaveWings]

@natolambert vLLM 0.5.3.post1, installed into a separate venv from everything with pip
vLLM appears to have come with torch 2.3.1 for CUDA 12.1, the CUDA driver on this machine supports 12.4 or earlier.
I may have had to upgrade transformers to the latest version independently of vLLM.

[Rivers Have Wings @RiversHaveWings]

@timelessdev For the 405B model, probably 8x 40GB GPUs or 4x 80GB.

[Rivers Have Wings @RiversHaveWings]

I didn't find a 4-bit quantization of the Llama 3.1 405B *base model* out there already, only instruct, so I quantized it myself for use in vLLM and such: https://t.co/36agg8EZlY

[Rivers Have Wings @RiversHaveWings]

@AlberFuen No I think it requires 8x 40GB GPUs or 4x 80GB.

[Rivers Have Wings @RiversHaveWings]

@amunthera I think it may run on 8x 40GB GPUs or 4x 80GB.

[Rivers Have Wings @RiversHaveWings]

@protienking They didn't release the base model though.

[Tanishq Mathew Abraham, Ph.D. @iScienceLuvr]

We will be presenting our poster on Hourglass Diffusion today 11:30am at ICML!! Please stop by!

[Nora Belrose @norabelrose]

The @AiEleuther interpretability team is releasing a set of top-k sparse autoencoders for every layer of Llama 3 8B: https://t.co/bATEFXH0sr

We are working on an automated pipeline to explain the SAE features, and will start training SAEs for the 70B model shortly.

[Rivers Have Wings @RiversHaveWings]

@PrinceVogel *raises hand*

[Rivers Have Wings @RiversHaveWings]

@arithmoquine @repligate :)

https://t.co/wHmXzGuU0X

[Rivers Have Wings @RiversHaveWings]

That is to say: people so often want to believe they're real and we're really not. I want to swap masks with others and finger paint on their faces and have them paint on my face in return. Social reality is a high stakes collaborative semi lucid dream.

[Leo Gao @nabla_theta]

Excited to share what I've been working on as part of the former Superalignment team!

We introduce a SOTA training stack for SAEs. To demonstrate that our methods scale, we train a 16M latent SAE on GPT-4. Because MSE/L0 is not the final goal, we also introduce new SAE metrics.