SFMB

Inventor | Global Labs MX/CA | Bizland Factory

[SFMB @smendozab]

@DAIEvolutionHub Send

[SFMB @smendozab]

@Riazi_Cafe_en Thank you, a great channel to follow, real value!

[Otros Diálogos de El Colegio de México @otros_dialogos]

La comunidad como rebelión. Un sílabo para sobrevivir en la academia siendo una mujer de color. Lorgia García Peña. Kianny N. Antigua (trad.).

#Reseña de Analoy Lafargue
https://t.co/YcZpPJvZbe

[SFMB @smendozab]

@newscientist What about the IP?

[JuliaHub @JuliaHub_Inc]

JuliaScript.jl makes running Julia scripts at max speed easier than ever! By automatically converting scripts into packages with optimal precompilation, it eliminates the hassle of manual #optimization. Explore how this powerful tool enhances your coding efficiency! https://t.co/UMRgGOaTJh #JuliaLang #Innovation #DataScience #juliacon

[SFMB @smendozab]

@kibancloud Buenas tardes, he tratado de comunicarme con uds, sería posible tener una llamada? Estamos en México, gracias @kibancloud !

[Darshan 🦖 @darshan]

This neuroscientist worked until she was 103.

She also:

• Won a Nobel Prize at 77
• Became a senator at 92
• Stayed mentally sharp into her 100s

Her secret? 5 daily habits that prevented brain aging: 🧵

[Miles Cranmer @MilesCranmer]

DynamicQuantities.jl has hit version 1.0!

What started as a dimensional analysis tool for PySR is now a mature physical units package for the Julia community. DQ emphasises type stability and can accelerate both compilation and runtime performance.

Thanks to all contributors!

[SFMB @smendozab]

A fantastic resource, we really appreciate it @SimonPrinceAI!

[Simon Prince @SimonPrinceAI]

FYI Understanding deep learning is currently on sale, reduced by 18% at Amazon US: https://t.co/EClBeXQOld
and by 31% at Amazon UK:
https://t.co/wcRomqdHQy

Bargain! Of course, the PDF is and will remain forever public at https://t.co/hqRA1xUPkk as well.

[Andrej Karpathy @karpathy]

To help explain the weirdness of LLM Tokenization I thought it could be amusing to translate every token to a unique emoji. This is a lot closer to truth - each token is basically its own little hieroglyph and the LLM has to learn (from scratch) what it all means based on training data statistics.

So have some empathy the next time you ask an LLM how many letters 'r' there are in the word 'strawberry', because your question looks like this:
👩🏿‍❤️‍💋‍👨🏻🧔🏼🤾🏻‍♀️🙍‍♀️🧑‍🦼‍➡️🧑🏾‍🦼‍➡️🤙🏻✌🏿🈴🧙🏽‍♀️📏🙍‍♀️🧑‍🦽🧎‍♀🍏💂

Play with it here :)
https://t.co/pFQGZIAW1k

[Andrej Karpathy @karpathy]

In 2019, OpenAI announced GPT-2 with this post:
https://t.co/jjP8IXmu8D

Today (~5 years later) you can train your own for ~$672, running on one 8XH100 GPU node for 24 hours. Our latest llm.c post gives the walkthrough in some detail:
https://t.co/XjLWE2P0Hp

Incredibly, the costs have come down dramatically over the last 5 years due to improvements in compute hardware (H100 GPUs), software (CUDA, cuBLAS, cuDNN, FlashAttention) and data quality (e.g. the FineWeb-Edu dataset). For this exercise, the algorithm was kept fixed and follows the GPT-2/3 papers.

Because llm.c is a direct implementation of GPT training in C/CUDA, the requirements are minimal - there is no need for conda environments, Python interpreters, pip installs, etc. You spin up a cloud GPU node (e.g. on Lambda), optionally install NVIDIA cuDNN, NCCL/MPI, download the .bin data shards, compile and run, and you're stepping in minutes. You then wait 24 hours and enjoy samples about English-speaking Unicorns in the Andes.

For me, this is a very nice checkpoint to get to because the entire llm.c project started with me thinking about reproducing GPT-2 for an educational video, getting stuck with some PyTorch things, then rage quitting to just write the whole thing from scratch in C/CUDA. That set me on a longer journey than I anticipated, but it was quite fun, I learned more CUDA, I made friends along the way, and llm.c is really nice now. It's ~5,000 lines of code, it compiles and steps very fast so there is very little waiting around, it has constant memory footprint, it trains in mixed precision, distributed across multi-node with NNCL, it is bitwise deterministic, and hovers around ~50% MFU. So it's quite cute.

llm.c couldn't have gotten here without a great group of devs who assembled from the internet, and helped get things to this point, especially ademeure, ngc92, @gordic_aleksa, and rosslwheeler. And thank you to @LambdaAPI for the GPU cycles support.

There's still a lot of work left to do. I'm still not 100% happy with the current runs - the evals should be better, the training should be more stable especially at larger model sizes for longer runs. There's a lot of interesting new directions too: fp8 (imminent!), inference, finetuning, multimodal (VQVAE etc.), more modern architectures (Llama/Gemma). The goal of llm.c remains to have a simple, minimal, clean training stack for a full-featured LLM agent, in direct C/CUDA, and companion educational materials to bring many people up to speed in this awesome field.

Eye candy: my much longer 400B token GPT-2 run (up from 33B tokens), which went great until 330B (reaching 61% HellaSwag, way above GPT-2 and GPT-3 of this size) and then exploded shortly after this plot, which I am looking into now :)

[MIT CSAIL @MIT_CSAIL]

This repository on ML operations has free talks, books, papers and more: https://t.co/wxiXkGWyq3

Developed by @visenger

[Steven Brunton @eigensteve]

FREE PDF for 2ND EDITION OF OUR BOOK!!!
https://t.co/t4kYJmgL61

We are committed to open science and open education:
Free book: https://t.co/t4kYJmgL61
Free code (https://t.co/QZFPbAa8ab)
Free lectures (https://t.co/2q9MFD6o4u)

https://t.co/53C3gdsJg6

[SFMB @smendozab]

@karpathy Agreed...

[Ray Dalio @RayDalio]

The organization, like the individual, has to push through to results in order to succeed—this is step five in the 5-Step Process.

While recently cleaning up a huge pile of work products from the 1980s and 1990s, I came across boxes and boxes full of research. There were thousands of pages, most covered with my scribbles, and I realized that they represented just a fraction of the effort I'd put in. At our fortieth-year celebration I was given copies of the almost ten thousand Bridgewater Daily Observations that we'd published. Every one of them expressed our deepest thinking and research about markets and economies. I also stumbled across the manuscript of an eight-hundred-page book that I wrote but then got too busy to publish, and countless other memos and letters to clients, research reports, and versions of the book you're reading now. Why did I do all these things? Why do others work so hard to achieve their goals?

From what I can see, we do it for different reasons. For me, the main reason is that I can visualize the results of pushing through so intensely that I experience the thrill of success even while I'm still struggling to achieve it. Similarly, I can visualize the tragic results of not pushing through. I am also motivated by a sense of responsibility; I have a hard time letting people I care about down. But that's just what's true for me. Others describe their motivation as attachment to the community and its mission. Some do it for approval and some do it for financial rewards. All these are perfectly acceptable motivations and should be used and harmonized in a way consistent with the culture.

The way one brings people together to do this is key. This is what most people call "leadership." What are the most important things that a leader needs to do in order to get their organizations to push through to results? Most importantly, they must recruit individuals who are willing to do the work that success requires. While there might be more glamour in coming up with the brilliant new ideas, most of success comes from doing the mundane and often distasteful stuff, like identifying and dealing with problems and pushing hard over a long time. This was certainly the case with the Client Service Department. Through a lot of relentless hard work in the years since the original problem turned up, the department has become an example to other teams at Bridgewater—and our client satisfaction levels remain consistently high. The great irony of all this is that none of our clients ever even noticed the problems we saw with the memos. Sending out work not up to our standards was bad—and I'm glad it was corrected. But it could've been much worse, tarnishing our reputation for delivering pervasive excellence. Once that happens, it becomes much harder to restore trust. #principleoftheday

[MIT OpenCourseWare @MITOCW]

Gil Strang. Free calculus textbook. Updated. Enough said. 🎤 https://t.co/SepNzY9ytw

[SFMB @smendozab]

Agreed...

[freeCodeCamp.org @freeCodeCamp]

Writing code used to be a distinctly human endeavor.

But with AI and code generation, the process of development is changing rapidly.

In this tutorial @SonyaMoisset goes over how to use AI-generated code safely while maintaining your skills as a dev.

https://t.co/UtEhI3355K

[freeCodeCamp.org @freeCodeCamp]

Writing code used to be a distinctly human endeavor.

But with AI and code generation, the process of development is changing rapidly.

In this tutorial @SonyaMoisset goes over how to use AI-generated code safely while maintaining your skills as a dev.

https://t.co/UtEhI3355K

[Viral B. Shah @Viral_B_Shah]

Thanks to @maleadt, we have CUDA.jl 5.0 for #julialang with an integrated profiler, which works in the same way as the regular Julia profiler. Also supports @nvidia CUDA 12.2, and uses CUDA to enforce memory limits. https://t.co/mLhIK3vFf0