Honored to be featured on @techiestory. Thank you for an insightful interview and sharing my experiences at major Tech company and transition to an ML startup.
Stories available:
#74 in English/Vietnamese https://t.co/BcEIqgwArw
#75 in English/Vietnamese https://t.co/PlimBXKQhg
Quora's new mission is to grow the world's collective intelligence. This company-wide mission encompasses both the Quora product and Poe. More below on what this means to us:
I’m #hiring for a Senior/Staff Software Engineer role on Machine Learning Platform team at Quora.
Details:
- Fully remote (anywhere in US and Canada)
- Exciting ML infrastructure work
- Interested candidates can DM me for more details
Link: https://t.co/4QL6ASKdGy
New: Poe Apps. Vibe code new visual interfaces for any combination of the 100+ models on Poe, and use others’ creations. You can make workflows, media editors, quizzes, and almost anything else. See thread in the linked post for details and much more to come in this direction!
Introducing Poe Apps: a new, easy way to create and use visual interfaces into any combination of the 100+ text, image, video, and audio models on Poe. (1/5)
🔥llm.c update: Our single file of 2,000 ~clean lines of C/CUDA code now trains GPT-2 (124M) on GPU at speeds ~matching PyTorch (fp32, no flash attention)
https://t.co/39G8UWusVp
On my A100 I'm seeing 78ms/iter for llm.c and 80ms/iter for PyTorch. Keeping in mind this is fp32, with no flash attention yet, and slightly stale PyTorch (2.1.0).
- It is a direct implementation of the training loop and backpropagation in C/CUDA.
- It compiles and runs instantly. No more "hit run then wait for tens of seconds for unknown reasons", for mountains of inscrutable abstractions to build a Universe.
- It deletes the need for the Python interpreter and a deep learning library.
- It allocates all the memory a single time at the start.
- It's pretty cool.
How:
Getting this to work required us to write a lot of custom CUDA kernels, and doing this manually (instead of using Tensor ops of aten/PyTorch and torch.compile etc.) is a bit like programming in assembly. And you spend quality time looking at more assembly (CUDA PTX/SASS). But this also means we get to hyperoptimize the code and possibly explore optimizations that torch.compile might find difficult to, which is awesome. Examples of optimizations that went in over the last few days:
- we're being clever with our memory consumption in the backward pass, only using a few buffers we need to propagate the gradients, saving memory capacity.
- one fused classifier kernel does the last layer forward pass, the loss, and kicks off the backward pass.
- many improvements to all the kernels involved, including e.g. gains from carefully constraining execution within the autoregressive mask in attention
- cuBLAS(Lt) calls for all heavy lifting matmuls, and fused bias accumulation
Big credits to two CUDA experts who appeared from somewhere on the internet to help this open source project, ngc92 and ademeure. We're hanging out of Github and Discords of CUDAMODE and my NN Zero to Hero.
Next steps:
- more optimizing of our (fp32) kernels, and especially switch to flash attention.
- mixed precision training (fp16 to start).
- multi-gpu training (DDP to start).
- data & evals to set up a proper GPT-2 training runs
- 🚀 repro GPT-2 (1.6B) training run.
- more modern architectures etc. (Llama 3?)
- writing, videos, exercises on building all of this from scratch.
Figure 1: eye candy: timing profile of the kernels (one layer). NVIDIA cutlass kernels with solid compute throughput taking up a lot of the running time => nice.
Useful Claude 3 trick to help you visualize code better.
Paste some code in, and ask it to make a flowchart.
Then, paste the flowchart code into a Mermaid viewer, and you'll get a nice, understandable visualization of your code!
Today we are excited to introduce a new bot on Poe: Web Search! This general-purpose bot is capable of conducting web searches as necessary to inform its responses, making it particularly good for queries requiring the latest information or specific facts. (1/5)
New: bot search! We are very excited to add the ability to search for bots on Poe on iOS, web, and MacOS today! You can search for specific bots or discover new ones tailored to your interests. This feature is coming soon on Android. (1/2)
We are excited to launch @Neeva's NeevaAI bot on Poe today. NeevaAI cites sources and includes up to date information in its replies. It is a much easier way to get a synthesized answer from an internet search.
Built an AI chatbot using @poe_platform yesterday.
It’s an Italian chef (Leonardo) that helps average cooks make above average tasting meals.
The bot gives users ingredients, recipes, calories and “beginner cooking tips”
Free to use: https://t.co/wbtNIQj19n
If you're looking for a productivity booster as a software engineer, look no further than Poe by #quora. As someone who has incorporated Poe into my daily workflow, I can attest to its significant impact on my coding-heavy day-to-day job.
https://t.co/W0VvTeznE9
When a famous Computer Architect ventures into YouTube and Only 42 People Subscribe. Please smash that like, subscribe button, and ring that bell notification so he can keep making more content about Low-level system! 🌟💻 🔨
https://t.co/VNhADtwtWi
#SystemsEngineer
Today we are launching Poe subscriptions, which will provide paying users with access to bots based on two powerful new language models: GPT-4 from OpenAI and Claude+ from Anthropic.
We are excited to launch Poe for desktop today at https://t.co/cMuZ6Cb3cY. With this launch, between our existing iOS app this new web interface, we believe Poe is now the fastest and easiest way to talk to AI across iOS and all desktop platforms.
Today, we are opening up public access to a new AI product we have been building called Poe. Poe lets people ask questions, get instant answers, and have back-and-forth conversations with several AI-powered bots. (1/n)