Jakub Narloch

[Jakub Narloch @jmnarloch]

@thewitcher @rohans62 Why?

[Jakub Narloch @jmnarloch]

Hey #Vancouver, you can issue an emergency alert about crime, but you can not issue an alert about upcoming earthquakes?

[Jakub Narloch @jmnarloch]

@pinskinator Wait until it will become a Senior.

[Jakub Narloch @jmnarloch]

@nickste

[Yann LeCun @ylecun]

I don't wanna say "I told you so", but I told you so.

Quote: "Ilya Sutskever, co-founder of AI labs Safe Superintelligence (SSI) and OpenAI, told Reuters recently that results from scaling up pre-training - the phase of training an AI model that uses a vast amount of unlabeled data to understand language patterns and structures - have plateaued." ...

https://t.co/wPzUOQ1Pb3

[Marktechpost AI @Marktechpost]

CodeMaker AI Breakthrough in Software Development: Achieves 91% Accuracy in Recreating 90,000 Lines of Code, Setting a New Benchmark for AI-driven code Generation and Fine-Tuned Model

Read the full article here: https://t.co/8uNEsf7eWi

@CodeMakerAI

[Genevieve Roch-Decter, CFA @GRDecter]

🚨Microsoft accounts for 19% of Nvidia’s revenue.

According to UBS.

This doesn't end well.

[Ronald van Loon @Ronald_vanLoon]

Automate #Coding

@CodeMakerAI, thanks for sharing

Don't miss out on the latest AI advancements! Sign up here to stay informed! https://t.co/UVx9GoUycR

#AI #Engineering #Automation #Productivity #EmployeeExperience #CustomerExperience

Cc: @jmnarloch

[Yann LeCun @ylecun]

Sometimes, the obvious must be studied so it can be asserted with full confidence:
- LLMs can not answer questions whose answers are not in their training set in some form,
- they can not solve problems they haven't been trained on,
- they can not acquire new skills our knowledge without lots of human help,
- they can not invent new things.
Now, LLMs are merely a subset of AI techniques.
Merely scaling up LLMs will *not* lead systems with these capabilities.
There is little doubt AI systems will have these capabilities in the future.
But until we have small prototypes of that, or at least some vague blueprint, bloviating about AI existential risk is like debating the sex of angels (or, as I've pointed out before, worrying about turbojet safety in 1920).

https://t.co/Qkyqo4WhO0

[Jakub Narloch @jmnarloch]

@russelljkaplan You are not wrong https://t.co/ukLpCqJQOD

[CodeMaker AI @CodeMakerAI]

CodeMaker AI Recreates 90,000 Lines of Code with 91% similarity

https://t.co/VBXJmJd1jY

[CodeMaker AI @CodeMakerAI]

Vancouver, BC. 1st August 2024.

Today, CodeMaker AI announces that it successfully recreated the implementation code of the 90K codebase using a fine-tuned AI model. The generated code reaches 91% to 92% similarity when compared with the reference implementation. The experiment was conducted on a popular Java library, Spring Boot. CodeMaker AI system was able to process 3,251 files and generate 90,063 lines of code autonomously in 1 hour and 42 minutes at a cost of $265.73. CodeMaker AI's unique ability to process source code at scale allowed us to demonstrate that it is possible to build a system that can combine autonomous source code processing that reaches a high level of correctness of the generated code. We also showcase the ability to fine-tune the model that can be used in various applications, including code completion, AI assistant, and fully autonomous code generation. We believe that this will open new possibilities for applying AI in software development at a large scale.

Reference:
Article: https://t.co/PslJY6Ukoe
Generated artifact: https://t.co/2W3xgw38cQ

[CodeMaker AI @CodeMakerAI]

CodeMaker AI adds multi-language support. When interacting with the Coding Assistant on the web or through the IDE, the assistant will prioritize using your native language as long as it is supported.

Learn more at: https://t.co/UtwhL0J1Yj

#ai #software #developer #productivity #automation

[Andrej Karpathy @karpathy]

In 2019, OpenAI announced GPT-2 with this post:
https://t.co/jjP8IXmu8D

Today (~5 years later) you can train your own for ~$672, running on one 8XH100 GPU node for 24 hours. Our latest llm.c post gives the walkthrough in some detail:
https://t.co/XjLWE2P0Hp

Incredibly, the costs have come down dramatically over the last 5 years due to improvements in compute hardware (H100 GPUs), software (CUDA, cuBLAS, cuDNN, FlashAttention) and data quality (e.g. the FineWeb-Edu dataset). For this exercise, the algorithm was kept fixed and follows the GPT-2/3 papers.

Because llm.c is a direct implementation of GPT training in C/CUDA, the requirements are minimal - there is no need for conda environments, Python interpreters, pip installs, etc. You spin up a cloud GPU node (e.g. on Lambda), optionally install NVIDIA cuDNN, NCCL/MPI, download the .bin data shards, compile and run, and you're stepping in minutes. You then wait 24 hours and enjoy samples about English-speaking Unicorns in the Andes.

For me, this is a very nice checkpoint to get to because the entire llm.c project started with me thinking about reproducing GPT-2 for an educational video, getting stuck with some PyTorch things, then rage quitting to just write the whole thing from scratch in C/CUDA. That set me on a longer journey than I anticipated, but it was quite fun, I learned more CUDA, I made friends along the way, and llm.c is really nice now. It's ~5,000 lines of code, it compiles and steps very fast so there is very little waiting around, it has constant memory footprint, it trains in mixed precision, distributed across multi-node with NNCL, it is bitwise deterministic, and hovers around ~50% MFU. So it's quite cute.

llm.c couldn't have gotten here without a great group of devs who assembled from the internet, and helped get things to this point, especially ademeure, ngc92, @gordic_aleksa, and rosslwheeler. And thank you to @LambdaAPI for the GPU cycles support.

There's still a lot of work left to do. I'm still not 100% happy with the current runs - the evals should be better, the training should be more stable especially at larger model sizes for longer runs. There's a lot of interesting new directions too: fp8 (imminent!), inference, finetuning, multimodal (VQVAE etc.), more modern architectures (Llama/Gemma). The goal of llm.c remains to have a simple, minimal, clean training stack for a full-featured LLM agent, in direct C/CUDA, and companion educational materials to bring many people up to speed in this awesome field.

Eye candy: my much longer 400B token GPT-2 run (up from 33B tokens), which went great until 330B (reaching 61% HellaSwag, way above GPT-2 and GPT-3 of this size) and then exploded shortly after this plot, which I am looking into now :)

[Andrej Karpathy @karpathy]

100% Fully Software 2.0 computer. Just a single neural net and no classical software at all. Device inputs (audio video, touch etc) directly feed into a neural net, the outputs of it directly display as audio/video on speaker/screen, that’s it.

[CodeMaker AI @CodeMakerAI]

CodeMaker AI now offers the Ultimate Plan, which speeds up batch operations up to 8 times compared to the entry-level tier. Learn more at https://t.co/aPRydgukkx

#ai #software #developer #productivity #automation

https://t.co/0ZzRmuMJw0

[Jakub Narloch @jmnarloch]

@donaldtusk @donaldtusk Panie Donaldzie, jedyna osoba ktora przejawia nie kompetencje dzisiaj, jest Pan.

[Jakub Narloch @jmnarloch]

@donaldtusk Strasznie się ośmieszacie