David Gill

CTO @ Hyperglance
🛡️ 10 years in Cloud Security | DevSecOps
💻 Developer - I build stuff.
🚀 Indie Hacking :: | npx davegill

[David Gill @DaveAGill]

@mpilo_person @AmenaiSabuwala This looks like they used a different technique: mesh gradient.

[David Gill @DaveAGill]

@championswimmer @levelsio And definitely gotta be running up to date PHP or it's game over.

[David Gill @DaveAGill]

@championswimmer @levelsio There's a little more to it than just SSL and SSH... gotta config firewall, close ports, avoid running as root but still listen to priviledged ports 80/443, etc.

If you don't know then it's easy to get haxored.

[David Gill @DaveAGill]

The fuck is this button? Why would I ever want to click this?!

[David Gill @DaveAGill]

@adamdotdev People saying "ternaries are not readable" -- only if you misuse them. Used correctly they aid readability.

[David Gill @DaveAGill]

@iupdate Not perfect: If they remove the notch and add back the touchbar it would be better.

[David Gill @DaveAGill]

@AetherAurelia Heh. I get strong XP/Vista vibes from the menu bar. From the thumbnail I actually thought this was a linux rice.

[Theo - t3.gg @theo]

"full-stack" devs are now more common than back-end and front-end devs COMBINED

[David Gill @DaveAGill]

@teodeleanu @adamc0dez @Firebase @supabase Not hard just annoying.

It has limited capabilities and not well maintained by AWS.

Works kinda ok if you're happy with their ugly, non-embeddable (full page) hosted UI for the login/signup screens.

You can build your own UI but then you lose the social login capabilities.

[Corey Quinn @QuinnyPig]

Rumors are swirling about the deprecation of VMware Cloud on AWS.

I'm proposing a helpful update for the VMC marketing page:

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

LLaMA-3 is a prime example of why training a good LLM is almost entirely about data quality…

TL;DR. Meta released LLaMA-3-8B/70B today and 95% of the technical info we have so far is related to data quality:

- 15T tokens of pretraining data
- More code during pretraining (leads to better reasoning capabilities)
- More efficient tokenizer with larger vocabulary
- Super sophisticated (including LLM components) data quality filtering
- Extensive empirical analysis of data mixture
- Focus on quality filtering of post training data (for SFT/RLHF/DPO)

All of the cool stuff in this report is related to how to curate data effectively for pre/post-training! This really shows that data curation/filtering is the most difficult and impactful aspect of training foundation models.

(1) Model architecture: Only 5 sentences are provided about the model architecture, which simply state that LLaMa-3 uses a standard decoder-only architecture with grouped query attention to improve inference efficiency (and a longer 8K context). It’s pretty clear that model architectures are becoming standardized, and most of the research focus is going into constructing datasets. In fact, the main architecture modification made by LLaMA-3 is a more efficient tokenizer!

“Llama 3 uses a tokenizer with a vocabulary of 128K tokens that encodes language much more efficiently, which leads to substantially improved model performance.” - from LLaMA-3 blog

(2) Better tokenizer: LLaMA-3 comes with a custom tokenizer with a vocabulary of 128K tokens (LLaMA-2 had a vocabulary of 32K tokens). This tokenizer is more token efficient (i.e., fewer tokens are necessary to encode the same piece of text relative to LLaMA-2), which makes inference more efficient. Authors also note that the new tokenizer improves performance! In other words, making sure that we are encoding the model’s input data correctly is super important.

(3) Massive pretraining corpus: LLaMa-3 is pretrained over 15T tokens of text (5% non-English), which is a 7X improvement over LLaMA-2 and even larger than the 12T pretraining corpus of DBRX. The pretraining corpus also has 4X more code relative to LLaMA-2 (this was a big criticism of LLaMA-2). With this in mind, it’s not a surprise that LLaMA-3 has strong reasoning/code capabilities—several papers have correlated pretraining on code to better downstream reasoning in LLMs.

“We found that previous generations of Llama are surprisingly good at identifying high-quality data, hence we used Llama 2 to generate the training data for the text-quality classifiers that are powering Llama 3.” - from LLaMA-3 blog

(4) FIltering pretraining data: Few concrete details are provided on the filtering process for the pretraining corpus of LLaMA-3, but it’s clear that a lot of filtering is done. These filters include heuristic filters, NSFW filters, semantic deduplication, and text classifiers to predict data quality. Plus, authors note that LLaMA-2 is very good at detecting text quality, so they use these models in the filtering process (see above). Authors also mention that they do extensive empirical analysis to figure out the correct data mixture (DBRX also mentions this is hugely important).

(5) Overtraining: Chinchilla proposed the compute optimal training regime for LLMs, but recent work indicates that pretty much everyone overtrains their LLMs relative to the compute-optimal ratio. LLaMA-3 is pretrained on two orders of magnitude more data (for the 8B model) beyond the compute-optimal ratio, and we still see log-linear improvements. Sure, we could train a larger model on fewer tokens and achieve similar performance while spending less on training compute. But, this doesn’t consider inference costs! We almost always will pay for more training compute if it means we can deploy a smaller model with the same performance.

“The quality of the prompts that are used in SFT and the preference rankings that are used in PPO and DPO has an outsized influence on the performance of aligned models.” - from LLaMA-3 blog

(6) Post training data quality: Even beyond pretraining, data quality is pivotal for LLaMA-3! The model is aligned with a combination of SFT, rejection sampling, PPO, and DPO. During alignment, authors claim that the quality of supervised/preference data is super important. In fact, the biggest quality improvements in LLaMA-3 came from curating this data and performing multiple rounds of quality assurance on humans annotations!

[David Gill @DaveAGill]

@Vjeux I know @VictorTraelin's challenge was to accomplish this in a single inference but I feel like as a human it takes me 2-4 mental inferences per step. I'm confident this would be trivial for GPT architectures if we could architect this as 2 to 4 inferences per step.

[David Gill @DaveAGill]

@thdxr brew, npx, docker or just a plain executable.

[David Gill @DaveAGill]

@webdevcody

[David Gill @DaveAGill]

@mathladyhazel "If she works just as fast" to cut a board into 3 pieces as she took to cut a board into 2 pieces, then the answer is still 10 minutes. I imagine she spends most of the time measuring.

[David Gill @DaveAGill]

@FrankZayasPhoto @MyPostyArt @timecaptales @fasc1nate Yes, clearly it's a Sony Alpha series which were introduced in 2006. The first fullframe (35mm as seen on the strap) in 2008.

The configuration of the mounting ring looks like an E-Mount introduced in 2010 for their mirrorless range.

[David Gill @DaveAGill]

@historyinmemes But how do we know that was the soundtrack?

[David Gill @DaveAGill]

@xsgames_ Because ‘i’ for int, ‘c’ for char, etc. I don’t always use ‘i’ though, sometimes it makes sense to give it a better name

[David Gill @DaveAGill]

@KenneyNL Qubits

[David Gill @DaveAGill]

@FreyaHolmer velocity is to speed as acceleration is to meth ? 🤔