CKtalon

[CKtalon @CKtalon]

@abacaj That’s if they even release a GPU

[CKtalon @CKtalon]

@RealJosephus @suchenzang Seems like the corpus used to train the tokenizer isn’t as clean as the corpus used to train the LLM

[CKtalon @CKtalon]

@drummatick @laurensweitkamp @suchenzang With 200k vocabulary, it’s entirely possible to have many full words

[CKtalon @CKtalon]

@SashaMTL Just stating facts. BLOOM having 1131 citations despite being released in 2022 while Llama2 having 3855 despite being released 8 months later. BLOOM was just severely undertrained with the amount of limited compute they had, with way too much ambition to do so many languages.

[CKtalon @CKtalon]

@dctanner @mov_axbx That’s a really expensive server on eBay considering its age and specs. It seems any used rack that can hold greater than 4 GPUs are highly inflated in price now.

[CKtalon @CKtalon]

@dctanner @mov_axbx What server rack model is that?

[CKtalon @CKtalon]

@realmrfakename @arpagon Not possible

[CKtalon @CKtalon]

@NVIDIAGeForce #RTXSUPER

[CKtalon @CKtalon]

@Yampeleg @abacaj Did similar trainings, and from some manual evaluations, the loss might have plateaued for hundreds of thousands of steps, but the quality of the generations are better given more epochs.

[CKtalon @CKtalon]

@charlieholtz @elevenlabs In the not-so-distant future, pairing this with the Meta Ray-Bans and have it narrate whatever you see will be mind-blowing.

[CKtalon @CKtalon]

@Suhail Helps when a ton of data is distilled from a powerful LLM? Phi-1.5 kinds of shows that generative data can produce a powerful model.

[CKtalon @CKtalon]

@BramVanroy OpenNMT does have most of those implemented since they are also now supporting LLMs. Marian looks dead, perhaps due to lowered importance by MSFT in preference of LLMs.

[CKtalon @CKtalon]

@Yampeleg Just the preview shows how dirty the dataset is…

[CKtalon @CKtalon]

@Science_boy_H @huggingface I’m suspecting QLoRA or LoRA doesn’t help for adding/increasing a model’s second language capabilities

[CKtalon @CKtalon]

@DanielSMatthews @nearcyan Text isn’t one to one. More of a translated summary

[CKtalon @CKtalon]

@_BruceX_ @ID_AA_Carmack Time is money

[CKtalon @CKtalon]

@e270889o @ID_AA_Carmack Plenty of ram, but slow compute-wise. Apple’s CoreML is too opaque to developers, so the Neural Engine hasn’t been usable in an obvious way yet.

[CKtalon @CKtalon]

@decryption @coxymla Follow this: https://t.co/ci3hBBpUry

[CKtalon @CKtalon]

@tmophoto @abacaj You split the layers across different cards. That’s why you need fast interconnects like NVLink so that the GPUs can process the computations quickly without bottlenecks.