matt

efficiently scaling recsys ml @pinterest, prev ml/math @Columbia

[matt @mattylawhon]

@zach_nussbaum Ooc did you try just using SigLIP?

[matt @mattylawhon]

@yacineMTB One use case is it's pretty important for certain kinds of architectures which otherwise produce high activations which disproportionately degrade training performance with lower precision training

[matt @mattylawhon]

@ZhaiAndrew Nice! I think I could really see myself using this but only if it works for all the entry points of finding saveable things that I already use, which is pretty much just apps

[matt @mattylawhon]

Just spent hours debugging a hanging distributed PyTorch job from accidentally having different dist.barrier() calls sync with each other... it was a good debugging lesson but also makes me think I should use barrier calls that are unique

[martindbp @martindbpcom]

@francoisfleuret Curse of dimensionality has been on vacation since 2012

[Andy Arditi @andyarditi]

New research post on refusals in LLMs https://t.co/GqOo2R6lDZ

[matt @mattylawhon]

@StuartReid1929 @jeremyphoward Any tips/details for getting this right?

[matt @mattylawhon]

@karpathy Out of curiosity, why fp16 over bfloat16 amp to start with? I would think bfloat16 amp would be slightly simpler to implement and a little more representative of what folks use nowadays afaik

[matt @mattylawhon]

@jacobaustin132 Interesting - that's pretty important perspective here, much more interpretable that way

[matt @mattylawhon]

@NicitaAlex @beffjezos Exactly

[matt @mattylawhon]

@Behzadhaghgoo @NicitaAlex

[matt @mattylawhon]

@NicitaAlex Exactly!

[matt @mattylawhon]

@francoisfleuret Weight/gradient distributions (sometimes by layer) are helpful, especially if you know what to expect. Another thing for distributed training specifically (not always needed) is having rough consistency of local metrics across nodes

[matt @mattylawhon]

@NicitaAlex 😲

[matt @mattylawhon]

The cost / data volume, concept and lack of shame here are all wild

[Susan Zhang @suchenzang]

so i guess this is a thing now

universities running ads to resell students' data

for training llms

💰💰💰

[matt @mattylawhon]

@EugeneVinitsky @frahs @jxmnop That plus the standard profiler run across batch sizes is gonna show a lot about which components see throughput improve with scale and which don’t.

[matt @mattylawhon]

@cHHillee @giffmana This might be different with torch2.0 but wasn't einsum kind of notoriously slow before that?

[matt @mattylawhon]

@yohaniddawela This dataset is at least 1-2 orders of magnitude smaller than used for small foundation models in the generic image setting like clip. I guess fewer zero-shot tasks we care about means way less data is needed here? Any other thoughts on what's going on?