Blair Johnson

ML Ph.D. Student @ Georgia Tech | Research Engineer @ GTRI

[Blair Johnson @oijna]

There's this copilot ad where the guy asks his computer how much thrust the Saturn V produced in hatchback cars (equivalent) and copilot says 90.

[Blair Johnson @oijna]

@juliendelavande @huggingface How is power consumption estimated?

[Blair Johnson @oijna]

@kalomaze @findboundary You aren’t in bad shape. A new driver is backwards compatible with old cuda toolkits. Torch doesn’t usually come packaged with nvcc, so it isn’t surprising that the compiler is out of date. Just look for a cudatoolkit-dev that matches your pytorch cuda version.

[Blair Johnson @oijna]

No notes.

[Blair Johnson @oijna]

My For You page was 37 elon musk tweets in a row today. I don’t follow him.

[Blair Johnson @oijna]

After dozens of visits, google still ranks the jax documentation below a pile of garbage about jacksonville, local businesses, and the singer whenever I search “jax”.

[Blair Johnson @oijna]

@StasBekman We usually see a ~10°C delta between the “front” A100s on our SuperMicro nodes and the “back” A100s that breathe their exhaust.

[Blair Johnson @oijna]

Jax is so fast.

[Mark Riedl @mark_riedl]

You can apply for early pilot access to the National AI Research Resource (NAIRR): https://t.co/cGrX4As7eo

NSF's Press Release: https://t.co/0w8vQR6yXB

[Blair Johnson @oijna]

Just tested and confirmed that copilot in outlook is vulnerable to the unicode exploit.

[Blair Johnson @oijna]

@DrJimFan I’ll be interested to see what happens after the first big leak of one of these kinds of models. A company puts a lot of potentially valuable data into a single 25GB crown jewel, but that data is inherently noisy and unreliable to competitors without ground truth.

[Blair Johnson @oijna]

@felix_red_panda @artificialguybr It looks like you can get ERA5 data for free (as recent as 6 days ago) from the Climate Data Store (seems to be an EU public service).

[Blair Johnson @oijna]

@Karmedge These benchmarks are for very specific formal reasoning tasks which GPT-4 is probably not heavily trained on (“all cats are blue, tigers are cats, Richard is a tiger, is Richard blue?”). It’s not that surprising that a fine-tuned task-specific model can perform comparably.

[Blair Johnson @oijna]

The only F1 stat that matters.

[Blair Johnson @oijna]

@jeremyphoward A bigger monitor and htop 😉

[Blair Johnson @oijna]

@TensorFlow Discovering that they need to read EXIF data for mobile phone images to be right-side up.

[Blair Johnson @oijna]

@OfirPress I also see a lot of people citing minor performance degradation on arc/winograde/hellaswag/MMLU. Very few of the tasks in these benchmarks are longer than a few hundred tokens, let alone the many thousands of tokens in the context lengths that they're trying to evaluate.

[Ofir Press @OfirPress]

@ggerganov @theemozilla @yacineMTB Watch out- models with different context window lengths aren't so easily comparable, and these results might be misleading. I talk about this here:
https://t.co/erEt0XtFt9

[Blair Johnson @oijna]

@theemozilla @teknium @ggerganov @yacineMTB How long are the benchmark tasks in these datasets? It would be great to rule out the possibility that the benchmarks aren’t really testing / don’t necessitate the full context.

[Blair Johnson @oijna]

This is the distribution of prompt+output lengths (in tokens) for the WizardLM 70k dataset. The examples >3k tokens seem to get there through listing lots of things, long junk code, and repetitive number sequences. Warrants filtering, and not helpful for evaluating long context.