Miha Jenko

SWE, ML & DS professional. #NLProc specialist.
Opinions expressed are not my employer's.
RTs don't count as endorsements.

[Miha Jenko @miha_jenko]

@EmilWallner I used ClearML in the past, it's battle-tested: https://t.co/iEDps6T1v3

[Max Hager @yachty66]

✨Introducing GPU Benchmark Tool

A simple tool to test the health and performance of your NVIDIA GPU and see its performance.

Let's be honest, FurMark and 3DMark suck 😂 and they are not really testing AI performance.

[Perry E. Metzger @perrymetzger]

Summary: Huawei trained a fairly big AI model entirely on their own hardware, with no Western GPUs. On the one hand, this shouldn’t be shocking to anyone. On the other hand, I know lots of people who claim that this couldn’t happen, so I thought it was worth reposting. (I expect a lot of people in the West will be even more shocked when the Chinese catch up on chip process technology.)

[Matthew Green @matthew_d_green]

You should use Signal. Seriously. There are other encrypted messaging apps out there, but I don’t have as much faith in their longevity. In particular I have major concerns about the sustainability of for-profit apps in our new “AI” world.

[Hamel Husain @HamelHusain]

. @sh_reya 's paper confirms what I see in practice

1) Automated evals don't work (without semi-manual human alignment)
2) Most tools don't provide this alignment
3) Automated evals add mostly noise
4) You can only write good evals by looking at data and reacting to failures

[Miha Jenko @miha_jenko]

@LeonDerczynski Researches can fall into the trapping offered by step-wise learning robustness. Since the algorithms are robust to lone errors in the training data, we develop habits to follow up on the training error whilst failing to apply the same rigor on test data too.

[Miha Jenko @miha_jenko]

@mtrc We are also going to need lossless video-audio on the internet, because these days it's hard to trust a low-res video with bleepy audio to not be AI-generated.

[Rachel Thomas @math_rachel]

Hype around AI in medicine often ignores two key risks:
- patterns of automation contribute to centralization of power
- medical knowledge is limited by the systemic refusal to trust patient expertise 1/

my new post: https://t.co/1nqX5bzP6x

[Jack Morris @jxmnop]

today i found out that this one australian guy has been toiling away making incredibly detailed Neural Circuit Diagrams with the vibe of a 1950s issue of Popular Mechanics, but content fit for the 2020s

behold. the Transformer

[Miha Jenko @miha_jenko]

@lvwerra This only works for you because you have some context already.

[Miha Jenko @miha_jenko]

@EmilWallner @patriciobcs Only for certain regions 😔

[Miha Jenko @miha_jenko]

@BlancheMinerva Feels like a Miro board to me.

[Jim Fan @DrJimFan]

A fact worth highlighting: NVIDIA is making its own *CPU*, and will increasingly excel at it. To max out GPU's performance, building CPU in-house is an inevitable path.

Below is GH200, the first superchip that includes all home-grown components: CPU (Grace), GPU (Hopper), and NVLink Chip-2-Chip (C2C) interconnect (delivers up to 900 GB/s total bandwidth). On GH200, CPU and GPU can access each other's memory freely and concurrently.

Whitepaper: https://t.co/oc6UKGtX53

[Kevin Fischer @kevinafischer]

I don't talk much about this - I obtained one of the first FDA approvals in ML + radiology and it informs much of how I think about AI systems and their impact on the world. If you're a pure technologist, you should read the following:

There's so much to unpack for both why Geoff was wrong, and why his future predictions should not be taken seriously either.

Geoff made a classic error that technologists often make, which is to observe a particular behavior (identifying some subset of radiology scans correctly) against some task (identifying hemorrhage on CT head scans correctly), and then to extrapolate based on that task alone.

The reality is that reducing any job, especially a wildly complex job that requires a decade of training, to a handful of tasks is quite absurd.

Here's a bunch of stuff you wouldn't know about radiologists unless you built an AI company WITH them instead of opining about their job disappearing from an ivory tower.

(1) Radiologists are NOT performing 2d pattern recognition - they have a 3d world model of the brain and its physical dynamics in their head. The motion and behavior of their brain to various traumas informs their prediction of hemorrhage determination.

(2) Radiologists have a whole host of grounded models to make determinations, and actually, one of the most important first order determination they make is whether there is anything notably wrong with a brain structure that "feels" off. As a result, classifiers aren’t actually performing the same task even as radiologists.

(3) Radiologists, because they have a grounded brain model, only need to see a single example of a rare and obscure condition to both remember it and identify it in the future. This long tail of rare conditions to avoid missing is a large part of their training, and no one has any clue how to make a model that acts similar in this way.

(4) There’s so many ways to make Radiologist lives easier instead of just replacing them, it doesn’t even make sense to try. I interviewed and hired 25 radiologists, whose primary and chief complaint was that they had to reboot their computers several times a day.

(5) A large part of the radiologist job is communicating their findings with physicians, so if you are thinking about automating them away you also need to understand the complex interactions between them and different clinics, which often are unique.

(6) Every hospital is a snowflake, data is held under lock and key, so your algorithm might not work in a bunch of hospitals. Worse, the imagenet datasets have such wildly different feature sets they don’t do much for pretraining for you.

(7) Have you ever tried to make anything in healthcare? The entire system is optimized to avoid introducing any harm to patients - explaining the ramifications of that would take an entire book, but suffice to say even if you had an algorithm that could automate away radiologists I don’t even know if you could create a viable adoption strategy in the US regulatory environment.

(8) The reality is that for every application, the amount of specific and UNKNOWABLE domain knowledge is immense.

LONG STORY SHORT: thinkers have a pattern where they are so divorced from implementation details that applications seem trivial, when in reality, the small details are exactly where value accrues.

Should you be worried about GPT5 being used to automate vulnerability detection on websites before they’re patched? Maybe.

Should you be worried GPT5 is going to interact with SOCIAL systems and destroy our society single-handedly? No absolutely not.

[Miha Jenko @miha_jenko]

@davisblalock d) The conservative position has not been equally reflected in internet data as the liberal position. The scrapped data, not the content filter, skews toward the status quo of the available data.

[Miha Jenko @miha_jenko]

@f4micom @Ianternn Passphrases are out.

[MMitchell @mmitchell_ai]

As I was privately babbling out my own diatribe on the FLI letter yesterday, I was honored to be pinged by @emilymbender and @timnitGebru about joining forces to put out coherent thoughts. Excited to be able to share it today.
https://t.co/WGVGf2S0EH

[Miha Jenko @miha_jenko]

@matthew_d_green This legislature might not make it in, but it was enough for R&D departments to pay attention and invest.

[Miha Jenko @miha_jenko]

@matthew_d_green That's what AI hype does. Pumps up the media and drives legislators and lobbyists into a frenzy. The industry and the academia supports it, of course, because mandates translate to contracts and grants.

[Miha Jenko @miha_jenko]

Someone mistook the @huggingface logo on my hoodie for Shell's. Too good not to share.