Daniel Galvez

AI Developer Technology Engineer at NVIDIA. Speech recognition and computer architecture nerd. All opinions are my own, not my employer's.

[Daniel Galvez @memorypaladin]

@ezyang Yeah, that's my concern. I worry that it's already pretty obvious which kernels aren't batch invariant in LLM workloads.

[Daniel Galvez @memorypaladin]

I'm thinking of internship projects for this summer. If we build a dynamic binary translation tool to detect whether any cuda kernel in your workload is not batch invariant, would that be useful to anyone? You would run it just like compute-sanitizer. https://t.co/Cjxisre23v

[Daniel Galvez @memorypaladin]

I'm speaking at GTC on Thursday at 2 PM about debugging cuda graph errors in pytorch. I will be talking about several tools in pytorch and CUDA that can detect and eliminate classes of errors for you. Will be fun.

https://t.co/0bJD9r54pt

[Daniel Galvez @memorypaladin]

@cloud11665 @tenderizzation It's more like... it's surprising how competitive eager mode run via a cuda graph can be with torch.compile with max-autotune at times.

[Daniel Galvez @memorypaladin]

@tenderizzation @cloud11665 There's also subtle issues with NCCL user buffer registration as well. All of the edge cases and the need to work with arbitrary kernels is the reason why no one has pulled this off for generic code yet. And some have complained at me about the need for torch.compile as well.

[Daniel Galvez @memorypaladin]

@tenderizzation @cloud11665 But anyway it is at a point where I would be happy to merge it as an experimental feature after a lot of code cleanup. (And of course we need Elias Ellison to say yes!) There's lots of interest, so it's a renewed priority for me. Feel free to AMA.

[Daniel Galvez @memorypaladin]

It's really incredible how much I learn every day from my colleagues at NVIDIA on mapping ML compute to hardware. It's a special place.

[Daniel Galvez @memorypaladin]

@mike64_t @samsja19 On this point, my dream project if there weren't more pressing matters is to work on the jit compiler infra in cpython. There's hardly any public data on why cpython is "slow" at a microarchitectural level AFAIK, but a lot of it is CPU frontend bound.

[Daniel Galvez @memorypaladin]

@mike64_t Need to learn the problems first before you can make a solution for it. There are lots of complications I've learned only by working in detail with torch.compile and with colleagues on Megatron. e.g., torch.compile cannot obviously enforce PGAS for "symmetric memory" right now.

[Daniel Galvez @memorypaladin]

Is Amazon DeepFleet actually a generative transformer model trained to solve NP-Hard problems, or is it just a classic Vehicle Routing Problem (VRP) solver being branded as "AI"? Would be cool if the former, but no real substance in the announcement.

https://t.co/sK4yMDW9Wd

[Daniel Galvez @memorypaladin]

@marksaroufim Oooh, lots of random things I don't know. Thanks.

[Daniel Galvez @memorypaladin]

FYI, when you're building pytorch at HEAD from source, the versions of optional dependencies like triton for your given commit are listed at: https://t.co/96Vr4z65Wl

[Daniel Galvez @memorypaladin]

@CUDAHandbook To be clear, the idea is that you would use this to infer for an arbitrary program what data is being used by any particular kernel as a form of runtime analysis., not to try to do cuda managed memory all over.

[Daniel Galvez @memorypaladin]

@CUDAHandbook A fun thing about virtualization technology is that you can presumably "infer" at runtime the pages referenced by each CUDA kernel if you intercept every address lookup, thus telling you the "segments" used. Maybe it's hopeless for data-dependent workloads, though.

[Daniel Galvez @memorypaladin]

Really appreciate how adding a comment to c10/core/Allocator.h in pytorch causes over 1000 files to recompile. Someone really needs to make ccache comment-aware, if that's not impossible or otherwise impractical.

[Daniel Galvez @memorypaladin]

If you want to get rid of the "I0702 10:00:03.704527 2389475 torch/_subclasses/_fake_tensor.py:3179]" prefix in your torch logs, you need to set the environment variable TORCH_LOGS_FORMAT="%(message)s".

[Daniel Galvez @memorypaladin]

@_rozzai I have not. It seems that the typical design pattern in CUDA programs is that you have a single CPU thread driving all GPU work. Pytorch uses a separate thread for backward by default https://t.co/gQR6WeQd3B but it never runs at the same time as the main thread.

[Daniel Galvez @memorypaladin]

Since ASLR makes the CPU allocations change with every process run, the location of the GPU virtual memory heap cannot be the same across every process run, as CPU memory might have been allocation already where your GPU memory previously was.