The @zymtrace proprietary algo detects the straggler GPU from NCCL on every rank and correlates it with the CPU and CUDA profiles, down to the kernel.
https://t.co/CbYrwkANdx
Introducing zymtrace GPU straggler detector!
In a multi-GPU workload, one slow GPU makes every other GPU wait.
ByteDance found stragglers slowed 42.5% of its large training jobs by 10% or more, mostly due to the code.
ML engineers into CUDA, PyTorch/JAX, GPU profiling, or coding agents: @zymtrace is hiring a Senior ML Engineer to build agents that diagnose and fix GPU bottlenecks. Fully remote, global. DM me for an intro or apply: https://t.co/bV4rDXO8IN
@typesafeai achieved 46% higher GPU throughput while training Jev using @zymtraceโs cluster-wide, continuous profiling and agentic optimization platform.
See how they did it in the link below ๐
And zymtrace doesnโt just surface the problem. It tells you what to do about it. If a process lands on the wrong NUMA node, for example, zymtrace gives you the exact numactl command to fix it.
Check it out: https://t.co/3YuQBaTzDN
We just shipped GPU topology health in zymtrace ๐
Zymtrace now connects GPU topology and health data directly to the workload profile and performance metrics, so you can quickly see when the hardware, or the code, is causing the slowdown.
@alanchanguk Re point 1, worth checking out @zymtrace. An eBPF-powered profiling platform which squeezes more FLOPs from your existing infra. From the team behind OpenTelelemetryโs eBPF profiler
Join me, @anbarth , @athre0z, and @israel_ogbole today to discuss AI and GPU System Perf Tuning with eBPF and flamegraphs by Israel+Joel @ @zymtrace https://t.co/PwP1drcWrL
Weโre attending @RaiseSummit!
Building or considering disaggregated infrastructure across NVIDIA GPUs, AMD GPUs, AWS Trainium, TPUs, or other accelerators?
Weโd love to connect.
@zymtrace ๐ @Walmart
We are excited to partner with Walmart to build a heterogeneous AI infrastructure observability layer tailored for general-purpose and GPU-accelerated workloads.
NVTX adds labels that describe application-level regions: model layers, inference phases, tensor shapes, #CUDA streams, and custom operations. Think of those annotations as receipts for code execution on the GPU.
Learn more: https://t.co/oHUSrMZqd9
๐ NVTX in @zymtrace AI flamegraphs.
For the first time, NVTX annotations can now be viewed directly inside an AI flamegraph, connecting application context to the GPU work it triggered.
๐ข May episode is out! ๐ข
OpenTelemetry Comes of Age: CNCF Graduation, Profiling and eBPF
With @horovits hosting @israel_ogbole, CEO & Co-founder @zymtrace, OTel Profiling SIG member.
๐ง https://t.co/s487tVy6iO
๐บ https://t.co/N4RknpOyrf
recap post:
๐ https://t.co/euLyVRXXcC
Zymtrace automatically detects MIG topology and attributes GPU performance metrics to each MIG instance, correlated with full-stack GPU flamegraphs.
https://t.co/fd23FNOF5w
Today, we're announcing the General Availability of MIG-aware distributed GPU profiling in zymtrace.
MIG makes shared GPUs more predictable. Zymtrace makes them optimizable.