Brady DeRousse

Low-budget tech guru that enjoys AI training/fine tuning and providing the best possible IT support experience.

[Brady DeRousse @BradyTechHobo]

@SJRonanMD Where do I go to join the group?

[Brady DeRousse @BradyTechHobo]

DGX Station arrived and I've been setting it up. I had a to order a different set of cables due to the switch not being compatible with 4x100gb, so I'm going to be limited to 400gb bandwidth. Should still be decent, but we'll see.

[Brady DeRousse @BradyTechHobo]

So I cannot run the base kimi k3.1 model on the 16x spark cluster. I'm going to need an fp8 or fp4 variant before I'm able to run it.

[Brady DeRousse @BradyTechHobo]

I have Kimi k3.1 downloaded and I'm currently distributing on the sparks. I'll have numbers in a bit.

[Brady DeRousse @BradyTechHobo]

I came across this gem using flare and it's pretty cray. https://t.co/KVUv0AUCcY running 4 instances across 16 sparks.

[Brady DeRousse @BradyTechHobo]

@billynewport It's mostly for fine-tuning smaller models for business use. Doing some inference on apps will also be way quicker. When doing inference I plan on pairing it up with the spark cluster.

[Brady DeRousse @BradyTechHobo]

I've finally received word back from EXXACT. The DGX Station with the GB300 I ordered back in July is finally shipping and will arrive Nov. 4th. Hyped to try out different projects I couldn't otherwise do.

[Brady DeRousse @BradyTechHobo]

What Kimi K3 looks like running on 16x DGX sparks. Currently trying different settings to see if I can improve speeds.

[Brady DeRousse @BradyTechHobo]

One thing I've found about these DGX sparks is that they run really hot without forcing air into them. If you don't have additional cooling, I recommend dropping the GPU clock speeds down to 1800-1900MHz when running massive tasks. Going to make a custom enclosure soon.

[Brady DeRousse @BradyTechHobo]

@Darkubbs Good morning/evening,

The dgx sparks use their own processors. Each has 10 Cortex-X925 performance cores and 10 Cortex-A725 efficiency cores with a built-in GPU that has a petaFLOP of NVFP4 performance.

[Brady DeRousse @BradyTechHobo]

I haven't been posting often due to work and personal life things that have come up, but moving forward I should be posting almost weekly now. I've been playing around with the sparks and I have amazing results. Fine-tuning and inference is not bad. I'll post projects soon.

[Brady DeRousse @BradyTechHobo]

It's taken a bit due to how busy things have been, but I finally got my spark cluster up and running. Currently serving GLM 5.3.

[Brady DeRousse @BradyTechHobo]

Purchased 16x DGX sparks and a 200gb switch. Once the cables come in I'll run benchmarks with inference and fine-tuning.

[Brady DeRousse @BradyTechHobo]

What I've found is that training across the ai max+ 395 machines is not viable with the 60~Gbps connection. Works great for inference to an extent, but it's not viable for training. @AMD anything in the works for similar machines that work better in a cluster for training?

[Brady DeRousse @BradyTechHobo]

@statusisdown Down in Virginia

[Brady DeRousse @BradyTechHobo]

One of the mlx5 cards is bad. Time to wait on more parts. I'm going to focus on building a 5 node cluster until the new card arrives.

[Brady DeRousse @BradyTechHobo]

Note to self don't get cheap risers. Can't continue project until new risers arrive on Thursday.

[Brady DeRousse @BradyTechHobo]

2 nodes out of the 6 configured. Remaining power cables arriving tomorrow and PCIe 4.0 risers arriving Thursday. I'll have the machines fully configured hopefully tomorrow or Wednesday night.

[Brady DeRousse @BradyTechHobo]

Update: Cards are good, it's just the cheap risers I purchased. Going to use these for now at PCIe 3.0 x4 speeds until the new ones arrive and then I'll release benchmarks.

[Brady DeRousse @BradyTechHobo]

Actually I think I messed up on the cards. I get the feeling these aren't PCIe 4.0 cards. lol