Jeffrey Castellano

Obsessively exploring simulations, graphics, machine learning & AI. This is my demo space & devlog for side projects in WebGPU, WebGL, Metal & MLX/CUDA.

[Jeffrey Castellano @CastellanoWorks]

1st real milestone of realtime threejs meshlets + path-traced lighting in the browser. V2 (soon) will use shareable assets, so I'll have a live demo then. Lots to fix: pop-in, blotchy light, fireflies, etc. Opus 5.5 got me over the finish line. #threejs #webgpu #pathtracing

[Jeffrey Castellano @CastellanoWorks]

@DennisSmolek Hope you do btw, I feel like OIDN has a lot of room to squeeze. Good luck! Sorry for the crazy response, ONNX is like one of my pet peeves haha.

[Jeffrey Castellano @CastellanoWorks]

Neural denoiser v0.4 is trained + multiple spp frame support lands milestone 2 for my #threejs real-time pathtracer. This version revisits regular meshes. Normal scenes work w/ many moving lights, glass, caustics, mats, skinned mesh, morphs, & GI. Captured w/ settings on 'ultra'.

[Jeffrey Castellano @CastellanoWorks]

@DennisSmolek I was literally only building this so that my webgpu agent swarm could have a nicer 3D home to navigate when showcasing its shared decision making and self weight mods through accelerated backprop.

[Jeffrey Castellano @CastellanoWorks]

@DennisSmolek I actually have a whole AI runtime I'm releasing. Like 95% done. That's my main focus but this was a side quest. ONNX has really limited use cases in my opinion. I highly recommend dropping it beyond PoC. I have a hard time imagining anything being better for having used it.

[Jeffrey Castellano @CastellanoWorks]

@sino1782013 Looks nice! How are you doing lighting?

[Jeffrey Castellano @CastellanoWorks]

@garrettkjohnson The meshlet pathtracer is really about handling big scenes over being accurate. Closer to a UE approach. It can REALLY handle some big scenes. Hopefully releasing demos this week that might surprise folks. I know it shocked me even when it ran poorly on my iPhone 13.

[Jeffrey Castellano @CastellanoWorks]

@garrettkjohnson The meshlet pathtracer is wavefront with streaming cluster LOD traced through the raster. Materials graphs & real-time complexity. 2 DAG & multi-level BVH w/ CLAS like structure. Trace is LOD aware & both versions work tightly w/ neural denoiser. Will converge both into one soon.

[Jeffrey Castellano @CastellanoWorks]

@garrettkjohnson My normal pathtracer is a megakernel with hybrid techniques... radiance cache, temporal, neural denoiser & sitting on a custom GPU built and resident BVH with full (but inaccurate) light transport. Big scenes get hurt but optimizing... Fixed frame costs are a double edge sword.

[Jeffrey Castellano @CastellanoWorks]

@garrettkjohnson Also there are three things happening here. The meshlet pathtracer, my normal pathtracer and three-gpu-pathtracer. Yours is wavefront sample accumulation over a queue focused on ground truth. Unbiased, good convergence. Actually correct. I assume by a mile.

[Jeffrey Castellano @CastellanoWorks]

@garrettkjohnson So I only knew your webgl version before & only glanced your webgpu version at release so I might be incorrect with this... but there are some pretty large architectural differences around tradeoff and intent.

[Jeffrey Castellano @CastellanoWorks]

@iBarakXYZ Most of it yes, as I get further along.

[Jeffrey Castellano @CastellanoWorks]

@LordoftheMounts Somehow I imagine it would take more than that.

[Jeffrey Castellano @CastellanoWorks]

@CodyBennett @jonathankennell By the way, not sure how your 2nd DAG is going but I suspect I'm going to deprecate in favor of more dedicated ray-only reps unless I can find a case where it really makes sense to keep for me.

[Jeffrey Castellano @CastellanoWorks]

Quick Correction - Video says meshlet pathtracer. This is not using meshlets, just left the title in by mistake. Two separate renderers but shared methods in many cases hopefully will have them combined at the end.

[Jeffrey Castellano @CastellanoWorks]

@mogmek Neural denoiser is overcorrecting right now and blotting out specular so I needed to combine my classic denoiser with the other for this capture.

[Jeffrey Castellano @CastellanoWorks]

@mogmek This is ramped up settings for sure... captured from my M5 pro on ultra settings. Not the earlier M1 Pro capture I was using for previous posts. But honestly, once I fix a few things on the neural denoiser this should be pretty close on the M1 as well at 30fps.

[Jeffrey Castellano @CastellanoWorks]

@CodyBennett @jonathankennell Also yes sorry to answer the implied question... two DAG, roughly same split and BLAS/TLAS still in there just avoiding when possible. Cache is already frail.

[Jeffrey Castellano @CastellanoWorks]

@CodyBennett @jonathankennell Your RC work is amazing by the way, wish I could have been at Siggraph to see your talk.

[Jeffrey Castellano @CastellanoWorks]

@CodyBennett @jonathankennell Everything's a trade-off as you know. Been trying hard to keep things honest in here.

[Jeffrey Castellano @CastellanoWorks]

@CodyBennett @jonathankennell yeah, I've been following your efforts. I avoided BLAS/TLAS (or rather tried it & agreed the refit is faster in many cases). I've got a similar hybrid approaches but mostly settled on a weaker version of NVIDIA's CLAS.

[Jeffrey Castellano @CastellanoWorks]

Also I find it handy to set up these scripts in a little tailscale synced socket server so AI can debug issues simultaneously across devices. Just a nice little workflow I found for me at least.