Building something new in AI compute infra, Nvidia alum, prev founder building AI for physical infra, author of AI books, prev infra investor at Moxxie
it's early days for open world models but licensing is a mess.
can't trust the headline "awesome permissive license" tag just yet. need to go all the way down yourself and verify all the components (encoder, decoder, adapters, training data).
it's not enough that it's an apache-licensed repo. it can have something like a non-commercial decoder, so now the whole thing becomes unusable. obviously you can still do research and experimentation, but the headline feels a bit misleading.
some research checkpoints forbid use at any lab owned by a company. some forbid using their outputs to train other models. some are void in entire countries and continents.
i found an interesting spatial bias while experimenting with the open video model Wan 2.2. if you ask it to generate "a mug to the right of a laptop", it often gets the placement backwards.
turns out the model seems to have a strong left-to-right ordering bias. the first object mentioned in the prompt tends to appear on the left regardless of the spatial instruction.
so i tried a simple inference-time workaround. instead of explicitly asking for the position, i rewrote prompts to name the leftmost object first. and the spatial placement accuracy approx doubled in my experiments. this is just rewriting the prompt.
looks like a lot of spatial instruction following can depend on something as simple as word order.
an interesting thing about visual and spatial AI is how heterogeneous the compute can get within the same trajectory.
for example, one trajectory might involve CPU preprocessing. and then 3D reconstruction on an L40S. world model inference on an H100. and rendering on another GPU.
so instead of running the entire pipeline on one expensive machine, you can route each step to the hardware it actually needs.
usually the gnarly part is managing state, data movement, and latency across these machines.
it’s usually intended to be a diss. now I’ve seen two flavors of this:
- when it’s just a bad product and doesn’t work as expected and has stupid vulnerabilities (like storing passwords in plaintext). it’s a fair criticism.
- when a person takes offense because someone else took a “shortcut” and didn’t follow the proper rules of building a product. but the product works nonetheless and there’s no flaw spotted just yet. this is mostly cope.
every now and then, i hear someone say “oh they just vibe coded their product”. okay but isn’t everyone doing that? in fact, not coding agents is just irresponsible now.
software engineering will increasingly resemble managing a compiler:
you specify the behavior, constraints, and architecture. and a machine produces an implementation.
automated systems verify its properties. humans intervene where verification is insufficient.
so perhaps “vibe coding” is either cope from people who haven’t figured out agentmaxxing or (hopefully) slang for “it’s a poorly engineered product”
@peteryantweets quality of verification is very much the engineer’s responsibility. how they do it is up to them and it’s all fine (manual, automated, hybrid, choice of tools, choice of evals). but they should be able to stand by it and defend it if needed
yeah something like codemode but with a render-and-evaluate loop. the agent generates a scene, renders it from multiple viewpoints. and also checks geometry, appearance, and physical consistency. it can then keep iterating. the point is to make verification part of the agent's workflow
agents are getting good at generating visual assets, but verifying what they've built is still kinda hard.
i saw an agent generate a 3D model and deliver it with the caveat "i haven't looked at the 3D model myself"
like yeah we've built benchmarks for 3D quality, geometric consistency, and human preference. but these evals aren't really integrated into the agent workflows.
e.g. a valid mesh isn't necessarily a good mesh, a photorealistic splat isn't necessarily geometrically accurate
so perhaps an important piece of visual AI infra is giving agents the ability to inspect and verify their own outputs.
@amirsadjady yeah but lot of advances happening though. perhaps agents will soon have the ability to open it first, visually inspect it, verify it, and then ship the output.
The open model world faces two main challenges in a post-Jensen world:
1. Political exposure. Open models’ allies are retreating in DC.
2. Investment. Open model builders need to find profitable products far sooner w/o his backstop.
Will feel it sooner than people expect.
ok who knew so many people still used outlook. maybe this is a good marketing strategy by microsoft. like take down outlook every few months and make people realize microsoft is used by so many people who clearly can't churn away