What cannot be measured cannot be improved! As @mecadoinc shows in our work with @huggingface, there's a whole lot of work to do to create AI systems that really work for mechanical design. And these are just single parts - no requirements or assembly context!
Introducing CADGenBench: measure how well AI systems produce engineering-grade 3D parts!
While current models can generate 3D parts, they are far from precise enough to build functional parts. We built a benchmark to systematically measure their capabilities on two tasks:
1. Generation from an engineering drawing of a part
2. Editing: given an existing STEP file and a requested change
The benchmark is tool-agnostic. It makes no assumptions about how you build the model. You can vary the LLM, and you can vary the environment. Use build123d, Onshape, Autodesk, or a model without an LLM entirely. We open sourced the scoring engine and a reference baseline on top of build123d.
A collaboration between Hugging Face and @mecadoinc!
Submission space: https://t.co/40kfjsd3Dv
Code repository: https://t.co/fvmMjzrIzp
@istvan_csanady@BrierRat@2112Power interesting. not sure that’s really true because of how certain features differ. for example, nx fillets can adjust to certain surfaces much better than in sw. model capability is limited by tool capability at eod in this case