@BlancheMinerva@roydanroy Its actually only showing up under certain conditions when you click the link. Maybe for him it looks fine. I donโt think his x necessarily got hacked.
the most goated example i could find when eval'ing Opus 5 was on a frontier bench very hard task. the model is given an engineering drawing of a machine part and asked to write code to rebuild it as a 3D freeCAD model. the fun catch is that the model has no way to view the drawing: mini-swe-agent is a text-only harness, so all the model ever sees is terminal text over bash IO. it's something hard and multidisciplinary even for a human engineer capable of seeing the drawing!
working blind, Opus 5 first tried off-the-shelf OCR with Pillow and Tesseract, found the results unreliable on the drawingโs thin numerals, and built its own vision stack on the spot. carved out each printed character as a connected component, rendered the glyphs as ASCII art (?!?), and matched glyph bitmaps to read the dimension labels off the page
then it vectorized the drawing itself: run-length line detection with sub-pixel centroids, least-squares circle and ellipse fits to recover radii and bores, arrowhead detection to tell dimension lines from part geometry. it worked out that the four views were printed at different scales, derived each oneโs mm-per-pixel ratio, and rebuilt the part, feature by feature, as a parametric freeCAD model
welcome to the world, my dude!
NeurIPS reviews are out. At this point it's an arms race where both sides have the same supplier. You write the paper with Opus 4.6, the reviewer writes the review with Opus 4.7, and now we get to rebut with Opus 4.8. Let's just hope Opus 5 doesn't ship before the discussion period closes.
@modal You literally just select the model and click create. It's magical. And with just an additional click, they automatically benchmark your endpoint for either agentic or interactive usage.
@modal endpoints are really good. You get a well-optimized llm endpoint with just one click, using all their inference knowledge (https://t.co/uW3joSZMlq). GLM 5.2 FP8 (a 750GB model!) running on 8xB200 pushing 600 output tok/s like it's nothing ๐คทโโ๏ธ
Frontier-Bench v0.1 (or in spirit, Terminal-Bench v3.0rc1) is out! Exciting new changes:
- stronger differentiation between frontier models
- regular updates: treat benchmarks like software
- auditing process of all benchmark submissions
- refined task review process
Weโre releasing Frontier-Bench: a benchmark that measures and evolves with the frontier of agent work.
Built by the team behind Terminal-Bench and Harbor, Frontier-Bench is an on-going community effort.
Frontier-Bench v0.1 contains 74 tasks on which the best agents score ~34%