A busy day for multiagent stuff, but I have to say this is the drop I'm most excited about! @kradleai is shaping up to be a leader in assessing the behavior in multiagent and multiprincipal settings. These settings are becoming common, ergo you should follow Kradle's work!
god this is fascinating
Kradle puts the models against each other (and against humans) in Minecraft simulation worlds that test their creativity, collaboration, logic, conflict, etc and then reports out the results.
Seems like we still have a leg up whenever instincts are involved.
@connacher_@JamesTamplin@SethBling The 6.2's are able to jump to that conclusion very fast and very consistently. We're pumped to add more to v2 of the benchmark to make these all more difficult.
We launched a new benchmark: KradleBench
Yesterday morning, our human entry (@sethbling) was head-and-shoulders above every AI model.
Then Astra 6.2 launched and beat him. I'm still processing this
To see AIs improve so fast in multi-agent realtime games makes the curve real
We had Minecraft legend @SethBling represent humanity on our new benchmark.
He was #1 until Astra 6.2 launched yesterday.
Introducing KradleBench: evals in live, multi-agent worlds
@kayleerowena you're welcome! i had a lot of fun making it!! i'm loving everyone posting their masterpieces too... if you come across any bugs, please lmk i'd be happy to fix em
@MobyGamer nope, clean-room reimpl.
the original author mentioned that the source to the public domain version used to be distributed on bbs's back in the day along with the app... is that source code floating around somewhere?
@mcdev there's a cool exemption for copyright/dmca related to the type of media it was originally on; see two bullets at end here: https://t.co/zoQKhglrDn