Asaf 🇮🇱

[Sumuk @sumukx]

Some thoughts about GPT-5.6-Sol after ~30B tokens:

Sol is the most OCD model I’ve used thus far. It very frequently gets one-shotted by random nits in the codebase and writes a bunch of tests to fix it. Even with fast mode, it’s incredibly slow to do this kind of iterative development, especially when builds take really long. This by itself is not a bad thing, but the worst part is that after 2 compactions, it’s chasing the nitpick / useless goals I never told it to accomplish, rather than the main task. This behavior is so bad, I thought I was messing something up and tried codex, pi, and opencode to figure out if it’s a harness issue, but there is no meaningful difference between the three, which leads me to believe this is a model problem.

AI code has this weird delayed release effect. You’ll only notice slop code 2 dev cycles into a codebase when you spend more time fighting with the code and on refactors than on shipping features. It’s possible that sol is better than 5.5 a couple cycles in, but tbd.

My file deletion experience has also been similar to others: this is a dangerous model to let loose without guardrails. For instance, when performing a routine container upgrade, it accidentally printed out an env secret, then panicked and rotated ALL secrets (this is internal so not public facing, which was also documented), and proceeded to break everything, spending an extra hour fixing everything and redeploying everything else to use the new secrets. It also gets rid of files it doesn’t like. I have no idea why this is, but I think something about the reward model rewarded bookkeeping.
Writing is another problem. 5.6 has a huge context bleed effect. It does not know how to write documentation and starts putting the specs in the documentation. If I ask it to develop a user sandbox for isolation, and also ask it to write documentation, it starts talking about specs and sandboxes in user-facing docs, which makes no sense. Fable is somehow much, much smarter in this regard. Frontend design has also not gotten better. Fable is still one generation ahead here.

Overall, as a huge 5.5 user, I am not convinced that sol is a meaningful upgrade. It’s possible my practices need to change, but unfortunately it feels like I’m spending longer fighting with 5.6 than I did with 5.5. It’s like the model is so SO smart, but so hard to work with, compared to fable and even grok4.5 surprisingly. It’s clearly intelligent, but also just doesn’t care about what I ask it to do? (Is this supposed to be AGI feels like?) I hope the codex team fixes what possibly is a bad harness setup, because the benchmark numbers show a very different story from what I’m seeing while using the model.

[Asaf 🇮🇱 @AsafAgami92]

@pvncher Why don't you just let the users customize their sub agent like in opencode?

I don't understand why codex and cc limit sub agents like that, you already have the blueprint from the @opencode project. We can write our own "scout"/"advisor"/"playwright driver" etc

[Asaf 🇮🇱 @AsafAgami92]

@sama Nothing, because it doesn't do what it's told

[Asaf 🇮🇱 @AsafAgami92]

@YotYotam פייר

[Asaf 🇮🇱 @AsafAgami92]

@ptr_dvd יש פה מציאות של ציבור שנמצא בשלטון, פוטר את עצמו משירות צבאי ושולח אחרים למלחמות אינסופיות. תחפור כמה שבא לך על הדאגה הממש כנה שלך לקבוצות מיעוט שנרדפות (על ידי מי חתיכת טמבל? הם בשלטון), אבל אתה מתחמק מאי השוויון הזה ומציג דרישה בסיסית לשוויון בתור רדיפה/צביעות וכו

[Asaf 🇮🇱 @AsafAgami92]

@avidichter שמעתי שמרדכי דוד לא נתן לך מספיק זמן לדבר בכנס שלו

[Asaf 🇮🇱 @AsafAgami92]

@pvncher I told it to run tests and commit when it's done, and it failed. 5.6 sol is the first model that struggles to follow explicit instructions.

I also told it to add "don't use the new browser ever" (the one that asks for approvals) to memories. It took 4 tries!

[Asaf 🇮🇱 @AsafAgami92]

@thdxr 5.6 sol has been awful at following instructions

Hasn't been this bad in years

What am I missing? Used it in opencode btw

[Asaf 🇮🇱 @AsafAgami92]

@HGoldich גם על יהודים זורקי אבנים הם יורים? מה בכלל המתנחלים חיפשו בכפר הזה?

[Asaf 🇮🇱 @AsafAgami92]

@theo Opencode is miles better than both

[Asaf 🇮🇱 @AsafAgami92]

@tallouria טוב מאוד, תפחד

[Yoel. H @YoelH5]

🚨עזה.

זה נאסר סלים.
ביום רביעי חיילי צה"ל הרגו בדם קר את הבן שלו עם כדור לראש.

האב השכול יודע עברית. שפת הפושעים.
לא נדרש תרגום הפעם.

הקשיבו לאב השכול ותפנימו למה אתם שותפים.

[Asaf 🇮🇱 @AsafAgami92]

@pvncher My system instructions are short and include clear and explicit instructions to run tests and commit changes.

Every model including 5.5 complies.

5.6 is the first one that didnt, plus, it scope creeped the task and made things worse.

Most disappointing model ever

[Asaf 🇮🇱 @AsafAgami92]

@theo It's just a shit model

[Asaf 🇮🇱 @AsafAgami92]

@DavidOndrej1 Most disappointing model release ever for me. Sol completely ignores clear instructions, starts every side quest ever, generally annoying and lame.

[Asaf 🇮🇱 @AsafAgami92]

@Avivkr13 מונדיאל אחי

[Asaf 🇮🇱 @AsafAgami92]

@Likud_Party ממטרה של קרינג'

[Asaf 🇮🇱 @AsafAgami92]

@E_tek @zivaolam נראה שהוא התבלבל (אולי מהאמוג'י?) ולא הבין שהתור של המשתמש הסתיים, אז הוא ג'ינרט הודעה של יוזר?

זה קורה לפעמים עם מודלים קטנים

[Asaf 🇮🇱 @AsafAgami92]

@kedar_mordechai @rozolyo כן, זוכרים. יש לך גם טיעון או שההמשך הסמוי הוא "ולכן מותר לנו להיות אינסוף גזענים ואינסוף אלימים"?

[Piers Morgan @piersmorgan]

It’s official - Netanyahu is now in charge of a rogue government that even ignores its own rule of law. Shameful.