Phillip Carter

Copilot Platform @GitHub. Formerly Observability @Honeycombio and F# lang/.NET tools @Microsoft

[Phillip Carter @_cartermp]

Work laptop has been properly adorned

[Phillip Carter @_cartermp]

@attunewise @HanchungLee as a token reseller myself the enterprise space understands there isn’t much of a technical difference between platforms, but they care about integrations with their other workflows (where there is some), custom work with customer engineers, and overall “trust” and relationships

[Phillip Carter @_cartermp]

fwiw the overall outcome wouldn’t have been that terrible in this case, but it’s definitely the sort of thing you’ll show your whole ass on down the road in some review where someone is caremad about measuring the right stuff

[Phillip Carter @_cartermp]

Almost fooled myself today how good Opus is for writing verifiably correct code and context switching over to SLO definitions and success/counter metrics.

The silly thing tried to jam two independent metrics together and say to measure that aggregate!

[Phillip Carter @_cartermp]

@kmett seems like the same room layout is somewhere in the weights

[Phillip Carter @_cartermp]

Follow @wormandwalnut for actually good and very strange AI art

[Phillip Carter @_cartermp]

This is how we do a substantial amount of work these days on the copilot team. I still prefer the richer desktop app experience when I’m focused on my own thing, but when collaborating with others this is how it’s done

[Mario Rodriguez @mariorod1]

Both Slack and Teams surfaces are GA and on a weekly release cadence. Keep your team in flow 🏎️

https://t.co/b6QkYHWaqp

[Mario Rodriguez @mariorod1]

Both Slack and Teams surfaces are GA and on a weekly release cadence. Keep your team in flow 🏎️

https://t.co/b6QkYHWaqp

[Phillip Carter @_cartermp]

@EnoReyes @poteto If I had my druthers we’d call them meta-harnesses or verifiable agent loop coalgebras but it is what it is

[Phillip Carter @_cartermp]

@EnoReyes @poteto The devopsification of the term is absolutely the future

[Phillip Carter @_cartermp]

@burkeholland @cbgrasshopper Heya, in repo settings, we have a toggle to increase the size. Default is small for the sandbox. Did you run with the default, or size up and still run out of resources?

[Phillip Carter @_cartermp]

Maybe. At work we kick off a bunch of stuff from Slack and debug in it too. But there’s so many configs and tools within tools, data sources, permissions, integrations with gaps, etc. that make various things drop off a cliff.

Even today we’re decades into CI being a thing and we have so many gaps and issues with CI systems industry wide. It’s a super long tail

[Phillip Carter @_cartermp]

Ehh, good taste shows up everywhere but bad taste is just more prominent. GCP has “taste” to some, but god help you if you dared use gemini via vertex for years. Or any other number of services that send you straight to hell if you have the audacity to try something simple.

Cursor has (had?) taste except for the massive pile of enterprise features that actually make it money. I don’t personally find Cursor’s OIDC setup for cloud agents particularly tasteful. And the field of AI-assisted IDEs-not-IDEs is enormous and full of stuff with more or less taste at a given moment.

I think TS and VSCode stand out because it’s just genuinely difficult and rare to make a developer product that millions of people use daily and generally find enjoyable. The dustbin of dev tech that did stuff well but failed for other reasons is huge.

But also RIP //xlab, the best internal-but-might-go-external tool for getting any VM you wanted on-demand, killed by a mandate to dogfood devtestlabs, which came with all the enterprise lockdown controls that buying customers love :(

[Phillip Carter @_cartermp]

@Aaronontheweb So I think most workers and even high level ones have been asking these things a lot. But leaders and investors in particular let hubris get the better of them, especially since 2024, to the point of even insulting people who said they were out of touch.

[Phillip Carter @_cartermp]

Lots of moral and economic reasons to curtail betting markets, but maybe my personal pet peeve is that they’re objectively no better than traditional polls and are likely just a downstream poll indicator. Now that we’ve libbed out can we stop with them?

[Phillip Carter @_cartermp]

@k_cieslak concerning

looking into it

gork is this true

[Phillip Carter @_cartermp]

This thread is incredible.

Anyways, there's a reason why my laptop has a sticker that says "AI at the keyboard, human at the deposition".

[Josh @joshmanders]

Hey @Railway, is there a reason employee's of yours are trying to breach my infrastructure?

Your employee Jake Runzer, just attempted to tonight.

[Phillip Carter @_cartermp]

@rickasaurus Was looking for the neel post

[Neel Nanda @NeelNanda5]

This is a cool paper, but I think it's best thought of as Astra is surprisingly capable of doing old school mechanistic interpretability research rather than anything spooky. The things it's doing are clever, but typically exploiting structure in the data and using known method mechanistic interpretability techniques (which Astra was surely pretrained on), though it is better at this than I would have predicted!

This is an exciting development for the prospects of ambitiously scaling interpretability or doing very laborious analysis like this, on more real models. However, I'm not holding my breath. This kind of analysis was pretty common up until about 2022, it largely fell out of favor because a lot of it is just exploiting fairly clean surface-level patterns that doesn't scale to the annoying complexity of real LLMs or which only works for the more simpler and superficial properties. And as far as I can tell, basically all the case studies here are just using a bag of interpretability tricks rather than anything spooky. The piece doesn't really update me on how far old school mech interp can go, just on how good LLMs are at it. It's very fun to see the creative solutions Astra came up with though!

[Phillip Carter @_cartermp]

@rickasaurus Yep. At least for now. Apple disproportionately invests in security engineering compared to everyone else at that scale. And there's just so many easier targets to go after

[Phillip Carter @_cartermp]

FWIW I think many many engineers will overindex on the latter and believe it can improve the former, and they will be proven wrong about that

[Phillip Carter @_cartermp]

I think the near-ish term for software will look like:

- Genuinely net-new feature work still bottlenecked on vetting ideas, SDLC, internal processes, etc.
- Radically faster time to improve performance metrics for that feature with a small agent fleet shortly afterwards