Marcello Herreshoff

[Marcello Herreshoff @mherreshoff]

@metaphdor Hey. I wouldn't give up on building things. Not during the prime years of the centaur era. Yeah, there's less time but there's way more tech progress years per sidereal year near the edge of the singularity

[Marcello Herreshoff @mherreshoff]

@liron Assuming you're the *only* contestant who dispatches to AIs that way! This should mean the bar for winning a one day Hackathon has jumped way up.

[Marcello Herreshoff @mherreshoff]

What if, even before Claude Code and similar, the bottleneck for building a popular useful app was having a workable and coherent vision for how the software would be used rather than the availability of raw SWE effort?

So, before Claude code, if someone had a brilliant idea for an app that they expected to get serious accolades for, the incentives were already such that they'd have built it the hard way.

Lowering the barrier to entry is still really cool though! It will mean that more marginal ideas will get implemented, and existing software suites will grow longer tails of features. In extremis, it allows for a proliferation of bespoke software, tailored to individual people's workflows. Those will mostly look like shovel-ware from the outside, because why put in the extra effort to sand down all the corners if you're just making something for yourself?

[Marcello Herreshoff @mherreshoff]

@paulg Write neutral prompts evenhandedly in a way that obscures your preferences. Broaden questions so the sycophant can't guess which answer would please you. You will still get fabrications, but at least it won't be because you accidentally nudged its steering wheel.

[Marcello Herreshoff @mherreshoff]

So, by the look of things it understands *that* you wrote in Tibetan but not *what* you wrote. That says "Surrounded by many hosts of ḍākinīs [or goddesses, sky-dancers, was khandro]". I don't see a host of goddesses with the possible exception of the indistinct figures in the lower left render.

[Marcello Herreshoff @mherreshoff]

@diviacaroline @LeahLibresco @michaelblume Caveat: the price of a house includes the land under it, and that can appreciate (e.g. buy a house at the edge of a city, wait for the city to expand towards you, then sell it to someone who tears it down to put up an apartment complex)

[Marcello Herreshoff @mherreshoff]

@raelifin The crossover continues (as of this morning there is also now a Sci Show video sponsored by Control AI, where Hank gives an AI safety overview) https://t.co/0YGEtmzgEr

[Marcello Herreshoff @mherreshoff]

@allTheYud @lennyeusebi I might call such people slop-drunk

[Marcello Herreshoff @mherreshoff]

@robinhanson Maybe we're just using words differently, but if you can take them off when it suits you, they're not your real values, they're just sparkling social strategies.

[Marcello Herreshoff @mherreshoff]

David Foster Wallace: In my book I invented the Entertainment, an infinitely addictive video, as a cautionary tale

TikTok, OpenAI, etc.: At long last, we have created the Entertainment from the classic novel Infinite Jest

[Marcello Herreshoff @mherreshoff]

Right. I didn't mean the corrigible system's goals were meant to be beliefs about its own goals; that would indeed be circular. But a corrigible system's goal is supposed to somehow magically be an external reference to (its beliefs about) what the operators want from it. The point I thought I was trying to make was that there are situations you might want a system's goals to be instantiated as updatable belief-like objects at all, not that those belief-like objects should recurse in an ungrounded way. I suppose you could couch that as a meta-goal to do what the operators want, but the system's beliefs about what the operators want seem like they'd act like interim goals, in a way we wouldn't see in a simpler example like chess.

[Marcello Herreshoff @mherreshoff]

Agreed WRT garden variety utility maximizers, even robust embedded ones. But imagine future-Dath-Ilan builds a corrigible AI. Intuitively, shouldn't that thing be able to execute a mental move that might translate as roughly "wow, action X has some drastic consequences, and if I'm wrong about which things are good, that would be terrible, so how about I don't do that"?

That thing's drives might actually need to be much more model-entangled than most maximizers' in order to produce the sort of normative epistemic humility we'd intuitively call being a good steward. Of course, this isn't to say I think anybody including Emmett actually knows a good alternate formalism with that property, or that such an alternate basin is at all easy to hit.

[Marcello Herreshoff @mherreshoff]

@eshear @AnnaWSalamon @ESYudkowsky Like, say, I could have a belief that "drinking water is good", but by itself that's just an inert string of words unless I also have a drive that makes me want to accomplish stuff I've labelled as "good".

[Marcello Herreshoff @mherreshoff]

@eshear @AnnaWSalamon @ESYudkowsky A belief is a fact about the world that I can reason with, whether or not it's represented in words. A drive, OTOH, actually *causes* me go do something, or push on the world somehow. They don't feel like the same type of thing to me!

[Marcello Herreshoff @mherreshoff]

@AnnaWSalamon @eshear @ESYudkowsky Agreed. Preferences don't feel like they're downstream of beliefs like that. It doesn't seem true for humans. A little kid can want a glass of water and try to get it without being able to articulate or have a clear intellectual grasp on what they want.

[Marcello Herreshoff @mherreshoff]

@ben_r_hoffman @diviacaroline @jimrandomh If it's the follow-up to a heavy-handedly normative thing, "Why?" can read like more of a pile-on demand than an expression of curiosity.

[Marcello Herreshoff @mherreshoff]

@ben_r_hoffman @diviacaroline @jimrandomh My straw-Divia has a norm against heavy-handed normativity. I predict if Jim had instead said "I find Facebook unusable without ..." and deleted the second sentence where he doubled down and called using FB without it a "mistake", this exchange would not have happened.

[Marcello Herreshoff @mherreshoff]

@Malcolm_Ocean I'd guess it's same reason Cybertrucks are only grey-area street-legal in the EU. Sharp edges on vehicles are more likely to kill pedestrians in a crash.

[Marcello Herreshoff @mherreshoff]

@aleksil79 @ESYudkowsky @vlad3ciobanu Learning a generalization isn't prohibited, but it is an accuracy concession. The actual probability distribution is a superposition over different types of human. So once a few tokens get sampled it gets handed more evidence about which type of human to pretend to be.

[qbolec @qbolec]

@ramez I try to distinguish what LLM wants (not clear as of today) from what the hero of the story written by LLM called "the Helpful Harmless & Honest Agent" wants