I personally strongly support many of the ideas in this report including all of the ones Sneha mentions, and plan to do what I can to advocate for them! Urgent action is needed and these proposals are very sensible responses to the current situation in my opinion.
- Many websites are deliberately inaccessible to the search engines that LLMs use by default. For example, a lot of important information is on Twitter but most LLMs (besides Grok) have trouble reading Twitter. I've added Exa and Playwright to my LLM but most people will use default settings.
- Some websites have important information in images, which LLMs skip over unless they're making an effort.
- What results appear in response to common queries absolutely impact the answers that LLMs provide.
- New website domains have limited visibility in search engines.
Don't just write for humans. Write for the AI agents that are informing the humans. Do SEO for AI agents. Info dump in a machine-readable format and don't leave essential info in images. Don't just build a microsite that AI search will never find.
Governor Gavin Newsom announced that GovOps, in consultation with Cal OES and four experts, has released the report "AI Safety in 2026 and Beyond."
It recommends mandatory baseline standards, independent verification, emergency shutdown capabilities, and the power to cap RSI. An important read for anyone working on the technical or policy side of frontier AI.
Next paragraph:
> Indeed, limitations call us to solidarity and openness to the transcendent. While the myth of self-sufficiency traps us in selfishness and leads us to cast aside the weak, recognizing our fragility makes us attentive to the needs of all. Saint Turibius of Mogrovejo experienced physical limits during his grueling journeys; he fell ill, suffered from exhaustion and faced the misunderstanding of the powerful. Moreover, it was precisely through a life so sorely tested that he transformed the ecclesial and social history of an entire country. Today it is up to us to follow his example!
That's the point here
"The weights also cannot reveal who owned the chips or where they were located. Blackwell systems can be rented from data centers outside China. The evidence concerns access to the hardware; by itself, it does not establish that chips were smuggled into China or that an export-control violation occurred."
right, existing export controls don't cover offshore cloud rentals
@Afinetheorem If you're writing about the virtues of Lakota science, I want to hear it in its own terms instead of through Western 20th century buzzwords like "multidimensional," "interdependence," "interrelation," and "interconnection."
🔐 Cal OES and the California Council on Science and Technology brought experts and partners together for the California AI Defense & Response Summit to strengthen cyber defenses, improve information sharing, and build California’s cyber resilience.
Currently I still only know of 1 alignment-specific RL environment company: making RL envs that aim to represent situations in real deployments in which alignment is challenging.
It seems like a huge mistake in hindsight that we have seen so much encouragement and funding to, for example, start 3rd party evaluators, but not comparable encouragement to start alignment data companies.
I do think a lot of traditional AI safety views underestimate the importance of data, so perhaps this is related. But the Hugging Face incident seems squarely downstream of both (a) reward hackable RL envs in general (b) much less effort and $ put into RL envs where alignment is a core component of reward.
A separate idea is alignment by default: the old idea of "find the persona basin which is aligned" and things will just work. This is clearly not guaranteed under increased RL. But my sense is there has not been a huge amount of effort to figure out how custom environments could help "nudge" the model back into good persona basins during extensive RL. Here too, a robust ecosystem of alignment data companies could help significantly.
In the RL data market, there is a trend for some labs to in-house slightly more RL env production. But I don't expect this to fully disappear, and I am convinced the market-based approach to sourcing RL envs produces the best overall volume and quality—for alignment RL, an ecosystem of startups would also greatly help make behind-frontier models safer.
When I'm thinking about pathway to extinction caused by AI – RAND examines pandemics, nuclear weapons, and geoengineering – I feel very thankful that AI agents so far have not been *that* misaligned. There's a significant gap in malice needed between the Hugging Face hack and human extinction. But we need much better alignment evals to be sure.
https://t.co/RddPe5W4yT
Apply to the Senior AI Fellow position at Cal OES! I'm hiring people with strong technical backgrounds to do the ambitious, thoughtful AI policy implementation that we need for America to survive a potential AI capabilities explosion. Priority deadline is Sunday, November 15.
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond.
Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks.
With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented.
That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions.
"Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.