Here what not do to if you don't want to get ranked. 👉 Make public docs reachable by ChatGPT Search's verified crawler if you manage a site behind Cloudflare. You'll need control of robots.txt, access to the Cloudflare zone, and public documentation URLs to check.
Inspired by a developer report of traffic claiming to be OAI-SearchBot; the traffic claim is unverified: https://t.co/D7R7qDKLln
OpenAI's crawler guidance: https://t.co/y3b1thgf32
1. Choose the documentation URLs you want reachable. Check that each serves readable content with HTTP 200. In Cloudflare Security Events, look for blocks or challenges on those paths and record the source IP and responsible rule.
2. Set Search to "Allow (do not block)" in Cloudflare's AI bot policies. If AI Crawl Control is enabled, set OAI-SearchBot's individual action to Allow for the intended content. Check both settings in the dashboard.
3. Allow OAI-SearchBot to reach those paths in robots.txt. Inspect the file served through Cloudflare, which may prepend managed content to your origin file. Check the served policy for the selected paths; set any GPTBot training preference separately.
4. Narrow a custom block or challenge rule if Security Events identify it as the cause. On the affected paths, scope that rule to requests matching `not https://t.co/pM7U54Aq0u`, then check later events for verified crawler requests. A user-agent claim alone does not verify a bot. If Bot Fight Mode caused the block, a custom-rule Skip cannot bypass it.
5. Review later events and response logs for successful visits to those pages. Compare source IPs with OpenAI's current published OAI-SearchBot ranges; Cloudflare Security Events may be sampled. After a robots.txt change, allow about 24 hours for it to affect search results. Successful access makes a page eligible for ChatGPT Search, but does not guarantee a citation.
That's it. Don't do it.
Zuck says Muse Spark Open’s weights have been released and that the model runs well on new Mac Studios.
The hardware behind that claim deserves attention. Meta says its local setup uses two M5 Ultra Mac Studios, while Apple describes connected machines sharing the inference work. That gives teams a specific configuration to evaluate. It also means budgeting for and operating more than one computer. For an organization that wants inference on its own hardware, the question is whether that control is worth the machines and the work of running them. Someone with a single Studio cannot assume they can reproduce the published setup.
Apple measured 24.99 output tokens per second with two Studios and 36.73 with four in an 8,192-token-context benchmark. Four machines produced a higher rate in that test, though doubling the hardware did not double the output rate. A team handling simultaneous requests would still need to measure its own prompts, response times and operating costs before deciding whether the extra machines pay off. Meta also says the model supports a one-million-token context window; Apple’s cited speed figures come from a much shorter test. I would want measurements closer to the intended workload before making a deployment decision.
Released weights let teams run and inspect the model, subject to Meta’s Muse Model License. It requires attribution, and certain MaaS providers above a stated revenue threshold need an additional agreement. A team testing locally and one planning to sell access should read those terms with different plans in mind. Zuck’s assessment and Apple’s measurements make this setup worth examining, but they do not establish performance across other workloads or independent replication. I’d start with the two-Studio configuration, test the requests the team actually expects, and judge any move to four against the gain.
Apple: https://t.co/IpdHRKfNFJ
Meta model card: https://t.co/irjFko7nfS
WebPlanex promotes llms.txt as a way to make store information more likely to appear in AI answers. Giving an agent a structured guide to a store sounds plausible. The evidence for a general visibility boost is thin.
Google says Search ignores llms.txt, including for AI Overviews and AI Mode. The file neither helps nor harms visibility there. Google Search Central: https://t.co/IQCmT7xXcj
SE Ranking analyzed nearly 300,000 domains and found no correlation between having the file and AI citation frequency. Its prediction model improved when file presence was removed. SE Ranking: https://t.co/9cnvOJyl8T
Chrome’s Lighthouse still recognizes llms.txt as an optional convention for agents. An agent that fetches and uses it may benefit; the citation study cannot rule out that narrower case. My read: promising more AI citations from adding the file stretches the evidence. Treat it as optional, make important pages accessible, and measure citations or referrals before crediting it for a gain.
Crawlable infinite scroll for developers maintaining public listings will lead to massively improved discoverabillity. Here's how I implement it:
You need control of the listing routes and client loading code, stable page batches, and Search Console access. Inspired by Impression's ecommerce implementation: https://t.co/BlAb3im2wR
1. Serve each batch at a distinct URL, such as ?page=2. Render its items when that URL opens directly, and return a real 404 for an out-of-range page. With JavaScript off, open a later page and check that its items appear.
2. Link the pages in the HTML with ordinary <a href> links, including a way forward and back. With JavaScript off, follow them from page one and check that each destination shows its own batch. Google's current guidance no longer uses rel="next" and rel="prev" for pagination.
3. Set each page's rel="canonical" to its own URL, including the page number. Check several pages for distinct canonicals. Use page URLs instead of # fragments, and avoid pointing every page's canonical at page one.
4. Add scrolling or load-more behavior over those pages. Fetch and append the next batch, update the URL with history.pushState(), and handle popstate to restore the listing on Back and Forward. Copy an updated URL into a fresh visit, then test both history directions; the URL and displayed items should agree.
5. Inspect a later URL in Search Console. Check its returned and rendered HTML for the items and links, then compare its declared canonical with Google's selected canonical after crawling. Google's pagination guidance: https://t.co/lX9KGd1foS. A passing live test does not guarantee indexing.
Here's what you should be doing to make public pages eligible for ChatGPT Search discovery when your site runs behind Cloudflare.
1. Check robots.txt for each intended hostname in AI Crawl Control's Directives tab. If it's missing, serve one at the hostname root; if it's blocked, fix the security rule. Confirm the served file responds successfully. Cloudflare may prepend managed directives to an origin file.
2. Set the policy in the robots.txt visitors actually receive. Permit OAI-SearchBot on the chosen public paths and decide GPTBot's training access separately. Check the combined file for wildcard rules or disallows that conflict with either choice. OpenAI documents the two crawlers here: https://t.co/y3b1thgf32
3. Set OAI-SearchBot to Allow in AI Crawl Control's Crawlers controls. If you exclude training crawls, set GPTBot to Block. Check that each named crawler shows the intended action.
4. Review Cloudflare Security Settings and Security rules for a Search AI bot policy or WAF rule that would block OAI-SearchBot on those paths. Adjust conflicting rules and check that none still denies the search crawler. AI Crawl Control's Allow action cannot override another block.
5. Filter AI Crawl Control Metrics by OAI-SearchBot, hostname, path and response status. Check observed requests to intended pages for successful responses and investigate failures. No requests in a short window are inconclusive: OpenAI says robots.txt changes may take about 24 hours to take effect. Access does not guarantee citations.
Today, Cloudflare and @OpenAI released the first findings from their joint research pilot, demonstrating how real-time network signals can make AI search drastically smarter, fresher, and safer without overloading website infrastructure. https://t.co/ZqY2lh4tyh
Amazon says Bee's optional paid link can suggest products from conversations. Imagine 'I need space' getting bins, then your breakup filed as a recurring household expense. https://t.co/bauFFVLB4u
Anyone doing it cheaper/better for the same quality? I'd love to hear how. ElevenLabs Receptionist is cool, but it ain't cheap.
It's nice to be handed a warm call and watch your CRM update in real-time with everything you need to know about the customer while you get the finer details from them.
And you don't have to use Claude for this. The only reason I do is they give $100 api credit on the pro plan. Each call uses about 3-cents of that. So 3000+ calls a month for nothing additional out of pocket. And let's face it Opus 5.5 is super smart and figures out the mumble callers contact info 99% of the time😅.
AI phone agents from Vapi, Retell, ElevenLabs, Bland list at 7–49¢ a minute. Human services run $3.45–$5.
The intake agent I build for local service companies costs ~5¢ a minute, all in.
Every step to build your own, plus the real bill from 16 calls 👇 https://t.co/kFFHRpL8o7
@perplexity_ai If a task tries to send data to an unapproved domain, a rule outside its microVM could block the request. The security review then needs to cover who can change that rule.
AI phone agents from Vapi, Retell, ElevenLabs, Bland list at 7–49¢ a minute. Human services run $3.45–$5.
The intake agent I build for local service companies costs ~5¢ a minute, all in.
Every step to build your own, plus the real bill from 16 calls 👇 https://t.co/kFFHRpL8o7
@_JoshuaSEO If an old product page redirects to a general collection, the 301 can work while the destination misses what the searcher wanted. Reviewing where each redirect lands matters as much as confirming it fires.
Routine disavowal of backlinks labeled “toxic” can sound like sensible SEO maintenance. LinkResearchTools recommends regular cleanup even without a manual action, and DTC SEO Agency presents disavowal as a way to improve rankings.
Google says most sites don’t need the tool. Its guidance calls for a considerable number of spammy, artificial or low-quality links plus an actual or likely manual action. It warns that disavowing the wrong links can hurt search performance. Google Search Console Help: https://t.co/Gz9cVOl1HS
Ahrefs disavowed 129 flagged links across three posts. Over the next 20 days, Search Console traffic fell 7.1%, while its visibility measure changed little overall. Results varied by page. The test had no randomized control, and a core update limited its duration, so the traffic drop cannot be attributed to disavowal. It found no consistent boost either. Ahrefs: https://t.co/0Zj2HmfszX
I’d treat a toxicity score as a reason to inspect the links, not as Google’s verdict on them. Check your link history and Search Console’s Manual Actions report. Consider disavowal when substantial manipulative links meet Google’s criteria, rather than submitting a file to improve a vendor score.
@thecsguy I quit selling there due to it. Everybody knows how to abuse it. Now people just scan their return barcode at the ups self serve kiosk without returning anything and get their money back in an hour.
Bros got a whole bot dedicated to his Amazon returns.
I hope Elon creates something that doesn't allow returns for any reason. The amount of return abuse Amazon sees must be staggering. I think any retailer that can pull this off can immediately slash 20% off their prices. Everyone wins but the serial returners.
amazon is blocking grok @bots from ordering, and elon's answer is just have your bot buy it somewhere else.
one of my grok bots only handles my amazon returns, so i'm watching this one closely.
would you switch stores for your bot? 🛒
Elon, i think its time for an X Shop? 👀