ArchiveBox

Self-hosted open-source internet archiving solution ⑊ Maintained by @theSquashSH ⑊ #webarchiving #internetarchiving #digipres #foss

[ArchiveBox @ArchiveBoxApp]

✨ Check out our new oneshot archiving CLI!

$ uvx abx-dl 'https://t.co/S7IoIUpftH'

Get the full power of ArchiveBox in a one-liner, usable by agents and humans alike.

🧩 Ships with 57 plugins to power extraction of many different content types:
https://t.co/eKLZ7PeQlV

[Charlie Marsh @charliermarsh]

pull_request_target, every time

[ArchiveBox @ArchiveBoxApp]

@daniel31x13 I mean I get where they are coming from, having too many bots degrades the value of the platform overall.

I just wish we skipped straight to the "money" option instead of going with options that ruin privacy. Arguably hardware attestation is the best we could ask for right now.

[ArchiveBox @ArchiveBoxApp]

This is just a stepping stone. The endgame everyone is moving towards is requiring:

- proof of govt Id (kyc)
- proof of biometrics
- money

[GrapheneOS @GrapheneOS]

Apple and Google are gradually expanding their use of hardware-based attestation. They're convincing a growing number of services to adopt it. Google's Play Integrity API and Apple's App Attest API are very similar. Apple brought it to the web via Privacy Pass, which Google intends on doing too.

Google's Play Integrity API requires hardware attestation for the strong integrity level and is gradually phasing in requiring it for the more commonly used device integrity level. Apple already has it as a requirement. Over the long term, this will increasingly lock out hardware and OS competition.

The purpose of these systems is disallowing people from using hardware and software not approved by Apple or Google. This is wrongly presented as being a security feature. Banks and government services are the main ones adopting it but Apple and Google are encouraging every service to use it.

Apple's Privacy Pass brought hardware attestation to the web to help with passing captchas on their own hardware. Many people saw that as harmless since few sites would be willing to lock out non-Apple-hardware users. Apple and Google are both likely to bring broader hardware attestation to the web.

Google's reCAPTCHA is planning an approach where they use Privacy Pass on Apple hardware, their own approach on Google Mobile Services Android devices and a QR code scanning system to require an iOS or Google certified Android device for Windows and other systems:

https://t.co/7rQnioRa8A

Banking and government services increasingly require using a mobile app where they can use attestation to force using an Apple or Google approved device and OS. Apple's privacy pass, Google's 'cancelled' Web Environment Integrity and now reCAPTCHA Mobile Verification are bringing this to the web.

Current media coverage for reCAPTCHA Mobile Verification misunderstands it and the impact of it. They're bringing a hardware attestation requirement to Windows, desktop Linux, OpenBSD, etc. by requiring a QR scan from a certified smartphone to pass reCAPTCHA in some cases. They could expand it more.

Control over reCAPTCHA puts Google in a position where they can require having either iOS or a certified Android device to use an enormous amount of the web. Google defines certification requirements for Android which includes forcing bundling Google Chrome, etc. It's enormously anti-competitive.

Google's Play Integrity API bans using GrapheneOS despite it being far more secure than anything they permit. It also bans using any other alternative. This isn't somehow specific to an AOSP-based OS. You can't avoid this by using a mobile OS based on FreeBSD instead. You'll just be more locked out.

Google's Play Integrity API permits devices with no security patches for 10 years. The device integrity level can be bypassed via spoofing but they can detect it quite well and block it once it starts being done at scale. The strong integrity level requires leaked keys from TEEs/SEs to bypass it.

It doesn't provide a useful security feature, but it does lock out competition very well. Services requiring Apple App Attest or Google Play Integrity are primarily helping to lock in Apple and Google having a duopoly for mobile devices. Play Integrity is more relevant due to AOSP being open source.

Governments are increasingly mandating using Apple's App Attest and Google's Play Integrity for not only their own services but also commercial services. The EU is leading the charge of making these requirements for digital payments, ID, age verification, etc. Many EU government apps require them.

Instead of governments stopping Apple and Google from engaging in egregiously anti-competitive behavior, they're directly participating in locking out competition via their own services. Requiring people to have an Apple device or Google-certified Android device is anti-competition, not security.

reCAPTCHA Mobile Verification will currently work with sandboxed Google Play on GrapheneOS but it clearly exists to provide a way for them to start using hardware attestation on systems without it. People without an iOS or Android device will be locked out when this is required even without that.

This isn't about security or any missing functionality. GrapheneOS can be verified via hardware attestation. Google bans using GrapheneOS for Play Integrity because we don't license Google Mobile Services and conform to anti-competitive rules already found to be illegal in South Korea and elsewhere.

Services shouldn't ban people from using arbitrary hardware and operating systems in the first place. Google's security excuse is clearly bogus when they permit devices with no patches for 10 years but not a much more secure OS. It's for enforcing their monopolies via GMS licensing, that's all.

[Nick Sweeting @thesquashSH]

whipped up a crazy little experiment: zfsbox

Run ZFS on macos and linux *without* needing to install the ZFS kernel module on the host.

(runs zfs in a vm and shares the volumes with the host via NFS)

[Nick Sweeting @thesquashSH]

Let's try to get a Targets.getForegroundTargetId method added to CDP!

https://t.co/6TQZTnawV4

[ArchiveBox @ArchiveBoxApp]

💸 ArchiveBox is now supporting ArchiveTeam Warriors on OpenCollective!

AT coordinates distributed archiving efforts and has helped save millions of URLs from Blogger, Vine, Imgur, Tumblr, Telegram, and more...

Consider supporting them too! https://t.co/CT22yJcKP7

[Linkwarden @LinkwardenHQ]

👋 Linkwarden 2.14 is out and we just announced it on Reddit, Lemmy, and our Blog! 🚀

This release brings:
• 🗂️ improved team collaboration
• 🏷️ tag pagination
• ⚡ faster UI
• ✨ better search and settings UX
• 🚀 Next.js 15 + Expo 54 upgrades

Read more 👇

[ArchiveBox @ArchiveBoxApp]

@2Aorg @EFF ArchiveBox is working on building this! We're eventually going to add P2P sharing of archives with a "pinning" feature similar to this, except using BitTorrent not IPFS.

We just have to solve this problem first: https://t.co/A4NgD75gUm

[ArchiveBox @ArchiveBoxApp]

@RelIrrelvantVIP @EFF ArchiveBox is working on this!

[EFF @EFF]

For nearly 30 years, journalists have relied on the Internet Archive to see how stories were originally published, before edits, removals, or changes. We need to safeguard that. https://t.co/WjLMSMhMnz

[ArchiveBox @ArchiveBoxApp]

https://t.co/sRnpAJItqU

[ArchiveBox @ArchiveBoxApp]

📑abx-dl and ArchiveBox now support both liteparse and opendataloader as built-in plugins to extract text from PDFs and other documents.

Check it out in our new plugin gallery (link below)!

[Tuana @tuanacelik]

We just open-sourced LiteParse 🎉

A lightweight, local document parser in the shape of an easy-to-use CLI. No API calls, no external service, no cloud dependency. Just fast text extraction from common file formats, right from your terminal.

It's built for developers who want parsing that stays on their own infrastructure and gets out of their way. Clean PDFs, DOCX, HTML: run it, get your text, move on. The output is designed to be fed straight into agents so they can read parsed text and reason over screenshots without any extra wrangling.

When you hit more complex territory like scanned docs, dense tables, or multi-column layouts, that's where LlamaParse picks up. Same philosophy, more horsepower for the hard stuff.

📖 Announcement post: https://t.co/VRJkLZYiwP
🔗 GitHub: https://t.co/PuSo8ZMFtl
🎬 Walkthrough: https://t.co/VPG8c3GLZ8

[ArchiveBox @ArchiveBoxApp]

https://t.co/SmiyhN5qXB

[ArchiveBox @ArchiveBoxApp]

📑 ArchiveBox and abx-dl now include an OpenDataLoader plugin, enabled by default!

This finally allows us to make archived PDFs and images searchable in the full-text index!

[Oliver Prompts @oliviscusAI]

someone just open-sourced a tool that converts pdfs to markdown at 100 pages per second.

100% free. runs entirely on cpu. no expensive gpus needed.

[AlternativeTo @AlternativeTo]

Open source self-hosted internet archiving tool to collect, save, and view websites offline, including HTML clones, screensthos, PDF, and WARC files https://t.co/mkYNWU2LhV @ArchiveBoxApp

[Charlie Marsh @charliermarsh]

@davegaeddert I also built this which generalizes the pattern though haven't mentioned it publicly anywhere: https://t.co/GywNk7cmvx

[ArchiveBox @ArchiveBoxApp]

Lots of massive improvements on dev recently, 0.9.0 may land by new years!

[ArchiveBox @ArchiveBoxApp]

Check out this excellent ArchiveBox setup guide for porting over Pocket data 🏷️ b4 OCT 1st SHUTDOWN

https://t.co/YO4tZvG0qX

[David Gewirtz @DavidGewirtz]

This was a fun project. https://t.co/Yh8UXW6rNl

[David Gewirtz @DavidGewirtz]

This was a fun project. https://t.co/Yh8UXW6rNl