Ivan Kwiatkowski

Security at a Big Tech company. Maintainer of Manalyze, Gepetto, and writer.
Trolling on a purely personal capacity.

[Ivan Kwiatkowski @JusticeRage]

I've been working on this for 5 years, and it's finally out! I wrote a dark fantasy book (no computers involved), and it's the hardest thing I have ever done. I'm extremely proud of the final result. (But it's in French, for now.)
https://t.co/EBRYfz6wXw

[Ivan Kwiatkowski @JusticeRage]

https://t.co/7cGwui9jsa

[Ivan Kwiatkowski @JusticeRage]

@ImposeCost I fully agree on this. What worries me is how practitioners will go from junior to senior reverse-engineers from now on.

AI powered RE will have to progress to no-supervision-necessary level, because I can't imagine many people will be able to do the checking in 20 years.

[Ivan Kwiatkowski @JusticeRage]

@ChrisAvellone @datboydnk I still remember Tyranny as one of the best CRPGs I ever played. It was so unique. I don't have high hopes to see this IP return, but I really hope you get to work on similar projects in the future!

[Ivan Kwiatkowski @JusticeRage]

@ostrag_w Bonne question ! Effectivement dans les "tâches" plus loin il y avait une demande d'argent.

[Ivan Kwiatkowski @JusticeRage]

Today, I earned 10€ by posting a fake Google review. Short thread. Sorry for the non-French speakers, I'll try to translate.

This is a random Telegram ping from a stranger. I express weird enthusiasm at the idea of writing fake reviews.

[Ivan Kwiatkowski @JusticeRage]

I actually get paid for my 5 minutes of hard work.
The end.

[Ivan Kwiatkowski @JusticeRage]

I *do* write the review: one star, mentioning that that the restaurant buys fake reviews, but also that their service is terrible and that I heard the chef make racist comments and spit in the soup.

[Ivan Kwiatkowski @JusticeRage]

The platform is built powdered IDA Pro, DM me if you'd like to know more!

[Ivan Kwiatkowski @JusticeRage]

I have been working on an AI malware analysis platform for a while now, and it works really well.
This is the output for a Winnti sample (0c45413122e68f4397fba9539fb74a67343a6496672c1a55862f95e2bcb105c3) posted by @cyb3rops recently.

Full report: https://t.co/dj6RXMhdP2

[VirLabs @VirLabs_AI]

VirLabs analyzed a Windows kernel driver that behaves like a stealth/cheat-support/rootkit-style component: 0c45413122e68f4397fba9539fb74a67343a6496672c1a55862f95e2bcb105c3

No IOCTL cmd surface. Instead, a covert local control channel hidden behind registry-set notifications.

[Halvar Flake @halvarflake]

I could not disagree more.

[Ivan Kwiatkowski @JusticeRage]

@halvarflake Thanks a lot for sharing this! I'm in this process right now and those thoughts are really enlightening.

[Ivan Kwiatkowski @JusticeRage]

This only makes sense if you believe SOTA models are a requirement to do impactful security research. Which is provably false.

If anything, this proves Anthropic scored an own goal when its marketing got accepted as a fact by US leadership.

https://t.co/U5EvxFWBfz

[Nando de Freitas @NandoDF]

No one should be surprised by this. The USA is doing what any self-interested nation state would do.

The real question is why are Europe, Canada, Australia, Korea, Japan and UK not able to compete seriously. That is the question everyone in government needs to answer.

And no, having a couple of startups that have raised $1B or $2B is FAR from enough to compete with $100B American companies. The scale matters. Imagine your sword’s length is 1cm and your rival’s 1m — no match.

Here is the harsh math (thanks to a poor version of Claude):

•10,000 GB200 superchips ≈ ~278 NVL72 racks.
•Each NVL72 rack costs roughly $3M–$3.5M.
•That puts the full-system total around $830M–$970M, before networking, power, cooling, and datacenter buildout.

That would enable you to train a model that was Sota 2 years ago. You need about 5 to 7 times this to compete today.
So the starting bill is $5B, but even if you have this, here is the reality: there’s no available chips. So when you hear someone raised $1B, remember this is going back to American compute, and is simply not enough.

The other two ingredients for AI are data and people.

American startups pay better than European ones, so the people vote with their feet so they can pay their mortgage and send kids to school. An experienced AI engineer makes double the salary in Europe by working for an American startup (like Anthropic) than a European one, and about ten times more if they work for a USA corporation. There are however amazing European startups, but the money and ambition is lacking.

The USA is far more relaxed with data and fair use - Canada is good too and @cohere is doing fine thanks to this. So American companies have a strong advantage over European ones. Brussels and the UK think they can hold the world to their questionable “ethical” views on data but they are just destroying the local AI industry, and in the process falling into a very precarious situation. They are partly responsible. Only the French minister has stood by their local LLM @MistralAI … and I guess more recently Germany has started to wake up.

The hope is of course LLM startups like @MistralAI and @cohere which are a year or so behind but can provide personalised services, and amazing startups like @cusp_ai @IneffableLabs @nscale @Orbital_Ind @bfl_ai and a few others. But for all these, it’s incredibly hard to compete.

[Ivan Kwiatkowski @JusticeRage]

@matrosov Definitely agree. Software protection needs a new paradigm. I don't think anybody know what it is right now.
It's way too early to tell, but we may have reached the end of closed source as a concept.

[Ivan Kwiatkowski @JusticeRage]

In the past months, I have advocated where I could to stop investing in obfuscation as a protection mechanism.
It was good at frustrating humans but machines just don't care that much. Packing is more effective so far but I don't expect this will last.

[Alex Matrosov @matrosov]

Previous generations of software protection (DRM perspective) have always relied on code complexity (for RE), compute limitations, and human limitations as the guarantees that kept hacking timelines reasonably long.

That's changed now. Beyond the acceleration in vulnerability research and malware analysis, the same new reality applies to software protection, and security by obscurity, or assuming the attacker is limited in compute and motivation, no longer works.

[Ivan Kwiatkowski @JusticeRage]

@0x464D @matrosov It's hard to represent how many tokens you can blow before getting remotely close to a senior reverser's hourly rate.

[Christopher | 🇺🇦  @d4rkshell]

Microsoft has banned Nightmare Eclipse from GitHub: https://t.co/EmeiJnJ0Ps

This is the researcher who disclosed several zero-days after Microsoft also deleted their MSRC account.

They have now moved on to GitLab: https://t.co/Npj0gplSum

(h/t to: @[email protected])

[Ivan Kwiatkowski @JusticeRage]

@ryanaraine @craiu @juanandres_gs @wearetlpblack My intuition is that the really serious adversaries should have the capability to run in-house unshackled models that are 90% as good as the frontier ones.
There will be no possibility of detection there, and IMO a lot of the threat intel work will go against distillation.

[Ivan Kwiatkowski @JusticeRage]

@GrapheneOS Bad actors don't buy old unpatched devices to exploit them and automate. They install custom ROMs that give them full permissions from the start. That's the type of threat I'm facing every day and that KA helps solve for me.

[Ivan Kwiatkowski @JusticeRage]

The truth is, when you're fighting bad behavior at scale, proof that users are on stock OS / devices where obtaining root privileges is non-trivial has value. You still have problems with keyboxes etc. but keys can be revoked as they become widespread until RKP is adopted.

[GrapheneOS @GrapheneOS]

Apple and Google are gradually expanding their use of hardware-based attestation. They're convincing a growing number of services to adopt it. Google's Play Integrity API and Apple's App Attest API are very similar. Apple brought it to the web via Privacy Pass, which Google intends on doing too.

Google's Play Integrity API requires hardware attestation for the strong integrity level and is gradually phasing in requiring it for the more commonly used device integrity level. Apple already has it as a requirement. Over the long term, this will increasingly lock out hardware and OS competition.

The purpose of these systems is disallowing people from using hardware and software not approved by Apple or Google. This is wrongly presented as being a security feature. Banks and government services are the main ones adopting it but Apple and Google are encouraging every service to use it.

Apple's Privacy Pass brought hardware attestation to the web to help with passing captchas on their own hardware. Many people saw that as harmless since few sites would be willing to lock out non-Apple-hardware users. Apple and Google are both likely to bring broader hardware attestation to the web.

Google's reCAPTCHA is planning an approach where they use Privacy Pass on Apple hardware, their own approach on Google Mobile Services Android devices and a QR code scanning system to require an iOS or Google certified Android device for Windows and other systems:

https://t.co/7rQnioRa8A

Banking and government services increasingly require using a mobile app where they can use attestation to force using an Apple or Google approved device and OS. Apple's privacy pass, Google's 'cancelled' Web Environment Integrity and now reCAPTCHA Mobile Verification are bringing this to the web.

Current media coverage for reCAPTCHA Mobile Verification misunderstands it and the impact of it. They're bringing a hardware attestation requirement to Windows, desktop Linux, OpenBSD, etc. by requiring a QR scan from a certified smartphone to pass reCAPTCHA in some cases. They could expand it more.

Control over reCAPTCHA puts Google in a position where they can require having either iOS or a certified Android device to use an enormous amount of the web. Google defines certification requirements for Android which includes forcing bundling Google Chrome, etc. It's enormously anti-competitive.

Google's Play Integrity API bans using GrapheneOS despite it being far more secure than anything they permit. It also bans using any other alternative. This isn't somehow specific to an AOSP-based OS. You can't avoid this by using a mobile OS based on FreeBSD instead. You'll just be more locked out.

Google's Play Integrity API permits devices with no security patches for 10 years. The device integrity level can be bypassed via spoofing but they can detect it quite well and block it once it starts being done at scale. The strong integrity level requires leaked keys from TEEs/SEs to bypass it.

It doesn't provide a useful security feature, but it does lock out competition very well. Services requiring Apple App Attest or Google Play Integrity are primarily helping to lock in Apple and Google having a duopoly for mobile devices. Play Integrity is more relevant due to AOSP being open source.

Governments are increasingly mandating using Apple's App Attest and Google's Play Integrity for not only their own services but also commercial services. The EU is leading the charge of making these requirements for digital payments, ID, age verification, etc. Many EU government apps require them.

Instead of governments stopping Apple and Google from engaging in egregiously anti-competitive behavior, they're directly participating in locking out competition via their own services. Requiring people to have an Apple device or Google-certified Android device is anti-competition, not security.

reCAPTCHA Mobile Verification will currently work with sandboxed Google Play on GrapheneOS but it clearly exists to provide a way for them to start using hardware attestation on systems without it. People without an iOS or Android device will be locked out when this is required even without that.

This isn't about security or any missing functionality. GrapheneOS can be verified via hardware attestation. Google bans using GrapheneOS for Play Integrity because we don't license Google Mobile Services and conform to anti-competitive rules already found to be illegal in South Korea and elsewhere.

Services shouldn't ban people from using arbitrary hardware and operating systems in the first place. Google's security excuse is clearly bogus when they permit devices with no patches for 10 years but not a much more secure OS. It's for enforcing their monopolies via GMS licensing, that's all.

[Ivan Kwiatkowski @JusticeRage]

GrapheneOS is right about Google / Apple's implementations of key attestation (KA) becoming dangerously close to vendor lock in.
BUT app devs can also roll their own KA and chose whose keys they want to trust.
Graphene does a great job publishing theirs.