We are deeply sorry for the disruption this incident has caused our customers and partners. We know the trust they place in our services and take that responsibility extremely seriously. We are seeing broad recovery and stability across our services.
This was a technical issue not related to cyber or AI / agents. The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.
Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration.
We will provide a full post-incident report once our review is complete.
System design is not important anymore - if you believe this, just wait until your 2-hour agent run fails at step 95 of 100 :) I hope it does not happen, but it actually might.
Agentic apps have two classes: short-running and long-running. When you are building short agents, a simple retry loop is enough. Each LLM call is isolated, cheap, and stateless. Fail, retry, done.
But retrying a long-running agent spanning hours is expensive in API costs, tool calls, and wall clock time, and retries can also have side effects like resending notifications. Some failures that a long-running agent might hit in production are:
- LLM provider rate limiting
- Network timeouts for downstream calls
- Pod eviction due to OOM kills mid-run
- Pod rotation during deployment
- Context window overflow
- Human-in-the-loop pauses for days
Notice how most of these are not AI problems. They are classic software engineering problems that are showing up in a new system.
This is where we need checkpointing, idempotency, durable event logs, and resumable workflows. These are backend engineering fundamentals where we built robust, fault-tolerant systems by applying hundreds of patterns.
Good software engineering practices do not go out of fashion. Because someday, you will ship your "tutorial" to production. That is when system design suddenly matters again.
Fundamentals remain fundamentals, even when the abstraction layer changes.
47 Simple Steps to Go From $0 to Selling Your Company (only takes 16 years to complete!)
1. Watch The Social Network on your 14th birthday, startups seem pretty cool
2. End up working at Chipotle because you don’t know anything about building
3. Finish high school early, intern at a law firm before college. On track to become good asian son
4. Hate it. Go to college to become a poet instead
5. Meet co-founder at UCSD. Remember love for startups. Convince him to build a company with you
6. Fail 5 startups in a row. Delete years of your life and exhaust all savings
7. Transition to part-time student and work at sushi restaurant to afford tuition
8. Convince random stranger on plane to invest $10k. Apparently you can just ask for things
9. Drop out of college. Startup #6 fails. Return the $10k
10. Start Jenni AI
11. Cold call 8 hours a day to hit $2k MRR
12. Get rejected by every accelerator, go completely broke, move back in with parents
13. Covid hits. Company going to die
14. Get $100k from Jason Calacanis. Immediately fly to Malaysia to extend runway
15. Run out of money again
16. Fly to Korea, win startup funding, almost go bankrupt anyway
17. Get emergency $250k investment from VC
18. Random viral Twitter thread 10x's revenue overnight
19. Pivot to EdTech, figure out how to go viral, hit $1M ARR
20. Get diagnosed with cancer
21. Get thyroid removed, cancer free
22. Get discharged, GPT-4 comes out, then Google releases AI on google docs, then Microsoft announces they’re adding AI to Microsoft word. All in same week
23. Go straight back to work
24. Hit $2M ARR on your 27th birthday
25. Hit $3M ARR 67 days later
26. Hit $4M ARR 87 days later
27. Hit $5M ARR 38 days later
28. Almost sell your startup. Change your mind at the last second.
29. Co-founder leaves the company
30. Girlfriend breaks up with you
31. One of your closest friends passes away
32. Startup stops growing
33. Feeling pretty bad
34. Retreat back into work, double down on Jenni
35. Acquire companies
36. Add key hires to the team
37. Hit $10M ARR
38. Meet Digital Science, realize they are the perfect buyer
39. Go through Due Diligence
40. Turn 30 years old
41. Spend an ungodly amount of money on lawyers and bankers
42. Jenni AI acquired by Digital Science 🥳
43. Coincidentally, Social Reckoning, sequel to Social Network, comes out 16 years later on the same week of your acquisition. Kinda crazy
44. Wire hits your bank account at 3AM
45. Stare at it for two hours
46. Text mom good news
47. Finally get a good nights sleep :)
@mmunozpz@invokebio Welcome to X, @mmunozpz
Great to see the intersection of AI and healthcare being explored to make a real-world impact. Looking forward to following your work and learning from your journey.
Wishing you all the best
I'm begging you to try systems programming.
I built tiny AWS - a cloud using no Docker, no VMs, just 10,000 lines of Golang and Rust code with this same mindset. Tiny AWS is legit just cli, and not even React or NextJS codes.
Stop spending all your time learning another framework, another library, another abstraction. Go figure out what actually happens underneath.
- Write your own memory allocator.
- Build an operating system kernel.
- Implement a TCP stack.
- Write a compiler.
- Build a database engine.
- Understand how Linux schedules processes, manages memory, handles interrupts, and talks to hardware.
Learn C. Get your hands dirty with Rust. Read source code. Read the Linux kernel. Break things. Spend time debugging a segmentation fault because you forgot how pointers work.
And then experience the satisfaction of finally understanding WHY something works, instead of just knowing WHICH API to call.
We're entering an era where AI can generate thousands of lines of code in seconds. Frameworks are getting easier. Abstractions are getting higher. The barrier to shipping software is collapsing.
But here's the interesting part:
The easier it becomes to generate code, the more valuable it becomes to understand what's happening underneath it.
Anyone can ask an AI to build an application.
But can you explain WHt happens when a process gets created? How memory is allocated? What happens when a packet travels through a network? Why a database quEry becomes slow?
Can you debug something when the abstraction leaks and the documentation stops helping?
That's the difference between knowing how to use technology and understanding how to engineer it.
You don't need to become a kernel developer. You don't need to abandon web development. And you certainly don't need to reinvent everything.
Just spend some time building things that force you to think.
Because there's something incredibly beautiful about programming when you stop treating computers as magical boxes that execute your JavaScript.
Don't just learn to build software. Learn how the machine works.
I'm telling you, try systems programming at least once.
I've spent 3 weeks connecting 40+ workflows and turning them into agents.
This company spent years automating everything they could. They have a buttload of workflows to do different things.
If you know the steps, automating a process is pretty simple. You can build a script that runs on an input and does whatever you want.
But agentic systems are so much more than that!
You don't have to orchestrate every step of a process. You don't have to plan every possible path. Instead, you give the agent a goal and access to the tools it needs.
That's it!
To this day, n8n remains one of the simplest ways to build automations, and they have now introduced Agents as well.
Basically, you can take any workflow you have and give it to the agent as a tool. The agent can now decide which workflow to run and when, based on the ask.
For example, imagine you have three workflows:
1. Retrieve customer information
2. Retrieve a customer support history
3. Email customer
The agent can pull a customer's information, check their support history, and put together an email that addresses their problems.
Before, people made decisions at every step and ran individual workflows.
Now, an agent can stitch workflows together and decide when and how to run them, but each workflow still runs the way you built it.
You get the best of both worlds: the flexibility of agents with the predictability of workflows.
Here is a link to read more about the new n8n Agents: https://t.co/nHESkwlRD6
Thanks to the team for partnering with me on this post.