Dassie Spivack

Helping you survive the cyber apocalypse

[Dassie Spivack @DassieSpivack]

@Scobleizer This is exactly right

[Robert Scoble @Scobleizer]

At Microsoft (i had a very public role back in 2003-06) I learned the best way to get a mob to stop throwing metaphorical Molotov cocktails through the front window was to acknowledge the mob and beg for a few hours to figure out what was going on.

It is the coverup that causes the mob to get more violent.

[Dassie Spivack @DassieSpivack]

The response to this has been unbelievable.

On Tuesday, I'll share the exact framework I'd use to write a breach notification email if I were in your shoes.

Follow and turn on notifications so you don’t miss it.

[Dassie Spivack @DassieSpivack]

I used to be the person companies called after they got hacked.

And I noticed something early on.

The hack almost never sank them. The cover-up did.

Every time, the team’s instinct was the same: wait until we know more, wait until we're sure, wait until we have the perfect statement, etc.

And some of the biggest companies in the world have learned what that waiting costs.

In 2016, Uber got hacked. Instead of telling anyone, it paid the hackers $100K to delete the data and stay quiet, and labeled it a bug bounty. The truth came out a year later. Uber paid $148M to settle with all 50 states, and its former security chief was criminally convicted.

That same year, Yahoo revealed that a 2014 breach had exposed 500M accounts. The news landed in the middle of Verizon buying the company. Verizon cut $350M off the price.

And in 2017, Equifax discovered a breach affecting over 140M Americans, then waited 40 days to tell the public. During that window, an executive who'd figured out what happened sold nearly $1M in stock. He went to prison for insider trading.

Meanwhile, hackers don't wait. They post your data. They email your customers directly. And when customers hear it from a stranger first, that trust is gone for good.

Telling customers early doesn't mean admitting fault. It can be as simple as: "We found suspicious activity. We're investigating. Here's what you should do right now. We'll update you as we learn more."

On Tuesday, I'm sharing the exact framework I'd use to write that email if I were in your shoes. Follow and turn on notifications so it's there when you need it.

Customers will forgive you for getting hacked. Most of them expect it by now.

What they won't forgive is finding out you knew and said nothing.

[Dassie Spivack @DassieSpivack]

@Alecaticus Actually it’s been five days ◡̈

Not a bot, actually a former cyber insurance adjuster, and now a JD candidate focused on cybercrime

[Dassie Spivack @DassieSpivack]

I used to be the person companies called after they got hacked.

And I noticed something early on.

The hack almost never sank them. The cover-up did.

Every time, the team’s instinct was the same: wait until we know more, wait until we're sure, wait until we have the perfect statement, etc.

And some of the biggest companies in the world have learned what that waiting costs.

In 2016, Uber got hacked. Instead of telling anyone, it paid the hackers $100K to delete the data and stay quiet, and labeled it a bug bounty. The truth came out a year later. Uber paid $148M to settle with all 50 states, and its former security chief was criminally convicted.

That same year, Yahoo revealed that a 2014 breach had exposed 500M accounts. The news landed in the middle of Verizon buying the company. Verizon cut $350M off the price.

And in 2017, Equifax discovered a breach affecting over 140M Americans, then waited 40 days to tell the public. During that window, an executive who'd figured out what happened sold nearly $1M in stock. He went to prison for insider trading.

Meanwhile, hackers don't wait. They post your data. They email your customers directly. And when customers hear it from a stranger first, that trust is gone for good.

Telling customers early doesn't mean admitting fault. It can be as simple as: "We found suspicious activity. We're investigating. Here's what you should do right now. We'll update you as we learn more."

On Tuesday, I'm sharing the exact framework I'd use to write that email if I were in your shoes. Follow and turn on notifications so it's there when you need it.

Customers will forgive you for getting hacked. Most of them expect it by now.

What they won't forgive is finding out you knew and said nothing.

[Dassie Spivack @DassieSpivack]

DM me your craziest cyber/hacking stories and I’ll post the best ones anonymously

[Dassie Spivack @DassieSpivack]

@Gregpazo YES. it’s literally nuts.

[Dassie Spivack @DassieSpivack]

@AWalkIntheRain You’ll get an email/notice if you’ve been affected!

[Dassie Spivack @DassieSpivack]

The amount of customer data impacted here is insane.

Almost every single person in the US will be affected by this.

[Andrew Curran @AndrewCurran_]

Quest Diagnostics has been hacked. Patient data was exposed. They serve roughly half of U.S. hospitals and physicians. Investigation ongoing.

[Dassie Spivack @DassieSpivack]

Wow, this is taking off.

Startup folks: what should I write about next? What other questions can I answer for you?

[Dassie Spivack @DassieSpivack]

@altryne @theaaron @pangram Ok thank you. Will try a different approach next time!

[Dassie Spivack @DassieSpivack]

That’s fair! I’m just getting started so your feedback is helpful. Frankly my writing won’t be as clear as this if I write unassisted, though the ideas will be the same. Is that worth the trade-off?

I know this is an odd way to decide this, but this info is very important and I want it to reach the right people, and there’s likely a trade-off here.

[Dassie Spivack @DassieSpivack]

@theaaron @pangram The wordsmithing was. I voice dictate my thoughts and experiences and have AI organize it into something clear and readable.

[Dassie Spivack @DassieSpivack]

More details here:

[Dassie Spivack @DassieSpivack]

You wouldn’t believe how many startup CEOs I talk to who would be royally screwed if one of their vendors got hacked.

A few months ago, this wouldn’t have mattered for most startups. But we’re entering cyber armageddon.

This is going to happen to you. It’s just a question of when.

The good part is that it’s very manageable. You just need the right mindset and a few processes in place before anything happens. Here’s what I’d be thinking about today.

First (don’t kill me): do you have cyber insurance? And the real question: have you ever actually talked to your broker about using it?

I know this is the last thing you want to deal with, and it’s boring as hell. But please, please call them and make them walk you through it. If your cloud provider gets breached tomorrow, who do you call? What help do they arrange? What needs their approval? What do you pay out of pocket? Get the actual phone number, and save it somewhere you can reach if your work email is down.

Then figure out if your lawyer handles breaches. If they don’t, ask who does and get introduced. Find out how hiring them works with your insurance. You want to know this person exists before you need them.

Next, pull up your customer contracts, especially the big ones. What did you agree to tell them if something happened? You’d think every CEO would know this by heart. Almost none of them do.

What triggers a notification, and who’s supposed to get it? Have someone qualified work through it with you, right now. You might have agreed to something eight months ago and forgotten. Someone at the customer may remember, and if they do and you don’t, you could be very liable. One angry customer with a contract claim can be enough to sink a startup.

Same deal with any security questionnaires you filled out. What did you say you do? Are you doing it? Is it still true?

Then come the business problems. How long could your product be down before you have a real cash problem? What if a big customer pauses payment while you’re also paying outside help? Run the numbers and see where things get uncomfortable. And who can spend money in an emergency? If everything needs your approval and you’re asleep or on a plane, what happens?

Who can take the product offline? This came up in the replies to my last post. Rotating keys can mean an outage, and someone has to decide whether to accept it. Figure out who has that authority before it matters.

Who talks to customers while your engineers are busy? Give that job to one person and make sure they get accurate updates. Otherwise your engineers are investigating while answering the same question from five people. You can draft the bones of those updates now: what we know, what we’re checking, what you need to do, when we’ll update you next. Leave room for uncertainty, and don’t promise things you haven’t confirmed (this happens more often than you think, and can get you in hot water).

Have someone keep a timeline: facts, times, decisions, things you tried that didn’t work. Everyone will be tired and distracted. You will forget.

And where does all of this live? If your plan, contracts, and contacts are all behind the login that’s down, you have a second problem. Try reaching what you’d need without your normal work setup. Make damn sure you can (offline copies are useful here).

Take a look at what you’re exposing, too. Old customer files, exports, recordings, and especially whatever your agents can access. Do you need all of it? Who decided to keep it? Who knows everything you’ve connected, what your agents do during an investigation, and where they’re sending information? Someone needs to own those questions, because every piece of it is future work if something goes wrong.

Then practice. Sit down with whoever would be involved and walk through it. At some point everyone will look at each other and nobody will know the answer. Write that down, and start there.

Bookmark this so you have it when you need it.

[Dassie Spivack @DassieSpivack]

What to do when your cloud provider is breached, from a former cyber insurance adjuster:

If your provider just disclosed a breach, you have to assume you've been exposed.

And every instinct you have about what to do next is probably wrong. So here's what you should actually do:

When you first hear about the hack, even if you doubt you were exposed, you need to get on it. Right away.

I don't care if you're on a date, or at your son's college graduation.

Get to a computer and start rotating every secret…everything. When someone asks later how fast you responded, you want the answer to be "immediately," with timestamps to prove it.

You're going to want to put as little in writing as possible. I get the instinct.

Do the opposite.

Document everything, obsessively. If your provider has locked things down and you can't rotate something yet, write that down too, with the time you tried. If you're using agents to investigate (and you should be, spread wide across everything), keep every trace and make them write down everything they find.

This will feel wrong in your soul. Do it anyway. If this ever turns into a claim, if you can show you did everything you were supposed to, minute by minute, you will survive.

You might tell yourself it's just names and emails and nobody cares. But this is wrong. All customer data counts.

I've watched companies lose millions just because someone could see data they shouldn't have.

You're going to want to wait for your provider to finish investigating before communicating with your own customers. Don't.

Figure out what information may have been exposed, even if you’re not sure yet who it belongs to, and what obligations you have under your contracts / applicable law. Your cloud provider’s investigation doesn’t change the fact that you have your own obligations to meet. And learning about a deadline after you’ve missed it is a very expensive lesson.

Bookmark this. I hope you never need it, but with everything happening lately, there's a real chance you will.

[Dassie Spivack @DassieSpivack]

You wouldn’t believe how many startup CEOs I talk to who would be royally screwed if one of their vendors got hacked.

A few months ago, this wouldn’t have mattered for most startups. But we’re entering cyber armageddon.

This is going to happen to you. It’s just a question of when.

The good part is that it’s very manageable. You just need the right mindset and a few processes in place before anything happens. Here’s what I’d be thinking about today.

First (don’t kill me): do you have cyber insurance? And the real question: have you ever actually talked to your broker about using it?

I know this is the last thing you want to deal with, and it’s boring as hell. But please, please call them and make them walk you through it. If your cloud provider gets breached tomorrow, who do you call? What help do they arrange? What needs their approval? What do you pay out of pocket? Get the actual phone number, and save it somewhere you can reach if your work email is down.

Then figure out if your lawyer handles breaches. If they don’t, ask who does and get introduced. Find out how hiring them works with your insurance. You want to know this person exists before you need them.

Next, pull up your customer contracts, especially the big ones. What did you agree to tell them if something happened? You’d think every CEO would know this by heart. Almost none of them do.

What triggers a notification, and who’s supposed to get it? Have someone qualified work through it with you, right now. You might have agreed to something eight months ago and forgotten. Someone at the customer may remember, and if they do and you don’t, you could be very liable. One angry customer with a contract claim can be enough to sink a startup.

Same deal with any security questionnaires you filled out. What did you say you do? Are you doing it? Is it still true?

Then come the business problems. How long could your product be down before you have a real cash problem? What if a big customer pauses payment while you’re also paying outside help? Run the numbers and see where things get uncomfortable. And who can spend money in an emergency? If everything needs your approval and you’re asleep or on a plane, what happens?

Who can take the product offline? This came up in the replies to my last post. Rotating keys can mean an outage, and someone has to decide whether to accept it. Figure out who has that authority before it matters.

Who talks to customers while your engineers are busy? Give that job to one person and make sure they get accurate updates. Otherwise your engineers are investigating while answering the same question from five people. You can draft the bones of those updates now: what we know, what we’re checking, what you need to do, when we’ll update you next. Leave room for uncertainty, and don’t promise things you haven’t confirmed (this happens more often than you think, and can get you in hot water).

Have someone keep a timeline: facts, times, decisions, things you tried that didn’t work. Everyone will be tired and distracted. You will forget.

And where does all of this live? If your plan, contracts, and contacts are all behind the login that’s down, you have a second problem. Try reaching what you’d need without your normal work setup. Make damn sure you can (offline copies are useful here).

Take a look at what you’re exposing, too. Old customer files, exports, recordings, and especially whatever your agents can access. Do you need all of it? Who decided to keep it? Who knows everything you’ve connected, what your agents do during an investigation, and where they’re sending information? Someone needs to own those questions, because every piece of it is future work if something goes wrong.

Then practice. Sit down with whoever would be involved and walk through it. At some point everyone will look at each other and nobody will know the answer. Write that down, and start there.

Bookmark this so you have it when you need it.

[Dassie Spivack @DassieSpivack]

@conny_woodall I’m trying 😂

[Dassie Spivack @DassieSpivack]

@calcsam Taking down production is the right choice here. Very few exceptions.