Twitxy Saved

Phil Venables

Verified account This account is verified. Learn more

@philvenables

All about cyber, resilience, risk, AI - at scale.

Partner - Ballistic Ventures / 4 x CISO / Board Director / Chief Risk Officer
USA philvenables.com Joined April 2009
595 Following 16K Followers 9.9K Posts

Latest posts 5

Phil Venables Verified account @philvenables
Coincidences Magnified - Are We Under Attack?

I will preface this with the humility that I might be totally wrong and all these recent events really are some type of coordinated cyber-attack, although I would say if it is it’s all rather pointless and unfocused. But, I doubt this is really an “attack” in any way. However, just because you’re paranoid doesn’t mean that they are not really out to get you.

No, the most likely explanation of what is going on is:

1. Fragility of increasingly large-scale and under-maintained, complex and brittle infrastructure.

2. Our increased digital dependence such that when these events happen we notice them more.

3. A few independent incidents coincidentally occurring together which causes the media and the rest of us to be more attuned to other outages that would normally go unreported, that we now bucket into some mythologized growing category.

4. Lack of resilience in the presence of failures so that their impact is longer-lasting and more impactful.

Let’s pick on a few of these. First, independent occurrences of events that seem to be connected. One example I remember was a day many years ago when the New York Stock Exchange was down for the day (software error), a US airline was ground stopped (hardware event), and on the same day a bunch of major bank CISOs, including me, were going into the White House Situation Room for a totally unrelated threat briefing (we had to hide from the White House Press Corp in case that was further coincidental evidence of “attack on America”.) After this I looked at what else happened that was unreported that day and there were dozens of other unrelated outages at hospitals, power plants, and so on. Perhaps even more happened that one day than has been happening this past week. It just went unreported.

Second, do we have any base rate data to understand if what of anything that has happened is actually so unusual or surprising? [Note: something being unsurprising doesn’t mean it’s acceptable, in fact we want such events to become ever more surprising of course].

- Microsoft outage: yes, this was unusually big, but a Microsoft outage isn’t a new occurrence. (Other large-scale tech providers have outages also, to be fair to Microsoft).

- FAA outage: maybe this was a minor change to one of 1000’s of cables that had previously been fine that had some weird accidental effect they’d not seen before hence this wasn’t the situation of “why do a production change on a Friday?” Organizations do changes all the time that they think are distant from impact risk. Perhaps this was like that. But it does seem their network should not have been so brittle in the face of this. I guess we will find out.

- NSA Net outage: do you know how many times NSA Net goes down per year? I don’t. But perhaps NSA Net isn’t a literally 100% available system (most things aren’t [literally], even stupendously critical things), so perhaps this was still within their four or five 9’s SLO. Perhaps it goes down several times a year but we just don’t normally hear about it. I don’t know, most of everyone else doesn’t either.
The list could go on.

The thing that should be disturbing to us though is the point on resilience.

I really worry when things like this happen together or in isolation and our response is: “This [bad thing] must never happen again.” and the organizations involved, sometimes when compelled to, make that system less likely to break. Then we build up more dependence on it, but again it will inevitably break in some new way for some other reason and the damage will be bigger because that increased confidence induced more dependence, because we all said “this must never happen again.”

The real lesson here is that we need to not just build resilience to failure in specific components but resilience in the overall “system”, like let’s always assume FAA comms may go down albeit at some increasingly low probability and have a true back-up emergency communications approach to at least sustain some operations. Maybe they’re working on this, I don’t know. And make sure that critical industrial systems don’t depend on alerts only being delivered by cloud-based email.

I’ll start where I began. It will be darkly funny if I’m wrong and this really is a set of related attacks. But when you look at how much breaks regularly everywhere all the time, I doubt it.

The actual need here is to reduce brittleness and increase resilience on the assumption that everything will always fail in some way at some time.
24 replies 104 reposts 387 likes 97K views View on X