Airports illustrate the potential impact of OT attacks that disrupt functionality. Dan Gunter, CEO of Insane Cyber, talks about how industrial environments differ from traditional IT, particularly in their reliance on availability and safety, where disruptions can have significant real-world and financial consequences.

[Heads Up: This transcription was autogenerated, so there may be errors.]

VAMOSI: In 2017, a sophisticated cyberattack known as Triton or Trisys targeted the safety systems of a power station believed to be in Saudi Arabia, marking a significant escalation in industrial cybersecurity threats. The attack focused on Triconex safety controllers, critical systems designed to prevent hazardous incidents. In other words, this is a safety system designed to warn workers when conditions are unsafe — such as the level of dangerous chemicals in the air.

Although publicly reported in December 2017, early warning signs appeared months earlier. In June, an unexpected shutdown initially attributed to mechanical failure raised little suspicion. However, a second shutdown in August prompted deeper investigation, ultimately revealing the presence of the malware. The malware — later named Triton — exploited a vulnerability in Microsoft Windows–based machines to infiltrate systems connected to the safety infrastructure.

Triton leveraged a zero-day vulnerability to establish persistent access and to manipulate safety controls. The attack exposed serious cyberphysical risks. Notably, Triton is considered the first malware specifically designed to disable industrial safety systems, potentially endangering human lives.

This is a story about the need to secure legacy cyberphysical systems, and what type of person might be best for that role.

I’m Robert Vamosi

This is Error Code

[MUSIC]

GUNTER: So my name is Dan Gunter, and my job’s title is CEO and founder of insane cyber, awesome. So insane cyber what we’re helping? We’re helping critical infrastructure and asset owners extend their security. And so a lot of great work has been done in this space on passive network monitoring. But one of the really big challenges of the industrial space, ot systems, factories, power oil and gas and other critical infrastructure sectors, is the sites are often spread out, so you have a geography problem. And then, from a talent perspective, the OT, the OT workforce, is a small, niche workforce of the overall Information Security workforce, right? And so our tool, also our tool helps with the scale of what an asset owner can cover, to cover the realities of their environment, but also help their team members, you know, already stretch teams to be more capable in this space.

VAMOSI: I talk a lot about cyber-physical. What does that mean? It means systems that have physical consequences in the real world. So a robotic arm in a factory. Or a conveyor belt in a store. You might not think of these as cyber, yet there are PLCs that handle the on and off, that measure the load, that are important to its function. So I’m just going to go right into it and say it’s 2026, why do you think it’s taken so long to recognize the importance and vulnerabilities in the OT space?

GUNTER: I think on one front, on it, it comes to budget, right? The operations budget at many asset owners is many multiples higher than the security budget, still, right? You have, you’ll have asset owners that are like, hey, you know, I need to invest in security. But just like other businesses, it’s a cost center, right? And so some of the resistance is sometimes how this story is told, like, right? Is this threat legitimate? Could this really happen here, right? And there’s skepticism on that side that cause people to go down that path of like, you know, let me just do nothing, right? The cost of doing nothing is, you know, lower than the cost of doing something I’m not actually having impact, right? There’s folks in the world that still do kind of believe that on the other side and this, you know, you can go into an individual asset owner, and what’s interesting is, they may be secure on their security program, on one part of the process, but not another. You know, these industrial sites, like, right? It’s not like, you know, it System Refresh, but it’s like, let me just go order from Dell. Like, we have asset owners that have, you know, they built a refinery 50–60, years ago. And every 10 years they built a new part of that refinery, right on the power grid. You know what goes into a substation. The life cycle for those systems is 15–20, years. And so from an IT perspective, right? A lot of folks will be like, hey, you know, after three, five years, you’re replacing this, right? And it’s like, no, like, because of how the grids built, like those asset owners thinking like, 1520, year windows, right? In addition to just the cost, right? It’s not as it’s not as easy to just plug a system in and call it a day. There is the cyber physical act part of the problem, right? This is driving pumps out in the wild. This is moving if you’re at a refinery, benzene, if you’re at a mine, you know, vats of hydrochloric acid, right? And so it’s not an easy the security problem is much harder because of that cyber physical and everything else that’s at play, right, right?

VAMOSI: I want to continue to explore that cyber physical disconnect. A lot of people think OT devices are just machines. Machines out in the field doing one task, but we’ve networked them now, and they become addressable from all parts of the world.

GUNTER: Absolutely. And it’s crazy, right? You look at the history, and in the 50s and 60s actually, where like when we talk about programmable logic controllers today, to your point, the computers, the industrial computers, that are driving things are the HMIs, the human machine interfaces that control those back in the 50s, 60s, like, you know, the first generation of these PLCs was built because General Motors. I think one of the car manufacturers was like, hey, right now, in this automotive plant, I literally have to run a wire from here to here. In every system required a wire. And in the 50s and 60s, they’re like, hey, like, it turns out we can run multiple of these wires into these industrial computers and just let the computer handle the switching and do things. So instead of my factory having hundreds of miles of wire, more like, we can do it with less. Right. The challenges like right over decades, those systems have changed, but to the point of legacy systems. You’ll still sometimes run into, like, old equipment that is decades old. We’ll, we’ll commonly see out there, like routing equipment that the company is not even around, and you see a stack of routers because the network’s literally still, like, tied to, you know, that legacy equipment that, luckily, they still have a stack of in there, right? And so it’s the problem in this space and that cyber physical side, because it is tied to the actually driving things. To your point, like, it can drive a lot right? It can move things around. It can open valves. It can turn engines on and off, right?

VAMOSI: Dan offers a real world example. Airports. If you step back, airports have a number of OT systems that could be considered life critical, although you might not realize it.

GUNTER: A lot of people don’t think about airports as critical infrastructure, right? I started my career in the Air Force, some of our Air Force bases, right? If you go to like, a refueler Air Force base or a cargo base that has, like cargo planes, like it literally takes dozens of trucks. If you were to send a truck out to fill that, it’s much more efficient to be like, Hey, let me have a fuel pump that’s pumping gas under the runway so I don’t have to go there, right? The closest civilian side is London, right? The London airport today, right? They are, I’m pretty sure they’re pumping fuel and doing things to get it there instead of rolling trucks, right? And so, to your point, like, what makes industrial fun is the purpose those computers are around, right? The last one, I’ll say, is like, baggage systems, right? We talk to folks and it’s like, what’s the like, what would be the worst way to disrupt an airport? And they’re like, well, all of the industrial computers that are moving bags around, right? Because then both the airport has a problem that people can’t get their stuff and to that point of like, PLCs and what it’s driving like, it’s it’s critical, and people don’t think about it till it’s down, right?

VAMOSI: What’s surprising to me is that it’s taken so long for not only for cybersecurity to realize, but for adversaries to realize that they don’t have to steal data. They cripple functioning systems. To your point, you take the baggage out of an airport, and it just disrupts everything.

GUNTER: Oh, absolutely, absolutely. And it’s, you know, and, and that’s where the field has changed, right? I mean, we’ve been back in World War One. World War Two, the military would more just bomb or blow up critical war supporting infrastructure, oil tank factories, other things now, like, it’s digitally connected, right? The world has changed to where, like, sometimes you don’t have to risk a person physical.

VAMOSI: A lot of these systems, to Dan’s point, were getting rid of the wires. They were making it simpler. They were using protocols that were designed to do just one or two things. Flash forward to today, and we have the IoT systems that are out there, and they’re saying, let’s take this really old protocol that communicates, and that’s all it does, and bring it into this device that I’m going to put into a home hub or something like that. They don’t realize that it hasn’t been vetted as much. It hasn’t improved over time. It’s still back in the whatever the 90s when it was first created, is that sort of the problem that you see in the OT world today,

GUNTER: That is, I mean, there’s industrial protocols to where, you know, you’ll have PLCs that are still on serial connections, still on the like, you know, more non Ethernet connection, and they’ll go through a serial to Ethernet converter. Well, when it comes to the IP side of the network, all that’s being done is it’s being put on that TCP or UDP packet. Or even in some industrial processes, it’ll actually get put on, like a broadcast or multicast, or even, like some industrial protocols don’t use TCP or UDP, it just throws on Ethernet. And those are fun, right? Because, but what’s fun, like, right, from like, evolution perspective, is those devices still talk to each other, and so there has to often be some interoperability, where you have, like, a IP, like Ethernet based network and serial, and you know, you’re exactly right. A lot of those protocols don’t have authentication, like, at best, they’ve just thrown it on and it’s like, okay, you know, PLC two on the digital network side is talking to PLC five on this side, and all that’s being done is thrown on that layer to your point, what’s interesting from the security perspective, right? If attackers then can spoof that right? They can take some of that advantage and be like, Hey, let me just stack a packet like this and make it look like it came from that other side, or work through it right, right? So a lot of these early systems had very limited chip capabilities. They were just binary, on, off, yes, no, whatever. Today, chips are so small and memory is so great. Is there a growing risk that some of these newer OT devices can become botnets, can become useful for other purposes?

VAMOSI: Dan mentioned IoT earlier. Like embedded systems are a challenge. From a security team, embedded systems to a stock are a bit of a black box. They maybe see communication, but on most of those embedded devices, you don’t get good host logs unless you kind of have fancy ways or know how to pull it off.

GUNTER: Right to that point too, the systems are becoming a lot more accessible. Some of the programmable logic controllers that are out there, some of the asset owners actually, on the efficiency side, are like, Hey, if you want to deploy Python code out to this PLC, here’s this way to do it right. And so it’s almost making it easier for folks that even like to pick up Python, which is pretty, you know, straightforward to learn. And you know, once you can get past the access point, use them for that. Use them to mine Bitcoin. Use them to, like, kind of live on, to cause and effect, right? And so you’re exactly right. Like, right, they’re becoming a lot more powerful just in that purpose, right? And on the flip side, on the safety side of it, like, right? They have to be mindful how engineers architect these PLCs, right? They need to make sure, like, the action that’s supposed to happen with the logic they put on there happens, right? And so there’s a trade off between going to secure versus like, Hey, I’m going to trust that what’s in memory or the code on here is what should actually be executed.

VAMOSI: Returning to something Dan said earlier, these systems, I’ve always said, are like a tower of Babel. Everything’s proprietary, everything’s small, and you don’t need to worry about them. But that’s not true anymore, and things are getting more and more connected.

GUNTER: Is that part of the problem? Then, for you going out there to try to secure these that’s part of the problem. It’s also a problem in the industrial space, like a lot of the OEMs have been bought a few times, and so you’ll have a major company like Emerson. Well, Emerson bought Westinghouse. And so when you look at some of the technology, when you look at the protocols like that was actually some of the technology came through acquisition, or came through others similar to that side, Schneider Electric has a safety system called triconix that was used in the or it was targeted in the

trisis at Man kind of intrusion, right? But it’s a safety system. They actually were not the inventors of that. That was IP. They brought, bought via acquisition, and came in the protocols. It’s funny, when you’re actually at the analyst level, you see that, right? So sometimes you still see like, Oh, hey, if I pull Westinghouse’s manual from 30 years ago to your point earlier, you were saying the protocol is actually very similar. There’s just changes maybe Emerson has done, you know, as they’ve modernized that product line as they push right. And so, you know, what makes it really fun for us is it is that box of puzzles? And sometimes it’s like, oh, like, hey, like, has this actually changed, or is this the same protocol that is in there? And you get into certain places, like where Westinghouse is interesting, right? The nuclear sector, right? Some of our most critical, critical infrastructure, a lot of it runs on Westinghouse, right? So you see, kind of not just across, like that. You know, changes as it changes hands, but it’s important to kind of know, like the hey, you know, how has the tech actually changed? How has this changed across time, right? And you talk change, can change control and change configuration and all that. Like, nothing is more critical than nuclear, right, right?

VAMOSI: Given the variety of older equipment, from companies that changed hands or went extinct, how does someone like Dan become a jack of all trades? How does he know that this was Westinghouse? This was originally Phillips, or Western Electric, or whatever it was back in the day. How do you get on top of something like that?

GUNTER: Some of it is weird, you know, OT is a niche for people that enter the field. You know, luckily, the OT talent market has grown, right? It’s the job market and the number of people in this space, we still need more people, good people in the OT security space, that can learn it. Right? We need folks that can kind of, you know, pick up the, you know, pick up the carpet a little bit and see what’s underneath it. A lot of our learning also will happen, like, we use some of the manuals that OEMs provide, and be like, Okay, this is how it said it’s produced. But we’ll also work with the asset owners we work with, and be like, Hey, let’s go to this power plant. Let’s go to this substation. Let’s start to pull packet capture, let’s start to pull windows, event logs and other data, and let’s see, like, how it actually works, and if we have to to work back to the functionality to know how to secure it, great. But it all starts with kind of that pulling that thread and, like, from my perspective, and having be in this space, like, there’s no better opportunity than when you can partner with the asset owner and be like, hey, like, let’s, let’s understand a little bit more on this environment, right? Because what we see is like, right? We’ll work with really smart electrical engineers, really smart mechanical, you know, Petrochem other engineers, and they really understand how the process works. Well, my background is mainly in computer science, right? And so when I come in with that computer science perspective, like bringing that in can help, because they’re doing great engineering work on that side. But it’s like, okay, how would a computer scientist think on this problem, right? And someone from InfoSec, and specifically OT space, and

VAMOSI: how, how often do you run across, like the white label problem, where you’re told that this device is X but then you discover that it’s actually mass produced as something else all the time.

GUNTER: And a fun one on that is for vendor remote access. It’ll be like, this is a vendor with so and so remote access, and it’s on port 3389, and it’s like, are you just calling Microsoft Remote Desktop Protocol? Your thing nine times out of 10, it is. And so, you know, sometimes you’ll see those you had that issue with, right?

VAMOSI: And so, apart from having the proprietary stuff, there are some commonalities. Like Modbus keeps coming up everywhere. It seems like that’s a functionality or a protocol that’s being used widely is that the case?

GUNTER: Modbus says that one actually the Modicon group, going back to the auto plant conversation earlier, Modbus actually came out of that auto plan. And so it is kind of the OG protocol in the space that was used to pull in and has been the history of a lot of modern day PLC and OT kind of technology where it’s fun today is you have kind of the base protocol that’s like, reads, writes, basic things you do to PLC, get things in memory, read them from memory, the latter logic That changes different values and drives, motors, drives, other things out there. What’s challenging is you take a protocol like that, and there’s actually like a vendor specific space, and so a vendor like Schneider Electric, a vendor like GE, or anyone else that’s using Modbus can say, Okay, I love this base protocol. I need to add more. Features, right? Because, to the point earlier, my PLCs are getting more complex, there’s things I can’t do just in kind of the 16 common or 18 common function codes. And so it’s also fun, because these are like, right, I triple E specs. And so sometimes you see a difference between, like, what is in the IEEE spec versus what did the engineer at the OEM actually intentionally implement, or what did they misimplement? And it’s not actually the protocol that you have from the engineering docs, right, right? So walk me through an example, and I’ll just pick petrochemical, but you can pick any industry that’s appropriate. How would you approach them? I mean, you’ve signed on, they said, Yeah, we really need this and everything but day one, day two, week, one month, year.

VAMOSI: How do you approach tackling their security system, given all these desperate systems set out there?

GUNTER: Yeah, it all starts with figuring out what’s important to them immediately, like, right? Priorities do move like, you know, what is the boardroom asking about? What is management outside of the operations group or IT group, or whoever is caring about security at the point kind of cares about, right? And so it starts with, really an understanding of, you know, what do you care about? Have you had ransomware lately? Has that ransomware activated? What’s interesting in this impacted you, right? What’s interesting in this space is when you talk about this space, like, instead of response, you know, bringing things back up from ransomware. Asset owners have those costs, right? But the bigger cost to most asset owners is the cost of downtime, right? You look at Ponemon Institute right, independent numbers on the cost of downtime, automotive is a $2.4 million an hour average downtime. Oil and Gas is $1.6 million an hour according to Ponemon Institute downtime. So not only are you losing from the cost of incident response and the stuff that your security program has to pay, but that other program. So where it goes back to that priorities is sometimes you come into a situation where they’re like, you know, hey, like the automotive things we saw last year overseas, right? Our plants were down for a few weeks. This cost us a significant amount, you know, in loss production, in loss of safety, right? So understanding kind of those we also start with just maturity, like, how much of your environment can you see? Do you understand what’s being told? How much of To the Point on Modbus? How much of Modbus, if you have Modbus in there, do you actually understand and how much is making it if you’re using a tool like Splunk or some of the other seams, like, what’s actually making it up to the sock to make decisions on right? Starting with that kind of understanding of where they’re at, then it is that side of, like, you know, you can’t boil the ocean, right? Where do we start to get some of that impact, right? Is it a visibility problem? Is it an access problem, like, right? And, you know, look, we are a vendor that has a product, but ultimately, at the end of the day, like the OT space is all built off relationships, right? And so it also comes down to you wanting to help folks with like that immediate problems, because eventually, like, it builds the relationship then to kind of get to maybe where we as a company want to get right. But it starts with that side of life, realizing that in these environments, right uptime is important, right availability, but the like, confidentiality and integrity, you know, can even be key, right? Especially availability and integrity are huge, right? So starting with those parts of the CIA triad, and, you know, having that scaled plan, right? How are we actually measuring that we’re not just doing a lot of activity and wasting time, right? There’s some good kind of industry wide wants on that. But sometimes it can get really personal, right? If people come in with needs and wants.

VAMOSI: So it sounds like it’s not too dissimilar to someone saying, secure my network. You immediately say, secure your crown jewels. What is the most important thing that you want to protect? You can’t, to your point, boil the ocean and do everything, so you focus on those core so that’s kind of actually comforting, because it’s, as I said, it’s kind of daunting to think about all the devices that you have out in the field or the mechanical things that are running your factory, and you’re just like, hey, no, narrow it down, yep.

GUNTER: And it goes back to where it gets more interesting to folks in this space is it goes back to that cyber physical right? So if someone’s able to cause an impact, it impacts actual operations, if that’s in a refinery and it over pressures the tank, or does something like that, and there’s not a compensating safety. Control that reacts to that. Like the other calculus in there is, is the reconstitution in that time? But also, like, do I have parts on hand to fix this part of the physical process that’s broken? Right if someone comes through and bricks a lot of PLCs or comes through and actually causes physical damage in there, like, do I have availability on a lot of that with industrial processes like the PLCs in themselves? The supply chain for those is easy, but when you get into some of the bigger parts of a refinery or a substation, like you’re talking about things that asset owners may not have in big quantities, right? Or they may be very expensive to get, or they have to hop on a boat to come across the ocean to be even be here, right? And so it’s that’s where the problem, I think, also, you know, really does differ a little bit from the IT side, but it is like you’re exactly right on what you were putting there, on Crown Jewels and some of that other approach.

VAMOSI: So Dan mentioned socks, and I hadn’t really thought too much about it, but are they paying attention to OT or is that something that needs to be stepped up and improved?

GUNTER: That’s one of the fun as you get to know companies, there are places that try to run separate it and OT socks. I’m of the opinion that that just creates two groups during incident response that have to work together. If you take, like, some of the NIST publications on how to build incident response, some of the other stuff, actually, a lot of the NIST they’re really good NIST docs, which are like, Hey, this is how the US government and the folks that participated on those NIST committees recommend to do the security program right. And, you know, right? You take like, plan, policies and procedures, right? You know, when you’re at your policy side, it is important right to if you have one SOC, you want to make sure, like, right, you give the authorization and the permissions and everything. So folks aren’t having to do it right if you have to. What gets challenging from that perspective is you have to make sure, like your plans, policies and procedures for incident response, for stock operations like that, like when something happens, you’re not picking up the phone and calling someone if they’re surprised that you’re even calling them right? It becomes even more challenging when you take, like, a Colonial Pipeline incident right when the East Coast pipeline was pushed out, right? You know, there’s a lot of speculation and folks that have guessed on what happens on the outside, right, but it sounds like, like there was some impact IT systems and others that do impact the other side right? And you even hit the side of any utility, right? If a utility can’t determine how much power you’re using and bill you for it, like we don’t know how that utility is going to react, right, because they’re a utility making money off of moving and selling electrons, right? And so you know where it becomes interesting, and on that stock perspective they have to work together in harmony. And we go a lot of places to where, like there is animosity sometimes between the IT side and the operation side. And people have been talking about convergence for decades, like the network has been converging for decades. Politics is the part that’s still being worked through now.

VAMOSI: Do you have any sanitized war stories that you bring into a discussion with a new client or something to underscore the importance of all of this?

GUNTER: Definitely, I think, like the big talk about Ukraine, 2015 and 2016 you know, their crisis, admin, whatever you want to call the safety system, those are key ones, right? And even in destroyer two is a key point, right? And the interesting things on that right is in the Ukraine, 2016 incident like, right? There’s videos out on the internet of folks moving the mouse around and opening breakers, right? It was a very kind of manual deal, right? And then you get into Ukraine, 2016 and they start writing malware. And then a few years later, right, we see malware that’s actually OT capable. The kind of scary thing for the OT space is going back to the Python example. There’s some really good Python libraries for industrial protocols, right? What’s short, luckily on the attacker’s side, they don’t know what to send or where to go, and so like where we start, some of that understanding, right? We have known cases of people moving a mouse, and videos out there of the mouse opening breakers and doing things like Team Beard or RDP or however they’re actually in. We also have like several examples now of folks actually writing code, actually doing that. What I’ll share on an interesting level is when you get into something like a crisis, I think some of the we’re. Reverse Engineering Report. So that used a very proprietary protocol that was, like, definitely a top shelf, higher end, harder thing for that attacker to do. There was a lot they didn’t understand about the protocol, and we’ve seen that across a few things, where the attacker either mis-implements it and it didn’t work as they intended, luckily, and they didn’t know what was wrong. And as a community, we never want to point to like, hey, attacker, here was the bug in your code that would have made this work, that didn’t go right. But I think that becomes more critical, right? Because, like, again, we have seen human driven attacks,you know, kind of code driven attacks. We’ve seen rants more on OT. We’ve also seen, you know, more nation state impact, right?

VAMOSI: Dan points to the 2010 Stuxnet attack on the Iranian centrifuges. A brilliant attack requiring both IT — windows vulns — and OT — PLC-specific malware. Go back to the OG impacting uranium enrichment and Iran, right? Like that was really, I think, the first known major, you know, nation state to nation state assumed impact, right? And to your point, that took a lot of study and understanding of the Siemens system seven protocol to really engineer that in a way that it was effective. And to your point, understand the entire stack there so that it would execute the way they wanted it to.

GUNTER: what’s interesting with that is you have a mix of like the industrial protocols being used. You also have a mix to where they were just doing things that you see when any Windows attack or do right? They were hooking DLLs. They were doing other things that are more like non ot attacks, that are still critical, but that’s how they right, were able to get the centrifuges to spin at a speed that was not what the computer was not what that engineering workstation or HMI was saying it was spinning at Right, right. It was that hooking is very similar between both. And I think that’s an interesting point when you look at ot malware, right?

VAMOSI: So what Dan’s suggesting is that Stuxnet is an extreme example of IT / OT conversion.

GUNTER: Yeah, it’s been converged since 2000 what? 2008 or 2007

VAMOSI: So how does one get into this line of work? Do you need to come from the CS side, or can you can in front of any background?

GUNTER: I would just say going back to the workforce. It’s critical for folks to get started right? And everyone has to start somewhere. You know, I started my career in the Air Force. I just started doing stuff in the Air Force too, going out, right? Some, some of the best talent in the world, whether it be in the US, whether it be in Israel, does start with us, military or IDF or other service, right to get that, I think, as a community, something’s big, and I’m curious on your opinion, talking to people on how getting in, like, you know, people can start military, there is an approach, I think, for folks to kind of start getting a job, right? Intern at an asset owner, become a part of the security program that has, you know, cyber, physical consequences on both sides, right? But what are your thoughts? Where do you see where we can solve some of the workforce gap that you’ve seen work in other areas? Oh, man, well, I don’t know how I feel about stuff coming out of the military, necessarily.

VAMOSI: I know there’s a lot of really good stuff that does come out of it. The Internet itself came out of the military. So there we go. But I think the problem still is that when I say that I have a podcast on OT or ICS, people look at me like you have a CISSP. What are you doing? What they don’t recognize is that this is a part of the fabric of, as you said, cyber, physical, or just cyber. And so I think the problem is there’s a workforce out there that doesn’t understand this space. And once again, when I was approached with this pitch, I was like, Oh my gosh, here’s a company that’s actually doing ot security, and they’re not it’s not an add on to the other things that they do. They’re just tacking it on because it’s cool and hip to do OT today. So I think just getting out there and saying that OT needs to be secured is probably the biggest problem for me.

GUNTER: I agree, I agree, and I think, I think a lot of opportunities are popping up on that internship side, on the side as we understand, you know, and more money is going into the criticality of the problem, right, right? What’s interesting for us is always seeing what the priorities are, right? What are the national funded priorities when it comes to OC OT, security out of DOE or scissor, right? How does that change, right? But globally, too, right? Because to the point of enabling even folks to start an industry like that, does. Doesn’t matter, right? How are we building the workforce? How do we prepare them? Right? How do we take someone out of school and, you know, get them in the problem or get them into the problem space without gate keeping, right? I agree with you in this space, we do. We do it often. I feel like over gatekeeper To where folks are like, Oh, start on the Help Desk. Start on others. And yes, like some of that experience is key, but you also have some smart folks that may be coming out of doing their PhD, may come out from some of the top tier universities, even in undergrad, you know, doing that interesting work, right?

VAMOSI: So a lot of you raise a good point. A lot of the stuff I’ve seen comes out of tinkerers, people who mod things, people who do this stuff on the side and fall into it because it’s interesting to them on that level, but they’re not thinking that it’s cyber related in any way. And there’s, there’s a disconnect. So is that something you’ve observed.

VAMOSI: So I’m always curious, if you’re not from a cybersecurity background, where has Dan found success?

GUNTER: Most talented people I’ve worked with, some have not actually had any previous info SEC experience, but they have come in again, like one, one person I worked with, they they were working with a company where they were doing, like, it was one of the automotive companies that you can like, open the door, rent a car, open the door, and use the car, and you know, their infrastructure to do that. He was more actually on the site reliability, on the IT support side of that, but on that, he had scripted a lot of efficiencies to kind of work through that side. And so, you know me, as I was hiring and working through it. It was like, okay, this person, like, has worked at a company to where, like, they’re unlocking doors and turning cars on and making sure, like, right? If you use your app and can do that to the car that hits the car, it should go to that there’s a lot of risk going there, right? Person ended up being an amazing security hire because he started with that tinkering side of just right, and working at the utility that has to do this, or a SaaS app that has to do this right, and it can start there to where. And he had no traditional education. I don’t think you know, beyond the high school level. It was all learned, like, you know, from that practicality side, right?

VAMOSI: To Dan’s point about gatekeeping and stuff, there was this barrier 20 years ago where it’s like, well, you don’t know enough. Or, as a journalist, I got a CISSP because I wasn’t being taken seriously. People were like, well, what do you know? You’re just a journalist and whatever. It’s like, no, let’s talk about this, the weeds here. So it’s nice to see that some of those barriers are coming down. And then, to my point, it seems like the people that I talked to in this space have been the garage tinkerers, the ones who decided to take a smart meter and tear it apart themselves and understand its internal logic and then rebuild it again.

GUNTER: Yeah. And I’ll point out too, like, what’s interesting in our space is some of our most successful, some of the most successful, like, Red Team engagements, where we’re actually breaking in and causing an impact, either on the lab environment or small factor safe way of doing it. It’s actually come from going to like, non computer science, non cyber like electrical engineers, and being like, Look, you sat at this console for the last 15 years. You’ve distributed power and done this like, what would you do if you were an attacker, right? And some of that conversation, honestly, as we work through the problem, led to like paths that we would then take to a red team engagement, or take to detection engineering, if we’re working on the blue side, and be like, Hey, let’s, let’s make sure we can’t do what that really smart power systems operator that doesn’t know much about cyber security, but he does know the power system, like, how do we incorporate and bring that in, right? And we’ve had a ton of wins, right? And so when we go on site, like, right? We’ll try to talk to both crews, sometimes in separate rooms, to be like, Hey, let’s, let’s make sure this isn’t being pulled into someone’s cognitive biases or into like, group think, right? Or that people are scared to bring up, like, Hey, this is what I would do to bring the group down pretty quickly, right?

VAMOSI: Well, that that is the thing, because the bad guys often come up with that thing that is so off the wall, out of the box, or back to your company name, insane cyber that it’s like, why didn’t we think of that first?

GUNTER: Oh, absolutely, absolutely. And that’s, you know, that’s why. That’s what I’ll say to anyone thinking, or listening to the podcast, thinking about a gun in this space, like a lot of people, like, that’s what drives them, right? Not just the like, you know, computer science side, but that side of like, hey, how does this work? How would like, because, like, right, as defenders, we want to find it before attackers do if we can’t patch it, at least get the Monitoring there, right? And when you talk about, like, validation at the end of the day, and feeling like you’re making an impact, like there’s nothing better of an impact than knowing that you either kept someone safe or kept that level right, because the consequences right, like you go to a refinery dealing with benzene, right? If that benzene gets out, like the cancer rates are going to go up within miles of everyone in that area, right? And the consequences are like you’d never want to overstate or fear monger, but the consequences are significant.

-30-

The surge in renewables and decentralized power is reshaping grids — and exposing them to new operational and cyber risks. In this episode, Rafael Narezzi, Co-Founder & CEO of Centrii, explains how rising connectivity widens the attack surface, leaving energy infrastructure increasingly vulnerable.

[Heads Up: This transcription was autogenerated, so there may be errors.]

VAMOSI: In late December 2025, a cyberattack struck roughly 30 electrical facilities across Poland tied to distributed energy generation. The operation — attributed to the Sandworm group — targeted communication and control systems at combined heat and power plants, as well as infrastructure managing wind and solar energy.

Exactly what the attackers intended remains uncertain. As the security firm Dragos noted, it is unclear whether they attempted to issue operational commands or focused solely on disrupting communications.

Further reporting from cybersecurity firm ESET indicates that the attackers deployed data-wiping malware known as DynoWiper. Although the incident did not cause power outages, Dragos reported that the breach allowed adversaries to access operational technology systems critical to grid operations and permanently disable key equipment at the affected.

This is a story about how the electrical grid is again being targeted and what the industry is doing about it.

I’m Robert Vamosi

This is Error Code

NAREZZI: Rafael Narezzi, Co-Founder & CEO of Centrii. We are the only one that can translate renewables, energy risks into a financial outcome from the cyber security perspective

VAMOSI: I’m glad Rafael started with that, because I’ve talked to a few other people about the impact of renewable energy on our existing electrical grids. On the one hand, it’s great that we’re moving away from fossil fuels. We’re getting all that solar and wind energy, but it impacts the older grids that are out there and the addition can also cause security concerns does it not?.

NAREZZI: it does. I think the field that we live today is a decentralized field before, I think, when I started internet in 1995 long time ago, there was not many generators at all connecting to the grid. In terms of a connection from the internet, managing managing remote locations. Of course, depends on type of assets that we’re talking about here in the in that situation, the only way for you to attack generators was from a war or terrorism or physical situation. So moving forward now to 2026 we are living a hyper connected world. Everything is IoT, everyone, everything is talking. You have a marriage from IoT, it to the OT, which becomes kind of a lot of soup of letters, right? So, and that’s where we are today, with a landscape of threats scaling, growing in the landscape of connection also huge. So today, even though, despite the many asset owners from the generation perspective, are setting fences in CCTV. What? Therefore, if is the risk is digital, right?

VAMOSI: And just to point to something that happened, I guess a year ago, we had the out outages in Spain and Portugal. And initially, people were like, Oh, that’s a cyber attack. And whatever, no, it turned out to be, if I understand correctly, an abundance of energy that just wasn’t compensated for.

NAREZZI: totally. And I think there’s two important connections to attribute to what you’re saying there, right? First, the balancing. Yes, it was a issue from the grid perspective. However, there is no explanation yet why two big plans of a Spain in peak time during 11 O’clock solar, big sun suddenly stopped to produce right? So we don’t have a conclusion of that. Despite that, we have a conclusion saying that we didn’t have enough energy. So which is the balancing situation? I’m not saying that it was a cyber attack, but is, we don’t have evidence to say that it is. It was at the time. However, I think what’s happening, and it’s been happening on the grid moving forward, Poland recently, I think a month ago, has received attack, which is the same situation, playing with 30 windmills in order to kind of the relay. So actually no stopping generation of the site, but actually playing with the balancing of the grid, because I think the cyber criminals or the state sponsor acts understand that if I play with the balancing of the grid, I will create a domino effect on the grid. So unfortunately, Poland was more prepared from the battery side, which Spanish didn’t have so many batteries to deploy fast in order to kind of a balancing back to the needed right? So that’s the kind of a scenario that we’re leaving. And I understand now that from the balancing situation, we will be able to see more attacks vertical happening. Why? As I just demonstrated on the beginning of our talk, as we decentralized energy, as we become kind of hyper connected, our landscape is pretty higher than anticipated before. In the targets becomes quite easy to do it. And if I can do a kind of a balancing situation, that I can cause grid outage. I don’t need to attack the big players. I can go to the small, niche players from the generation side, build a capacity, and then, as I call a DDOS from the energy field, which is multiple generation, small sites, which and then I can create a kind of maybe offset demand, or more demand the grid needs, and then you have the same situation that happened in Poland or even in Spain.

VAMOSI: So you have these, these batteries that we talked about, but there’s also the generation aspect, like at the windmill at the solar panel, there are components there that have been installed without the security concerns that are necessary.

NAREZZI: So you could have a widespread attack of solar panels across a residential area, or windmills up on a hill somewhere. Could you not? Yeah, you could. And I think just to illustrate that, who doesn’t remember the ocean’s 11 movie, when the guy has a white van, goes to the top of the car park in Las Vegas, and then he press a pinch device, which is actually, if you compare to the likelihood of today, you’re going to see a battery a best so if I dispatch a large volume of battery, which is today, by the way, controlled by using an password, into a grid, what do you expect That will cause it to the grid? If it’s not prepared to handle that kind of a discharging in that time they need it definitely going to have a balancing situation. The same thing will happen in your house if you start to consume a lot of energy in your your your plug is not up to kind of feed all the energy your digital tour goes off. So a similar situation. And the aspect is, how do we control the multiple proliferation of assets that has been connected to the grid? That’s one aspect. The other aspect that you’re saying, Yes, we are growing agro tech, vertical farming, cannabis farm, that we already help data centers, and why we are helping, not helping them to produce whatever they produce, we are helping them to protect the energy, because the decentralized, decentralized field as well, becomes opportunity for business to be in a remote location, actually producing what they do the best, and actually ensure they can actually do what is more essential for them, which is energy. Without energy, there is no business, and that’s why we are here for now.

VAMOSI: You mentioned Bess, if you could deconstruct that for people who don’t necessarily know what that means, yeah, it’s a battery, right?

NAREZZI: It’s a battery storage in order for you to charge when the brigade re demands more power. So we have cars. It’s kind of a best if you think about it. However, the cars today only receive the energy eventually, as we go more along with the proliferation of smart grid, you might going to be able even to trade your your energy of your car back, back to the grid. I think the grid is already at for that. But I think we see this happening. However, when we see a best scenario, which is very, very common use on the trading aspect. So I I charge my battery on the on the off peak time, and then I discharge on the peak time, and I make, kind of my my, my money from there. However, we have optimizers, as is called a software, a software that actually connects multiple batteries together, even the software perspective, which connects to multiple batteries. So I can, instead to attack individually asset owners, I can attack the supply chain, which actually connects all the batteries together and then dispatch all together. So the scenario of the landscape of a connectivity in remote dispatch is out there, however, is how we are protecting, how, how much visibility you have and and how, how confident are you that is actually who is dispatch? Is the person that should be dispatched in the battery.

VAMOSI: So are there regulations and standards in place for this, or is it just sort of a free for all?

NAREZZI: I think start to have certain regulations needs. What that 2.0 in Europe is one Merc SIP is another one. However, there’s different aspects of the regulations. I think the regulations are getting soft. So nice. 2.0 I think has been better one, because they make the director liable in the company, even to lose the license. I think next step is being quite more soft, understanding that they have time to address that. Kind of just giving a penalty on the money we see that we think is important, I think is very valid, but doesn’t give kind of the really the penalty for maybe bodies to move. What we see from the knees 1.0 to the 2.0 knee is 1.0 is start, very similar to the neck SERP, similar like we find there is just kind of a issue of a fine amount, and that’s it. And then when we see the situation of the needs 2.0 where they kind of first increment the scope and then also penalize directors to be reliable, they are not expecting you to be the best on cyber. But at least the basic is necessary, and that’s what we need for and also the company that is generating the energy. So if I have a company generating and I get a cyber attack, am I going to compromise my entire license of operation? Is that worth it? Should I invest a little bit more? And I think that’s what I think need to happen more. I’m not saying that we should break the penalty at all to individuals, not at all, because in Japan, for example, there is a regulation there, and there is no penalty, but it’s a they take as a very kind of a discipline. Is a family shame if they don’t do it, and you don’t need to have a penalty. So once again, it’s more a way of operations then, rather than a conscious like as a culture, I would say

VAMOSI: Electrical grids, they are kind of brittle in some ways. We have this rapid technological growth going against that. Are these regulatory bodies and whatever responding fast enough, because in the past, they could go years and years and years and come up with a new standard, and that was fine. The iteration now is much quicker, much faster, and the response needs to be there. Is that happening?

NAREZZI I think is. But knowing the speed that we need it that’s that’s what’s happening. I think technology is moving faster in every single direction. When I say technology assets, deployment of generation, I consider as a technology, and that’s why it’s getting hacked. Many people say, Oh, but this asset is in the middle of nowhere. Is is it just a windmills? Just a solar panel? Yes, they are. But also behind of those are softwares. And softwares, as we know, get hacked on the it is very, very common, but OT is falling apart if the same part of a kind of vulnerabilities. So I think the end of the day, whatever produce electrons, if it’s a windmill, if it’s a battery, if it’s solar, is a software and we know, in fact, software can get hacked. And if you can get hacked, then you can affect maybe a company. However, is a big difference between it and OT. The difference is it, if I have a cyber attack, I can stop I just unplug my computer, my server, I take it out, I put offline. Of course, I’m not saying that, because probably the IT folks is going to say, hey, these guys, these guys crazy. I’m not going to do that, right? I understand. But you have this option, how do I mitigate a issue or an incident on the energy which I cannot stop the energy? That’s kind of a different matter where I see it’s completely different environment to deal with.

VAMOSI: So what I’m trying to figure out, what solutions do you bring to the market that would would help here. I mean, how do you play a role?

NAREZZI: Well, I think in the market, there’s a lot of solutions in place. We do all our part as well to offer, like a scanning asset inventory vulnerabilities, which is, I think if you go around, they’re going to be, hey, this guy, this guy is late. The market is already involved for 10 years ago. I understand we can do that as well, but we can also integrate to all those vendors. So we are agnostic on vendor. What we provide better is actually the financial in the risking back to the company taking the KPIs of cyber. So instead for me to go to a top executives and tell, tell them about CVS, how critical is a sev, you know, how vulnerable can be, I go to the executive and say, Look, here we go. The 300 400 sites that you manage. There are like 10 here that you have to concentrate right now do the fact, if it’s a hack happening here, your exposure of financial will be like 2 million. Would you like to invest on the stand sides or the 300 sites? So I think we do more kind of a pragmatic approach based on the understanding the environment, understanding the site, understand the technology that generate the energy. Therefore, as I said to you, solar is eight hours. Wind is 24 hours. Battery is twice a day. So there’s a lot of components of risk criteria that we taking in place in order to create the right profile. So when I come to the executive, I’m not giving him KPIs on cyber, which everyone is doing, and the executives are looking and say, Okay, what do I do with that? Which is red is the last red of the red that I need to focus my attention on, because everything is going to be red, as we know. So I think what we try to do is say, look, those red flags that we see here actually is your your kind of hit pot where you have to invest right now, because I think the consequences were quite significant on your on your budget, or on your impact financial situation. So that’s what we try to do very well. So that’s why, I think, distinguish us from the market as well. Because also one is our most important aspect of OT environment, is we are become verticalized. And this is a bit is a big impact. Why? Because we verticalize on energy. We are experts on energy. We cannot be energy, manufacturing, pharmaceuticals and everything, because and then we are not understanding the financial impact. Do the fact that i becomes the expert on the energy, then I make sure that I speak the language, by the way, even our prices per megawatt on the subscription make sure they understand our language as we understand their language. So it becomes a very common system to be used because they know how much it’s going to cost. Doesn’t matter if it’s a site one mag or 500 Meg is priced accordingly to the size and the needs of each site.

VAMOSI: So what are some of the issues that you bring to the risk table when you’re discussing with them the risks that are different from just saying, hey, there are these priority CVEs out there. You really got to hit those first. You might bring something totally different to the table. What might that be?

NAREZZI We do? We break into categories of quick wins, big projects, and also what they need to do in order to have a fast execution. So when I present the numbers, besides of the numbers, there is action plan for them to follow. Normally. They can do that internally. They don’t need to invest another company to do the industry today, what they do, as I call it, a spray budget, so they pick one item they believe that is be across the board. They buy this as a budget. They say, this year, I’m going to invest in patch management. Let’s imagine that is the case, and that’s going to be the flavor of the year. So we can do patch management across right in is the Patch Manager is the solution for everything? Or are you covering the most impactful side from the financial implication for your client? And I think when I start to talk to the client about compliance, the financial aspect, and also the risks associated to the criticality in the CVS in translate, using $1 sign in the front, he understand that if he invests like maybe 100k in return of a 2 million makes more sense than invest like 2 million across everything that maybe is not covering the aspect of diploma that we suggest in the beginning. So I think that’s what we are trying to do, is giving more framework where they can base the risk on the financial rather than the CV.

VAMOSI: I am wondering what particular weaknesses in OT that Rafael foundt, as opposed to the general IT level of risk.

NAREZZI Yeah, well, everything is in part of the same scenario today is the industry has changed. Very few industries still have air gap, if I can say still having air gap, the situation is pretty simple. Cost of operations on air gap is pretty much three times more than, you know, remote situations. So therefore, everything reflects money. Even then the company might say it’s not that safe. The money will always be in the front of the aspects of decision. And I think even if it’s as I said it ot IoT, whatever talks on the network have their way, including all the protocols that we cover, I think at the end of the day, what matters for any company is the financial impact and exposure, and that’s what I think they need. They are able to take the best decision when they are provided with the financial guide that is in front of them. So when I sell, sometimes our product, yes, the Caesar is part of the process. However, who buys more our product is a CFO is the CEO? Is our people that are more driven by financial risk rather than technical risk. I think it’s important, that’s for sure. You have the technical, have the compliance, you have the financial but we combine everything under one umbrella and then say, Look, that’s that’s your exposure, that’s your criticality, and that’s your actually a number that you have to reflect on. And I think that’s why we’ve been very successful on our journey.

VAMOSI: We talked a little bit about some outages in Portugal and another Poland recently. How secure is the grid in general? Because Hollywood has this impression that it’s so easy to bring down, like all of Los Angeles, all of New York, whatever. How, in your perspective, secure is the grid?

NAREZZI . Well, grid is kind of safe, in a way, but you have to think about this being viewed for, you know, one of the oldest infrastructure of our times, right? When they build it, they didn’t think about, you know, how many generators are we going to have, how many points are we going to kind of interconnect the demand and supply in terms of requirement is coming from the data center, for example, right? They couldn’t predict all these situations that we are we are seeing today. And also, I think what has speed up was to do with the pandemic as well that we just passed by, right? So that I know for enforce everything to be more digitalized, to be more remote. So before I used to control my generation under my radar now is everywhere, and I need to kind of control eyes on that. I think the situation of smart grid what they call as a new name for that, but I give the same comparison of smart cars. You can have a smart cars, but you don’t have the smart roads, so you don’t have sensors on the road. We’re still using cameras. So it’s quite interesting. We are trying to try to try to talk about smart cars, of autonomous car, which is driven by cameras and sensors, sure, but the road is still the same road as we started when we started. So the roads are not ready for that, even though that is despite is working, but it’s, you know, having a sense on the road will be much better than actually having cameras on only on the car and the dependency on the car. So I think what I’m saying is, is the grid. Is the grid ready for the supply of energy that we are seeing in the in containerization of a mitigation of attack. Well, we recently see that just some balancing depending on the frequency. We are not right. This is what’s happening. And I think is, how fast can we improve that? There’s a different order that I think is going to take time. But I think as the companies are getting tired to depend so much on the grid, they become decentralized. So as a data center, the data center is not going to wait for the grid. They’re going to move and actually get their own decentralization field. They own generation side. And that’s what I think we are here to help as well. So yeah, I think is the smart grid still not so smart as we think.

VAMOSI: You touched upon something there. The data centers, the rush to get them built in time. There’s the likelihood that they’re going to cut some corners and not necessarily follow everything directly just to get that data center online.

NAREZZI Yeah, no, and there’s a big demand coming on that I think data centers will be moving to, as we call a multi grid. So it’s a multiple technology together to accomplish the decentralization of the energy and be less dependent on the grid. Because the timeline for connections on data center in us, for example, is five to seven years for if I’m not mistaken, so they’re not going to wait for that time, right? And if you see the last post that I did on LinkedIn, I think yesterday from the Bloomberg, the renewables, or the decentralization of energy, must double or triple from the demand they expected to have. So the numbers are going gigantic. However, it’s still a lot of companies decided to do and why to do under decentralization, I think is still a little bit of a scenario that the growth is needed. Energy is not there in AI only going to happen if you have energy. So I think time is running fast. And if you compare to other countries, they are kind of a fully on board with the der right. This is central, decentralized energy field. So I think it’s a lot of things for us to see in the market happening. See in the market happening there.

VAMOSI: Yeah, so I guess to leave on a final thought, we’re talking about the idea that there are decentralized energy systems entering into the grid, and that they may or may not be following all the rules. And this really broadens the whole landscape for attack, does it not?

NAREZZI: It is totally, I think, as much connectivity that you have across so you open more, you know, as a landscape for attacks, not only attacks, verticalizations, also you have now state sponsored acts that is looking to create any kind of a damage on the critical infrastructure. We are leaving a scenario of a geopolitical now that every single country today has to be, you know, attention and taking the precaution. Manager. However, as you state, we move very fast into decentralization. We move very fast with the technology. But I think the democracy or the politics hasn’t moved as fast as we expect to do to actually control the dynamic of growth that we had on the on the market today for the energy, which is the essential product that everyone consumes every day.

-30-

Many devices on modern networks aren’t what their labels claim. This episode, Rob King, Director of Applied Security Research at runZero, explores white-labeled surveillance and IoT hardware, why some vendors are banned by governments, and how hidden risks can spread across enterprises. Discovery, device fingerprinting, and protocol analysis reveal what’s really connected — and why knowing your true inventory is now essential for security, compliance, and trust.

[Heads Up: This transcription was autogenerated, so there may be errors.]

VAMOSI: In October 2016, a large botnet brought down Dyne, an internet provider that supported Netflix and other large streaming services. Botnets were nothing new. They’ve been used to target companies and then extort fees to get the attacks to stop.

Basically, a denial of service attack works by sending a website or server requests, then not following up with typical responses so that the server is opening more and more requests, such that it handle any new requests from users legitimately trying to access the site.

Obviously one computer can’t send all those requests, so distributed denial of service attacks are used where compromised computers all over the world target a specific server to flood it with requests. And yes, content delivery networks can shift the address of the server to mitigate the attacks.

That’s not what’s important here. Here the DDoS attack came from ordinary street and home surveillance cameras. The Mirai botnet used a vulnerability in a white label chip that was sold in white labeled camera. White label means that someone produces the camera and then others buy it and slack their name on it.

This is a story about white label devices that are in your organization, and why it’s important to identify them before it’s too late.

I’m Robert Vamosi

This is Error Code.

[MUSIC]

KING: So hi there, everybody. My name is Rob King. I am the director of applied security research at run zero. And run zero is the, in my personal opinion, the world’s greatest exposure management platform, vulnerability discovery platform, vulnerability management platform and asset discovery, we really do, and I’ve been doing this a long time, we really do the best asset discovery in the world. We really do the best exposure management of the world. It’s super neat. Free to try. Run zero.com/try so check it out. Cool.

VAMOSI: If I haven’t mentioned it before, SecTor is one of my favorite security conferences. It’s held in Toronto, and it’s been under the radar for years, yet it attracts some of the best presentations. I always look forward to it. Ironically, Rob and even RunZero are American. His talk on white label devices at SecTor was really good. So perhaps we should start with Rob’s definition of what a white label device is.

KING: sure, so it might be a surprise to some people, but most of the time, in some for some types of technology, when you buy it. The brand that’s on the label is not who made it. The example I always like to give is essentially every microwave oven sold in North America, regardless of the brand, General, electric Sears, Whirlpool insignia, whatever. They’re all made in the same factory in Guangdong. So there’s they just come out of the factory. A retailer puts their label on it, puts a fancy brand on it, and then sells it. And that’s really true with things like security cameras. Security cameras are extremely commoditized. So for example, if you look at a company like Lorex, or even Honeywell, their cameras are actually made by Dahua. So dah Hua is a Chinese company, and I’m sure I’m saying it wrong, it’s dah Hua. There we go, still saying it wrong, but less wrong. And they they make cameras for tons of different vendors, Honeywell, Lorax, activecam, Amcrest, ikagami, hiltron, KV vision, all these different cameras. When you buy them, you’re really buying a dot block camera.

VAMOSI: So when I when I hear you talk about white label surveillance cameras, I’m immediately going back to the Mirai botnet and the fact that a vulnerability and a white label camera propagated through a whole bunch of systems and allowed a botnet to take over. That’s kind of an extreme example, but something like that could happen with these devices as well, certainly.

KING: And there was one, and it wasn’t one of the ones I just named, but we did have an interesting example of one of these cameras that recommended that you download a root signing certificate from the camera and install it across your organization so that your traffic to the camera was encrypted and secure, right? And never mind, you could also use that certificate to spoof any website in the world. So little bit of a potential backdoor there, right?

VAMOSI: But the thrust of your your talk was more on the legality of having certain devices in your system, because there are certain devices that are now banned by various federal and state and provincial governments. Could you talk a little bit about that?

KING: Exactly. So DAW was one of the big ones named also hic vision, which some people say, Hi K vision, I think it’s hick vision, but either way, I’ve heard it both ways. But those devices are banned by the United States federal government in any situation where it could be considered a secure environment. So observing federal buildings or installed in, you know, organizations, installed in locations where federal data is processed, that sort of thing. And then a lot of state governments also have that ban. And in fact, a lot of state and local governments in the United States will base their own technology lists on the federal government’s list. So it tends to be banned just about everywhere. And then in Canada, same sort of thing, there’s a federal ban across all Canada for certain vendors. And then also several individual provinces, Quebec, especially, have banned certain devices and certain vendors for security reasons, for national security.

VAMOSI: Okay, so I want to dive into this whole idea of how you discover these devices, as Rob was saying, if I buy a microwave and it says Amana or something on it, and it’s not deep down inside. It’s actually mass produced somewhere else in the world. It’s white labeled. Same with a network device. Let’s talk a little bit about the discovery and how you would go about finding this out.

KING: So we there’s lots of little telltale signs. And I love, you know, I got into this industry because I just love dissecting protocols and fingerprinting devices. And this is not a joke. I really love doing this. I would do it even if I wasn’t getting paid. I’m very lucky that I get paid to do it. So there’s lots of little bits and bobs on these devices that, if you know what to look for, you can, you can kind of tell. I mean, sometimes they’ll just straight up tell you, like, if you go to, you know, if you open up one of these cameras or devices or whatever, in your web browser, it’ll say somewhere in the corner, like, oh, it’s tick vision, right? Or oh, it’s Dahua. But most of the time, they’re not quite that, that obvious, right? And in fact, what you’ll often see with these things is that they’re almost, unusually, almost, almost pathologically generic, like you can go to their web interface and there’s literally no mention of a brand, there’s no copyright message, there’s nothing. It’s just, you know, this very generic username, password, login prompt. But that generic, this generic city is is still common across all of them. So you can say, oh, you know, I, I can, it’s really fun at parties. I can look at a random camera web interface and be like, Oh, actually, that’s a big you know, that’s like, said fun at parties. But you can also look at little individual protocol works, which I think are fun. So for example, a lot of times these devices will register a name with DNS. So you’re if you have an environment where you’re getting an IP address or whatever via DHCP, and you can do dynamic DNS, it’ll say, oh, yeah, this the host name is, pick vision, dash, 1732, or whatever. So it’s pretty telling that, Oh, I thought that was a Honeywell camera. No, it’s actually. And then you also have, you know, some obvious stuff as well, like MAC addresses. So every device in the world, almost, you know, there’s a million people in the audience saying, Actually, okay, there’s a MAC addresses. Are the hardware addresses of these devices, right? They’re supposed to be unique across all devices, and they’re maintained. That uniqueness is maintained by manufacturers asking for blocks of MAC addresses from the management authority, which is the I triple E. And so you can just go and look at the MAC address and go look up the vendor, which is based on the prefix, and it’ll say, oh, it’s actually, you know, the Hua technology code limited, right? Which is always fun. So there’s some really obvious things, and then there’s also some more, some more, more subtle things.

VAMOSI: And among the more subtle things, you talked about how even the documentation is generic,

KING: yes, yes, which is, which is always fun. At one point, I had a slide where I had a side side by side comparison of the documentation from Dahua for that one of their network video recorders, and then one from Mountain View. And it’s very much like, you know, Hey, can I copy your homework? Okay, but don’t make it obvious. And so there’s, you know, the blue is a slightly lighter shade of blue, and the model numbers are slightly different, but it’s the exact same, you know, the exact same page, the exact same documentation, more or less just the logos change, which is always interesting, and then sometimes the documentation is also aggressively generic. Where they don’t even mention the name of the product, they’re just like, Thank you for buying my network video recorder. And it’s like, which one did I buy? I don’t know, though, right? Always, always useful there.

VAMOSI: So yeah, in your presentation, you talked about the login screens, and how, you know, just looking at the login screens, you might not be able to tell that it’s a generic one, but sometimes it’s blatantly obvious, because there’s no branding on it, or, to your point, copyright, etc, etc. And then right, and, okay, no, you, continue, sorry,

KING: oh, I was just but then sometimes they will put the logo there, right? They’ll say, Oh, it’s a, you know, an Intel Ross or, or big vision, of course, you know, that’s one of the obvious ones. But then even the some of the subtle stuff in if you go look at the source for the login page, you say, view source, which, for those of us of a certain age, that’s how we learned how to write HTML. Was viewing source. It almost felt like and by breaking the law, if I view the source of this web page back in 1996 but you can go and look, and it’ll just be like, you know, pick vision.js, or something like that. And so it becomes obvious there as well, if you just even peek a little bit behind the curtain. And then also looking at just the HTTP headers. So you know, when you talk to these systems, the system will send back HTTP headers saying, you know, my server is whatever. And the server for hick vision cameras is often a web server called hick vision web server. So regardless of what kind of camera you’ve got, if it’s running pic vision web server, you can be relatively assured that it is a pic vision camera or device.

VAMOSI: So right now, we’re talking about a specific example. But if you’re running, you know, critical infrastructure, if you’re running a utility or factory or whatever you have 1000s of these devices. So how do you begin that discovery process to find out what you have on your network?

KING: I would recommend that you get run zero.

VAMOSI: Yes, okay,

KING: it is truly it is truly quite useful. You can search for, you know, all these different First off, we’ll do the fingerprint for you, and you can say, hey, please show me any dotwa cameras or hit vision cameras. Here’s the, you know, we even have a built in query for different vendors and devices banned under various regulatory regimes. So they’ll just pop up on your dashboard and say, Hey, by the way, here’s all these, you know, big vision devices. But if, in a bout of peak, you just decide you can’t possibly use run zero, but you should, but you just want to be recalcitrant, I would recommend starting to look through your inventory using the discovery protocols for these these devices themselves, a lot of them would have, will have proprietary protocols and open source, you know, standardized protocols that they use for discovery. And what’s interesting is, all of you know, all of the Dahua devices will respond to the Dahua Information Protocol dhip, even if they are not branded Dahua, right? So you can download the Dahua DH IP app on your phone and run it, and suddenly your cameras that aren’t labeled Dahua will start responding to to this, you know, these, these probes, which is like, oh, okay, maybe those are Dahua kind of things, which is fun.

VAMOSI: So explain to me a little bit about this discovery protocol, the proprietary ones. I hadn’t really heard much about that.

KING: Yeah, so there’s been with the advent of IoT and everything’s a computer now, your refrigerator, your microwave, your washing machine, there’s been this huge push to make them configurable and standardized on the network using protocols that anybody can talk to. You know they want to be able to be configured remotely and talk to the Internet. In fact, some of them will even do nice things like punch holes in your firewall, which we had a nice discussion about that on our blog the other day. But these discovery protocols are designed because these are consumer These are pieces of consumer technology, and they want consumers to use them. So they want to make it easy to show up on your phone and look how high tech they are. And they don’t want to, you know, long gone are the days where they expect you to, you know, laboriously type in an IP address on the printer’s front panel, or, you know, flip some dip switches or something like that, so, or just have a static IP address that is always there that just conflicts with everything. So now they’ve got these discovery and configuration protocols, and a lot of them, so a lot of them, for discovery are fairly standard, like multicast DNS is almost universally supported. UPnP, quite popular. But for configuration, the protocols tend to be a lot more proprietary, and these proprietary protocols will also give a lot more detailed information in some cases. So like I was saying with Dawa, they have the hip, and then hick vision has their own called, I forgot what hick visions protocol is called. It is called sadp, which is the something something Discovery Protocol. But these protocols provide, you know, a mechanism for discovery of these devices, and often, a way of configuring them that would just wouldn’t be available with something like mDNS, you know, and these protocols are one of my, one of my joys, one of the things I really enjoy doing is reverse engineering these protocol that I can, you know, send out pro packets myself without relying on a third party tool. And it’s amazing what you find when you do that.

VAMOSI: So just to abstract it, to make it friendly for everybody, you buy, say, an LG washing machine, and you download the LG app and it says, Oh, now, now you can configure your wash load on your phone, but you also have an LG refrigerator. Would you like to configure that as well? Etc, etc. So that is an example of LGs Discovery Protocol going out there and fingering your network and making sure, like, all the devices that they have control over are pulled into the app for your convenience,

KING: exactly, exactly. And yes, interesting use of the word control there, but yes, absolutely right.

VAMOSI: And then I just imagining, though, like, are these, like the Dawa and so forth? Are they for consumers, and you’re using them in a network, and, sorry, an enterprise or industrial environment, are these actually industrial discovery protocols?

KING: So a lot of these cameras are what you would get for home security, which there’s been just an absolute explosion of home cameras recently. I remember, you know, thinking it was so weird that my friends were like, Oh, I’ve got cameras on every corner of my house. And now that’s that’s almost common. Spoiler alert, if you see the movie weapons, that’s a major plot point. But a lot of these devices are also industrial and prosumer and designed for outdoor use, heavy weather use, secure use, monitoring businesses and gates and in fact, a lot of them even integrate with things like door controllers and gate controllers and people counters, which is a whole class of technology that will actually use facial recognition and AI and say, Hey, this is, you know, this person came in, this person, the same person left. Two people came in, but one, only one left going out, you know? And that’s a whole other fascinating piece of technology that a lot of these devices integrate with them.

VAMOSI: So a moment ago, you laughed when I used the word control. Are you thinking nefarious purposes, or is it just something as simple as, like, Oh, my washing machine needs a software update and, Oh, it got pushed out automatically. I don’t have to think about it.

KING: I mean, it’s definitely, that’s part of it, the nice part, but the reason why a lot of these devices are banned in government networks is simply because the powers that be are worried about how much control these companies and these vendors can have in known secure or supposed to be secure environments. We don’t, you know, I’m not going to say that we have any evidence, because I don’t think we do. But theoretically, speaking, you know something, something nefarious could happen with sending out a software update that sure it improves the code, but also it adds a little bit of surveillance tech that maybe you don’t know about. So it’s, I don’t want to say that. It’s I don’t want to, I don’t want to impugn anybody. But yes, there’s, there’s a little bit of nefarious potential there, I suppose.

VAMOSI: So it’s a bit of you’re assuming good intent by having the vendor push out the code for you and update the device for you, you don’t have to run around on Saturday and do patch Saturday of all your electronic devices.

KING: I’m assuming good intent is not necessarily the right word. I’m assuming consumer convenience, right? And also, by the way, you should patch all your IoT devices. When was the last time you checked to make sure your printer was running the latest firmware? Maybe not ever. Yeah.

VAMOSI: Well, that is the IoT problem. It’s like, the stuff that businesses were kind of good at consumers are really bad at, and often they don’t really think of the two things together. It’s like, yeah, I can turn on my dishwasher remotely with my app, but it’s like, on the same token, what software is it running and how what was the last time it was updated? You know, nobody thinks in those terms on their own has to be right, pushed down from somewhere.

KING: So what was it the I’m flying 35,000 feet, but thanks to the wonders of technology, I’m annoying my family by making my car horn deep in the garage, like exactly that’s it’s great.

VAMOSI: So going back to the white labeling in businesses and so forth, having these prohibited devices, are there consequences for businesses that are unaware and haven’t scoped to see whether they have these devices.

KING: So, you know, I am not a lawyer, so don’t, don’t take anything, I say, as legal advice. But these devices, yeah, you it can put contracts with federal business and or government public business entities in jeopardy. You can face fines. You could face, you know, up to including, I guess, contract cancelation if you can’t prove that the environments that you’re using are free of these vendors. And we even had a famous example I like to give run zero in, I can’t remember where it was, but it insisted that there was, you know, we were absolutely sure that there was, in fact, a band vendor at this site. And everyone was like, No, there’s not. We checked. We looked everywhere. We looked everywhere there is a demand vendor on this network. And the guy was like, you know, if we don’t find this soon, if you either you’ve got a bug or something else is going on, anyway, in the vendor’s pocket was a phone from event from a band vendor that had connected to the Wi Fi. So, you know, the call was coming from inside the house, literally.

VAMOSI: Yeah. So this is then, just part of a regular security assessment that would be performed. It’s one of the things that would be scoped into that assignment.

KING: Yeah, indeed, and it’s generally considered part of the due diligence before you enter one of these contracts, right, right?

VAMOSI: And so you attest that your network is free of these devices, and the contract goes forward, hopefully, hopefully. Yeah, I noticed also that there was no grandfathering in of these devices. It’s pretty, pretty strict. It’s like, if you have them anywhere, they must be removed at any cost.

KING: For a lot of them, yes, and in at least one of the provincial governments in Canada, they did give a, you know, no going forward. But the ones that were there were okay, but that was the exception to the all the other ones I read about were, were definitely, you know, nope, you know, these must be removed by this date or else, right?

VAMOSI: And that that is a problem with this whole IoT conundrum, because there’s, you know, I’ve seen estimates, billions of devices out there. Yeah, the problem is out there in the world, and now you’ve suddenly got to rein these devices in. And oftentimes they’re really dumb and they don’t have a lot of memory and they can’t be queried or whatever. But you know, so be it that’s that’s where your business comes in, finding good asset discovery system, right? Exactly. So you guys don’t do just devices, right? You’re also doing all sorts of network discovery, right?

KING: Yes, it is. I take it personally. If there’s a device on your network or a service on your network that I can’t fingerprint, and I really do like, I get mad. I’m like, no, no, no, no. I’ve never, you know, we’ve never seen this before, but by the end of today, I’m going to tell you exactly what it is or die trying, right? So it’s a lot of fun. So we can detect any any piece of software that’s listening on a network socket. We will try to identify that software, any device that’s on the network, including stuff that speaks relatively esoteric protocols or even tries to kind of hide itself. We tried really hard to detect it. We also will detect devices that are acting as network bridges. So, you know, there’s a famous example of the aquarium pump in the casino where their network was completely locked down and everything was secure. But then they brought in a smart aquarium pump and connected it to the Wi Fi, and it was bridging the public, the public network, with the secure network. And attackers were actually able to ransomware a casino by going through the aquarium, home and run zero can say, Oh, by the way, did you know this device is acting as a router? You should probably look at that.

VAMOSI: It’s funny. For years people in IoT have cited this example to me, yet other than a brief mention in DarkTrace’s global threat report in 2017 — which is now offline — there is very little documentation that this event even happened. And I haven’t found anyone who can provide details about the attack other than an attacker used the online aquarium system as a way into an otherwise secure casino. I am not saying it didn’t happen; I’m just saying for the last ten years you’d think there would be more detail about this event. From credible sources. And there is not.

KING: Although it is incredible, like, Why do smart light bulbs act as routers. They really shouldn’t be doing that, but they right. And then one that our illustrious founder, HD Moore has brought up a few times, that I think is really interesting is even people, you know, in the industry, they don’t realize that oftentimes certain bits of tooling, like, if you’re running, you know, a large number of containers, or, I’m sorry, any number of containers on your on your laptop, or, you know, you’re running some virtual machine software or something. Oftentimes, those configurations will turn your laptop into a router because they want to be able to route to those machines, but they don’t say only route on this one local interface, because they want to be able to talk to be able to talk to the rest of the world or whatever, and so you’re suddenly your router becomes a bridge. Yeah, I’m sorry, your laptop becomes a router and a bridge between potentially many networks.

VAMOSI: I hadn’t considered that. That makes total sense, because you want your VM to talk to the internet. Yes. Okay, so for those of us at home that are just playing around with this, you mentioned in your talk Wireshark, I have something on my phone called thing, and it allows me to fingerprint networks, and I love going to like public spaces and just seeing who’s connecting and what’s connecting and all of that. How is that? I mean? Is that a good entry into this space?

KING: I will tell you that you it is the most fun you can have at a coffee shop to open up Wireshark. And I’d recommend anybody do it, even if you’re not really interested just to see how the sausage is made. It’s the amount you’ll learn just by kind of watching it flow by is amazing. Years and years ago, I, for several years, I taught a class on network forensics. And you know, we would have people come into the class and they never even heard of Wireshark. Some of them didn’t even understand. And I don’t want to make this sound like you should just, you know, nobody’s bored understanding this. But they came to this class with no foreknowledge of even the concept of, like, packet oriented communication, like, you know, breaking things up into packets and then sending them across the wire or the ether or whatever. And by the end, they were like, This is great, you know, I can see exactly how it works, and I can, I’m going to go home and I’m going to run Wireshark on everything. And I was like, yes, yes, I have succeeded lots of fun.

VAMOSI: So neither of us are lawyers, but so the Look, But Don’t Touch rule, you’re looking at these packets. What can you glean from that, legally or successfully?

KING: So, in a lot of networks, the packets are just there anyway. It’s not like you’re doing anything wrong. You know, your computer is receiving them kind of, sort of anyway. And a lot of cases and they’re just dropping so just looking at opening up Wireshark, there’s, again, I’m not a lawyer, but really, there should be no harm we have gotten as an industry. We have gotten better about trying to make things encrypted by default, zero trust by default, and all that. But there’s still an immense amount of traffic out there that is unencrypted, and you can just see information flying across the wire. And with the advent of all this, these IoT devices who want to be discovered, right? They want to be easy to connect to. There’s just a regular pulse of discovery traffic going out saying, you know, oh, by the way, I’m here in DNS advertisement. Hey, you know, look at me. I’m, I’m here to be, to be connected to and printed with. Or, you know, whatever this device is. So even if we’ve started encrypting most traffic, the IoT wave has made discovery so much more popular, so much easier. And then also, if you get into these sort of, I don’t want to say like retail, but you know, if you know, if you go to like a place of business, oftentimes, you know, the coffee shop doesn’t have a dedicated IT staff, right? And so a cash register is probably not going to be going to be set up once and never updated again, you know, or you have a contractor that only comes out once a, you know, once a quarter or something. And so these devices often have some very interesting bits and bobs running on them. You can even see things flying across like you may not. Hopefully it’s encrypted, but you still see things like, you know, the cash register is very talkative now somebody, somebody is running a job or something, which I think is, is fun, and then also just host names like, oh, I now know that the guy sitting over there must be named John because there’s a device on the network named John’s iPhone. A little bit of PII just sitting there for the picking.

VAMOSI: Yeah, that’s what I have done, is just like, Oh, you’ve named all your devices. How convenient. I know who you are. Awesome, exactly. So you also mentioned in your talk that SS seven is gone away, and now it’s SIP its voice over IP. It’s how are devices taking advantage of that? We just talked about Wi Fi, but now we’re talking about, like, other forms of communication.

KING: Yep, and yeah, SS seven, man, that’s Wow. Thank you for Thanks for the memories I have. I’ve totally never done anything illegal with the phone network ever so and that’s my story, and I’m sticking to it anyway. Yes, so sip, which is the session initiative, Session Initiation Protocol, is what’s used to control voice over ID calls in most situations. And I remember, this was in the early 2000s wired or some other technical application said, oh, you know, by the by the 2030s you know, 50% of phone calls will be going over an IP network. And I was like, that’s never gonna happen. That’s impossible. And of course, you know now, everything is voiced over IP. And so SIP is interesting in that it will, if you’re, if you’re talking to a SIP endpoint, it will tell you quite a bit about the device. Can tell you quite a bit about the device that it’s running on. So a lot of these, these, you know, potentially banned vendors, will have interesting, you know, sip headers like, oh, it’s, you know, the hospital server, or whatever. So there’s yet another way of potentially identifying, identifying these devices.

VAMOSI: But what are they using those for? If they’ve got a web server, if they’ve got other things on there, what is the SIP server doing for them.

KING: So a lot of them are also intercoms. So if you want to talk to somebody over there at the gate, that’s often set up using sip and then a related protocol, sort of related protocol, RTSP, the real time streaming protocol is what’s often used for shipping video data and audio data back from a camera or capture device to a recorder. Okay, so lots of RTSP, going on, right? If we’re talking about cameras, that makes total sense. If we’re talking about washing machines. Made me not so if you see your washing machine running, RTSP, yes, I would start. I would start, I would start asking questions.

VAMOSI: One of the cool aspects of digital fingerprinting is that you can find regional variations. For example, the same device except the firmware has differences because of local regulations or requirements. That allows someone like Rob to look for the odiities– the needles in the haystacks. That one device that doesn’t belong.

KING: So yeah, so one of the things that we did was scan the entire internet, at least the entire ITV four space, which, again, if you had told me in 1997 you’re going to be scanning the entire Internet to look for things. I would have laughed in your face, but no, you know, it’s something that I get to do for fun now, and I get paid for.

VAMOSI: So I want to stop you there. I’ve talked to people that have done this before. When you’re saying you span, you scan the entire IP before you’re talking about all the open ports, or you were looking at a particular port.

KING: For this case, we’re looking at specific pieces of information, SNP, Telnet, banners, things like that. And yes, people still do some devices do run telnet in 2025 that’s that’s always fun.

VAMOSI: And how long did that take for, say, Telnet, or any of the protocols that you were looking at, or open ports that you were

KING: looking at, excuse me, so to be polite, you don’t want to scan because you don’t want to upset a, whoever’s hosting a scan and B, you don’t want to be rude. And, you know, just hammer a bunch of IP spaces. You also want to try to, you know, kind of spread the spread the spread the love a little bit. And don’t, you know, don’t do things sequentially, because if you hit a bunch of ideas in the same in sequence, you might bump into you might, you know, exhaust a router, a small routers, a state table or something. You know, be nice. But you can scan the entire internet, picking your speed in 48 hours, if you want to be super slow, which is, again, just mind boggling.

VAMOSI: Yeah, agreed. I’m sorry. I interrupted. There you were. You were saying that you scanned the internet, right? So one

KING: of the things that I wanted to learn about with these, these white label devices, is not, or it’s, it’s not necessarily white labeled in this case, but different regional versions of the same product. So a lot of vendors will have a specific version of their firmware for a specific market. One of my favorite stories there was, there was a tool, way back in the day, called HP open view, and it was a network management discovery piece of software. It was super cool. And in the background of the main sort of interface was a world map. And the world map just had all the countries colored in different shapes of gray. And they had to have a special release of this software specifically for the People’s Republic of China, because the island of Taiwan was a slightly different shade of gray in the worldwide release, and you cannot possibly imply what that implies. So special version of software there different countries, different polities will have different regulatory requirements, including the strength of encryption, or theoretically speaking, whether there’s a backdoor enabled or not. And so what I wanted to do was see if we could form some sort of clustering of here’s interesting versions of software in different parts of the world. This seems to be this version, or this telltale sign seems to be highly correlated with this particular region or location. And so what we found was it’s pretty interesting that you can, you can see that there’s a strong correlation for some devices. For example, in Cisco IOS, there are 51 and this is just what’s publicly visible on the internet. Obviously, there are 51 distinct versions of Cisco IOS visible in Canada on the public Internet, whereas in the Russian Federation, there’s 282 distinct versions. But even though there’s 282 distinct versions, Russia still has one version that makes up the plurality, whereas it’s much more fragmented in Canada, right? But what this can tell you is, if you see a version, you see a telltale sign of firmware that’s extremely common in one location and then only pops up once or twice somewhere else in the world. What does that mean? And what that usually means is you potentially bought something off eBay and had it shipped to you or somebody moved from, you know, Indonesia to the United States, right? One of the things we found that was interesting is there’s a very, very common version of my protect router OS that seems to be handed out by some ISP in Indonesia. And so they’re just everywhere in Indonesia. The specific version of microtik, there’s only one instance of that version in all of Canada. Now, microtik is not super popular in Canada to begin with, but there’s, you know, there’s several 1000 that you can see on the public Internet, but only one instance of this thing that’s extremely popular in Indonesia. So what does that mean? It means either somebody moved from Indonesia to Canada and took their router with them, or somebody in Canada ordered a router off of Ebay or somewhere else, and it was shipped from Indonesia. And you know, you might say, oh, that’s just an interesting bit of trivia. And you know, maybe it is, but because of these regulatory requirements, because of the different update cycles that different regions get, that router might not get updates. That router might have a configuration option that isn’t available or doesn’t have something that you need for regulatory compliance in your country, certain encryption algorithms, certain protocol protocol support, default settings, things like that. And so it’s just something to be mindful of that, okay, this piece of hardware is or this piece of software is different, and that’s, you know, things that are different on your network are always things that should be investigated. Wow.

VAMOSI: So it could even be like geo fenced or something where it doesn’t get updates, if it’s outside of Indonesia,

KING: potentially, yeah. And we have one of the things we have in run zero that I always one of my favorite features, and it’s such a it’s such a small thing in terms of the UI, right? But every asset has a an outlier score. So if you’re looking at your network and run zero, and you can say, Oh, here’s all, here’s all my assets. And if you reach a certain number, and there’s a certain bit of mathematical chicanery, you know, there’s this large enough sample size to make this chicanery possible, assets will get an outlier score that say, Okay, this outlier is or this asset is extremely different on this one dimension from everything else. Why is that the case? And sometimes it’s obvious. It’s like everything in your network is running this version of Windows, but this one is running not that version of Windows. So does that mean you missed an update, or something like that? But sometimes it’s more subtle, like the default TCP window size on this system seems different from every other system. That seems weird. Why is your TCP stat acting odd? You should really look into that.

-30-

From Black Hat USA 2025, Dan Berte, discusses the successes and failures of ride-sharing autonomous vehicles in San Francisco, and how these lessons might help design better IoT integrations of cities and AVs in the future.

[Heads Up: This transcription was autogenerated, so there may be errors.]

VAMOSI: On the rainy afternoon of December 20, 2025, a utility failure plunged 2/3rds of the city of San Francisco plunged into darkness.

Here’s KTVU: Amanda, Thousands are still without power though progress has been made. Power has been restored for about 9000 customers. About one third of San Francisco remain without power. Waymo suspended service.

VAMOSI: More than 60,000 residences and business lost power. Worse, street lights and traffic signals were dark. The city shut down its municipal rail service — not because the trains couldn’t run. Rather the stations were dark. In addition to dinners ruined, and street lights out, there were traffic jams as drivers tried to figure out how to navigate dark intersections full of cars. It was challenging as a human. Imagine how it was for an an autonomous vehicle.

Here’s ABC News: Waymo’s self-driving cars were put on pause by the company on Saturday in San Francisco after the autonomous vehicles were baffled by the lack of traffic signals due to a widespread power outage in the city.

Power continues to be restored to tens of thousands of San Francisco businesses and residences following the outage that affected some 130,000 customers. At one point at least 30% of the city was in the dark.

And here’s ABC7 BAY AREA NEWS: the impacts are widespread both inside and of course, on the streets as well. the power outage reportedly disrupted Waymo service, numerous people seeing cars and shared videos like this of the vehicles just stalled at intersections right in the middle, apparently confused because the traffic lights were out. waymo later shared on its app that it had paused service and when it would resume.

VAMOSI: I am fortunate to live in a part of the world where it’s common to see twenty Waymos threading through neighborhood streets. And now we have Amazon Zoox driving around as well. I’ve ridden autonomous vehicles; I support them. They are the safest drivers on the road.

So what happened on December 20th? As a human driver, when the traffic signals go down, you revert to the rule of the right in the US– which means you treat every intersection as a four-way stop sign, and yield to the driver on the right of you. I was driving in Golden Gate Park that afternoon. Cars, with human drivers, were stunned–what should have been aten minute drive turned into a forty minute drive as not one but three intersections lost their traffic lights. Cars slowly navigated these intersections, but of course, humans being humans, the car went at various times. Stuck in the middle was a Waymo.

Understand that it follows the rules, which, in the case of any stop sign means that it waits 3 seconds before preceding cautiously through the intersection. That's how it’s suppose to work, but humans, sitting in traffic for forty minutes, don’t couldn't to three and don’t necessarily yield to others. The poor Wymo was confused — why weren’t the other drivers behaving?

But this wasn’t the first time autonomous vehicles stalled on the streets of San Francisco. In 2023, GM’s Cruise offered a taxi service. Without a power outage, in what might be argued were fairly return circumstances, a fleet of Cruise vehicles shut down in the narrow streets of North Beach.

Here’s KRON 4: four’s man, harry talk to witnesses and explains what cruz says caused the Cruz says this glitch in their system was caused by all the people on their wireless devices at the outside lands music festival in golden gate park. and some people are saying this shows the technology is not ready to expand its use. Traffic near layers street and grand avenue at a standstill Friday night. crews vehicles stopped for no apparent reason with their hazards left. people unable to go anywhere. is that lined up and then lined up with a lab-grown as you know, just a procession michael was going on. Stephen Giddings is visiting san francisco from new york city. he was near the intersection when the backup started to free. Bill Bree was inside of his apartment. he heard the commotion here. all horns blowing and i look and there’s all these driverless cars lined up all the way lay hole and there’s one on her knee that san francisco board of supervisors president Aaron Peskin says this problem is becoming all too common and now have over 50 documented instances of autumn autonomous vehicles getting in the way of fire, a ambulance and police so we stress as concerns the cpc.

VAMOSI: I say all this because I subsequently read several articles, by people who were not in San Francisco, who have never ridden or driven alongside a Waymo, offering their knee jerk analysis that Waymos aren’t ready for the real world. They are, and right beside them, Amazon’s Zooz are testing with human drivers, ready to become the next autonomous vehicle on the roads.

I’ll say this about Tesla– they are no where near ready for autonomous driving. They have about 30 vehicles in a confined space in Austin Texas. And in the state of California they are required to remove the name Full Service Drive and Autopilot from their vehicles. Without Lidar — I don’t see how anyone can certify a Tesla for autonomous driving anytime soon. Seriously. They aren’t ready; they don’t have the road miles that Waymo and Zooz currently have. But Full Service Drive is a topic for another episode.

All this does raise some serious questions about the next steps for autonomous vehicles, now that we have a number of them on the road, and we have see different scenarios playing out. In a moment, I’ll return to my conversation with Dan Berte of BitDefender who is thinking ahead on this, and other topics. And we’ll hear about a new protocol designed to make all our IoT devices work better together, perhaps necessary for true autonomous services in the future.

This is a story about … the next steps in autonomous driving, and for that matter, how our IoT connected world can get even better.

I’m Robert Vamosi

This is Error Code

[MUSIC]

BERTE: I am Dan Berte, and I am IoT director for Bitdefender,

Bitdefender is a global cybersecurity leader that’s very well known for its consumer products, but it’s of recent years, has been applauded with accolades for the new IoT strives and the enterprise foray as well.

VAMOSI: I talked to Dan at Black Hat USA in August of 2025. The first part of that conversation can be found in episode 75, where we talked about IoT-based Living Off The Land Attacks and Air-Gapping Solar Systems. In a nutshell, Living off the Land means exploiting what’s on the device already and not adding code. The thing is Dan is an incredibly interesting person and he had several talks at Black Hat and DEF CON. His company is also part of the MATTER Consortium, which strives to improve interoperability and compatibility between different manufacturers and security.

BERTE: kind of takes us back a little bit into the IoT realm, which is something that I’m very fond of. So, to make it simple, you know how we’ve had these ecosystems where, you know a particular product would work with Amazon, Alexa, or would work with Google Home or is Apple Siri, Apple home kit, and all these interfaces sometimes cross pollinated, and maybe you could use one particular product with one particular protocol into another, and then you have some sort of a management but It was finicky and it was confusing, and consumers didn’t really love it. So a lot of organizations, a lot of big, large companies, and so I’m going to point people to to looking into the members of the matter protocol, which is all the biggest tech companies have decided to standardize in our operation of connected devices. And being part of that group means bringing you know anything from standardization to security research improvements, new ideas, just management to make sure that all these devices that are compliant are seamless and safe. And it’s a noble and an extraordinary, complicated process, and getting on one of those group calls is sometimes confusing or overwhelming. However, having all these companies, all these minds together to self coordinate into a protocol that’s safe, reliable, easy to use, is enormous. So now you can bring any device into an ecosystem with the easiest steps, and they just work as you’d expect. So we’ve done research into cybersecurity blind spots and in the protocol.

VAMOSI: So MATTER is not new, however the talk at DEF CON is special.

BERTE: So we’ve had some talks at large conferences, but we also, and this is by the way, this year at DEF CON. It’s the 10th year of the IoT village. They’re celebrating that decade, which is crazy, and we’re going to celebrate alongside them with a CTF on the meta protocol, where we’re going to help people understand, or, you know, the researchers and the hackers, we’re going to visit the village. We’re going to help, if they’re new, to matter, understand how matter works, and understand some of the limitations and so many things that we’ve seen. And basically, we’ll challenge them to break into a CTF that we have for a interesting reward at the same time

VAMOSI: Of course the new research includes new vulnerabilities to report.

BERTE: So this is a dish, a few takeaways from this latest research. Nothing new. In fact, it’s all new, all in New at the same time. So we’ve identified buffer overflow, a buffer overflow attack permitted the researchers to, or, you know, a bad actor, to install any unsigned firmer they wanted to this camera and Take control. It was obviously a root access and would compromise the whole system. Of course, the scale of the deployment of these products is always a factor, so we always look at these things. However, there’s a few takeaways here. Nothing new on that side, and neither local access to attack. So it’s all great there. But however, we’ve seen three things that we want to salute. So the vendor was one of the best vendors we’ve ever worked with in terms of their response time, patching and coordination with us.

VAMOSI: This is refreshing and welcome news. Vendor disclosure and how vendors respond to researchers such as Dan is a big part of the problem. It’s good to hear that’s changing.

BERTE: So over the years, you’ve heard me and other occasions as well. My colleagues complain, the biggest issue we’ve always had wasn’t finding vulnerabilities, wasn’t coordinating patching vulnerabilities. None of this stuff is actually coordinating and being able to reach the vendors. So as I said, one of the major issues that we had was reaching vendors. But in this particular case, this vendor was really responsive, really professional, and we salute that. However, they haven’t followed a manual to do it. There is no rules or a guidebook that other vendors can follow as well to be as responsive, as coordinated as this vendor. So we would love it if we could replicate or pass the word out. So it is why we had so high hopes of the systems and protocols such as, or sorry, the regulations such as the new cyber trust mark that we’re still waiting for, that we can also talk about, if you’d like, because there’s new developments. There is, ultimately other protocols as well that are trying to regulate the space. Because a year later, here we’re sitting talking about this, and we’re seeing no additional regulation, which I know a lot of our listeners are going to be shrugging they don’t want more regulation, but it’s in this particular case, is one of those exact cases where regulation is welcome, and I don’t mean regulations as constricting or restricting progress. It’s just bringing everybody into some sort of a report of a common understanding of what needs to be done in order to handle vulnerabilities and hunger and handle life cycle, end of life situations and so on. So going back to our research, we’re fortunate to be able to deliver a presentation on on the on how we found vulnerabilities on the cameras. However, there’s a twist to the presentation, which I’m really excited about. It’s not particular to this vendor, but it’s actually an example of trying to understand and the doc actually is about handling of the cryptography and handling and management of firmware on devices. So the presentation is a little more general. It’s not really related necessarily. It exemplifies this recent findings, but it also gets a really good insight on insight on management of cryptography on and handling the firmware on connected devices, which I think is a fantastic lesson that can help researchers and vendors alike improve on their processes. Because we’re seeing a recurrence here, and it’s where we see recurrence that we can make an impact by changing one little thing.

VAMOSI: So, how does Trust Mark factor into this?

Well, cyber, the US, cyber, trust mark in this particular example, and we have some additional red regulatory bodies across the world. This is the US mark, initiated by the US government through its NIST government body. There’s others around the world as well. The EU is doing their own thing, and Singapore has been quoted a number of times where they’re successful mark, and there’s some, there’s some private initiatives as well from organizations such as the CSA, CSA that’s behind matter because I’m confused. Hold on, this verified, right? I’m sorry. It’s okay. My brain just, yeah, I think it’s it. So the CSA group is behind matter, right? I can look that out anyway. That’s not getting charged. But yes, there is, there is some private initiative as well, which is, which is exciting. Now, what’s interesting about the US Cyber trust mark is that it’s trying to establish some guidelines that vendors ultimately will follow to just like in the in just like in the in our conversation about the living off the land attacks just kind of limit some of the most obvious, egregious ways that some of these devices can be compromised from the get go, because it’s such a, you know, the the IoT space is such, sorry, because the IoT space is such a fragmented space, and even 10 years later, we’re we’ve been struggling for this for a decade. We’re finally seeing some sort of guidance. Because I don’t think of it as more as regulation, but more as guidance. It’s just showing people how to do things in a way that works cohesively, in a way that it can be translated into better products. So what’s interesting about cybertrustmark is that it just kind of lays the foundation of what needs to be done that shaves off a lot of the most egregious issues. And for those who would look at it critically, it might seem like a toothless initiative. However, let’s just say that it’s a evolving process, right? You start off slow, you take everybody on board, and then, as you know, we’re growing and evolving. We’re going to have more and more things added to it that would make more sense, you know, having, you know, my favorite, for example, imposing some sort of a SLA on vulnerability management and coordination, right? Just given, you know, having you know, the specific time when things can be like, right? Now, the whole industry kind of relies on an invented and gentleman’s agreement, 90 day notification, right? I’ll find something, I’ll let you know, and I’ll say, Robert, we found this thing, and you have 90 days to patch it, and then we’ll release it to the public. And you’ll be like, Okay, I’ll try to do it. But then there’s no law that says we’re just kind of evolved into this. However, the government has decided to make this a very functional program. So they said, well, let’s get somebody who’s better at implementing these things than we are. So they found a lab to coordinate the program further. But in the news recently, we’ve we’ve learned that the coordinating body is presumed to have ties to what’s coded to be an enemy state, which is China. Now it’s all speculation, stuff that I’ve read online, but why I’m mentioning this is we’re seeing further delays of implementing something that we’ve needed to have implemented for far longer. So we’re in a bit of a conundrum here. We’re trying to educate the market and trying to make sure that the vendors are getting up to speed on the minimum viable things that we want them to, hey, respond to things really quickly. Patch links really quickly. Let’s coordinate this and that, and then we shouldn’t think of the cyber trust mark as a prize. It’s some sort of a, you know, nice accolade. It’s a conformity. It’s, it’s just a, it’s a common sense denominator that we should all use, right? So I hope the issue gets resolved soon, and the separate Trustmark becomes the guiding principle for our global think it’s a global effort of standardizing and improving on the IoT products, which, again, I’m going to go back to our research from last year has long stopped being a consumer or an enterprise play in separate worlds where we talk about different conferences and we’re different kind of people. They are now intertwined, and they work together, and they all look to the bad guys, like that unloved baby monitor that was developed by a company that didn’t have the resources to put in cyber, right? I don’t know how bad that is. No,

VAMOSI: One of the problems with IoT is that it’s been a tower of Babel with everyone using their proprietary protocols and languages. MATTER seeks to address that. What’s important is that we should have had this years ago because we’re seeing life critical services that could benefit from it today. Such as the rapid development of autonomous vehicles on our roads.

Dan is also in the Bay area of San Francisco so we both have autonomous vehicles at our doorsteps. Naturally, we fell to talking about one service in particular and what might be learned from it’s failure in 2023.

Cruise was a Bay Area independent autonomous vehicle company and in June 2015 it received a permit to test self-driving vehicle technology from the California Department of Motor Vehicles. Cruise was purchased by General Motors in 2016. And in September 2021, Cruise received a permit from the California Department of Motor Vehicles to provide driverless taxi rides in the state. So far so good, right?

Very quickly, there were problems with the service. In January 2023, the city of San Francisco wrote to the California Public Utilities Comission, requesting that the Cruise applications be denied. They cited “in the months since the Initial Approval, Cruise AVs have made unplanned and unexpected stops in travel lanes, where they obstruct traffic and transit service, and intruding into active emergency response scenes, including fire suppression scenes, creating additional hazardous conditions.” A local activist group, Safe Street Rebel, then published a viral video to social media, showing that the Cruise autonomous vehicle could be disabled temporarily without permanent damage by simply placing an orange traffic cone on it. Putting an orange safety cone on the hood of a Cruise made it look like a unicorn, stuck in traffic, with its lights flashing in distress.

Then, in August of 2023, the Cruise taxi fleet failed massively during a major three day rock concert in Golden Gate Park. On the second night of the concert, when Cruise shut down in downtown San Francisco with the company claiming it had lost cellular signal, I remember swerving on Geary Street to avoid a rogue Cruise driving down the wrong side of a very busy road until it encountered oncoming traffic, stopped, and went into panic mode. And then two weeks later, a Cruise stopped in the middle of an intersection ahead of me, blocking several lanes of traffic as it lost signal with the mother ship and got confused turning left on a green light.

Then in October, things got dire.

Here’s KTVU: We have an update now on that breaking news out of San Francisco where a woman has been seriously injured after being hit by an autonomous vehicle. KTVU’s Zach Sous has arrived on the scene on Fifth Street just south of market. He joins us live with the very latest. Zach, what can you tell us there? Yeah, this is still very much an active scene, as you can see, the section of fifth between market and mission still blocked off. To my left over there, you can see the cruise vehicle that was involved in this. We’re told that around 930 this evening, a female pedestrian was crossing the street when she was struck by that cruise vehicle. We are told that no one was inside the cruise vehicle. There was no driver behind the wheel and there were no passengers inside that cruise vehicle. But the female pedestrian became trapped underneath that vehicle, trapped under the left axle.

VAMOSI: This was the incident that ended Cruise service in California, and ultimately the rest of the country.

CNBC: General Motors taxi service cruise now no longer allowed to operate in california after the state suspended their licenses and more details are emerging the question being asked today by the dmv and public officials is this. was there a cover-up the incident under investigation involves a driverless cruise vehicle that initially came to a hard stop running over a pedestrian the dmv alleges that cruise only showed footage up to that stop and left out subsequent footage of the vehicle pulling over a distance of about 20 feet while dragging the pedestrian under the car the entire time. we’ll show you an image of the fire department physically lifting the kaur off aft woman cruise disputes this and says it shared the whole video, but the investigation has much broader implications for the entire autonomous industry. It involves the biggest names in tech cruise is backed by general motor. its name robo taxi exert is waymo. others include amazon, which bought zooks for $1.2 billion in 2020 apple’s rumored project, titan, and tesla’s own very early plans for a full robo taxi fleet robo and cruise have been taking paid rides since august. waymo has the edge cruise’s suspension now likely to widen that gap further.

VAMOSI: I mention all this because the 2023 Cruise experiment on the streets of San Francisco makes for an interesting IoT case study. This was a life critical vehicle that wasn’t necessarily ready for prime time, and the company itself may not have been mature enough to handle the real world consequences.

Dan has been looking into cruise, which is coming back. GM has resurrected the program, with scaled back intentions. It’s now a Level 2 driver-assist system allowing hands-free driving on mapped highways with an attentive driver. Super Cruise more capable on more roads and in more complex urban settings, aiming for Level 3 “eyes-off” driving by 2028. Cruise previously had been billed as a Level 4 — autonomous vehicle.

BERTE: I was fortunate to ride in some of their initial rides in San Francisco. I believe you have to, and there is an immense excitement to the kid in me to experience self driving vehicles. And I’ve talked to friends. I’ve spoken to friends over you know, who visited San Francisco, visited me there, and we’re going to ride into Waymo now, because Waymo still operating. And they’re like, I would never get into that car, and I would be whatever reason. I’m not saying they’re any sort of fundamentalist. However, I think there’s some sort of a, you know, fear of lack of understanding of how anything works. It’s like the fear of flight is, you know, I don’t understand how the planes up in the air. So I have a fear. I personally don’t, but I understand whoever does. And so going back, I think a lot of people have told me, Oh, I would never ride into into a Waymo. But what I’ve noticed is, as soon as everybody gets into the ride and we go past a couple of blocks or traffic lights, we carry on the old conversation about whatever else we were talking about, and they forget, yeah. So it just becomes so normal so quickly. So I But however, you know, being the kid that I am into the excitement that I have having ridden in cruise before they went away, you know, prompted me, of course, to, you know, like other researchers, to look into more, maybe into, you know, how the systems work. And, you know, of course, so we were looking through the white papers and the documentation that Cruise has made available. And you know, one thing that stood out, you know, the car makes decisions based off of, you know, they make three scenarios for each participant in the traffic and traffic, you know, for example, there’s a pedestrian, there’s a cyclist, there’s three vehicles. You know, based off of what is expected, they all do. The car devises three scenarios, and then it makes a decision by jamming all of them. And you know, it knows, okay, that car is going to make it right. But it can also can also make a left, but it’s signaling right so that pedestrian can cross at any time, and then basically they jam that and it’s, it’s neural capacity, you know, capability to inform the car what to do next. Now. But what struck me as it struck you, is that the way they managed, you know, emergency vehicles, like you said, they would use the microphones to inform the vehicle of emergency vehicles approaching, so you would hear the sirens. The car would pick up on that, and then we start looking with its cameras for the emergency emergency vehicles to handle that. Now, if you rode in one of these self driving vehicles in San Francisco or Arizona and Las Vegas and LA and elsewhere where they are currently under operation, and you’ve ran into an emergency vehicle, you probably found it very interesting that they had, you know, had to stop, but knew to swerve around or get out of the way. However, what I found interesting is, you know, this mimicry of human management of driving, okay, we will hear the siren and we when to me, as a technologist, it’s always like, well, those vehicles could communicate, directly to these autonomous vehicles that there are inbound their route, so it would never have to listen to a siren to make a call to make a decision as to avoid the area. But you know, I mean, just think of this. These two machines, they each run on a vehicle to vehicle system, which there are many in development, but none that have taken on yet. And then you would, you know, you would know exactly that the fire ladder is going to, you know, three blocks down to the right of this particular building. And if your autonomous vehicle was going that way, it would just never need to be in a way in the path of that emergency vehicle would just get reroute and go completely around the roadblock that that could create. And it just makes so much sense that that’s how it should be handled, however, and I think that’s kind of slowing the whole thing down. And I’m not, with all my due respect and admiration to the efforts of these wonderful people are putting into the research here. Of course, they know better than I do, but I think we could help the industry as well by improving infrastructure, instead of putting the burden all all on those on those systems to improve and mimic humans, because in a future, and I’ve wrote this in a piece many years back that you can probably Google somewhere. And I said there, my kids won’t know how to drive. You know how you if you were, you know, if you drove a Tesla, you know how they stopped equipping them with a shift selector, and they made it autonomous, it would guess you’re in a garage. You’re definitely not going to go backwards. You need it. So it’s going to automatically be and drive, or basically what they were doing. And a lot of people don’t see it as of course, it’s a potentially cost saving measure, but it’s also a removal of the control surfaces that we wouldn’t need when the cars become fully autonomous. So when you get a shift lever gone, you know, the steering wheel becomes a yoke and maybe just becomes a joystick, maybe it’s just a few buttons. So what we’re being taught now, you know, the cockpit, the instrument cluster gone, could be a cost saving measure, sure, but it’s also a sign that what we’re really wanting our vehicles is to just hang around and watch a big screen, akin to our living room, not look at a you know, how we had, you know, up until what is it? The 80s, maybe we had a two pilots, the captain, the first officer, and then a mechanic of the flight. They had their own panels, and now that Job’s gone, because we don’t need a mechanic to check all the dials see if the engines are working. So the same for cars. You know, these gages, they’re not of interest because we’re not, soon we’re not driving anymore. So I was thinking a lot about this. And you know, if you know, if we’re not there yet, we will be all these we will have to deal with a vehicle to vehicle system that’s probably going to start in an environment where it makes sense in fleets, semis, emergency vehicles and so on so forth that, you know, can be standardized, and then that’s going to spill into consumer vehicles as well, and they’ll be able to coordinate, and they’ll be able and we’ve, you know, just like we read about, you know, being able to do a 3g call, and then maybe 10 years later, it was reality, and it wasn’t interesting anymore. It will happen. It’s coming. There’s multiple systems being worked on. But however, what I’m saying is, I don’t want us to be taken by surprise as researchers, when these things are available and deployed, and you’re going to bring a whole host of new risks from a cybersecurity standpoint as well, right? We all know about the stories of you know, you know, def cons and others. Where we’re seeing the presentations were, you know, different self driving vehicles were being attacked by, you know, painting different lines or exactly. So that’s a reality. So in my, you know, as I’m exploring this, as I’m looking at the space, I’m thinking a better way to secure the future of autonomous vehicles isn’t to make them mimic human drivers. It is now, and it makes sense that it’s that way, but it’s also trying, and this is me gently nudging and urging, you know, potentially administrations to consider building infrastructure that meets self driving in the middle. And we design systems and you know, we help vehicle to vehicle coordination systems get deployed, it’s going to make everybody’s, you know, life easier and safer, you know, I imagine, you know, instead of the carpool lanes we have on the one on one, those, those will become autonomous vehicle lanes. Can you imagine what would happen if you all started crossing the highway, autonomous vehicles being on the highway, they will all have to stop, and that we can go around this. So, you know, we have to, you know, welcome autonomous vehicles into the infrastructure. And don’t get me wrong, for those who say, Well, I like to drive. I like to drive. I mean, it’s not, we’re not talking politics here, you know, I have a blast, you know, you know, driving my ice car on many roads on the weekend. Of course, I’m not however, I’m looking at the future. I’m trying to be the best. Acknowledges that I am to be realistic, that my dad loved riding a horse, and I don’t know how to and that’s fine. Driving is not a skill that humans need to have. It’s just something, honestly, if cars were gone, can we use those skills for anything else? Probably not right steering and all these are invented for that specific task. So those things are going to be gone. This is not a skill that we have to hold on to so anyway, it’s food for thought, but, but again, as we’re seeing, you know, the new autonomous vehicles trying to cohabit, cohabitate in our urban environment. It’s just a reminder every day for me to try to stay ahead of the curve and be there where the puck is going, where the technologies that are coming that are going to make these systems coordinate with existing, let’s call them legacy vehicles in the future, but also, you know, from from an OT standpoint and an IT standpoint, these will have to coordinate as well, potentially with More wearables and stuff with pedestrians and cyclists as well,

VAMOSI: So maybe this is an opportunity for a MATTER protocol to come out where everything can converge, and you can take the ADAS systems and ice vehicles and make them communicate with the fully autonomous or EV vehicles coming down the pike. Does that seem like a solution? Absolutely

BERTE: Absolutely. I mean, of course, you know, as we’ve seen capabilities in the I mean, you know, I’ve had my share of looking into the space and going to different conferences, talking to companies that are involved in automotive and, I mean, it fair. The CAN bus is a really old system, and it’s binary still at the core of most vehicles today. So that’s also that’s a limiting factor.

VAMOSI: And that, I’m sure, will be the topic of another podcast with Dan. I’d like to thank for taking the additional time to talk with me at Black Hat USA and look forward to future conversations.

-30-

The growing importance of OT security, highlighting overlooked risks in critical infrastructure, legacy systems, and supply chains. Through real-world examples, Eric Durr, Chief Product Officer at Tenable, shows why OT security differs from IT, emphasizing visibility, resilience, and risk prioritization to protect safety, operations, and business continuity.

[Heads Up: This transcription was autogenerated, so there may be errors.]

VAMOSI: Operational technology or OT security has become a critical concern as organizations increasingly recognize that risk extends far beyond traditional IT systems. Modern enterprises rely on complex physical and digital environments — data centers, industrial machinery, energy systems, and supply chains — where components such as HVAC, fire suppression, and legacy control systems are essential to safety, uptime, and business continuity. When these systems fail or are compromised, the consequences can escalate rapidly, threatening human safety, massive financial loss, and organizational liability.

OT and critical infrastructure expand the organization’s security attack surface in ways many leaders underestimate. Real-world examples — from data center HVAC failures that can destroy billions of dollars in assets within hours, to aviation manufacturing systems where incorrect torque settings could endanger lives — demonstrate that OT security is inseparable from operational safety and compliance. Even when attacks do not occur, organizations must be able to prove that systems operated within strict specifications.

In a moment, my guest will highlight growing awareness of supply chain risk, both in software and hardware. As companies increasingly depend on third-party components, understanding the provenance, configuration, and integrity of devices and firmware becomes essential. At the same time, legacy OT systems — often unpatchable and built on insecure protocols — force organizations to rethink traditional security models.

This is the story of how OT security is fundamentally different from IT security yet effective protection requires visibility across IT and OT, contextual threat intelligence, and pragmatic risk management that prioritizes resilience and accountability over perfection.

This is the story of how OT security is fundamentally different from IT security yet effective protection requires visibility across IT and OT, contextual threat intelligence, and pragmatic risk management that prioritizes resilience and accountability over perfection.

I’m Robert Vamosi

This is Error Code

[MUSIC]

DURR: I’m Eric Durr. I’m the Chief Product Officer at Tenable. Tenable is really focused on exposure management, which is taking all of the different risk areas in your enterprise. In this case, we’re going to talk a lot about OT and help organizations manage the risk, both within that specific domain, but also across their whole estate, so that they’re as protected as they can be.

VAMOSI: So before we started, Erik mentioned that Tenable has a position on OT and I would love to hear that.

DURR: So I think that, you know, my point of view of this personally, and I think tenables point of view are consistent in the sense that, you know, I think there was a time period where most ot was fairly isolated, both, you know, air gapped, you know, fairly not simple in the sense of, there are lots of complexity in OT, but kind of not a lot of interconnectivity into the IT systems, not a lot of touch from people that could bring risk into that environment. You know, usually a standalone team that’s dealing with that in the organization. And it’s not that that never happens when we talk to customers these days, but it’s sure a lot less common. It’s much more common that you’ve got a mix of OT and IT devices in the environment. You’ve got more and more interconnectivity to the corporate network, the cloud, all you know permutations of that you know, and and more and more and more people who have access to those environments in various ways which can bring new risks and things like that. So, you know, as that reality changes, I think that our point of view is, you got to think differently about how you secure that and, you know, there’s, I’d say, a point of view that you, you know, let’s just think about you’ve got an OT program and system and set of tools, and you’ve got an IT system and set of tools, and you got a cloud system and a set of tools, and, you know, we’re pretty far on the extreme of saying that’s a bad way to think about the world. The world is way more interconnected than that. And attackers, of course, don’t care how you’ve organized your security team, they don’t care, you know, they’ll go after the weak spot wherever it is, whether it’s a weak spot in your organization or a weak spot in your environment and so and so. We care deeply about the OT domain, and there is a bunch of depth you have to have in, you know, all the weird arcane things of HVAC systems and controllers and this and that and the other thing. And at the same time, we kind of think you’ve got to take a blended approach to this to actually secure your enterprise.

VAMOSI: So with this blended I hear a lot about it, ot crossover and so forth. Erik’s not necessarily talking about that.

DURR: I think it’s the way I think about it is, there’s ot specific software, right? We have pieces of OT specific software. There’s scanners that are scanning, you know, in certain ways for certain kinds of equipment, and you’ve got to, you know, understand the equipment and know what it means. And, you know, there’s a lot of that kind of complexity that data then comes in, has to come into a data lake that also pulls in data from your IT systems, and also pulls in data from your identities, and you know, just and your network, and you know, a bunch of other interesting areas, and this is where I mean, not all customers are equal. It really does depend on how your environment is set up, and it does depend on what kind of organization you are. And you know, an oil and gas tends to look very different from, you know, a retailer that’s got a bunch of OT and, you know, stores, you know, around the world. So but I would say the general trend for us is, hey, more and more of this stuff is getting interconnected, and that’s both important on the attack or, sorry, on the attack side, that there’s attack vectors that exist because of that, but it’s, I’ll say, even more important on the mitigation side. So you know, one of the things we can do, because we kind of have the broad sensors that allow you to go pull a bunch of this data across your enterprise and then bring it together into one data lake, and then, you know, analyze it and reasonable about it and things like that. You know, there’s a bunch of really common situations where we say, okay, there’s a known vulnerability that’s on this controller. You know, it’s not, it’s not feasible to go update that tomorrow. You know, it’s gonna go down for service three months from now, or six months from now, or whatever the normal window, or maybe never, in some cases, that may never have a planned, a planned update. But if we understand the interconnection, you can say, hey, is there a networking change we could make, or is there, you know, is the is the risk exposed because of a privileged identity. You know, can you very easily go tweak that privileged identity to restrict the attack surface and and so I think it this kind of cross visibility matters a ton on understanding the different kinds of attack, but also then very quickly mitigating and kind of, I think this is another shift, kind of broadly in the industry, is going from, you know, we can list all of the things that could ever be wrong, which no security team I ever met has the budget or the time to go mitigate every single one of those things, certainly not at the same time, to, hey, not just prioritize, you know, Yes, you need to do that, but, but also, you know, figure out the how you break the attack chain, and if you can break the attack chain, you know that certainly buys you time. And for a lot of organizations, that’s all I care about. Just I broke the attack chain. I’m not interested in theoretical risk. I’m interested in concrete risk.

VAMOSI: So does tenable have a timeframe where ot is becoming more relevant? Is there a point when it became more relevant to industry that they need to start focusing on this, as opposed to just network attacks.

DURR: I mean, I think it’s a if I zoom out for a second, the there’s always this interesting relationship between kind of what’s been in the news lately? What are the common kinds of attacks, or, you know, attack targets, if you look broadly at, you know, a large set of attacks over a period of time, and then the third dimension, what a responsible organization should do to protect their crown jewels, right? And they’re not unrelated, right? But, but you know, if all we did is say we’re just going to go protect the things that are most of the attacks, we’d still all basically be doing MFA, and that’s it, right? And sadly, there’s still a whole lot of organizations out there that don’t have MFA completely rolled out, and they’re thus at a whole lot of risk as a result of it, right? But, but I think, you know, you look at cloud security as an adjacent but I think interesting example relatively few cloud attacks, or at least public cloud attacks, but you look at how much spend there is in cloud security, it’s many billions of dollars of spend on cloud security. And I think that’s appropriate. I think that, and it’s appropriate because of the criticality of that piece of infrastructure to the functioning of the business you can I mean, insurance is maybe a dirty analogy, you know, but, but at some level, it’s like you’re irresponsible if you, as an as an organization, are not protecting some of your biggest assets. Zooming into OT, again, it depends on the company. But, you know, the an oil and gas company, oh my gosh. You know, if their ot network goes down, that’s everything you probably for that, for that company, you know, if you’re a retailer and you’re and things go down. I mean, one of the things I was going to bring up is we’ve seen a tremendous amount of traction over the last couple of years in data centers. And yes, you know, I don’t think most people think about data center as an OT battleground, you know, but it turns out they’re a really complex mix of OT and it and IoT and and, of course, you know, the data centers have been exploding for a long time, and then this little AI fad we’re in the middle of. It sure seems to be accelerating that more. So we were working with a whole bunch of different customers of various types, you know, and bringing in, you know, our OT solution, but also, really this comprehensive solution, because you say, look, great, you protect the OT of the data center, but you don’t protect the it. How does that help? Right? We’re talking

VAMOSI: We’re talking about HVAC, we’re talking about fire, we’re talking about all these different things that are part of the data center, right? That a lot of people don’t think about right, being a security risk.

DURR: That’s right, but that’s right. You take out the HVAC, right? And you don’t have a data center, yeah, two funny stories related to that, the, I mean, funny, and the perverse security mindset of what’s funny, right? But a bunch of years ago, I worked at Microsoft, and I was never particularly close to the data centers at Microsoft, but at one point, someone shared a nugget with me, which was, you know, they asked me, kind of in the form of a riddle, you know, what do you think happens if the HVAC systems fail for, you know, an hour? And I was like, I don’t know. Probably nothing, you know, probably not much. Turns out, no, that’s not the answer. And I don’t remember the exact you know, but it was like, in an hour the temperature in the data center goes from, you know, a cool 65 degrees to 95 and in another hour it goes to 120 and guess when all those parts fail in those machines? And it was like, you’ve got an hour or two of grace to resolve something. And it was such an eye opening thing for me to kind of think about, you know, oh, man. And these, you know, billions, 10s of billions of dollars of assets for an organization. You know, that’s just such an interesting thing. The second one, I was taught, I have to generalize this to protect the innocent. But the I was talking to CISO for airline type, you know, made parts for airlines and and they were, you know, interested in a bunch of the previous company, but I was, they were interested in a bunch of kind of analytics and tooling and things like that. And they were kind of describing some of their ot challenges that they had, and talking about how they needed to, you know, have, you know, they have these powered, I don’t know what to call them, like a wrench, basically, but, you know, a socket set basically right to, you know, put some of these complex pieces of, you know, airline machinery or airplane machinery together. And he was describing a bunch of the requirements of what they need, you know, we, you know, they have to be adjusted to this tolerance, and we need that data, and it needs to go in this system. And, you know, and, and I was kind of just really scratching my head, and kind of like, why do you How is this a security problem? Like, what, you know, Ed, why are you so worried? Like, it might be interesting, but it was a security problem. And he just said, Look, you know, if a plane goes down, you know, there’s tremendous amount, obviously, a horrible thing, but, you know, tremendous amount of liability associated with that. We have very clear specs on how tight each bolt needs to be made. So there’s kind of two things, one attack vector, if someone gets in and messes with it. So now these bolts are tightened to the right thing, that could really put a lot of people’s lives at risk. That would be horrible, right? We don’t want to do that. Okay? And I’m like, Okay, I understand now I’m a board, right? The anybody else has said, but let’s say that doesn’t happen. But something bad happens, we need to be able to prove that it was actually that wasn’t the problem. We did tighten everything to the proper tolerance and things like that. And so I think just these, these examples were very, very helpful for me to kind of think about just how complicated the risk profile and attack surface is in OT and and, you know, just makes me want to, you know, another example of where the more you can bring that data together, the more you can reason about and better protect, you know, your enterprise.

VAMOSI: I often think of automotive where everything is known down to, you know, the very specific nuts and bolts in the car, and all of that comes out of liability from the mechanical days of the car. But, yeah, airline industry the same way you tighten the bolt a certain amount and influences the tension on that. So are other industries starting to pay attention to the supply chain, the fact that they don’t make everything, but they consume everything yet, ultimately they’re liable.

DURR: Yeah, yeah, I do think so. I mean, obviously software supply chain is something that, you know, people have talked about for a long time. You know, it’s funny. I was just earlier today talking to somebody about in a previous life, I I led the incident response team for Microsoft and and led Microsoft through the solar winds, how Microsoft reacted. And one of the things I was talking about in sharing that story was how, you know, that wasn’t the first, you know, supply chain kind of attack, but it certainly was one, I think, that got the industry to wake up more and say, you know, we need to increase our aperture of what, what we should be worried about from a risk perspective, and start having strategies and mitigations for for how we’re going to manage that. You know, it’s on the physical side. You know, not a topic I’ve thought about as much. You know, but, well,

VAMOSI: there’s H bombs — Hardware Bill of Materials.. I don’t know how helpful they are, but at least knowing, yeah, the pedigree of the the pieces coming into your environment, whether the firmware has been checked, is a totally different matter.

DURR: No, that’s right, that’s right, the the but I guess if, if I, if I generalize it, you know, and of course, I’m biased through the lens of kind of how we help customers today, but you know, a lot of the things you do in a good security program also provide you some protection from just failure of one sort or another. I mean, a weird example is, if you have a good backup and restore plan that maybe you put in place because you were worried about identity takeover, that can also help you in a whole bunch of other circumstances, including just catastrophic failure that has nothing to do with you know, like could be a hardware failure, right? Grades out of catastrophic failure, but, but I do think it, you know, you don’t get perfect protection from that. So you don’t have to reason about how much you know, where are you putting your money and what, what threats are you trying to mitigate, right?

VAMOSI: So in the OT space, or just in general, critical infrastructure. There’s 16 different areas. We’ve mentioned oil and gas. We’ve mentioned aviation. You mentioned briefly, retail. Where are some of the more interesting stories for ot coming about?

DURR: what’s Yeah. I mean, at some level, I would say, just like I tell my children, I love them all equally. I think I have to say that I love every aspect of critical, critical infrastructure equally. You know, the certainly, we’ve seen a spike in data center, as I already mentioned. We’ve seen a spike in kind of retail warehousing, kind of, again, a place where you have a whole lot of automation, a whole lot of a mix of OT and it kind of stuff that people and very central to that business, so something that they really feel like they have to have a really good strategy to protect that we’re doing a lot in energy. So one of the things, as an example, we have a partnership with Siemens energy, where some of our solutions for ot protection kind of come as part of the various controllers and and things that they sell. And so we got some good traction there. But you know, it’s, it’s each one of these aspects of critical infrastructure, you know, has some stuff in common, and then some unique stuff that you have to to worry about, right? So, you know,

VAMOSI: And so what you are implying is your solution is kind of security controls for the existing ot equipment, because legacy ot lots of problems. There may be open protocols that were never intended, or maybe the protocols don’t even exist because of resources.

DURR: I think that the kind of the way we think about the world, and our part of helping the world is giving you the broadest and deepest insight into places you’re exposed, and prioritizing those exposures so that you can spend your energy wisely and wherever possible, giving the best, I’ll say, advice on how to mitigate that, right? Again, I gave the example of the, you know, thing that’s hard to update. A more extreme example is the 30 year old HVAC. You know that the manufacturer is out of business, but the thing’s still working great. Probably has 20 or 30 more years of you know, why would you replace the thing? And yet, you know, if there’s some risk there, and we all have heard these stories and know about these issues, you know, if you can recognize that exposure, understand, you know, and if it’s unreasonable to directly mitigate the core issue, you know, you can just say, I’ll just design my security perimeter so that I’ve got, you know, right the right controls, then, you know, it’s not perfect. But you know, people are running businesses. They’re not trying to build perfection. They’re trying to be responsible and and manage to a budget and all that good stuff.

VAMOSI: So I’m going to come at this at a different way. But how are you seeing the uptick in OT interest. So for example, I’ve known of your company for a long time. So, you know, I’m a security professional, I’d be like, of course, we’re going to bring tenable in, whatever. But how do you introduce that ot conversation? Or are they already there and they bring you in specifically for that ot conversation,

DURR: yeah, I think it’s a mix. There’s certainly and some of this is a shift. So our a couple of years ago, we started this shift to say, you know, prior to that, you know, we kind of thought about these different pieces as relatively standalone. You know, we do some ot security stuff, we just identity security stuff. And then about two and a half years ago, we said, you know, yes, we do those deep things, but we also see that there’s this need and these issues if you don’t bring this stuff together. So that’s why we launched tenable one back then, with a vision of bringing this data together. And then we’ve been hard at work at, you know, making that better and better. The one thing we that’s new this year, we just announced a couple of weeks ago, which is important piece of the puzzle, and I’ll tie back, is we announced that tenable has a philosophy of being an open platform, and because we recognize that really, when you the data you need is much broader than any one company is ever going to provide. And so you need, you know, in some circumstances, data coming from endpoint or coming from the network firewall, or coming from, you know, your human capital management system, like a workday or something like that. And so, so we’ve built a bunch of partnerships and integrations with, you know, the popular providers, and we just announced a milestone of 300 integrations. And of course, we have more to do, but the and so tying it back. The the, the thing that you know, we really think about is, how do we make it easy for people to stitch this information? And in some cases, someone says, you know, I have an OT problem. I hear you have an OT solution. We’d like to talk to you about it. And we say, Great. In a lot of cases, we connect with an enterprise. And they say, you know, I know about you. Tenable for one area, you know, often vulnerability management, which is, of course, the thing we’ve been doing longest, right? And, and then we say to them, Hey, tell us about your overall security program. Like, what are you, you know, where do you have problems? Let’s see if where there’s other places we can help. And in a lot of those conversations, someone will say, you know, oh, I’ve got ot. I’ve got this ot problem. Here’s what I’m doing, here’s what I don’t like about it, you know, right? And so, you know, it’s a real mix, but, but I kind of, and of course, customer is always right. So, you know, if the customer says I don’t buy into this, or, you know, or my environment really is so air gapped and so, you know, I’ve designed, I’m really confident in how I’ve designed this, so that I don’t worry about the intermixing of then, great, we can still help you, right? It’s just increasingly that’s not the norm in what we talk about.

VAMOSI: So it sounds like so when they say like they’ve got an OT problem abstract, and tell me what you hear when they mention that.

DURR: Something like that, but I think so. I mean, kind of obviously there’s two categories. There’s the people who don’t have an OT security program or set of tools today, and there’s still a lot of that, right, right, which is a little surprising to me, but you know, it is what it is. And so, you know, there, there’s a, there’s a, sometimes an education component of, hey, here’s how to think about it. And here’s, you know, depends on where they are in the thought process of that. And then there’s a lot of cases where people say, Hey, I have this tool. Here’s what I’m struggling with, you know, can you help me with that? Because it doesn’t do OT, it can be both, right? Okay, yeah. I mean, you know coverage of, yeah, you know, certain systems, you know, you know, depth in, you know, threat intelligence, and correlating that with, you know, like, one of the big things here in OT, like in a lot of parts of security, is it’s super helpful to know, you know what kind of attacks come from, what kind of you know, actors like, you know, obviously, Iran’s been the news a lot. Iran is relatively well known as liking to target, you know, OT and, you know, we like, we actively track, you know, various actors and active, actively track, you know, these things. There’s about 30 vulnerabilities that you know, known vulnerabilities that the Iranian threat actors are known to use. And you know, and so we have that data and bring that data into the product experience and contextualize it for the customer. And you know, shockingly, but not shockingly, you know, a whole lot of bunch of these vulnerabilities have been known for a long time, but still, you go into a new customer environment, and, you know, run the, you know, run the analysis. And, oh, you know, here’s a bunch of things that, you know, you probably should go take care of. You know, we also have surprising amount of, I’ll say, relatively basic misconfigurations that we see where, you know, it’s not uncommon that will go into environment, you know, run a bunch of scans, you know, as we’re working with a customer, and find, you know, a few, you know, open connections to the internet, or a few, you know, overly permissioned roles or things like that. And so, you know, it’s, it’s, it’s sad that it’s that some of the basic things are still pretty prevalent, but, you know, at the same time, it just, it’s a real problem. We can help people with it. And, you know, I kind of think one of the thoughts about security overall is, you know, a lot of what we’re trying to do is make the bad actors work harder. Yes, and actually changing the economics of cybercrime is a super legitimate way to decrease the overall impact to the world, right, right?

VAMOSI: So would Erik say that ot security has to be different in some ways? In other words, if I’ve got an IT security program, can I think, well, I’ll just bridge it over to my ot environment, and I’m set that you really do have to come to the table and say, Well, no, ot has these unique values, and we are positioned to do that is that

DURR: it’s, for sure, different. I mean, you know. And of course, the different there are, what I should say is there are some very distinct differences, and one we already talked about, some assets cannot be patched, right, right? You know, there’s not really an example like that in the IT environment, or at least not a common example like that. Whereas it’s pretty common in OT deployments, we do that a customer will have some asset or set of assets that just, it’s impossible to catch it the and so I think that, you know, a lot of it carries over, but, you know, but I don’t think it’s universal. And this is where having kind of the right mix of, you know, expertise in an organization, and, you know, that is helpful.

VAMOSI: I’m going to bring it back to that it OT, yeah. So on the one hand, we’re saying that they can be together. Yeah, we shouldn’t think of them as being that different. On the other hand, the solutions have to be because of the very nature of OT, slightly different from each other. They can’t be similar.

DURR: But again, I think that’s that’s not unusual in the world of security. I think it may be, in a way, OT or is newer in the psyche of the average security person, or, you know, like, even if you look at, does the OT security is there an OT security program, and does it report to the CISO? Right? Right? You know, that’s, I think, been shifting over the last few years, where maybe three or four or five years ago, you know, a whole lot of companies didn’t have an OT security program. They might have an OT maintenance like, but it wasn’t a dedicated we’re thinking about security, and even if they did, it was relatively rare that it reported to the chief security officer. And of course, these things are all you know. There’s no gross generalization, but I would say in general, it seems now like it’s more and more common that you know, if you have a significant ot piece of your business, that you’re going to have at least a small team of people who are worried about this, and more often than not, they’ll report to the CISO and and I think that’s really good. I think it’s healthy that that’s happening, but, but my point is that, oh, that’s newer, whereas, you know, we kind of take for granted that in a security team, again, at a complex organization, you need a networking expert, and you need a, you know, a data analysis expert, and you need a cloud expert, and you need, you know these you do need that expertise right to help design your program, to help think about what’s common, what’s not common, and so. So in that way, I don’t think ot Is that unusual. I think it’s kind of the norm that these really complicated areas of technology need and deserve some expertise.

VAMOSI: Well, you just said ot maintenance, and I think the common perception of that is that, oh, the actuator broke and, you know, burnt out, and we need to replace it. There’s no security awareness about what caused that to burn out. Or could it impact other systems?

DURR: That could be a bootstrap method to get into your system, or, or, if I can, if I can jump on, please pick the or the maintenance person who shows up every you know, X months to do some scheduled maintenance. Was just in a, you know, Starbucks cafe on the public Wi Fi, doing who knows what on that laptop, and then plug that laptop into you. You’re nicely air gapped or nicely protected ot environment, right? And what? What happened, right? And you know, that’s both a theoretical risk, and we do sometimes see stuff like that being a problem, right? So,

VAMOSI: so short of having actual demonstrable damage as a result of attacks, yeah. How do you guys raise awareness about the OT risks? Because, again, I think there’s a mindset that it’s the conveyor belt, right? How could that harm my company? Yeah,

DURR: yeah, I think so. First, philosophically, I’m anti ambulance chasing. So you know, there are certainly, you know, security vendors who who have the attitude that creating fear and panic is a helpful way, and, you know, it might be sometimes a helpful way to sell some stuff. I just think that’s, you know, it’s not a very it’s not my cup of tea. It’s not tenables cup of tea. The, you know, the, I would say, for the most part, we don’t really worry about telling people they have a problem that they are don’t already know that they have a problem for if that makes sense, you know. And a little bit of that is philosophical, a little bit of that is, you know, we got lots of great customers, engaging, and more engaging every quarter. And so, you know, why would we bother going and telling someone you should worry about ot when they’re not already worried about OT, right? You know, the, you know, and it’s always a little, you know, there’s always a balance of, you know, we’re here to help, right? We’re not here to tell people what their security program should look like or what their priorities should look like. So, you know, there’s a, there’s a fine line there. If someone, oh, I have a lot of OT Well, are you worried about that? Do you feel like that’s under control? Where does that fit in your overall, you know, priority list of things you’re trying to work on, you know, but, but, you know, we’re not gonna, we’re not gonna go much further than that.

VAMOSI: One thing we haven’t discussed is the involvement of state actors in OT attacks. I’m seeing OT as a new low hanging fruit for the state actors, like they know the networks are fairly robust. They know that recording is there and whatnot, but, oh, there’s this blind spot over here.

DURR: Yeah. I mean, I certainly think that. Let me pull the thread of what? Because I think you pulled a good thread, which you didn’t touch on much, because I think in most of the conversation I was talking about it through the lens of, you know, kind of a public business that, you know, is thinking about dollars and cents and ROI and things like that. And, of course, that is a big chunk of the market. But you know, you’re right to talk about, you know, more narrowly focused on critical infrastructure, where, of course, there are still budgets and there’s still reality of you can’t do everything and you need to make trade offs. But there’s also the the existential threat to real safety and real human lives, and, you know, national security and a bunch other stuff like that. The the, I mean, I do worry about that, right? I do. I think the good news is that the better the tools get, and the better you know, the understanding of how to secure these kind of environments get democratizes the solutions to an extent, and so, you know the, and I don’t think the, I don’t think there’s a meaningful set of differences between how you secure, you know, water district and how you secure, you know, An oil field, right? Yes, there are individual, specific differences, right? But I think you know the same core things of, do you have an accurate inventory of all the things that you’re trying to protect? Do you understand the potential vulnerabilities of each of those things? Do you understand the intersection of how those interact with each other? Do you understand identities and network you know, like all those things are kind of the same elements and and maybe one thing I also haven’t touched on enough is how critical it is that it’s all dynamic. And because the threat landscape is changing all the time. The understanding of vulnerabilities is changing all the time, but also your infrastructure is changing all the time. And you know, whether it’s from plant maintenance or a part failing, you know, or something changing, you know, like, there’s just a million reasons why you know it’s a, it’s a it’s dynamic. And you know, certainly, kind of our design pattern is built towards near real time. Everything, right, right? And so, and certainly, what customers ask us for, they ask us lots of questions about, you know, if something happens, how long until I know about it, you know, and, and, and and so I think that’s an important dimension to the problem as well. Just, you know, this isn’t just once a month. I’m going to take a look and then see what I need to do, right? You know, it really needs to be, you know, running all the time, and, you know, with the right triggers and thresholds and people who are paying attention, and stuff is changing.

VAMOSI: Since I’ve got an expert here, I’ve heard a lot about potential ransomware, of like elevators and, you know, very specific niche systems. Any evidence to that? Or is this still just a theoretical or, hearsay?

DURR: Yeah, yeah, I have not heard of actual, you know, actual attacks. It’s, I would say, generally, with ransomware, it’s one of the hardest things to get great data on, because people don’t tend to want to the people who pay ransoms tend to go to great lengths not to tell people that they pay to ransom, right? And so, but, yeah, I’m not aware of any. I think if you generalize it, though, you could say another element in a good solution here is recognizing that not all, not all assets are equal, right, right? And I think there’s a very good line of thought in a, you know, mature program around ot that would say, you know, these are, you know, highly sensitive assets, you know. And there’s multiple, safety is one of those dimensions of why something may be highly sensitive. And so, you know, the right program and the right set of tooling can have, you know, respond differently depending on, you know, okay, this is a, you know, a new attack that’s possible because of whatever, because of a new threat that’s been discovered, or because of a change in your configuration that that now puts this in this example, the elevator, this is now a target that it wasn’t yesterday. You should really do something about that, you know, right?

VAMOSI: And you mentioned no FUD. And I do marketing. I get that. I try not to do FUD as well, but you kind of have to go there to open up the landscape to get people to think about that elevator. Because that’s right. 15 minutes ago, they were not thinking about their elevator. That’s right. Critical, right?

DURR: That’s right. So that’s, you’re right. It’s a fine line. It’s a fine line between ones, hey, I mean, yeah, you know, with the concept of a threat model, right? Like, you know, it’s, we’re gonna think about all the things that could go wrong, and then we’re gonna build a plan around, you know, which we think are most likely or most bad if they happen right now.

-30-

Ad fraud driven by both humans and AI agents require new signals beyond traditional bot-vs-human checks. Gavin Reid and Lindsay Kaye from HUMAN Security discuss how monetization includes ad and click fraud (peach pit), selling residential proxy access, and operating botnets for hire and preventing harm requires dismantling criminal infrastructure and collaboration across industry, since many infected devices cannot be practically cleansed by end users.

[Heads Up: This transcription was autogenerated, so there may be errors.]

VAMOSI: In October 2016, a group of gamers released code that leveraged vulnerabilities in devices, specifically surveillance cameras. The cameras were white labeled, meaning that one factory could produce the vulnerable chips that went into many different brands of surveillance cameras.

Gamers are … well, competitive. And in the world of gaming, if you can take your opponent offline, then you get ahead. So what started as code to do just that, knock an opponent offline, blew up in the real world. Suddenly, the code, which took advantage of a vulnerability in a protocol, could be used against surveillance cameras.

And someone did that. And instead of taking a gamer off line, they took DYN, an internet provider that supported brands such as Twitter and Netflix. Those companies and others, along with swaths of the Eastern United States, all suffered a massive denial of service attack from bots on … IoT devices.

The Mirai botnet was the first major attack by a device-driven botnet as thousands of surveillance cameras, each with internet connections, were leveraged. While the damage cost the respective companies millions of dollars in losses, the act itself was relatively innocent. Similar to DDoS attempts in the past.

Mirai still lives on, with variants causing local attacks. The thing is it is not alone. In a moment I’ll introduce you to a world of active cybercrime fueled by devices in your home and in our world. And, in the case of BADBOX, there is no patch, there is no easy fix.

This is a story about how cybercrime is blurring the boundaries between humans and bots when it comes to click fraud.

I’m Robert Vamosi

This is Error Code

[MUSIC]

REID: I am Gavin Reid. I’m the CISO of human security, and at HUMAN we’re tackling sort of a different kind of cyber crime, and this is one that blurs the line between bots and humans automation intent, and we protect organizations

KAYE: I am Lindsey Kaye and I’m Vice President of threat intelligence here at human

VAMOSI: It might be hard to discern what it is that they do. They look at devices out in the field, IoT and OT devices, and they discover ways that criminals are using these devices for automated attacks. And for profit.

REID: We’re like a platform that protects other large platforms from the sort of automated attacks, right? And, of course, now, with the advent of, you know, agentic in your the bots that actually can do some level of decisioning, you know, things have sort of ratcheted up a lot. So we’re in line with, you know, the major platforms that you’re familiar with that you’re probably using on a day to day basis. And so we have to not only be able to identify if it’s human, if it’s not human, what the intent is, where it’s going, what it’s doing, but we have to do it kind of at the speed of light, right? Because no one wants additional latency.

KAYE: I am the person who runs the team that is responsible for this research in coordination with our research team. Is more kind of a data science approach.

VAMOSI: Gavin mentioned agentic AI. How is that fitting in? HereHow is HUMAN using the AI to assist with threat intelligence, or is HUMAN combating against the AIs that are perpetrating these whatever frauds?

REID: Yes, yes and yes. So you know, like you know most companies or like many companies in the space, we have a long history of algorithmic development using machine learning, and so, of course, AI and agentic AI has allowed us to do some different things with that. But at the same time, those same customers that we protect against automated attacks, they you know, we had to pivot a little bit, because not only do we need to be, you know, the expert at bad bots and being able to identify and protect against them, but we also need to have that same give our customers that same level of visibility and sort of trust in the good bot. So we’ve really, as an entire company, sort of pivoted against one that was mostly, you know, look and block to one that now can decide. You know, it’s an agentic AI bot. What do you want to allow that bot to do? Is that bot okay with scraping your site? Is it okay with creating logins? Or do you not want that? Is it okay with logging in and buying stuff and using credit cards? We have to get, like, much more, into the nuts and bolts of what you know, what the bot is doing, and whether we want to enable that, whether we want to enable that and monetize it. You know, we have partnerships where people, if you’re a content provider, you may want your data to be scraped, but you feel like people should pay for that. And you know, same with these large language models. They live off of the data that they’re scraping. That’s what really makes them smart. It’s all the human data that’s put inside there. And so there is obviously a much bigger pull on sites where it used to be that an individual would log in and look at maybe Atlantic Monthly and read some articles. Now you have bots that are doing this continuously, 24 hours a day, seven days a week, and the business model is shifting.

Unknown 9:48

VAMOSI: So Lindsay, I imagine just determining that humans are doing fraud is one thing, but now you enter in the agentic AIS and so forth that are doing bad things as well. Do you differentiate? Are you looking at how fast keystrokes are committed, or where the input is coming from and the traditional things? Or do you have to have new tools today to detect the AI part of it?

Unknown 10:17

KAYE: So one of the things that we have is a whole different set of detections that humans uses to determine whether it’s a bot or a human. And historically, we’ve been doing that for a long time, but now the agentic AI and all those different AI tools are coming out. So you don’t just have agentic AI, you have scroll crawlers and scrapers and things like that. So being able to kind of use what we have learned from how we, you know, tell some things about both are human. Now we have to kind of build upon that and understand how, how are these AI agents, and how are these AI tools crawling and scraping? So some of the research that we’ve been focusing on more recently is really trying to understand just that. So looking, for example, at, you know, are threat actors trying to emulate popular AI tools in order to conduct different types of fraud, as well as if a threat actor were to be using AI agents to do fraud or any sort of cyber attack. You know, what does that look like? And humans data really kind of allows us to sort of start understanding some of that. I believe we have a couple of research pieces where we’ve covered those topics, as well as one that is going to be coming out in the future that I think really kind of let you see sort of similarities and differences between general bot activity, AI related bot activity, and AI agent bot activity.

VAMOSI: So when I think of bots in this space, particularly device oriented bots, I think of Mirai. And Mirai was famously used to bring down the Dyn DNS server. So what types of fraud or things are being committed with, with the bots that you’re seeing today?

KAYE: So I think Mirai was one of the first examples that really made you know, the general public in 2016 understand, you know, what is a bot? What is a botnet? Oh, my God, it’s this thing that’s out there. But the one that we have most recently uncovered, at least discussed publicly right now, is bad box two.

VAMOSI: So I have to ask in naming threats, was this something that was in the code itself bad box, or was it referring to the fact that it used devices, or is it playing off of Busy Box or some of the protocols.

KAYE: So oftentimes, you’re right, we do try to name threats after the different sort of identifiers in the code, or something that would make sense. But here very simply bad boxes. These were boxes, and they were bad. So while it is not necessarily kind of the most creative origin story that we’ve ever had, we’ve had some really great ones. It is, in fact, kind of how we named it. So originally with peach pit, which was the ad fraud component of the first bad box campaign, there was an identifier in the code that lent itself to using the word peach. So that was a peach pit with a bad box. However, that was the name that, since we used the first one. It really just got, you know, so much coverage and people were excited about that game. And it really does very aptly explain what exactly you’re dealing with with these bad backdoor boxes that we kept kind of rolling with it.

So this is a build upon the original bad box research we did. And if you’re not familiar with that, the general premise of that included all of these Android open source project devices that had back doors installed on them. So these were things like CTV devices, projectors, other kind of these lower end tablets and phones that sometime between when they were created and when they were actually purchased by the public in brick and mortar and online stores, you can probably think of a back door was installed on them, and this back door enabled the threat actor to do different types of fraud. So in the original bad box, we focused mostly on the ad fraud module called peach pit, and I believe we found about 74,000 suspected bad box devices. So fast forward a couple years, threat actors went away, and now they came back. So that helped us find bad box two. And bad box two, we found over 1 million devices that we believe have a variety of the backdoor installed on them. So some differences now include different types of fraud. So just a couple include click fraud, two types of ad fraud, and then creating an operating a botnet with this device, including providing residential proxy services so other threat actors can buy access to conduct their own attacks, as well as several different types of backdoors. So in the original campaign, we saw just one called Triada, but now the threat actor has more complicated, more complex, and then just overall, more backdoors.

VAMOSI: This is important. This crime is spreading because it’s using something known as residential proxies. What’s that? Well, it’s that cheap smart TV that promises you all these premium channels for a low low price. I mean, you know it’s got to be illegal.

REID: yeah. So, you know, there, it’s become a big sort of, you know, there’s a big market of selling things that help, you know, these fraud actors, you know, make their multi million dollar fraud deals. There’s a number of different sorts of components, and one of them is residential proxies. And these go from the maybe slightly sketchy to the almost 100% sketchy, because there are, there will be some people that say, Well, there’s a use for residential proxy. I want to see what my site looks like, you know, from all over the world, in different residences throughout the world, and I can do that. And so there could be some potentially legitimate use, but much more, it is fueled by illegitimate use. And going back to what I said earlier, about you have, you know, with this device, that’s your house, and most of the time it’s doing normal things, and occasionally it does something bad. This is very, very useful if you’re sending spam, if you’re doing ad fraud, whatever you know, type of fraud you’re doing, a big proponent of it, like most of the stuff that we see ends up now being funneled through residential proxy. So it may start in a cloud based service in China or Romania or whatever, but when it actually contacts you or the site that it’s doing the fraud, it’s actually coming from a residential proxy network. And these networks can be connected to TV devices like the bad boxes. They can be your phones. They can really they can be your laptop. You’ve downloaded this app that allows you to VPN wherever you want for free. But what they don’t say is that they’re also setting you up as a VPN. And so, or if they do, it’s very deep in the, you know, user license agreement that you never really read. And so, you know, to answer, you know, even another one of your earlier questions, what, how much of this fraud Do you see like when we sit on these residential proxy networks, which we did with bad box two, we saw it all like, literally every time you would see some sort of major hack show up, you know, on, you know, on the internet, or people talking about it. We’ve seen components of it run through these residential proxies, because they’re so widely used. So this became just a huge, you know, way that they could make money. That’s what they pivoted to. And we knew that we were gonna, we can’t really stop that as easily. And so we took a slightly different tack with bad box two, and working with some of our industry partners, we worked to get the command and controls that were managing all of these bad boxes sink hold, because those once we shut them down, they could no longer access and send new instructions so that they would carry on with their Old instructions for a while, but eventually they got disconnected, delinked, and we see now millions of these boxes connecting to these sinkholes that you know, you know, through industry partners again, like the shadow server Foundation, they were really great at working with us. We got the infrastructure that protects or that manages these systems completely shut

VAMOSI: I’m interested in the malware being on there, like again, I’m going to use Mirai as my basis, because I understand that one, there’s a vulnerability, and code was added to take advantage of that vulnerability. Is that something similar here? Or is it something that was injected into the Android open source project, or

KAYE: So with Mirai, you had your kind of version of the malware that was able to find other IoT devices that tried to fall down in passwords, log in and sort of self spread. So the bat box is a little bit different. So you do have that same concept of a backdoor itself, but it can’t, at this time, spread to other devices. So the way that we believe that these devices are being infected is sometimes at the factories, we believe that someone is installing these backdoors on the device itself. So it’s not necessarily a vulnerability, per se. It’s something that is a modification, either in the firmware or in bad box. Two, what we saw is there’s several different kinds of app level backdoors as well. So in addition to the back door being installed at the factory, you can also kind of get this back door installed on your device at sort of that application level, really, any way that you can think of that people get malware. So drive by download malvertising, you know, clicking on that wrong type of link. So they’ve really kind of diversified it that way. But it’s a little different from Mirai, because there isn’t that sort of self spreading notion to it.

VAMOSI: Okay, so Gavin with a network attack, you could put controls, you can block it, and so forth. When we’re talking about devices, devices in homes, what sort of mitigations are possible with that?

REID: Yeah. So it’s really hard for the end user to know that their box has been commandeered by the bad guys, and that’s particularly why they’re doing it. If you think of, you know, a spread of millions of these devices, and they’re, they’re global, right? And so if you’re, if you’re a bad guy, and you say, pivot to using one of these devices for a few minutes to do whatever fraudulent attack you’d like to do through the residential proxy network. The majority of the traffic on that box is normal traffic. It’s, you know, you watching Netflix. It’s you you know, going in and watching the sports and and so there’s a very small amount of it that is actually doing the fraud traffic. So it’s really, really hard for the end user to sort of figure that out or to mitigate it. And that’s why resident one of the reasons residential proxies are so successful is it’s hard for even the good guys that are trying to mitigate these things at a high level, to know, okay, well, 99.9% of the time there’s good stuff, but occasionally there’s some bad stuff there. And how do I know when to block and when not to block? And this is where things get very this is why you have companies like human security that, you know, we’ve lived at sort of modeling this over the last 10, you know, 15 years. So we have really good data to use in our machine learning models to be able to help understand when good connections go bad, like so, you know, to answer your question, it’s really hard. The only way you can mitigate this is by taking the device and, you know, putting it in the trash, right? There is no Oh, like, I’ll just run this, you know, Service Pack, or I’ll install some antivirus on it, and it’ll be fine. It doesn’t work, you know, especially for these devices that have been hard, you know, coded firmware. There’s no way you can fix that. Like, as an end user, you don’t know how to, like, wipe out firmware on your connected TV device and find the firmware for the very specific chipset that you have, and then reinstall it. You would need really specialized knowledge and tools to be able to do it. So there isn’t any fix. And then what we have been doing is, because there isn’t any fix, we’ve been you know, the mitigating factors that we’ve been looking at is okay. Well, how do we dismantle the infrastructure with a bad box one that was turning down how they were monetizing it, which was the peach pit model that Lindsay talked about, we had millions and millions of these fraudulent ads going through that were basically just making money for these guys. And so what we did is identify it. We worked with our industry partners, you know, where did, where does this ad fraud? You know, where do you come from? Who’s instituting it, who’s doing it? So we could shut down the entities, and so they could no longer do it, and we could block it, and we could make sure that, okay, we can’t stop them from owning these bad boxes, right, but we can stop them from making money. And so as soon as, like, we did this literally, and, you know, there’s a bunch of modules on this device, when we did this literally, like, a week later, we saw that the command and control centers basically went and wiped them all out. So they became normal connected TV boxes. And I think at that point they realized someone’s paying attention, the game is up, and they didn’t want it to connect back to them, because they want to do it again, because they’re making millions of dollars out of this. So they just, you know, basically wiped the machines, wiped the machines of all the badness that they put on. So even though we couldn’t do it, they could because, you know, they’re controlling these things at the root level, and they know what to you know, they put it on there, they know what to do. Now, with that box two, it was a little different. We thought, okay, you know, this isn’t, you know, they’re, they’re, they’ve got multiple different models for monetization, as Lindsay very correctly pointed out, like we could block them, and we could identify them really easily, because everything was sort of hard coded into the firmware. And once we found out the IOCs, we could share those and get them taken care of. So what they did is said, okay, one methodology for getting malware on these, one methodology for monetization isn’t going to do it anymore. We’re going to do many of these. And so even if they block one, we can still do another. And as we did exactly that with adware, they just really pivoted to selling these as residential proxy services.

VAMOSI: So earlier it was mentioned that there are primarily two types of ad fraud. Walk me through each.

KAYE: Sure thing. So I can start with the first one. So the first one was involving hidden ads, very simply, hidden ads are ads that the user can’t see, but that are claiming to be displayed. So that’s considered fraudulent, because people are being charged to show the ads, but no one’s actually seen the ads, which is not intentional. So this ad fraud component was responsible for 5 billion fraudulent bid requests per week when we caught it. So when we caught bad box two, we believe we caught it during sort of a testing stage, as evidenced by seeing the word test in some of the malware and other indicators that they’re still kind of testing things out, spinning them up. So while this doesn’t seem like a whole lot of fraudulent bid requests. This is something that we think could have gotten bigger had they been able to fully, actually, you know, productize their campaign. But the way that this happened was through something called the evil twin method. So this is something that we had seen once before with a campaign that we had uncovered and disrupted, called confetti. But the basic premise is that there are two sets of apps, so a decoy twin and an evil twin. So the decoy twin are these benign applications hosted in the Google Play Store, and they don’t do any sort of fraud. They just simply have to exist. So they have sets of identifiers, so ad identifiers, package names, other things like that, that then the evil twins can actually take so these evil twins use those identifiers, and they’re the ones actually doing fraud. So the way that you get an evil twin on your device, you cannot get it from the Google Play Store. You get it from different ways that you can be infected, right? So any sort of way to download malware, but very important, is not in the Google Play Store. So this is where all these sort of hidden ads fraud was coming. And then the second type of ad fraud was hidden web views. So really, what this is is very similar to a hidden ad, but it is a web view. So if you ever have used some sort of, you know, application on your phone and there’s some sort of Internet access to it that’s not through the actual browser, so goes to some sort of URL, displays it in the app. That’s simply probably a web view. So the hidden web view is a web view that the user can’t see. So what the device did was, in the background, without the victim’s knowledge, is it would have one of these hidden web views, and it would actually navigate out to a set of html5 games. And what it would do is look at the ads there. So pretty much how this works is that the user would be using a device, they would have no knowledge that this is actually happening in the background. So these games on these html5 game websites, they had ads popping up every couple seconds like there is no expectation that an actual human would have eyes on these games. That would be a hugely frustrating experience. I’m sure if you had you’re trying to play a game, and ads would come up every couple seconds. It’s just unplayable. So we believe that these games existed solely for the purpose of conducting ad fraud with these devices viewing these ads.

VAMOSI: wanted to follow up on what you said, these games, the hidden games, no human would tolerate all the ads going through it. So the point being, for every click, there’s monetization. What is the game? There was no pun intended. What is the goal of a criminal organization if they’re getting clicks on these ads or these ads are being visible to them?

REID: So this is potentially a way for cash out so generally, when somebody views an ad that’s worth some kind of number of cents, but once somebody clicks on the ad, that’s worth a whole lot more. So we saw a click frog component to this as well. So that’s very similar. Again. There are these websites that the devices would visit in the background and click on the ads there, but this is some kind of cash out mechanism. So if you are having your devices kind of go and then view these ads, then the criminals are actually kind of making money off of that. So if that makes sense to you, partially we believe that the click fraud component, and at least some of the ad fraud component, was the way that the threat actors from monetizing the other parts of the operation. So as you kind of understand, this is a whole cyber fraud ecosystem. There’s a lot of components to it, a lot of pieces, parts, threat actor groups that are all sort of interwoven here. So if you consider the ad fraud portion, you know a great way for them to make money, and potentially is a way that they can fund the other parts of the operation. That’s how I’d encourage you to look at it. And

VAMOSI: these ads are legitimate, or these ads are equally criminal oriented. So a little of each.

KAYE:I mean, they are ads, but whether or not they are good ads, they are not good ads, right? If that makes sense. So it’s like, in order to cash out, then there has to kind of be these games that have these ads that need to be looked at, but no actual people are going to look at them. So they’re never intended for actual humans. If that makes sense.

VAMOSI: I guess I want to clarify when you say cash out. So we taking, like, virtual money and make it Fiat. Or what do you mean by cash out? So Gavin

KAYE: Gavin might be able to explain this a little bit better, but the basic premise is that there is money that they need to then take from point A and put to point B. And if you view your ads, and you’re hosting your ads, then you can, kind of, if that makes sense, then you can, sort of, yeah,

REID: yeah. So Lindsay got it perfect there we saw with, you know, we’ve suspected that these have been used as a way to wash money. We don’t have actual, you know, the financial tracks transactions behind to prove that, but we do see exactly what Lindsay is saying, that people are buying their and paying themselves off with, you know, one form of Fiat and getting, you know, another form of Fiat out of it, so that, you know, it’s one level abstracted at the same time. They’re also at, you know, they’re putting these spaces up for anyone to bid on. So legitimate people can bid and say, oh, I want an ad on that website. You know, I’m going to spread my ads everywhere. And so they are getting a mix of legitimate ads in there, of people that are just, you know, the advertisers themselves are getting ripped off, which, again, is sort of where, you know, our company comes in, these advertisers, you know, buy us to protect against that sort of stuff happening.

VAMOSI: So Lindsay’s team comes up with bad box evidence. Says this is out there. This is real. What type of disclosure process is involved here, since you’re talking about a lot of devices out there that are potentially, you know, white boxing, the same software and hardware, how do you how do you start to surface up to the individuals that they need to do something, or that we need to be aware of this?

REID: Well, it’s hard, right? If they’re in the millions, you know they’re and you know, typically, you know, why did they why? You know, the another question would be, why did they buy these in the first place, right? And that’ll help you understand maybe, how difficult it is to stop it. But instead of buying an Apple TV for what 95 bucks, they buy this, you know, Android, open source, black box thing, and it’s like $30 or $25 and so, you know, often they’re being sold in areas where people just don’t care a lot. Also there, you know, we saw a lot with bad box two, where they’re being packaged and resold as kind of pirate boxes, where you buy the box from this strange third party and it’s pre loaded, and it like shows you all the Netflix and all the, you know, all the different sports channels and all the movies and everything. And you pay one, low 599, a month, and you get everything globally, worldwide on this box. And so, you know, they’re already, you know, they’re, they’re purchasing this box specifically because it’s, like, super cheap, right? So they don’t necessarily, there’s no nest, there’s no, like, Registrar of everyone who owns one of these boxes, and there’s literally hundreds of different manufacturers behind them, and then sub manufacturers, or, you know, people that buy them and resell them underneath them. And so, you know, we reached out to the normal folks that we saw these. I’m not going to name them, but you can imagine who they may be. And you know, they were concerned, and they didn’t like it, and they said, you know, show us like, you know, which where is this for sale? Say, on my website, and we’ll get these taken down. And it just became, you know, there were so many of them that it was really, really hard for us, you know. So what we do, like we did this, in fact, this got pushed by, I think we presented on this at a security conference. And after presenting, the ETF sent a note to the FCC and said, Hey, what’s going on here? Why aren’t you doing something? And then some of those same people who may have not been as interested in following up, like followed up and said, We want to get every one of these and make sure that we’re no longer selling them and we help them. And they took them all off, right? Literally, two weeks later, the same boxes were on different sellers and they had, you know, there’s like, I think the one of the main boxes was like a T 95 now, again, not all t 90 5s were bad, but a lot of them were bad. And so these sellers just changed their seller name, and they would have the same box, the same advertisement, but they just airbrushed out the T 95 and this is now a connected TV box. So it’s an ongoing problem, right? Because we have folks in areas that maybe care a little bit less, like if this was, if these were all being manufactured and sold by a US company, we could probably do exactly what you said. But because it’s so diversified, we had to think of other ways that we could just make it, not as you know, you know, they wouldn’t be able to make as much money out of it. That’s the only thing that really stopped them from doing it, or and or dismantling some of the infrastructure so they could no longer monetize the boxes themselves.

VAMOSI: So maybe this is a question for Lindsay, we talked about shutting down the infrastructure. We talked about not selling these devices and so forth, but is there sort of a root vulnerability or something that can be addressed to mitigate this going forward?

KAYE: So this isn’t necessarily something that was built on the back of a vulnerability, right? So somewhere in the factory, someone is maybe associating with a group that is installing these back doors on the devices between when they’re created and shipped out. So that’s not really a vulnerability, more like a bad set of threat actors, right? And then you have this sort of second component, which is a little bit more controllable, in that, you know, people go out and they download apps from all sorts of third party marketplaces, potentially ones that you know may or may not be sketchy, then you have sometimes people click on suspicious links. So while, obviously, we can’t blame victims for installing these things on their devices, that’s something that you know, with a little bit more education and just kind of knowledge of the fact that that is risky we always talk about, you’re only downloading apps from official app stores. You know, being pretty careful about what you click on. If your device is behaving funny, you’ll consider chucking it. That’s kind of the other aspect of it. So that’s not perfect, but that does sort of help address at least that component of it.

VAMOSI: And this is limited to the Android universe, correct?

KAYE: Yep. So specifically, only Android open source project devices. So these are the ones that are not Google Play protected certified there. As you are aware, Android is an open source operating system. People kind of just install it on devices that they want. So to really separate out these devices, the AOSP ones from the official Google ones. It is only the AOSP devices.

VAMOSI: So, hearkening back to my questions around Mirai, Mirai is still around. It’s not going away. Are the projections that bad box two is going to be around for a while, and we’re just in a mitigation mode.

KAYE: So one of the biggest things I believe that keeps Mirai around is the fact that Honest Empire actually open sourced the code, and now we see people pivoting on it, making new versions of it, right? So that’s kind of one of the biggest pieces of what makes Mirai kind of come back and be successful. In addition, if you look at the bad box, we don’t see that open source back door. We see the threat actors pivoting on the back door, making new versions of the back door. But outside of that, it’s something where we demonetize them. The first time we’ve demonetized them again, they’re always going to come back. Right? A cyber criminal’s job is to make money. They want to keep making money. We have prevented them from doing so. So like all cyber criminals and generally, all threat actors in general, they will pivot, come back, try again, because it really is important to them to keep victimizing people to, you know, pad their wallets. If you think about

REID: it, I would add onto what Lindsay said is that it’s not a cyber vulnerability. The vulnerability is in the supply chain. Vulnerability is, you know, you can create these things for $20 and you have this wide audience that you can sell globally, that until that gets fixed, until people say, Okay, well, I’m, I’m really concerned about the device that I plug into my network. I want it to be something that I’m not concerned with. And so there’s like, a huge sort of communication and sort of learning here, where I think people, if you really sat them down and talked to them and showed them what’s going on, they don’t want people committing crimes from their home networks and stuff, but they just don’t know. They don’t know what the, you know, the so, you know, it’s really important. They don’t know what the risk is. It’s really important that, you know, we have folks like you that are doing these podcasts and talking about it, because it just doesn’t get, it doesn’t get to the level of visibility, you know, we saw. We had another. What was it? Fish and chips that was around, like, shopping and stuff, and how these fake sites were up, and that, you know, the actual, you know, real news, you know, wanted to do interviews on, but they haven’t seen that same interest, really, with bad box,

VAMOSI: so kind of niche. So it sounds like this is a supply chain issue. There’s a lot of talk around hardware bills and material and software bills and material would something like that start to address the problem.

REID: I think we’re so far away at the $20 mark for s bombs and stuff that it really wouldn’t matter. Like, by the way, they would just stamp on whatever s bomb you wanted. Like, yeah, no, that wouldn’t. What would, you know, change this is, is people like, you know, maybe the major, major menu, you know, people that are reselling these things say we will not resell stuff unless it’s past some level of certification that it’s okay. You know, Lindsay’s mentioned the Play Store protect a couple times, like Google does have devices that are certified, that are totally safe to to use. And so, you know, until customers say, I want to use that device and not another, or the government gets involved and does something like they do with cars, right? You can’t buy a totally unsafe car, supposedly, or, well, you can’t drive it, right? So the other do that. But, you know. Think of, it’s one car or two cars per family. You know. Think of how many IoT devices An average home has, and who’s going to fund testing. And like, we’re in a, you know, we’re in an area where it makes it easy for the fraudsters to continue to exploit this. I was

VAMOSI: just going to say, if we’re talking about a $20 internet connected toothbrush or maybe less $10 nobody’s going to care. They’re not going to pay the $50 one that’s been certified and run through the rigors of testing. So I get your point there. I’ve asked a lot of questions. Is there anything that I haven’t brought up that you’d like to introduce into the conversation. Either of you

KAYE: Sure. So one of the things that I want to highlight here is bad box 2.0 really does highlight how much more sophisticated these ad fraud and cyber criminal operations are getting. So we’re used to all kind of seeing those more simplistic ad fraud operations where it’s just kind of a bad set of apps or something like that. But, you know, it is just one component of this huge cyber fraud ecosystem here. So we have the ad fraud, we have the click fraud, we have the residential proxy modules. We have the back dooring of these devices with more than one back door. So it’s something where, when we look at how cyber fraud is going, it is getting more sophisticated, and this is just one great example. So we can certainly expect to see campaigns of this level of sophistication in the future. So I guess that’s what, partially what keeps my job super fun, but it is something that is certainly worth being aware of.

REID: I think, like with the badbox one, it was, you know, was bad. The boxes did bad things, but the size was very different. With Bad box two, they’re bigger and a lot better. And, you know, I look at this as you know, these are kind of like sleeper cells all over the world, just waiting for the bad guys to do whatever they want, you know, it should be, this should be a big issue. The Press should be interested in this. This should be something that people go like, if there was a way that you could just get into every home in America, or South America or whatever, and commit crimes there, stab people in the backyards and stuff, people would, would not stand for it. But somehow, because it’s, you know, happening in zeros and ones, it’s okay. And what we’re doing is we’re setting up this amazing framework. We were part of it. We buy these devices for the bad guys to continue to persist and to continue to be effective. And there isn’t really a lot doing, you know, there’s, you know, we’ve seen great I mean, props to Google. They’ve done tons to try and shut this down, really out of the goodness of their heart, right? That, you know, they’re and, you know, we’ve seen a couple of other folks you know, that have been, you know, I think team Comrie in particular, you know, we mentioned the shadow server foundation. There are a few folks that are going, Hey, this isn’t right, and should do something to stop it, but it really needs more.

VAMOSI: I’d like to thank Gavin and Lindsey for talking about the BADBOX 2.0

-30-

Operational technology (OT) systems are no longer limited to nation-states; criminal groups and hacktivists now actively target these systems, often driven by financial or ideological motives. Kurt Gaudette, Vice President of Intelligence and Services at Dragos, explains why these systems might not even be the primary targets.

[Heads Up: This transcription was autogenerated, so there may be errors.]

VAMOSI: Ransomware. What started off as attackers hitting individuals at home has ballooned into an international, multimillion dollar industry, targeting large corporations. Even targeting parts of our critical infrastructure.

What’s interesting is that there are Ransomware as a Service vendors, sites where you can contract for someone to hit a specific organization that you define. So large are these organizations they are even franchisees — you know like McDonalds, independents who resell the main brand — and sometimes … sometimes, even though the main ransomware organizations say they have ethics, such as we won’t target x, y, or z industries … Sometimes, the franchisees don’t give a damn who they hit, or recklessly scope their campaign to pick up collateral damage.

Such as attacks like on a children’s hospital, where apparently a franchisee made that mistake. That’s what happened in December 2022, right around Christmas, Toronto’s SickKids hospital was hit with ransomware. A hospital. For sick kids. Over the winter holidays. Here’s the CBC.

In this example, we have a ransomware gang apologizing. Here’s a decryption key. We’re not going to do that anymore. The problem is that it already did happen.

Last summer there were several small water utilities attacked. Why? I wanted to find out so in a moment I’ll introduce you to an expert.

This is a story about … what’s behind bad actor motivations against utilities and industrial control systems.

I’m Robert Vamosi

This is Error Code

[MUSIC]

Vamosi: Years ago, in 2009, I sat down with Howard Schmidt, the first US Cyber Czar under President Barrack Obama. I’d known Howard for years before that interview at RSA (as it was known then) and so we were comfortable just talking. As we started the interview I remember Howard looking down and asking if I was recording. I assured him that I was. Afterward, he asked again. It wasn’t until I got back to the office at CNET that I confirmed what he’d said, that I hadn’t recorded a single word.

Fortunately, I had taken paper notes, and this wasn’t an audio interview, it was in print. I was nonetheless able to write my article.

A few months later, I interviewed Schmidt again, this time for my book, When Gadgets Betray Us. And this time I made sure I had not one but two recording devices handy. I always do that now. In fact, the video of that interview is still on the internet somewhere along with an excerpt of the book featuring Howard that appeared on Salon.com.

Long time listeners to Error Code will note that I usually open with my guest stating their name and something about their company. Well, in the blur of doing multiple Error Code interviews in various locations — some in crowded hotel lobbies — at this year’s RSAC I forgot to hit a record. And I forgot to hit record on my backup device as well. And I didn’t realize any of this until after we’d completed our introductions and launched into a discussion of motivations for hacking ICS. So, my apologies.

My guest is Kurt Gaudette, Vice President of Intelligence and Services at Dragos. Dragos is of course an industrial cybersecurity company that focuses on products and services for the non-standard part of cybersecurity. So things like SCADA and industrial control systems critical infrastructure instead of normal enterprise computing.

The interview starts with me asking Kurt about which of 16 critical infrastructures his company supports.

Gaudette: Well, we started out in electric. That is, that is where we started. We really cover all the major verticals. Electricity is our bread and butter, but oil and gas, different regions, mining, we kind of do a little bit of it all. But I think electricity is where we’re rooted. So from the manufacturing side of the house, we cover manufacturing in a big, big way. That’s one of our major verticals. We also cover building automation, right? So think of HVAC systems. Think of HVAC systems, supporting data centers, anything supporting a data center. From an OT infrastructure perspective, we also go down that lane as well. So we’ve got pretty good coverage across a lot of the

Vamosi: Last summer there were a series of attacks on small, rural water districts and I’m wondering what’s the monetary value for some ransomware group to go after a small rural water district. Whereas nation states have political reasons, or military reasons for going after the targets, why target a water utility in rural Texas?

Gaudette I would say like hacktivist, political reasons, right, or emotionally driven reasons, criminal groups. It’s about the bottom line, right? And the cash and the easier it becomes, the more capabilities that are spread around, and the more awareness they have that OT is a target of opportunity, they’re going to take advantage of that. And so that’s, that’s the piece that really disturbs me, is it’s, it’s not just a state actor tonight, and I say this loosely, that kind of have a code of conduct, or at least some left and right boundaries that they act within. And I go back to my time when I did space technology and and, and, I think you probably would remember this from a tech perspective, that Russia and China were launching some things up into space, right, like positive large debris fields, and that was a big step over the line.

VAMOSI: What Kurt is talking about is an event in 2022 when Russia blew up a satellite in orbit and created a very large and uncontrolled debris field as collateral damage. Here’s KSAT 12 with more:

KSAT 12: If you missed it earlier this week, that was big news. The U.S. State Department reported that 1,500 pieces of debris are now orbiting Earth after Russia recklessly launched a missile that destroyed one of its own satellites.

The lives aboard the International Space Station were at risk, including the four American astronauts on board.

For more explanation, an astrophysicist from the Harvard-Smithsonian Center, Dr. Jonathan McDowell, provided insight:

“The Russians were testing out their new anti-satellite system. This is a new version replacing older satellite weapons. They’ve conducted test launches before, but this was the first time they actually targeted and destroyed an old Soviet-era spy satellite from the 1980s, blowing it into thousands of pieces.

These fragments are now whizzing around the Earth at 17,000 miles per hour. The International Space Station and other satellites are orbiting through this risk zone.

In the immediate hours after the test, the debris remained relatively close together. NASA reported that it passed uncomfortably close to the ISS, prompting astronauts to take precautions, including closing hatches in case any compartments were hit.

They’ve since reopened the hatches as the debris has begun to spread out. While it’s no longer an immediate localized threat, it represents a 10 to 20% increase in overall space junk, which increases the long-term risk to satellites and space missions.”

Gaudette: I think you have a similar type of thing, if you said for what we’re seeing now, and that is, but, yeah, so I go back to the Russian China thing, but there were still thresholds, right? They knew they were going to get them marched, and then they kind of pulled back. And, you know, it was a one time deal, or maybe a two time deal, but there are still boundaries. What I’m concerned with is kind of these lack of boundaries with these happiness groups and these criminal groups now, and that’s the scary part, right? There’s nothing necessarily holding them back.

Vamosi:What I’m inferring from what you said was that these low-funded utilities aren’t necessarily a primary target, that the bad actors may be going after specific electronics or devices that’s within that organization, and it becomes collateral damage that they are affected.

Gaudette: It could be collateral damage. The other thing is, is that sometimes these smaller utilities or smaller businesses, I think sometimes people overlook those and go, Hey, that’s really not part of the political infrastructure. But if you really were to kind of do the network diagram of, hey, what are the things that are supporting military capability? What are the things that are supporting key transportation for movement of military capability? And you can make these ties a number of different ways, right? But if you really take a hard look, they become a really critical part of critical infrastructure, like a key component, and it’s not and size doesn’t necessarily matter. It doesn’t have to be the biggest thing, right? It could be very small, localized but they provide X amount of power to a particular human enabler for something the United States needs from a national security perspective, and that’s where it becomes very interesting. So I don’t want to ignore the smaller utility companies or the smaller businesses, because they also play very critical roles. And then again, there is the opportunity for collateral damage, and one could envision, like an aggregate, it could be demonstrably more damaging than going after one large target. In fact, the barrier to entry is probably lower in that case, because they don’t have the resources they have. It guy who also doubles as there as the lawnmower, etc, etc, right?

VAMOSI: In episode 47, I spoke with Leslie Carthart of Dragos, and she mentioned that with the smaller utilities it literally comes down to the one person who is both the IT and OT professional, as well as the one who mows the lawn and pays the bills. If you haven’t already heard my interview with Leslie, she’s a masterful storyteller. In fact, she told the story of a moth that set off alarms in a remote power plant in the Dakotas. I know we talked a little bit about Leslie’s great moth story. I bet you have a great story from your past.

Gaudette: I don’t have a Leslie moth story because that’s probably one of my favorites, that favorite stories of all time and Dragos lore, right? We tell that story over and over, and I think it’s, I think that’s a great story to identify the type of people that we have working at Dragos. They’re very innovative, they’re very creative, and they think outside the box, which I think is fantastic. I think I guess the story that probably I highlight, and it was when I first came out Dragos. It was my first year, and it was very much about the public, private relationship of things, right? So, as you know, the National Security Agency via the cyber collaboration center, there’s CISOs, there’s a number of different oriented Department of Energy, but we do a lot with the cyber collaboration center. They’re the ones that are very much reaching out with information and that will bring up one good use case, and that was the Rockwell Automation and vulnerabilities kind of use case. And this is very much about getting reached out to by the cyber collaboration center saying, Hey, we have some information. We may have a partner that may want to work with you to solve a problem. Of course, we accept it because we’re always in that lane, right? It happened to be Rockwell. The CCC had information about what they suspected. There was intelligence associated with some capability that could be leveraged to take advantage of these vulnerabilities. You know, they can provide us only so much information that they can declassify, etc. So we took whatever information they could give us, pretty much reverse engineered it, built some analytics around that and and they’re very much aware that we’ve got a pretty wide reach into the community via our tech. And some great things about our tech is wherever we have a platform and whether we have to watch where we can hunt an environment we can run these analytics and see whether that capability, at that time, was active in the community. We also have something called neighborhood keeper, which is our anonymized resource that sits in the platform as well, that basically sifts through a bunch of metadata in an anonymized fashion. And that’s another way for us to go and run the gamut, again, a global gamut, where we can say, hey, if we run this analytic in this environment, is this capability out there in the wild somewhere? And it just gives us a good sense of, is it or is it not? We don’t know anything more about the customer, except for maybe it might be in North America, right? And depending on the density of the numbers of platforms that we have, right, we can say, hey, maybe it’s in the northeast side of North America, or it’s maybe in Asia, or maybe here. And then we can send an anonymized message across the net and go, Hey, we know there’s three or four instances of this going on somewhere. Phone home, ET, can, can anybody want to provide more information? And we do have people that phone back and go, Yeah, this is what we’re seeing. Can you help us out? Right? And we’ll work with them to kind of get a better picture of what’s going on. And kind of back to this Rockwell thing we did just that we, you know, we got some hits. We worked with the community to kind of determine what it is. We worked with Rockwell, you know, a good chunk of time in advance of the release of those vulnerables publicly. And so by the time they actually released those vulnerabilities, and I’ll be fair and say, hey, it was just not Dragos, it was a few different companies working in a common space to really prepare ourselves for that release. And so by the time they released it. We were, you know, a good chunk of the community was already aware and kind of working in the background to get ahead of it in case, in case, the adversary did decide to take advantage, right? Good. This is how we do business, right? Good, yeah.

Vamosi: So you mentioned public, private, and I find that interesting. I come from a background of working with ISACs and so forth, and that’s just one example. But what I heard last year, and what I’m hearing again from you, is that you’re actually getting some of your own customers independent of ISACs coming forward and saying, “we would like to educate the community as well.” I’m seeing this sort of bubbling up from the industry, like, do you think that that’s a maturation of the IoT or OT space?

Gaudette: So there is maturation going on. It’s not going on at the rate that you need it to go on. And I’m sure other people have said this, right, but I’m heartened by the fact that I’m seeing it right. I’m seeing the awareness level rise for operational technology and exactly what it means, but I would really like it to be part of the public conversation space where they can just go, oh, operational technology. That means my lights, my water, my, you know, what I buy at the store. And when people can make that connection, then it means so much more. But I think a lot of people automatically default to the IT side to housing. You go, Oh, it’s just information they’re stealing. And of course, people are concerned about getting their credit card information stolen, or this information, or that or intellectual property. But I also look at the intellectual property of the OT space, right? So when you have a strategic entity coming into these environments. And here I’m talking about volt side, or what’s common in the known as Voltaire, right, sitting in an environment maybe, you know, in the case of some of the customers that we’ve recently dealt with lately, 300 days or more conducting espionage, and really kind of going, Hey, how does this particular entity have their operational technology set up? And think about if they did the same thing and they do in a manufacturing way. Hey, how do they set up in a manufacturing to get the most out? So there’s IP in that space, not only from a hey, how do we do business and conduct business, but boy, how do I attack this later on, and how do I shut it down? How do I manipulate it? Disrupt, disrupt, whatever. Right? So I think that part of that intellectual property space is also key, and it’s very much associated with operational technology as well.

Vamosi: So the broadening of people’s perspective of what OT means, and it’s important, it’s important, and you talked earlier about the connectivity to other things. For example, I’m just going to throw this analogy out there and tell me if I’m off base, but like the interstate freeways in the United States had a military design behind it to move materials from one country to the other fast, so you have a lot of flat roads to be able to do that. It’s that type of awareness that might need to happen where this common device in your environment could be leveraged by a nation state, and people would be like, What? No, it’s an internet connected toothbrush. What are you talking about? Is

Gaudette: The type of Yeah. So I want people to be aware enough that you know, from a national career, this is asking a lot, right? So I want people to be aware of what national security needs. Just the fact that you brought up the interstate, I don’t know how many people are aware of why the interstate system was built, or, Hey, why is it? Why is our rural structure built the way that it is right? Why is the shipping and the port system built the way that it is built for both defense and to project power, a lot of that right national security with national security in mind. So, yeah, I would love for that awareness level to go up so they realize that these small things, or in aggregate, a lot of small things, can lead to secondary and tertiary effects that people aren’t even considering.

Vamosi: So what do you say to a prospect who is struggling with their IT world, and you come walking in the door going, Oh, but wait, there’s more. There’s this whole ot world you got to deal with too. I mean, do you coexist? How do you start that conversation? Am I asking for a sales pitch here? No,

Gaudette: No there’s, there’s no sales pitch. I think how we start that conversation is about taking a journey from right? And it’s about understanding, what is it that I can do to make a difference in a customer environment, you know? And from a customer’s perspective, is, hey, what are the few things that I can actually do that makes a difference on things we always lead with the five critical controls. It’s a Sans written paper. Rob Lee Tim Conway wrote it together, and basically they took all of the activities that they had seen to date and said, What are the common factors, you know? What are the things that these adversary groups can most easily take advantage of? And then, what, from a customer perspective, can they do to better secure their environment. Just from a What are the five basics, right? Have a plan. Design, you know, secure architecture, its visibility and monitoring, secure roadmap, remote access. It’s, you know, risk based, vulnerability management. Those five same build a defensible architecture. I think I flipped something there. That was the second one, right? But if you do these five things, and these are not an overnight thing, these are, these are, these are a multi year effort, right? But you have to start somewhere. So we give people some place to start, and we are, I’m also very open. It doesn’t have to be with us, right? But hey here and Sam’s rights for the community, right? So, hey, community, these are the five things that if you do them, you know, this makes defense doable, and I think people don’t realize that it’s doable, but you make it harder for these adversaries to get into your environment, and it’s less likely they’re going to get into your environment. And for those people, for those companies that we see, implement those five critical controls, they’re pretty successful, right? So if my my pitch, or my getting off the stage line is defense is doable, I think that that is, that is the one I’d want to

Vamosi: That seems to be a theme this year, rediscovering that SANS paper, because you’re the third guest that I’ve talked to that has evoked that paper for brilliant. So it hasn’t all been Dragos, right?

Gaudette: No, and that’s a beautiful thing, right? Because that’s about the community you learn. Well, it’s also SANS.

Vamosi: In addition to those five things, do you have any other guidance for utilities and ICS that you provide. Any I know you didn’t like the word preventative, but any best practices that they should engage,

Gaudette: well, those best practices are rolled back, back into my five critical controls, right? Because pretty much everything that we do is derived off of those, right? We use those as the basis for how we do business. Fair enough.

Vamosi: Fair enough, I guess,my final question then would be, how concerned should utilities or ICS in general be about supply chain attacks?

Gaudette: I think we should be. Supply chain affects everything, right? So you should be aware of what you’re dependent on, what comes into your environment, what the age of the things are in your environment, what the updates to those things are be very aware of the vulnerabilities. And some vulnerabilities are supported. Others are not right. So not all companies are made the same. So some original equipment manufacturers won’t support continuing to build patches for that gear after a certain amount of time. So it’s just be aware of all of those factors, right?

Vamosi: And in that OT world, with those devices that have gone beyond their end of life but are still in use, there are controls that can be put in place. There are ways to mitigate vulnerabilities that may come up, yes,

Gaudette: 100% and that is something that Dragos is very serious about, right? It’s about identifying what mitigations you can do. And we found, like the vast majority of vulnerabilities that are outlined in the different variety, excuse me, and the advisories don’t come with mitigations at all. I can’t remember the percentage right off of my head, but there are a lot that do not. And what we do is we roll through the operational technology associated vulnerabilities and we provide those mitigations back into it. Not only we provide those mitigations, we correct them for errors, and then we categorize these vulnerabilities in the three basic categories, what we call now, next and never. So for companies that are not as mature or even mature companies, you get a deluge of here are all these vulnerabilities in your environment? Well, what do I do about that, particularly from an operational technology perspective? Because it’s hard to shut down my manufacturing line, hard to turn off electric distribution, right? So I have to plan and work to find a time to do a patch. And then, out of all of those ones that are out there, out of all those vulnerabilities identified, which one should I do first? So we spend the time at Dragos to go now, next never. And in the now category, it’s 6% 6% are in that category of these are the ones that really matter. These are the ones that are associated with the vault typhoons or the vault sites of the world, right? So giddy up on those 63 and 31 are the next two categories, right? So you have a little bit more time to deal with this. Deal with these as you can but these ones get after now, and I think that really helps the customer and the community writ large to just deal with this overflow of information and just help them start getting after the problem

This is a story where one maritime company found multiple vendors maintaining unrestricted VPN access to systems across a cruise ship, exposing safety-critical functions to potential compromise. Bill Moore, CEO of Xona Systems, returns to Error Code to talk about how that company and others, such as data center operators, are recognizing their latent multiple-vendor OT exposure and learning how to address it today.

[Heads Up: This transcription was autogenerated, so there may be errors.]

VAMOSI: Ah, summer time. Perhaps you’re thinking of taking a cruise [sound of a ship; seagulls]. If you’ve ever stood beside a cruise ship, they’re massive. They are literally a city on the water. They’re a SMART city on the water.

Think about it. A cruise ship consists of several life critical systems, and Original Equipment Manufacturers install and maintain these critical systems — such as navigation, engine controls, or water treatment. Because they are specialized, often the individual manufacturers retain remote access for support, often through always-on VPN connections. This creates significant risk, as a single compromised OEM access could jeopardize entire operational environments of the ship.

Maybe you’re not taking a cruise, maybe you are driving and you see a vast data center just off the road. That data center has ancillary systems like HVAC and fire suppression. What if a third party that controls those gets compromised?

Really the question is what happens when a cruise ship gets hit with ransomware in the middle of a journey? What happens when the HVAC or fire suppression system at a data center is compromised? And do we really have to wait until that happens to find out?

Perhaps we need to rethink the OT aspects of this. Instead of providing individual vendor access, what about having the Cruise line handle the communications with the devices?

This is a story about OT systems you might not be thinking about and what security controls can be put in place.

I’m Robert Vamosi

This is Error Code

[MUSIC]

MOORE: Okay, Hi. I’m Bill Moore. I’m founder and CEO of Zona systems,

Zona systems, you know, we were founded about seven years ago, really, for the sole purpose of providing more simple and secure access to your most critical infrastructure systems, you know, typically operational technology, HMIs, data systems, PLCs and so forth that exist, you know, to run the world, whether it’s in energy, oil and gas, manufacturing, transportation and so forth.

VAMOSI: I previously spoke with Bill in Episode 53 where we talked about how you can virtualize the OT network and interact with it, adding 2FA and encryption. The security world spends a lot of time focusing on the IT networks and IT security, and all the while in the background, there’s been OT. OT predates IT. So is that kind of a fair way of looking at the history that you know? We’ve put a lot of emphasis on networking for a decade or two, and now we’re just getting back to looking at the OT world again.

MOORE: Well, I Yeah. I mean, I think, look, so networks, you know, became the internet very quickly, you know, over the last 20- 25 years, right? Really, back in the late 90s, 2000s when the internet really, really took off, right? And everybody was connecting, you know, to wider networks and sharing information, you know, remotely and everywhere, right? So OT really kind of didn’t move like that, right? OT, sort of sat there in the background, for the most part, completely disconnected from it and disconnected from the internet. It was, you know, running, you know, just about air gap for the most part. You know, it’s only over the last 10 years that I think, you know, these connect these networks have started, you know, either you know, and it started out really pushing some data to some application that would sit in the cloud, that you could analyze some of the data coming out of the OT network. But, you know, no inbound connection. So when I, you know, I started the company seven years ago, the first two or three years, you know, I talked to a power generation planet, you know, control system supervisor. And he would say, you know, we will never, ever allow remote operations into a power plant. Just will never happen, right? And then, you know, fast forward to today. You know, you had COVID in there. In between, it changed even for, you know, very, very critical systems that, you know, at a power plant, right and and now we see on manpower plants, you know, where they’re controlling it, you know, from another power plant, or remotely, or whatever. So it’s, it’s moved pretty quickly, and you have this kind of, you know, a lot of those assets, too, are being digitized. So there’s this big digital transformation going on. So OT is blurring into looking more like it. Because in the past, you know, you’d have, you know, PLCs and things like that, and these kind of obscure HMI systems that maybe ran some really strange version of Linux or Windows or something. But you know, now there’s a lot of, you know, regular systems that are in the OT network that are really more IT systems, right? And those IT systems are talking, internet protocols are talking TCPIP so, so I think there’s been this slow you know, I always talk about our networks. It’s kind of the the networks at times for God. I mean, you know, you go into some of the networks that are now just starting to connect. And like, you know why you’re running telnet on this network, right? And because, you know, telnet disappeared 20–25 years ago in IT networks, because it was just so insecure, right? That, oh, well, that’s the way we connect it like you can’t use that, you know, it’s just things like that.

VAMOSI: Right? And to your point, you can put security controls, you can turn off those insecure protocols, you can you can do things at a network level, but still, it’s the device itself that’s that’s needing to be secured from the beginning,

MOORE: right? Right, exactly. So, yeah, I think it’s and it’s really about, how do we, how do we put modern controls, you know, privilege, access management, modern multi factor authentication, you know, modern encryption, all of those things to make sure that the connections to those critical systems are protected, right? And not not a lot, not expose those, those antiquated systems that can’t be patched anymore, that are running, you know, insecure communications, and keep that kind of in a controlled environment so that they’re not accessible or easily accessible outside of that critical network.

VAMOSI: Yeah, I’m kind of interested. We talked last time about the usual suspects, energy, oil and gas, transportation. What are some of the peripheral areas that you’re protecting?

MOORE: Yeah, I mean, those are kind of the heavy, heavy industries that you know there, but we also have customers in the Department of Defense as well as and I actually look at DOD is kind of a horizontal, not as a vertical, because it can, it can include a lot of the heavy industries as well. But we’re starting to see a lot within data center management. You know, with all the AI coming out, we’re seeing more of an interest in sort of a more secure way of accessing data center infrastructure. Maritime is an area where we’re seeing quite a bit. We have a very large maritime customer, where we provide secure, OEM, third party access to critical operational technology on cruise ships.

VAMOSI: And then with maritime, are we talking about the ports? Are we talking about the actual ships?

MOORE: So we have customers in both. We have customers that are protecting, you know, systems where, you know, there’s, obviously, you have automation around unloading and offloading cargo in a port. You know, a lot of those are automated systems now that, you know, obviously is critical infrastructure, and obviously there’s, there’s safety and availability concerns, you know, with those types of systems, yeah, and so with ships, you know, we see that with, you know, both cruise ships and and cargo. We have, as I mentioned, one very large cruise customer, and then we are working with, actually several other right now, other maritime customers that are more cargo oriented.

VAMOSI: So within transportation, you were saying there’s some attention being paid to supply pipelines,

MOORE: Yeah, I mean, well, I start with maritime, you know what we’ve so one of our customers, what drove them to our solution is, and this is kind of a trend, not just in maritime, but I’m seeing also manufacturing and other industrial sectors. There was this idea that the OEM puts in the system, whether it’s a manufacturing line or, you know, automation on the manufacturing floor, whatever it may be in that particular sector. And they own that. They own the access, they own, the support they own making sure everything’s running right, but, but, you know, up until recently, there wasn’t this. Hey, I also, you know, we, we actually own these assets, even though they’re being operated by our OEM. And this is, you know, whether it’s Rockwell or GE or Siemens or ABB, it doesn’t matter, right? It’s, it’s, you know, do we know what kind of access they’re using, and is it secure to our critical systems. So in maritime, what happened is, they had many, you know, we’ll say, more than a dozen different OEMs that we connected different types of critical systems on on the cruise vessel, right? So one may be a merit, maybe, uh, the navigation system right one could be the distributed control system that provides visibility into all subsystems. You could have an engine control room. You could have water and wastewater, you know, because it is a city, floating city, right on the water. So there’s a lot of operational technology and critical systems, and each OEM would maybe have a piece of that right so they were compromised through an always on VPN connection because they didn’t recognize that owning the asset meant they also owned the risk that of compromise of a group, you know, compromising and hacking into those systems, right? And so that obviously got up to the board level, because you have the safety of passengers and not, you know, with compromise of those systems and so forth. So they decided, hey, no more, right? We have to tell the Siemens and the GES and the ABBs and the Emersons and all of these big OEMs, you can no longer use your own solution to connect to the network, right? You have to come through our solution so, and we’re going to, we’re going to audit it, and we’re going to, you know, be able to go back and watch a movie of what you did. And, you know, they’re very transparent on that. And some of them were like, hey, great. I don’t have to worry about it anymore. I don’t have to worry about my own access. And others are like, no, what do you mean? You know, I I don’t want to, I don’t want to use your system. So it’s it, but that’s a trend that I’m seeing is is super important, because you know the supply chain as security aspect, when, when the asset owner or the people that are running these industrial systems, whether it’s maritime or manufacturing or or oil and gas, whatever it may be the there’s massive safety involved, right? If they don’t have visibility and forensic knowledge of who accessed what, when you know how, long and so forth, especially if there is an incident, you know, cybersecurity incident, you know, being able to go back and track that is, you know, as I am vitally important. So that’s a trend that is happening as we speak. Across the board, there’s a recognition, you know, we’re seeing not only directors of OT cybersecurity roles that have been put into this is mainly medium to large, you know, enterprise companies, but also security architects that are looking at this problem and making sure that, you know, solutions are being put in that, you know, reduce the attack surface, reduce the risk, reduce the exposure of these critical systems. So, so

VAMOSI: that kind of sounds to me, like automotive as an analogy, where, you know, the OEM sells the car, but the pieces of it are, you know, provided by tier one, tier two, tier three vendors, and ultimately, I Think the security is with the OEM automaker you’re suggesting that, like with maritime, that that’s a model that they’re moving toward, yeah,

MOORE: that’s, that’s a, that’s a great analogy, right? Because, you know, if you have a car manufacturer, and obviously there’s been stories about, you know, folks hacking, you know, into car and unlocking the car, whatever you know, use, using computers or whatever that that’s not, yeah, that wasn’t the OEM that provided that to the you know, that wasn’t the tier two OEM that provided that to Hyundai, air, Toyota, or whoever it is, right? That you know, that’s not their problem, right? Because the consumers, you know, the brand is really the is the OE is the tier one, right? It’s going to be, you know, you know, Toyota can’t sit there go, oh well, that was a system that we bought, you know, and by XYZ, oh, yeah, they should have put security in it. Nobody’s going to care, right? They’re going to say, no Toyota, or you’d know on day, you’re, you’re the problem, right? And so that’s kind of, you know, that’s sort of what we’re seeing here with these large companies, you know, manufacturers, or across the board, right? Whether it’s energy, manufacturing, transportation, yeah, they, they recognize. They can’t. They can’t point the finger here that it’s, it’s their brand at stake. All right,

VAMOSI: So the data center part is very interesting to me, and we’re talking OT, so we’re talking about, like the HVAC system we’re talking about, you know, controls like that that keep the facility up and running. Is that correct?

MOORE: Yeah, that’s correct. I mean, it could be HVAC systems. It could be, you know, could, could be fire suppression systems, things like that, yeah, a lot of that is now automated, and you know, you know all the climate controls and things like that are, are certainly automated, and computer systems generally manage those systems.

VAMOSI: So where I’m going with this is, I find, I find these OT systems are overlooked a lot when we’re talking about, say, threats to the critical infrastructure, people still think of, you know, the grid, and only the grid. They don’t think about the other things that are going on behind the scenes. So staying with the data center, you could cripple the HVAC system, and, you know, destroy the data center fairly easily in a cyber attack, in some ways.

MOORE: Yeah, yeah, that’s true. And, you know, with with data centers, there’s, you know, really a lot of it, you know, there’s a lot of automation and understanding, because a lot of this is scaling, and there’s, they’re, they’re really building out and adding a lot of compute and storage and so forth, you know, to handle the increase in requirements around AI, right? You know, I saw, might have been a podcast, actually, where they talked about the difference between doing a chat GPT search versus a normal Google search, even though now Google is implementing AI as well. But initially it was, you know, you know, 20 times the amount of processing power to ask, you know, AI, a question, versus the search, you know, on something you know in a search engine. So obviously that is going to need, you’re going to need, eventually, at least 20x the processing capabilities that you ‘ve had in the past. So you know that that goes into energy requirements. So we’re seeing a lot of, you know, also additional backup storage, you know, across the board, right? More capacity in in sort of gas power generation, but a ton more in these like new battery technologies, battery backup, you know, for buildings and and certainly for data centers as well.

VAMOSI: So I know this is going to be hard to abstract or come up with a quantitative answer for, but I’m curious, how do you feel the critical infrastructure is positioned today against, say, nation state attack?

MOORE: Yeah, I think, sorry, yeah. I mean, I don’t, you know, I don’t want to paint a complete bleak picture here, because I am seeing, you know, folks starting to pay more attention to it. And you know, some of the compliance is getting teeth in it. You know, think of NERC sip and some of the new regulations for not just high what they call high impact systems. And we’ll just talk about energy for a moment. You know, that’s generally around power generation where, you know, high impact would be 100 megawatts or more of power Gen and, you know, so that’s, that’s, they’ve had pretty stringent requirements on, you know, separation of the network there, you know, obviously, you know, multi factor authentication, all of that, and and encryption and in, you know, really taking more of a zero trust approach to it for high impact. But the new regulations coming out next year, in April, also, I think 0039, include, you know, medium to low impact. So what’s interesting about that is that’s now wind farms, right? And some of these other dual use sometimes they’re a combination of solar, battery and wind, but, you know, wherever there’s, like a, you know, a generating facility in renewables, or peaker plants, you know, that may be on temporal and maybe smaller plants that may be less than 100 megawatts. Mean, these are now all following on, falling under what’s called medium and low impact. So, so I, you know, so that’s the good news. You know, at least in the energy sector, I think there’s a lot of fragmented policies across, you know, all of the different industrial sectors. You know, they all have kind of their own things, and they’re all, you know, at different phases of that. And, you know, I think they’re still in reaction mode to what’s going on in the industry in terms of, you know, the nation state, you know, apt threats from that, or just, you know, really garden variety ransomware, right? And some of the ransomware service groups and so forth. So, I mean, it’s there. They seem to be constantly playing catch up. But I think at least on the compliance side, it seems to be moving in the right direction, you know, kind of having, sort of all those fragmented into one sort of standard, you know, I don’t know if we’ll ever get there, but, you know, IEC 64 for three has been it’s been interesting. Just recently, Australia decided for all their industrial sectors to standardize on that, which I think is really good, you know, because they’re recognizing, hey, you know, we need, you know, one kind of source of compliance, truth, if you will. So, yeah, so that’s kind of the good news that you mean. The bad news is, you know, we’re still dealing with massively underfunded security for very legacy systems and it’s very, really hard to upgrade. And, you know, we start talking about utilities like water and wastewater, you know, energy pipelines. You know, all of these are running generally older control systems. You know, the HMIs and the SCADA systems. And the underlying Windows and Linux are generally not being patched very regularly. It could be very old operating systems that are not even being supported anymore, you know, like with Microsoft, for instance, with Windows, old Windows systems. So, those are some really, that’s very problematic. And you know, so the challenge there is, you know, how do we move to a place where we can be not just, you know, hey, we got visibility. We know what they are, that’s great, but can we start implementing more active defense, where we can make sure that who’s accessing is using the right level of MFA, strong authentication and very granular authorization, and then being able to audit that access and be able to have the right safety controls, human in the loop, and things like that, you know, to make sure that it’s the right person accessing these systems. And we’re not allowing, you know, people to not have, not have the validation right from folks outside accessing those critical systems, right, right?

VAMOSI: So, I know last summer there was a lot of discussion around some of the smaller water districts that were being hit. And in one case, it was the cyber Avengers out of Iran. It’s more like, are we going to see some of the smaller players impacted first with some of these nation state attacks, simply because, to your point, they have a legacy system. They may not have been updated. They may not have, you know, the vigilance to monitor their systems as well.

MOORE: Yeah, I think it’s, it’s generally, I mean, you know, it’s, it’s sort of low hanging fruit, right, right? A lot, a lot of these, you know, especially rural systems, you know, they’re exposing their networks, you know, to the internet. You know, because, mean, frankly, you may only have one person that’s responsible for the SCADA systems, the IT systems and the cyber security for some of these small water utilities, right? And they’re trying to figure out a way of reducing the amount of time, you know. And they’re like, oh, somebody comes in and says, Hey, you know, you can just check all of your your treatment plants, you know, for this cloud system. And hey, you know, everything will update. And if there’s a problem, it’ll tell you which, which plant you got to and whatever you need to do, as opposed to driving out to every site, blah, blah, blah, okay, that’s attractive. And they’re not going to ask, you know, they might be a skate engineer. They’re not going to ask, Well, what kind of cyber security? I mean, are you following IEC 64 they don’t. Probably. You don’t even know a lot of that stuff, right? And some of these, you know, these smaller county you know, municipal water utilities, for instance. So, I mean, I think, you know, those are, those are gonna be low hanging fruit, because what will happen is somebody puts in something that maybe helps them be more efficient, be able to have better visibility, but it’s also sharing their critical infrastructure, and those critical systems, you know, to the internet, right? And so somebody, and it doesn’t have to be the group in cyber Avenger group. It can be, you know, you know, a board freshman in high school that just likes, you know, playing around and hacking around on the internet, right? He’s got on showdown, and he’s seeing these systems, and he goes, Oh, let me see if I can, you know, maybe look at and look up the default password for this piece of equipment that’s showing up and goes right in, right? Does stuff so, you know, yeah, those are going to be low hanging fruit, right? All day long, you know. But some of the larger water utilities, you know, they’re, they’re also could be exposed here, because, again, you know, some of the aging infrastructure, and, you know, there’s always things that get misconfigured and things like that. So if there, if there’s folks that are more sophisticated and they want to get into maybe a larger municipality, you know, they can focus a lot of attention to that, right? And and try to find you know through social media and spear fit, you know, fish, different emails and things like that, to try to get you know, to get into the network and try to live off the land there. So, so I mean it, there’s different approaches, right? I think smaller utilities are just going to be easier, for sure, just because they’re under resourced and they’re not paying attention.

Unknown 16:44

VAMOSI: So I want to flip that around and speak to the motivations again of these attackers. Could it be that they went after, say, smaller utilities to flex and sort of say, like, look what we can do, but in reality, it’s like, well, those are kind of outliers, and you may not have this massive strength that you’re projecting. Doesn’t exist.

MOORE: It’s, it’s, it’s interesting. I mean, I guess I could speculate

VAMOSI: If you don’t feel comfortable doing that, that’s fine with me. That’s fine.

MOORE: Yeah, no, I, I kind of, I kind of feel like it’s it, you know, and not take diplomatic approaches, but it’s a little bit of a mixed bag. I think it could go either way, right? I mean, I think, yeah, there could be certain groups that are going out and just trying to, you know, create, you know, some some, some buzz around what they’re doing for whatever reason, you know, I can’t speculate to, you know, all the motivation behind that, you know, other than you know, to to make a point, but at the same time, you know they could be, you know, one thing that they could be doing is focusing on one area, but they, but They have kind of, you know, they’re running multiple tracks, right? They could be focused on a much more complex, you know, major metropolitan area utility, you know, and whether water energy at the same time. And they could be creating noise in these rural areas just to kind of throw that up right while they’re doing, you know, living off the land somewhere else, right? Distract away from others, maybe another industrial sector. I don’t know. You know, hey, let’s, let’s pay attention to water. We’re going to be over here and in pipelines or water, you know, or in, you know, in energy or something, and energy gem or something, you know? So I don’t know. I mean, it’s an interesting question

VAMOSI: that, yeah, that reminds me of when people would throw up a DDoS attack to cover some other data breach that they were doing at the time, or now, I guess it’s a ransomware attack to cover some other data breach that they’re doing in the back, yeah, yeah. Over in the telecom space, quite the opposite. Apparently, there’s some stealth stuff going on there where malware and whatever is being placed but not activated for whatever reason. Are you seeing examples of any of that in the telecom communications industry?

MOORE: Yeah, you know, it’s interesting. We haven’t really spent much time, you know, in the telecom area, we’ve had some discussions. But you know, really, for, you know, most of my focus has been in sort of the utilities, energy, you know, oil and gas and manufacturing and something like I mentioned before, transportation, maritime and so forth.

VAMOSI: no, that’s fine. That’s fine. So we talked earlier about NERC and FERC. Those are industry driven standards, and to some degree that makes sense, that they would know the energy business better than anyone else and be able to come up with those baseline standards. Are you seeing any other industries going down that path, or are they kind of expecting some sort of blanket ISO to come out that will cover a lot of different industries?

MOORE: Yeah, I see it. So the answer is yes. And, you know, and again, it’s somewhat fragmented, because there’s, there’s certain, you know, both geographically, like in Europe, there’s news two, that’s a standard. It’s similar in many ways to IEC, 6443 I think that one which kind of is, you know, initially was Isa 99 I believe, and then it kind of the IEC, 6443 took over. I mean, I think that’s probably the closest we have to an international standard for industrial control systems, you know, and security around that. But, you know, again, there’s going to be this sort of, either industry specific stuff, like NERC sip and which is more North America for energy, and then there are two, which is more of a European standard. And there’s, there’s ones in Australia and Saudi as well. You know, I’ve also seen things that are very specific in transportation around TSA and the security directive there that seems to be getting a lot of the pipeline people moving, you know, in getting, you know, shoring up their defenses and security. So, I mean, fortunately, some of these standards are pushing these companies to, you know, to adopt better security, right, better, better defense. You know better visibility, and you know a better understanding of their vulnerabilities. I mean, it’s unfortunate that it takes compliance, you know, with potential fines, to get them moving. But it is what it is, right? You would think it would just go, wow. You know, seeing a lot in the news, we need to fix this now in manufacturing, I see there seems to be a more pronounced focus, not just on the compliance side, but also because they are very, very worried about ransomware taking down manufacturing lines. So manufacturing, it seems to be kind of, you know, operationally driven, as much as it is compliance driven.

VAMOSI: Where are we then in terms of an outlook in the Summer of 2025. Are we still seeing legacy systems getting connected to the internet with telenet still enabled?

MOORE: You still see that, even today, it’s sort of, you know, as these things open up and more digital assets and more data, you know, more analysis, more analytics. There’s so much great operational data that can be extracted and so, I mean, there’s certainly business reasons for it, but they’re fragile, right? Because they’re just, you know, set it and forget it, right? You know, their systems that run have been running for decades. So, yeah, and they’re not upgrading overnight now they’re like, Oh, now we’re connected to the, you know, the internet or it. We got to upgrade all our systems. It’s not working like that. It’s, it’s a very, very slow process, right to kind of upgrade those systems and put in secure by design system.

-30-

This is a story about a Chief Hacking Officer who draws on his expertise in physical and virtual security assessments — along with some intuitive AI-driven coding — to safeguard Operational Technology. Colin Murphy of Frenos and Mitnick Security talks about how some of his early assessment work with Kevin Mitnick is helping him with OT security today.

Vamosi: In technology, job titles are pretty crazy these days. In fact there’s a silicon valley job title creator on Tumblr. Like Reddit Superintendent, or Information Pioneer, or Sales Qualification Sage. Silly. Random. That’s Silicon Valley, right?

There used to be a structure, a reason. For example the C suite was limited to the CEO, CFO, CPO, CMO and CSO. Then there’s Chief Information Security Officer, and CISOs are another group of acronyms.

So what would be a very cool C-Suite title? How about Chief Hacking Officer. That would be cool. You know, I’m John Smith, Chief Hacking Officer at the Acme Company.

Well, what do you do, John, and the Acme Company as Chief Hacking Officer? Well, I hack.

Okay, and what else do you do?

What else should I do? I just, know, hack.

Actually, the job would be a bit more than that, and I’m sure most people have heard of at least one real life Chief Hacking Officer. Kevin Mitnick used this as his job title at a few companies. And it fits. Kevin was gifted at the art of hacking. Running spreadsheets, probably not so much.

So, at RSAC this year, when I was offered the chance to talk to a Chief Hacking Officer for a new OT Security startup, I jumped at the chance. Hacking OT as your job title. How cool was that? And did it have anything to do with Kevin Mitnick. Turns out it did.

This is a story about a Chief Hacking Officer who draws on his expertise in physical and virtual security assessments — along with some intuitive AI-driven coding — to safeguard Operational Technology.

I’m Robert Vamosi

This is Error Code

[MUSIC]

Murphy: I am Colin Murphy, Chief hacking Officer for Frenos and Mitnick security.

Frenos is a digital twin AI enabled adversarial threat modeling tool, and we’re ingesting information across our networks and products, and doing what a red teamer would do at a magnitude, you know, 1,000x 1,000,000x really, there’s no limit. And Mitnick security is Kevin Mitnick’s consulting company, boutique. We’ve really selected on those customers, and after he passed, we’re continuing the legacy. I’m working closely with the same team, and his wife.

Vamosi: We’ll start with Frenos; I want to get to Mitnick Security in a bit. OT runs the gamut from things that were installed in the field 20 years ago and have very limited resources to a modern appliance today, which is probably running a commodity operating system and has a lot of robust memory. How do you how do you span that in your assessments?

Murphy: You know, we bring in segmentation data so we understand that basically, the topology across all of all of the systems we bring in asset information, vulnerability information, and so we are, we’re partnering with the tools that already exist. We’re not trying to reinvent the wheel across the entire industry, but we want to solve that single pane of glass and the threat analysis and model, right? So we take all of that in, and that’s, that’s what we call the digital twin.

VAMOSI: A digital twin is a virtual representation of a physical object, process, or system that can be digitized, mirroring its properties, condition, and behavior. It uses real-time data, simulation, and modeling to reflect the performance of its physical counterpart.

Murphy: We create using graph, right? A model of the segmentation, we put the assets, and then we have vulnerability and system information. Now our AI agent models, attack scenarios trained on adversarial groups, TTPs, everything.

Vamosi: So with OT segmentation, you’re doing security controls,

Murphy: We’re not doing security controls. You’re not, yeah, okay, so we’re, we’re, you know, the operations aspect of it is handled in other tools. We just want to look at the adversarial tap pass I see.

Vamosi: Okay, so you’re only focused on the threat, okay, exactly, and trying to identify where it’s coming

Murphy: from. Assume, yeah. So attribution, attribution, I’m a big proponent like I love assume, assume breach. When people started implementing assumed breach in their route, their tabletop exercises, their security plans. It became more realistic perimeter, it does a lot of the work, but the minute someone opens an email that they shouldn’t or has an unpatched device, they just brought the threat over them,

Vamosi: and the TTPs that you’re identifying, is that coming from, like lighter attack, is it a variety of sources? Is it proprietary?

Murphy: It’s all of the above all. So we will get, we have partnerships, and we get information where it’s easily available, but then we also do our own research, and we go find things that are out there in the dark web or other communities sensor information, and we’ll build our own proprietary, adversarial fingerprints. So

Vamosi: are there particular industries that are focused on this. Now, in other words, are you finding, you know, a particular sweet spot, energy, energy, and is that, do you think the result of a particular uptick in attacks on energy, or is it just they’re more advanced because they’re more advanced?

Murphy: I think I would have answered this differently 10 years ago. 10 years ago, I was like, the energy’s way, like they’re so legacy. I think that they are responding to new threats. They are moving towards the bleeding edge of technology where they can and trying to figure out ways that they can protect and identify those 20 year old systems or fortinets from three years ago that never

Vamosi: got right? So you just introduced the name of a router. There’s a couple of them out there that have been identified. CISA has called it out in particular as having issues. Are these unexpected things, or is this like because we haven’t really looked at this industry. Now it’s bubbling up.

Murphy: I think it didn’t have the attention it deserved. So now I think there’s more focus on it, where it’s bubbling up. The threats and the vulnerabilities have always existed, sure,

Vamosi: Yeah, not like the vulnerabilities were created yesterday. Yeah. No, no. But the expectation was different. The threat model was different. When those devices were implemented, would begin, yeah,

Murphy: you know, they had a I feel like the industry had a false sense of security because it wasn’t well understood. So kind of like the Apple Microsoft argument, right? It used to be, everyone thought Microsoft had all the vulnerabilities just because there was more of them. So it space. We thought in this industry, IT space is where those attacks and the focus was going to be. But OT is massive, right? So,

Vamosi: yeah, security by exterior, yeah, does not work. We know that. So you did some research on routers, on firewalls, yep. So tell me a little bit about that research and what you found,

Murphy: yeah. So in mid January, 15,000 Fortinet firewall configurations were released on the dark web. They got posted within days the credentials from them. Yeah, your VPN users in those configurations, almost all of the credentials were printed, so they’re hashed in the in the policy or in the configuration. So not only did they post the configurations that then started posting the credentials for these firewalls, VPN access, etc, we got, we got a hold of those configurations. I started digging into them and going, what one was the attribution? Where do these belong? What business was impacted by this, what industry was impacted by this? And so I started looking specifically for OT. And, you know, surprisingly, there was a lot of power generation, DCS, mod bus protocol built into these policies. And then I started seeing things like power and generation, and it went deeper and deeper and deeper. We normally wouldn’t care about that. That’s not something that they’re going to expose to the internet. But if I have configuration data, VPN information, you cracked credentials, and now, reading the policy, I know what’s behind the perimeter.

VAMOSI: What Colin is saying is that by looking at the dumped credentials, he can find the industries that are affected by the breach. And in this case they were power and energy related.

Murphy: So now, now you kind of build this attack chain and go, I know who my targets would be. So that’s where the research has taken me.

Vamosi: So if you’ve identified the victims, what about the vendors who made these leaky devices? How are the vendors receiving this information? Has Colin been able to work with them? Or is it still kind of difficult because they’re not used to it,

Murphy: Disclosure is still tough. I feel like that is a problem that should have been solved by now for us, and it has. As a researcher, I have a problem every year. I’ll give disclosure, put it out there. No one doesn’t favor it. So I think that they’re doing a better job. Like Fortinet has tried to contact customers they could identify and say, Hey, your firewall is exposed. It’s been back door. You were in a breach. But that’s only if the customers are keeping in touch with or providing information that identifies the firewall configuration of activity. A lot of these devices are outdated or unpatched. The licenses are probably three years old, five years old, so out of service as well,

Vamosi: And the end user probably doesn’t want to engage with the vendor at that point.

Murphy: Yeah?

Vamosi: For some it’s gonna cost me money to call them and say, Hey, am I one of these people impacted? It’ll be like, sure. For a service agreement, we’ll get you one pass. It’s a cross sale.

Murphy: It’s a cross up sale, upsell.

Vamosi: For some reason at this year’s RSAC, people pulled me aside and wanted to know more about Kevin Mitnick. I spent about three years working with Kevin on the Art of Invisibility, which is a timely book about how to disappear online. Maybe people are interested in that now — disappearing online. Not calling attention to yourself. I get asked about that a lot.

It’s also coming up on three years since he passed. I suspect that’s an appropriate amount of time to reflect upon his work and his enduring legacy. If you didn’t know, Kevin had his own company for years, doing boutique security assessments. Many of the stories in Art of Invisibility were inspired by real life events, and while his work with Mitnick Security was under NDA with those clients, Kevin provided vague outlines which I was able to embellish and then provide color for the book. So the work that you’re doing for Frenos, is it different from what you’re doing with Mitnick, or was it

Murphy: It’s very different. It is. So when I joined, I joined free, knows, four weeks ago, that’s the irony before that I was with though before, what’s nice with working with the Mitnick security team and another business is there’s always things I can apply, both, like there’s nice overlay, and so the research and Red Team engagements I do at MIT security translates usually into something in a product that I’m helping develop. So at frinos, is Chief hacking officer. I straddle engineering, product and operations, and I get to kind of play in all those areas, and tilt the hat to Kevin Mitnick, since that was his title at KnowBe4 and Mitnick security. So it’s kind of a he was a important part of my life, and you know what I’ve done in this industry, and so I want to show that respect. I

Vamosi: I think a lot of people don’t know about Mitnick security. They know about know before, but Mitnick security has been around for a long time. Maybe just tell me a little bit more about, like, what its philosophy is and maybe why it hasn’t been so well

Murphy: Yeah, absolutely so bit Nick security was Kevin’s idea of how to legally hack and get paid and contribute back to the community. That was his original idea. And so he would find interesting groups and companies to work with and engage in in technology and industry. And because he was selected, we kept a very kind of boutique operation. And his, I mean, he would work 20 hours a day, seven days a week, he had, he had no need or reason to, but that was his passion. And so we always made sure that we delivered a a robust, you know, engagement we covered. The scope was one thing, and then we’d go back to the customer and say, Hey, we’re done with that. We got in. We have the we, you know, we have domain, but you have these other systems or these acquisitions, and they’re like, yeah, absolutely, let’s do it. And so we always ended up giving a report back. That was, you know, so, so, so much larger than what originally we were engaged with, and they came back every year, and we probably got the most satisfaction out of getting harder and harder, right? You do a red team engagement, you try and you try again in two years, and they’re like, Oh, I haven’t had time. I had no budget. We would keep in touch. They joined the slack, and sometimes, way before the next engagement, they’d say, Hey, I have this idea that zero trust. What do you think about this? And so that was kind of why we stayed boutique, because that doesn’t scale well, right? And so we, you know, we made sure we had good customers and all the different industries we liked. So oil and gas, manufacturing, automotive, healthcare, SaaS companies who bring us in to test their web applications, their networks.

VAMOSI: So this is where our conversation gets really interesting. Colin has boots on the ground experience hacking large retail chains. He cites an example.

Murphy: I have some really fun stories on social engineering engagements where the scope was everything, but it started with me pushing a stroller through a mall to a retail store with my kids and installing a device in a public area so that my team around the world could tunnel through that device, do a wireless attack against the storefront, travel from there flat, non segmented, flat network. They fixed that beautifully after the fact, but we went from that storefront, and here in the US, we were able to go anywhere else in the world, to hops. So we were in their London storefront. We were in their AWS. We were in their enterprise. We had their ERP CRM systems. I mean, it started with my kids in a stroller, and we and they’re up. They’re a very large organization. They got that report back, and they made significant changes. We tested them two other times, and it kept getting harder and harder. That’s that’s what we want,

Vamosi: I hear these stories, and I think that’s just Mr Robot ish — that doesn’t happen in the real world, but you can attest that as a pen tester, you did these things in the real world. Did the social engineering. Those hot boxes were put wherever it kind of sounded like Target. But anyway, flat network, yeah, that’s everywhere. Yeah, I wouldn’t attribute it to that, but so your job, your role, wasn’t just behind the keyboard.

Murphy: It was, it was a bit of everything, jack of all trades, yeah.

Vamosi: And so would you, would you recommend that to somebody like, is this a cool job, or is it, like the best, a burden to get up from your keyboard and actually go into a shopping mall and,

Murphy: no, it’s amazing. I think that if you’re really passionate about reverse engineering cyber security, like over the top, passionate, I think it’s the perfect job. If you want to write exploits and reverse engineer web applications, then you have your niche, right? You’re gonna you’re gonna stay in that, and that’s gonna be your career, and you’re gonna contribute and do talks on it. But this broader approach, I mean, it also the social engineering in going in person uncovers threats that get missed a lot of the time with red team engagements,

Vamosi: right? Because you’re in the real world.

Murphy: You’re in the real world. You won’t see, you know, these are, these are, in some cases, there’s hundreds of 1000s of assets in these networks. You what you see with code and scanning is completely different. If you walk up and you see, you know, a hit card reader or Ethernet port on the wall, you go, Wait, what is that? Where does that go? Right? What’s on the other side of that? In my background as a CIO, I know that it probably goes somewhere that someone forgot to unhook it from, and it’s on the wrong VLAN, and there’s no knack on it. And so now you know an attacker could plug into then suddenly they’re taking you into the CEO’s office virtually.,

Vamosi: Oftentimes they forget about things. Oh, do we have security cameras? I forgot about that or whatever? Yeah, you see that on prem as part of your reconnaissance, and then you bring that into the pen test, that’s right. And then you go back to the client and say, We found this, this, this. And they’re like, did we have these things that does happen in the real world.

Murphy: Oh yeah. A great example is that this was a little bit different, because they came back and said, Oh, wow. Thank you for telling us also that it’s no longer in scope. This was going to make the test too easy to jump from that into where we wanted to go. But it was a digital reporter that was open to the public. Public win default credentials, and you could do a code injection on it and get a reversion. So now you’re already on the network. You have a place to move, you know, horizontally from.

VAMOSI: And then there’s good old social engineering, which is definitely an in-person assessment.

Murphy: We had an engagement where I wasn’t going in to test the technology. I went in to an employee that had just been terminated a week prior, and I showed up with a fake badge and a lanyard, no polo, and I said, Hey, I’m really sorry. What happened? I’m here. I’m a courier. I’m going to take your assets back to the company. She gave me two sheets of paper with every credential she could remember from the business and the laptop. And I had her sign a fake form. I left with it, and the customer came back. They got the, you know, they got the end engagement report. They were like, well, that’s a different problem for us to solve, yeah. How did you know for LinkedIn? And she happened to be local, so it was easy for me to just go there and do that,

Vamosi: I wanted to know a little bit more about Mitnick Security. I mean, I worked with Kevin for three years on his book, but I didn’t necessarily know that side of it.

Murphy: the thing that, yeah, I mean, something I could expand on that I hear in the community a lot. If you don’t know Kevin, and you look from 10,000 feet, you go, he hasn’t really, he hasn’t done much, right? Because he’s not really hacking anymore, which we don’t see. And I didn’t know that or see it until I worked directly with him. You couldn’t be further from the truth. He didn’t see it, because he was always working. He was always hacking. He was always in those systems. And anyone that’s worked with Kevin over the last eight years, we all have the same joke. Are you there? Because if you got that message on your phone and you responded, you were going to go either into an hour conversation or a 24 hour conversation every single time.

Vamosi: he would call me from a taxicab and be like, Hey, I had a great idea. We should add this in the book. Are you writing it down? I’m like, yeah, no, I’m recording it. God, yeah. And then we get this awesome story, which, in fact, made it into the book. So yeah, I heard it when my agent approached me and said, Can you finish this? Because his previous ghost writer threw up his hands and said, I can’t write this book. I said, all right. And then I turned around and I asked people in the community and like, am I going to regret working with Kevin Bentley, yeah, nobody told me to walk away. Everybody was like, this is the best thing in your life. Go for it. And I love that. I must say, he irritated the hell out of me at times. But, you know, in retrospect, it’s like I had an opportunity to work with somebody like Kevin.

Murphy: I give credit to this industry, to people caring about security, to what he did. I mean, it was early days, but it created what we have today. And I remember as a young kid going down a similar path. Was cabin here in the Bay Area where I grew up. And when he went away, when they caught him, I went, crap. How do I keep doing this? But for good, there is no path to this. At the time, you were a computer science major, a Cisco engineer, your business degree, right? Like, or you just wrote them. And here we are a massive industry with so much opportunity. Yeah, and it wasn’t

Vamosi: just, it wasn’t just Kevin. There was Kevin Poulson. There was Laurie love. There were a bunch of people who were similarly entrapped by the system of the day that didn’t understand personal hacking from the type of criminal hacking we see today. Anything else that you’ve experienced with the OT world as Chief Hacking Officier?

Murphy: I’m very passionate about this, so when I’m not working, I’m working, which is nice. It’s all played for me, okay, no matter what I’m doing. So the Fortinet research kind of led me down the path of going, what are what’s out there, what protocols are used to manage all these different types of devices in these networks. So not looking at the hardware, but Erlang. OTP, yep. Okay. Another CBE, I turned five under and under five minutes. I weaponized that into reverse shell. Those types of things are now what I’m looking into I have a I haven’t disclosed it yet. I’m still doing the research on it, but I have a zero day for drop, air, SSH, 19 million devices exposed in Shodan. It is in modems in people’s homes. It is in industrial control systems. It’s everywhere using quad and and kind of knowing where to look in these libraries. I had a working exploit in under two hours. Wow, I found the bug and had a working exploit in under two hours. And it’s a memory corruption, right? So I’ve been in four weeks. This has all been in four weeks, and it’s, it’s amazing, like I I’m excited to be part of the OT community, and I’m learning from so many great people already, it’s different, which is, it’s challenging for me, which I think really drives me to find ways to contribute and build really good products. And that’s, that’s really what

Vamosi: it’s now. Let me see if I understood correctly. So you using vibe coding to do some reverse engineering? I am and that’s going, well, I guess you’re prompting it, you know, in a good way, and that you’re not seeing weird stuff come up, not

Murphy: yet. I’m not using deep seek. So that’s probably, that’s probably helping

Vamosi: AI, models will vary, yes, but so that’s going to speed the process I was thinking about, not that you know a lot about Capture the Flag world, but I was thinking how this is really going to change that. Because when they’re doing king of the hill and reacting in the moment, they can just be throwing it off to their AI model and coming back with

Murphy: so I do know about that. Yesterday I was at Hack the day. Okay? And while I was watching some great speakers, I went, I never get to do CTFs. I’m gonna do CTF. I was able to do one thing. I don’t. I’m not a developer. I can hack code, I’ll reverse engineer, I’ll write attack tools. But sitting down and writing a robust application is not my skill set. So I went through each challenge and would get the binaries and really set up the prompt to do what I would normally do, but it would take me hours and tool sets and, you know, a path, a sequence automation. Instead, I told the AI agent what to do and said, Go, Go, run on this. This is what I expect to happen. Or, Hey, I see these combinations of files. I think it’s, you know, it’s a block cipher, and it would come back in what would take me hours. I Of all of all of the challenges I think I got to then I had to leave. I got through about 90% and so and some were very complex. Now you have to have the background to prompt it. You can’t just paste it in there for the harder ones. But this is the, this is probably the most impactful thing, because as I left there, I went, what does this mean for the industry? What does this mean for for the past 20 years, I’ve been saying, I think everyone’s been saying, we’re lucky. There’s not enough bad guys, because the world is vulnerable. Systems are vulnerable. We’ll always have these types of things. There’s just not enough bad people out there exporting them where we’d be critical. Infrastructure would just fail. Now, with AI. It just took 10 bad people with bad intentions and made them as strong as 100,000 right? And as every day, these advances make it easier for a very small group to become as powerful as a nation state. That’s what we’re up against.

Vamosi: Okay? You’re talking exponential growth here. Yeah, so the window is not like 10 years. The window is like two years. It’s sooner than that. It’s approaching as those skill sets are developed. And

Murphy: It’s our responsibility as citizens, as a community, as cyber workers, to use AI now the blue team, now the defenders and the engineers have to keep pace. We have to keep that momentum. So what comes with that for us is we don’t just get to do whatever we want and break systems. We have to have proper GRC and compliance, we have to trust and understand the model we’re using. Whereas the bad guys don’t have to do that. They don’t have those controls in place. Sure, we have a tough job.

Vamosi: I’d like to thank Colin for coming on the show and talking about his work at Frenos and also at Mitnick Security. So being a Chief Hacking Officer is not all fun and games, but it’s also not sitting behind a keyboard and staring at pivotal tables and budgets. It’s a bit of boots on the ground, and a lot of curiosity. A lot of poking around and finding out what works and what doesn’t. I’m reminded of that song by Cole Porter and George Gershwin, it’s nice work … if you can get it.

ROI is always a tricky subject in cybersecurity. If you’re paying millions of dollars in securing your OT networks, you’d want to be able to show that it was worth it. Andrew Hural of UnderDefense talks about the need for continuous vigilance, risk management, and proactive defense, acknowledging both the human and technological elements in cybersecurity and how just because something didn’t happen doesn’t mean that it didn’t..

VAMOSI: ROI is always a tricky subject in cybersecurity. If you’re paying millions of dollars in securing your networks, you’d want to be able to show that it was worth it. Just because something didn’t happen doesn’t mean it didn’t. Well, try explaining that to the bean counter in the CFO’s office.

At RSAC in 2025, there are growing concerns surrounding operational technology (OT) security within organizations. To date there haven’t been large scale attacks on OT — and, strangely, that’s kind of bad. Why? Well, it highlights how many companies remain complacent due to a lack of OT attacks. These organizations are failing to recognize that OT systems are just as vulnerable as traditional IT infrastructure.

As a result there’s a lack of motivation or budget to proactively address these risks, with only compliance regulations like those in the EU helping driving action. Which gets us back to the difficulty in quantifying return on investment for cybersecurity measures. Perhaps the best analogy is to insurance — valuable precisely because it prevents visible failure. If only it were that simple.

Supply chain vulnerabilities are now a critical threat vector, with attackers increasingly targeting third-party products and services to infiltrate secure environments. AI used in cyberattacks enable sophisticated phishing campaigns. Looking forward, there’s concern over dormant threats embedded in OT systems, lying inactive for years until triggered. This long-game strategy, while plausible, may be disrupted by rapid technological advancements that render such threats obsolete. Want any FORTRAN exploits? Lastly, there’s the human element. A disgruntled employee can still sabotage your OT environment from the inside.

This is a story about … the need for continuous vigilance, risk management, and proactive defense, acknowledging both the human and technological elements in cybersecurity.

I’m Robert Vamosi

This is Error Code

[MUSIC]

HuraI: I am Andrew HuraI am managed detection and response director with UnderDefense cyber security company. So UnderDefense is a cyber security company. We do focus on managing detection and response, so that’s our primary focus. We do have penetration testing services. We do have incident response services. So this helps us to stay on the verge of these modern threats that emerge, because having the expertise in blue teaming and red teaming help us to stay on top of those emerging threats at the moment,

VAMOSI: Last year at RSAC I spoke with Andrew in episode 40 about Hacking IoT Surveillance Cameras For Espionage Operations. This year our discussion was more general around the difficulties in doing security assessments on OT systems — both the technical and the human.

HuraI: I would also like to add that we are keen to do our business personally. Often in security companies, forget about this personal attitude and personal approach, and that’s what we tend to do. So it’s not one size fits all. We will do our best to fit the needs of the company that’s in front of us, and it’s a partner, it’s not a vendor type of relationship that we do. So

VAMOSI: What’s the idea behind this service that you offer?

HuraI: So we want to know the customer. We want to know their pain points. We do want to understand their infrastructure, that’s no it’s essential. It’s like going to a doctor, and if you have your personal doctor, he knows everything about you and that that’s the chance for him to step in and help you right when you need it. You do not go over the screening. You do not do additional tests. That’s what we aim to do. So coming over, doing the assessment, talking to the customer, to their teams, making sure we understand what they do, making sure we don’t have blind spots. So I would, you know, I would hate to say that, Hey, John, I’m doing security for you, but I’m blind. I want to have telemetry. I want to have data. And that’s when we come to a stage where some customers would want to refuse showing you everything that they have, giving you all the data, all the information. There could be various scenarios in this case. I mean, you know, there could be a compliance where you just cannot share data outside. There could be an option of them protecting their often, I know personal information, maybe keeping their secrets, trade secrets within their company. But sometimes, and I would say it’s not that sometimes, in our case, talking to vendors and companies around RSA or here, I see that in general, there is a good chance that CISOs cybersecurity engineers responsible for security within the company, they just don’t want to admit the risk. So it’s easier to stay low, it’s easier to hide than to actually face the threat that is in front of them. And I do understand that sometimes it works out well. I mean, it’s like an insurance on only car that you buy and you can get it for 10 years in a row. Nothing happens. But you know that it might be a chance of one day that you didn’t get it, you might be in trouble. So that’s what we aim for. We want, as I mentioned, to study a company, to make sure we understand them, to make sure that we can offer something when the big one strikes.

VAMOSI: I want to be clear here. Are you talking about both the IT world and OT world, or just the OT world?

HuraI: the moment, I was like, generally speaking, about the security principles that we would want to leverage within the companies and but as we go in more particular details, as you mentioned, it’s easier to do it at the moment, at least, it’s easier, in a way, to show the value to the customer. It’s easier when the customer takes this report, samples whatever you want, whatever we call it, and brings it to the C level, to the board, to show the differences. But when it comes to OT, it’s quite a different story, right? You cannot just say that you know we are secured or not. We can do testing. That’s something that helps. We can do drills, attacks, scenarios. We can do something like bringing the company down for a short period of time, just to show them the risks. But no one actually wants to go this hard way, it’s just easier to, as I mentioned, to stay off and continue doing the business as they were doing, because my father did this way, because my grandfather did this way, everything worked out. Well, why should I be right?

VAMOSI: Is it because so the boards and the C level, they see the news stories and because there have been network breaches, because there have been network attacks, it has gotten into their vernacular that they can talk about ransomware, they can talk about things on a high level, mind you. But nonetheless, the OT world, we have very few examples that are publicized to your point.

HuraI: You’re absolutely right. But let’s, let’s go back to the animal world.

VAMOSI: Okay, you’re going to want to hear Andrew’s stories. He’s very good at creating security metaphors … and ones that stick.

HuraI: So there is a typical story of a lion hunting on the spray. Where does he go? They usually go to the place where they have water. That’s where they drink. That’s what happens in a cyber world as well. So water in the hole, that’s what you want to do. The attacks when you try to penetrate the company at their very first or even zero stage of their engagement, that’s something that you do, and we’ve noticed a lot of cases, a lot of examples, but malicious actors, especially from I’ve been studying Chinese and Russian apt groups, especially those countries, they do want to get into your environment at a very low level. And so why not try to penetrate a company? I mean, when you have an access, when you have a chance to go in, then you decide if you do or not. When you have a lot of keys, you can decide which room you want to open or maybe not. It’s, you know, it’s always a chance that they might not open it. We had an instant response case, by the way, just just adding here, it’s very small to infrastructure. But what happened there was that actually a company wanted to stay low, as you mentioned, they didn’t want to share any information outside, but they had to, because malicious actors, they publish information on their website. So it happens that they actually were using water in the hall. Vulnerabilities, cyber, cyber attack types. They just had a malicious link on the website where most users of this organization visit. So let’s say a typical news website, or maybe something that is more familiar to their departments that they work with. They had these malicious links, so users followed the link, eventually their credentials were compromised, and now, you know, it’s so much easier to hack people than hack tools. Getting within the infrastructure with legitimate access was just a piece of cake, and that’s what we emphasize to all of our customers, you do need to have not only MFA, but also security monitoring on top of this. MFA, having thresholds, having good security walls, is just not enough. Rome fell down one time ago, but it has, like, beautiful walls. It was protecting the city for a long time, but, you know, eventually it happened, right?

VAMOSI: What I’m seeing in the OT world is the attacks still have that IT component to it. They’re going to go after a business related software package or vulnerability in that, and then piggyback their way into an OT environment. Is that generally the case? They’re not going directly to the OT or are they?

HuraI: I would assume that we do not need to outline and say that there is no chance they are doing any attacks. I think they do. I do believe that it’s really easy for malicious actors actually to get into this software, because it’s, you know, as we mentioned before, vulnerabilities issue, light chain issue, outdated software issue that you follow right but now, why don’t you think that having a plant somewhere outside of a city, outside of a country, outside of a region that you work for that you don’t have an ingested malware at this stage, at the very beginning, and maybe malware is just too complicated word for this misconfiguration would be better to use this word. So having a misconfiguration that can give you a key to this environment in future is something that you want to follow. I would say that you cannot influence regulations, or the GRC of other countries, right? That means that you cannot actually do 100% audit of these devices. What you can do, of course, after you receive the device, you can test it, you can make sure that you understand how it works. But it’s too complex. Like, do companies have time? Do they have money and budget? Do they have enough personnel to do this? And should they, I mean, imagine yourself buying again. I’m doing a lot of analogies today, but imagine yourself buying a bread somewhere in the market, going back home and testing if it’s really a bread or not, you should probably just get it, get your sandwich with ham or something else, depending on what you love, and that’s how it goes, right? So I would say that in the modern world, as you noticed, it’s easier for malicious hackers to hack it. It’s easy for them to use this. It ot way other than background, but I would not eliminate the one. I mean absolutely, especially in the critical infrastructure. We’ve seen the example of Prussian backed, Chinese backed, malicious acts attacking infrastructure in us, right? The water facilities, the electricity grid, the one happened in Ukraine, long ago, not long ago, indeed, the recent beach in Spain.

VAMOSI: On April 28, 2025, large parts of Spain and Portugal were hit with a large power outage. Here’s the BBC:

BBC: but what’s been happening in Spain and Portugal well electricity has mostly been restored a day after a massive power cut it caused disruption to millions of shops homes and businesses it paralyzed transport networks in both countries the operator of Spain’s national grid says power has been restored to 90% of the mainland but the effects of the blackout are likely to be felt for days the Spanish power company Re has ruled out a cyber attack as the cause of the mass blackouts

VAMOSI: At the time of this podcast, the exact cause of the Iberian power outage remains unknown, although early speculation has suggested that a “rare atmospheric condition” might be a factor.

HuraI: The first thing that I was thinking about was a cyber attack. Of course, that’s because, you know, of the sphere that I worked for. But anyways, that was the most obvious one, other than hearing so forth about national disaster. Come on, nothing happened like this for 100 years before it just happened exactly in this time span. No, I don’t believe that maybe I’m too, you know, superstitious. Maybe it’s me being too into details, but I think that someone is hiding something from us in this particular case? Well,

VAMOSI: the details will come out at some point in time, but I also step back from events like this and hope that the government agencies are not saying much because official misinformation can quickly spread. For example, there was a blackout in Brazil years ago and rumor quickly spread that it had been a cyber attack. When all was said and done, however, the finaal report released months later identified the cause as soot — common ash — that had shorted out a transformer. Okay, not as sexy as a cyberattack. Not even as good as a cybersquirrel. That said, more often I hear about scenarios where a bad actor comes in through, say, enterprise software — think Colonial Pipeline — but I have yet to see a full on direct to OT level attack. I do know that a lot of OT now is interconnecting with the cloud. I wondered if this is what Andrew is seeing as well, if the Cloud is becoming another vector for OT attacks.

HuraI: I would say that it’s so much similar to network one, you know, it’s a matter of you having your resources in a separate way and doing the proper configuration and, well, maybe misconfiguration of this network. I’ve been to a couple of meetings with the cloud providers, and they are saying almost the same thing. We are doing cloud security, and you guys need to do security in the cloud. That means that the company needs to be responsible for actual, proper fine tuning of their infrastructure that they use within this cloud solution. Now it’s, you know, again, it’s just a part of this scenario, because you can still hack or find vulnerabilities in the vendor’s cloud of the vendor. The other scenario that you mentioned is not only it or cloud or network, why don’t we eliminate the physical one? Again, we’ve talked about the human aspect here for a long time. Why don’t I just find someone who is not satisfied with his position, offer him something valuable. And it’s not always a budget or money that you need to give a person. Maybe it’s good motivation, or maybe something else, or maybe he’s just angry at his boss. So be kind to your employees. I guess now you have a person that is an insider, that is helping someone else that is able to the easiest one to just put some USB stick in the server or in the device, and that’s it. I guess all these firewalls roam walls and everything that you mentioned, you just kind of pass by at lightning speed, right? But again, we are here to make sure that incidents happen. They happen all the time. We go to the doctor’s office because we are human and we have some vulnerabilities. Now, how do we respond? That’s a question, and I would say that each and every company, each and every CISO, anyone responsible for security within the organization, needs to think about, what do I do in case something happens? Am I prepared? Do you have a plan? Did I have this plan tested? Do I do tabletop exercises? Sometimes it’s even a matter of you testing the plan. When you have a switch in your teams, I mean, you have an employee that just left, and he was, or she was, responsible for the essential part of this response. Now that they left and you want to be onboarded, but they are not even aware of this. No one actually gave them the idea that they should be doing this. So they are in the list. They should be doing it, but they are not what happens within the security incident. That’s, you know, when the bottleneck happens, that’s, that’s when you have the real problem. I would assume that we need to focus on this, and we need to pay attention to the response exercises as we go because things have happened.

Unknown 15:36

VAMOSI: So you mentioned, people do things because things have always been this way. That’s a conversation I’ve been having a lot lately with practitioners. Is it a big leap to go into a company and to point out that just because your ot has never been attacked, that it is just as vulnerable as the servers, as the cloud, as the other things that they’re protecting, or are the organizations that you talk to, at least aware and open to it, and want to have that ot world exposed as well.

Unknown 16:11

HuraI: It is a challenge. It’s like someone’s doing exercises. They go into the gym, others, not other person is not and you ask, why are you going to the gym? Like, you know, it’s something you need to do. And you’re like, Well, I have some other things I don’t have time for. I have some other tasks to do. I’m busy with my work, blah, blah, blah. I think this is actually the same that happens within the infrastructure or within the organization, within the departments. There is a need to do something, but there is not always a will to do this. And this will, it can be, not only based on the budget, it can be motivation, internal or external. And I’ve seen that something like GRC or compliance helps to build this will a lot. I mean, take an example of the European Union when they had this Data Protection Agreement implemented, it was a mess at the beginning. It is a mess at the moment, in a lot of cases. But as for me, it’s an essential step that we need to take. And I think that you know, doing all these regulations might be helpful. On the other hand, you do know that it slows down the business process. Slowing down the business process means that the company is moving slower. They are making less money. They’re not developing as they should. So we are here on the verge of pros and cons of both solutions and as for me, critical infrastructure is something that is, you know, related to the national interest. This is something that we need and the government needs to put their attention into now, if it’s something less important, then I would assume that we should try to give a willingness to the CISO, to take this chance and take this business risk. It’s always about managing risk, right?

VAMOSI: It is, and you said earlier, and I agree the ROI is hard to see. It’s like a 10 year insurance plan. And just because nothing happens over 10 years doesn’t mean that nothing happened, but that ROI is hard to demonstrate. I would think that the ROI on ot security would be even harder if companies aren’t embracing the IT world, it’s more abstract and harder. I would think,

HuraI: absolutely, what do we measure as an ROI? Usually it’s, you know, the investment that you spend on something. And now you can say that there was a time span of 10 years. Let’s use 10 years that you had some outcome, nothing happened. Well, eventually we are happy that nothing happened. And this is our ROI. This is something that gives us the strength to say we did not just spend this money. We invested this money, right? Because if we did not do this, there is a high chance that the company wouldn’t survive. It stopped. The whole department is not working. Or even less, you know, or even more, depending on, again, on the company size, right?

VAMOSI: the other trend that I’m hearing in the conversations that I’m having here at RSAC is the attacks on the supply chain, the devices that are manufactured elsewhere and brought into critical infrastructure that may escape,

HuraI: Okay, I’m bringing up the sticker from my pocket at the moment and says, supply chain innovators, absolutely. This is the, I would say it’s the easiest. Yeah, easiest. Things. It’s a pretty easy way for a malicious address to get within the organization. Sometimes you feel that you pay a lot to your personal security as well as build a new body in the gym. But then someone else comes in, and then someone else happens to be working with your company that is not as secure as aligned with what they do. But you do need those services. You kind of use those tools to this machinery, use this shadow IT, for example. That means that you know you kind of expose yourself. We don’t have a chance to go into it again. I’m just throwing back the jersey aspect in my head. Sure we cannot just go back and say, Hey, you’re not ISOs up to whatever compliance, just because we do not want to stop, stop the businesses from the development cycle. I don’t know. I’m pretty sure that supply chain attacks are something that is to be leveraged as we go, especially with modern technology, with L and M and AI usage. We’ve seen on the penetration test example, that I think 150 or 200% rise in the use of AI tools, and as a consequence, as a result, fishing well compromised business accounts, trying to get within the systems. Now you can do the same with the supply chain. What I think the example of one of the government us, government infrastructure that we had in breaches recently, when they had Microsoft Account compromised and they managed to get within the accounts of the government organization. Oh, come on, that’s definitely not something we want to have. No and I love Microsoft. It’s not about, you know, blaming someone. It’s just about being cautious. It’s about making sure that we stay aware.

VAMOSI: with the AI attacks that you’ve seen so far, are they the easy ones, such as crafting well phrased phishing attacks, or is it more subversive where they’re going after like poisoning the data lake,

HuraI: we usually see something easy and fast. So this is probably the example that you mentioned first grandson, wire fishing, something that you can use and leverage. Have an ROI first. That’s what they have as well. That’s a business structure that operates within the dark net, within the dark world. Let’s call it this way. I’m more than sure that we will hear about the sophisticated AI attacks in future, it’s just because it’s a matter of building the trends, finding out the possibilities. I’ve heard about the language model that is writing a new ransomware on its own. So typically, why not now, we should think about having another AI, another element that will be defending companies. So maybe in future, we’ll go to the stage when there is an AI that works as the entry point to the organization, so it checks all the vendors, all the employees, everyone. What do you do next? It’s malicious actors. You hack this AI. That’s it. As for me, it’s important to use the new technology. We do need to focus on it. It brings a lot of attention because it’s a powerful tool. You know, with great power comes great responsibility. That’s one of the famous phrase that one of my heroes used when I was a child.

VAMOSI: Oh, Spider-man’s still my hero.

HuraI: So I do believe that we just need to be cautious with

VAMOSI: sure and going back to supply chains, this concept of lying dormant for an indeterminate period of time, it seems like within the OT world that this is something that is not only done but advantageous to the bad actors out there, that they can plant the seeds and maybe harvest them many, many years later. Any monitoring, anything that you’ve seen that suggests that that might be a strategy.

HuraI: I do know about the cases when malicious actors were present within the infrastructure, it and OT for three consecutive years, and they just, it was an example of, actually the Russian Black apt group, and they kind of just exited the organization because of the inside management problems within their organization. Someone just wanted to put a new star to their forehead to get promoted, so they decided to do some low level thing and kind of show themselves that they are within this infrastructure. But what if not? What if they decide to go further and longer? I do believe that, you know, when you stay low for a number of years, you get a chance to be unnoticed. Let’s look at the practical example we had the, you know, the external systems like the spice.

VAMOSI: So let me see if I understood correctly that first example. So someone went through the effort of seeding OT devices with something and went on for about three years, and it was only because the internal organization collapsed that it didn’t get activated. Nothing ever became of it.

HuraI: So the first example was that for three years they’ve been present within this infrastructure, in this company, and I’m more than positive they would probably continue doing this. But this apt group, they had a management change, so someone new, comer, came to, you know, to be in charge of this organization, and they wanted to show the quick results of their activities. So they kind of burned this, this bridge, and they showed that we are here, and this helps defenders to notice this type of activity, just just because they wanted to, just because apt group management wanted to be promoted. It’s challenging. It’s something that we but on the other hand, we need to pay attention to these facts, because humans are on both sides, on the offense and on the defense side. So they make mistakes. Most teams can make mistakes. Most teams are, you know, responsible for doing their own actions. And in this case, it’s just an example. Maybe there are more organizations when they still have this bridge on, when they have the connection, when they have telemetry, when they have data, but they’re not using it right? Why not? Right?

VAMOSI: We’ve human examples of this, like during the Cold War, right?

HuraI: But let’s think about a spy that is trying to get intelligence from another countrty. What do they do? They’re sending a couple abroad, they’re having children being born in this country. So these children are being raised for 20 years or so. Why don’t they have the same thing with oti? Why don’t I plant a seed and just wait when I need to use it? And that’s definitely something that is important for me to understand. On the other hand, there is a chance that someone will forget about this. What happens with, you know, archives, they usually get lost. So maybe they planted something. Maybe there is a vulnerability in OT, but maybe there is no one that can leverage this vulnerability anymore, because everything changed, because it’s not a matter of business risk. At the moment, it’s been secure enough, so that’s perfectly fine. We’re going around the question of, are systems compromised or not? I would say that we should assume that systems might be compromised, right? And then taking this into account, we can put some thresholds. We can build security monitoring systems around it. We can, we can do everything with Pinson’s response. Otherwise, if we assume that everything is fine, why doing this? No need to spend anything.

VAMOSI: We haven’t yet seen that in the IT space, at least we don’t know that. And probably it comes down to what we were just saying, at a certain point maybe the nation-state actor loses focus, doesn’t see the ROI, and simply abandons the long-game campaign entirely.

HuraI: You’re right. The other thing I was thinking of is that technology develops very fast. That means that, well, we do not have a lot of plans working on 1980s technology, or like even the millennium. They are still advancing. They are putting a lot of effort into this. And this could be a key for, for this change, for this unnoticing, long time effect of all these tools. But, you know, we have the International Space Center flying over there. There is a lot of technologies over the our heads, and it’s away from the last century.

VAMOSI: I’d like to thank Andrew for coming on the show and talking about the ROI with OT security, and the idea that someone could be playing the long game by planting something through the supply chain, and letting it lie dormant for years. Again, even if there hasn’t been an OT attack, that doesn’t mean there won’t someday be one. Organizations need to be vigilant and prepare– not just for IT systems but for OT systems as well.

-30-