Water system controllers don't belong on the internet, says ex-NSA chief

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh.

Let me give an example: I once worked with an integrator who was working on an AHU feeding an extremely critical portion of a datacenter (I was a lead by this point and mostly played babysitter). During certain points of the day you couldn't open the door to this room due to negative pressure because the logic was over-ramping the exhaust fans. As I watched this contractor work, I saw him open his laptop, with Windows on it (because Microsoft has had a death grip on this industry for decades now), and proceed to backup the PLC program into a massive folder with God knows how many other "customer projects" he was carrying around in this thing. He then proceeded to go do some physical checks in the field, came back, and prepared to upload the fixed program. As I watched, I noticed he _grabbed a backup from ANOTHER customer_ and I immediately had to intervene. Who knows what untold damage I saved from that single move.

I tell this story to demonstrate just how far into the dark ages this industry is. I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID and why they might want to consider a disaster recovery plan for their multi-million dollar mechanical plant.

This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper.

So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.

Well yes this is true but air gapping isn't perfect security, it's important to understand that too.

And true air gapping isn't possible because it'll need to be monitored somewhere central so there'll have to be some vpn or mpls whatever. Meaning it can be hacked.

Having it exposed to the public internet is not good practice but should be far from the only layer in its security.

And really, when a PLC is found unfirewalled on the public internet, you can bet that's far from the only security screwup in that infrastructure. If they won't even handle the low hanging fruit.

There are many wireless pump-and-reservoir systems that while not internet connected, use insecure RF links. These local RF (and casting a wider net, Bluetooth) interfaces are also ripe for abuse.

At modern population densities, basic infrastructure breaking on down on a large scale can kill millions in a matter of weeks.

The largest threat isn't bombs falling on our heads, it's incompetent fools leaving the door open to their enemies.

These aren't mistakes that can be excused. Failing in one's duty to steward important infrastructure must mean immediate replacement of leadership.

With coding agents now being used for hacking, there's a decent chance we'll see a 9/11-scale hacking incident as a result of NSA/DHS negligence in securing American internet-connected services. Similar to how the CIA's negligence allowed 9/11 itself.

The USG should be deploying thousands of security engineers armed with the latest coding models and agents, in attempt to secure systems before they're hacked. A few billion dollars spent here could save us trillions.

He is wrong and right. They should be connected to the Internet when they aren't 30 year old PLCs ripe for abuse. Until then, cut the data lines and do water monitoring the old way.

Don't put your PLCs directly on the internet. In fact most industrial stuff is very not made to be directly connected to the internet. But interpose a firewall+VPN solution and you might be ok, if done competently.

And remote access to hardware definitely makes management and maintenance a lot easier and quicker. (else you need to drive out for every minor issue)

We could say that about a lot of infrastructure that has been recklessly placed on the open Internet because it was cheaper than more secure solutions. I would say the same about home security systems, for instance.

what does belong on the internet in a post-mythos world?

one argument: only services which need to be available to unauthenticated endpoints should be default reachable.

all other services should be default unreachable (no data plane until authorized ...then use internet and other networks to establish the connections).

yes, that is not always easy. it is much more possible than it used to be.

and arguably we now need to commit to the tradeoffs of default unreachable services.

>Other countries start securing their water

>nsa: what no, stop that

I mean, this statement shouldn't be shocking to anyone. Critical infrastructure should be air-gapped if at all practical.

>Water system controllers don't belong on the internet, says ex-NSA chief

sounds like cope for a bunch of felons that management, its director and congress can't get a handle on.

you reap what you sow