Managing Director at SevenShift GmbH
Pablo Endres – Personal website
Hello Pablo, can you tell us about your background and your current role?
I’m originally from Venezuela, where I studied informatics engineering. As a child I also spent six years in Canada. For the last 14 years I’ve lived and worked in Germany.
When I was in college I had a side job as a network administrator. In 1998, we were hacked for the first time and I had to deal with it. In those days we had a bastion host – a Solaris 6 – with Telnet and FTP servers. No encryption (SSH and “modern protocols” were brand new). We got to the bottom of it all and I have loved security ever since. After I graduated, I started working for a telecoms provider and shortly afterward I founded my first company.
I am the founder and managing director of SevenShift, a real boutique firm. Right now there are five of us on staff. We focus on IoT security from hardware, firmware and backends in the cloud to apps – essentially the whole IoT project. We offer pen testing, security by design, and training. Ideally we are involved from the beginning of the project so we can keep security in mind from the very start. I also collaborate with Justin Searle and his ControlThings team so that we can offer ICS/OT offensive security courses in German and Spanish, for example. (We met many years ago at a Black Hat conference.)
What’s a typical work day like for you?
Usually I start by checking who on the team needs something from me. But then I’m happiest when I can really spend time on customer projects: things like aligning on requirements or conducting security reviews and hardcore security testing. We also run training programs a few times per year. I love teaching people and really enjoy it when I get to present the material.
Fortunately we still spend 70 percent of our time on “doing” – the rest is coordination, Excel and PowerPoint, and so on.
What interesting trends do you see today in OT and IoT security?
There’s a lot happening in terms of norms, standardization, and regulations, such as ETSI EN 303 645. These developments will certainly be helpful for us. For example, MQTT has established itself as the quasi-standard on top of TCP. NBIOT and LoRaWAN are now used as network protocols in almost all projects.
At some customers we are seeing a kind of “shadow IoT” – for example, in the form of sensors – in parallel to IT and OT.
Without naming any names, could you describe an actual pen test on an IoT device or connected system?
I can tell you about a smart city project. We received a variety of products from different manufacturers and were systematically testing them.
How did you structure these tests?
We used the same standard phases that take place in every IT pen test:
- Pre-engagement actions (such as site surveys) – very important in OT
- Threat modeling and vulnerability identification
- Resolution and retesting
.Of course, the boundaries between each phase are fluid.
So what did you do?
Our job was to analyze several smart city solutions. The end customer drew on our report to choose which product to buy.
We started by understanding the solutions, obtaining the hardware, and getting access. We generally prefer a white box approach because that allows us to put the customer’s time, energy, and of course money to the best use.
Our first steps, for example, involved simply reviewing the documents and testing normal user interaction with the products the “golden paths” – in functional tests. Then we began sniffing and reverse engineering activities. In parallel, we ran classical tests on the backend and apps.
Did you find any major problems in the end?
We discovered quite a few problems that are familiar from the OWASP Top 10 List. In principle, all the tested products were insecure – even those from professional providers.
How did the customer identify and prioritize actions based on your findings?
First we had to build the solution manufacturers’ awareness for these issues. Sometimes they didn’t realize that security measures were really necessary. Some thought that they didn’t need encryption because they weren’t storing or processing any PII (personally identifiable information).
As soon as the problems were recognized, we could help the customer to improve solution security. Steps to do so included simply turning on existing SSL hardware support, adding a secure authorization process, and configuring ACLs in queues to prevent access to other devices or backend areas.
Did other cases present difficult technical challenges and how did you solve them?
I could give many examples. Each project has its own challenges – but that’s precisely what makes this work fun. If we only had standard projects we would get bored fast and wouldn’t learn so much.
Maybe an example of “security through obscurity” would be interesting: we had a customer who changed the pins of standard plugs to prevent outsiders from connecting to the nodes over the LMT (local maintenance terminal). And that was supposed to replace real security measures, which the customer didn’t take. Using social engineering, we got a suitable cable and built a copy of their setup.
It’s very common for standard tools to not work with proprietary protocols; then reverse engineering and software programming can help.
We’ve had similar cases with wireless connections. We used SDR (software defined radio) and in just a few days we had the so-called proprietary protocol. Once a customer changed the WLAN’s radio frequency and thought that would protect his system. It didn’t, of course.
Are you aware of any attacks that could have been prevented with better OT security? (Examples are welcome!)
At many locations, remote access is not really monitored. This is increasingly true thanks to mobile devices like mobile phones, tablets, and so on. But plenty of “old” modems out there aren’t on anyone’s radar screen either – that’s why “war dialing” works.
For some IIoT devices, a connection was sometimes even made from the OT to the new dashboard.
What common OT/IoT security misconceptions or falsehoods have you encountered?
“Everything is secure here – we’ve got firewalls.” Or “We only use X network technology, so we’re safe.”
We also often determine that security measures have been understood or implemented incorrectly. One typical mistake, for example, is failing to isolate processes from one another when segmenting networks. Or not activating encryption, especially for wireless applications like ZigBee or HART-IP. OT protocols usually can’t do so, and IoT product developers don’t want to for efficiency and cost reasons.
Outdated asset inventory lists, site assessments, and communication matrices are also extremely common. We won’t conduct any active penetration tests if we don’t have these documents – and we always start in a testing environment.
What do you think is often the most effective technical action to take for the security of IoT devices?
Over-the-air update functionality. I can use it to add patches at customers even after the fact. And to validate input – so I can prevent injections, for example.
Why not Multifactor Authentication (MFA)?
MFA is also important and very effective – but often it’s implemented wrong. In that case, there are tokens that you can push through with “brute force.”
What’s an important takeaway on this topic?
Companies should view security as a digitization enabler rather than something that blocks it or drives up costs.
Thank you, Pablo – we wish you lots of continued success!
Are OT and IoT security issues for your company? As an independent entity with a portfolio of proven security providers, CyberCompare can provide you with comparative offers at no charge and with no obligation. Reach out to us or use our diagnostic to learn more about your cyber risk profile.
Please remember: this article is based our knowledge at the time it was written – but we learn more every day. Do you think important points are missing or do you see the topic from a different perspective? We would be happy to discuss current developments in greater detail with you and your company’s other experts and welcome your feedback and thoughts.
And one more thing: the fact that an article mentions (or does not mention) a provider does not represent a recommendation from CyberCompare. Recommendations always depend on the customer’s individual situation.