Discovering Tofino Xenon

Tofino, apart from a lovely area in the west coast of Canada is a security company that develop products like Tofino Eagle or Tofino Xenon.

The first one is the name of a series of industrial specific routers and the second is an industrial firewall. I was lucky enough to get in contact with a Tofino Xenon and could test its characteristics. The purpose of this post is to capture my experiences with it.

Tofino Xenon

Tofino Xenon is an industrial firewall with state inspection (Stateful firewall - SPI) and Deep Packet Inspection (DPI), that allows the user to filter both Modbus and Ethernet-IP industrial traffic.

Among its characteristics, we should mention that it has two Ethernet 10/100Mbps port and that it works in a transparent on the link layer OSI 2 level, as if it was a bridge device. On the other hand it has the capability to be mounted in a rack through a DIN rail.

The installation is very simple. The manufacturer recommends to install it "just in front of" the PLC or field device that we need to protect and "install a Tofino for each device we need to protect". Taking into account the price of a Tofino Xenon, that's a huge economic effort and not all companies can afford to follow these instructions...

We tested a Tofino in a testbed at the CICLAB laboratory in the University of Leon (in Spain). We connected it, on one Ethernet port to a typical 4 port non-industrial switch and on the other Ethernet port to a Schneider-Electric M340 PLC.

Our objective was to understand how the Tofino Xenon worked. We were not prepared to not even being able to connect to it. Our first difficulty was to discover the MAC address of the Tofino Xenon. We knew that the manufacturer provided a discovery application (Tofino Configurator) that allows to operate with the Tofino Xenon. However we wanted to test on our own, mainly because Tofino Configurator only works for Windows systems... After a few minutes we realized that Tofino Configurator is actually a Java development which is been later compiled into a .exe file.

So, we connected to the Tofino Xenon as explained and tried to discover it on our own. The device itself comes with a sticker which stated the "Tofino ID”. which appeared to be a MAC address. However, the more we tried to "NetDiscover" that mac address (or any other), we didn't find any network device connected to the network:

Eventually we got to find some documentation mentioning that the Tofino ID was actually the MAC address, but this device kept invisble for us. Finally a youtube video ( TV201: Tofino Orientation - https://www.youtube.com/watch?v=gigtan466rA), shed some light into our situation. We then realized that device (as it's stated in the documentation) works at Level 2 OSI, and therefore it never adquires any IP address.

We then realized that Tofino Configurator has a peculiar way of discovering Tofinos:

1. Tofino Configurator only works if there are two devices (or networks) connected to the Tofino, one to each Ethernet Port.
2. With this configurator, it sends a UDP packet simulating to be a device connected to one of the ethernet ports, and that packet is sent to another device which is in the network connected to the other Ethernet port of the Tofino Device. This UDP packet will be sent to the 6689/UDP port of the "destination device".
   
   Tofino Xenon discovery performed with Tofino Configurator

3. Tofino Xenon detects this packet and returns a new UDP packet to the originator, apparently from (and therefore faking) the address of the UDP destinee. However is actually Tofino Xenon who is replying. This UDP request never arrives to the destination.
4. When this UDP packet is returned, a TCP conversation starts between our computer and the the Tofino Xenon.

The following is a Wireshark screenshot of the whole process traffic:

As you can see from our IP address (10.10.13.136) a UDP packet is sent (packet 24) to a device which is located on the other network (10.10.13.200). This IP address belongs to a computer connected directly to the Tofino on the other Ethernet port.

The UDP packet is sent against port 6689 of the destinee. As there's nothing listening on port 6689 of the destination computer, this returns an error ICMP reply (packet 25) -in black-, but, at the same time, the originator receives as well a UDP packet (packet 26), apparently from the same destination IP and from the port 65002. Packet 26 is actually sent by the Tofino Xenon, when it detects a the previous UDP packet which travel between its ethernet ports. This UDP packet is clearly parsed by the tofino Xenon, which clearly shows, it's a stateful firewall.

Tofino Configurator sends next, from the origin IP (10.30.13.136), a second -slighly longer- UDP packet (packet 30), which is never replied by the Tofino Xenon.

Finally Tofino configurator starts a TCP conversation (packets 36 on) where the Tofino Xenon configuration information is interchanged.

Information interchanged next through a TCP (encrypted) connection

Our first objective was trying to understand the UDP "handshake" sent between Tofino configuration and Tofino Xenon, but we were not able to understand the UDP payload. It sends a 68 byte payload. From thos 68 bytes, bytes 21 to 36 (16 bytes) were always the same, in all discoveries we tried. It could be any license key from the Tofino Xenon.

The rest of the bytes are clearly encoded or even encrypted data (no bytes 00 or FF were found):

We tried to write our own script in order to be able to connect and configure the Tofino Xenon. We used python and Scapy to do it:

Even when we followed the steps previously mentioned, the Tofino Xenon did not reply to the UDP packet on port 6689. We eventually tried to connect a second Tofino Configurator software to try to work in two computers at the same time. And then we realized that we could connect to Tofino Xenon from towo machines at the same time.

Somehow, Tofino Xenon maintains a connection status and the MAC address from the "owner" client of the firewall. It seem to store any kind of state table that stores which MAC address is the "Master", and only this Master has the capabilities of access and modification over the Tofino Xenon configuration.

So to confirm this hypotheses, we resetted the Tofino Xenon. Again, only the first "client" was allowed to configure the Tofino Xenon. It seems that once a "master" takes control of the Tofino, no other device is allowed to even "discover" it (UDP packets to port 6689, do not work any more).

DoS

Another test we performed was to send it a hugh amount of UDP packets to port 6689. We detected that, if Tofino has already been discovered, Tofino simply discarded and relied UDP packets to port 6689. But if Tofino had NOT been already discovered and a 6689/UDP packet flooding was performed, a Denial of Service occurred, as no other IP could discover Tofino, because it was working on parsing all malicious UDP packets.

"Pickpocketing" Tofino

Then we deciedd to try to "pickpocket” Tofino Xenon to a legitimate Tofino Configurator who had legitimely discovered it.

We tried hard and harder but we were not able to accomplish this task. Once a UDP to port 6689 passes through Tofino, the device stops replying to any request not coming from the owner/master who has discovered it. Only when we modified/faked our attacking MAC address to be the same of the legitme master could we receive any reply from Tofino.

Next test was to try to understand the “protocol” used between the Tofino Configurator and the Tofino Xenon. We obtained a pcap file with the traffic interchaneg between both of them. We found that:

• Tofino Xenon comes with an embedded OpenSSH server. it's banner says it's a OpenSSH 6.4 (from 2014)
• Tofino Configurator uses a library named Ganymed to connect to Tofino Xenon. This library was implemented in 2006.

All the configurator initialization process, the Tofino rules which are sent to the device, the reset process and many other functions offered by Tofino Configurator are performed through encrypted commands sent via SSH.

A second script was implemented with Scapy that emulated that encrypted communication, from a previously obtained pcap file. However, although the connection was established, logically, the encryption key need to be different to the one used in the captured pcap, so we could not continue on this path.

This way, after a promising initial connection establishment, the Tofino cutted of the communication once the key interchanged process was not able to finish properly.

After a number of further tests we realized that the encryption key negotiation was RSA-SSH. We tried to modify the encryption algorithm for this negotiation. As it can be seen in the following capture, the negotiated protocols go in plain text and we could modify them, removing all but "None":

Eventually we discovered that the Tofino Configurator library allows the "None" cyphering algorithm, however Tofino Xenon does not. At this point, we decided to give up in our attempts to compromise Tofino Xenon.

Summary

Tofino Xenon is a secure Industrial Firewall. It includes a number of security features (it is only discovered at level 2 OSI, all communication is performed via SSH, and therefore encrypted, etc). However I wouldn't say it's unbreakable. From our point view, it's been only the lack of time that has stopped us from compromising it. It's not easy, but it does not seem impossible.

Several tests could not be performed:

• Continue working on the encryption and encryption key negotiation process
• Decompiling Tofino Configurator in order to understand how the Configurator work

However one possible vulnerability was found, as the device stops responding when it has not been discovered and it's flooded with UDP packet to port 6689. This cannot be considered a vulnerability, as the first step an operator does when installing the Tofino Xenon. And once a PC connects to the Tofino, it cannot be used by other "client" unless it's reset.

Given the price of the device, the manufacturers should have put stronger efforts on securing point like:

• Not using Java for Tofino Configurator Development
• Adding more protocol that can be inspected by the firewall
• Avoid flooding attacks

Surely inside Tofino there's a Linux OS, or maybe a VxWorks OS. But surely it implements a iptables firewall inside it.

From our point of view, it would be much cheaper to implement a industrial firewall using a computer with a well-tested set of iptables rules.

P.S: These tests were performed by the members of the CICLAB Group at the University of León (Juanma, Jacinto, Borja and me).

Resource exhaustion vulnerability on Schneider-Electric M340 PLC

Introduction

Past May, while working at the CICLAB laboratory of the University of Leon (Spain), I came across with a vulnerability on the PLC (Programmable Logic Controller) I was working with. For a research paper, which is currently under review, I had to deal with the configuration protocols used by the different PLCs I had in my testbed. The testbed consists of a number of PLCs from different vendors, among them Schneider-Electric.

Schneider-Electric PLCs configuration protocol is an evolution of the well-known modbus protocol, but using only the 0x5A Function Code. This protocol (or subprotocol), named internally "UMAS", has not been publicly specified, and therefore it had to be reverse-engineered.

After decoding this "UMAS" protocol (which I will explain in a future entry of this blog), I tried to send some invalid traffic to test the robustness of the PLCs.

From the Schneider Family of PLCs I had two different models in my testbed, a M580 PLC and a M340 PLC. After several tries, I was not able to compromise the Schneider-Electric M580 PLC, however when tried to send specially crafted UMAS packets to the M340 PLC, I found it stopped replying to ping commands.

This behaviour was tested using three different firmware versions. The M340 PLC seemed vulnerable, even when working with the latest firmware version (V2.70 at the moment).

Schneider Electric was contacted. I sent them an explanation of the problem together with a "pcap" file that showed the problem. Very quickly they let me know they had done their own tests and that it was a real vulnerability. It was finally published as the security notification SEVD-2017-048-02. The ICS-CERT has not released the public advisory yet, but they're on the way.

I must say they have been very very professional.

The Schneider-Electric M340 PLC

Explanation of the vulnerability

As mentioned earlier, this vulnerabilities is based on the modification of a series of legitimate "UMAS" packets sent in a specific order. "UMAS" is a configuration protocol that allows the software clients (Unity Pro and Vijeo Citect among others) to connect, configure and monitor Schneider-Electric PLCs. More than 20 different subfunction codes have been determined. These function codes allow the client to perform different tasks (Check keepalive, start/stop, upload/download strategies, read/write memory, etc).

One of these "subfunction codes" allow the client to "read" from the PLCs memory asking for a number of bytes to be read, starting from an offset of a "Memory block" (specified by a 16 bit number).

These memory blocks store different types of information (unlocated variables, system words, function blocks, etc). Every memory block has a different size (or top offset), depending on the data stored (sometimes it's 0x5B00, sometimes 0x3F00, sometimes 0xF500...). When trying to read memory over the "top offset" of a memory block, the PLC got locked. This happened with different "memory blocks", always that you sent a UMAS_read_memory request reading over the top offset of the block.

That's what, in other words Schneider-Electric explain with the following sentence:

The software does not properly restrict the size or amount of resources that are requested 
or influenced by an actor, which can be used to consume more resources than intended. 

After sending this series of packets, the PLC got freezed and only could be recovered by approaching physically to the device and resetting it manually.

The following image show a wireshark capture at the moment where this packet (#133) is sent. Before the packet is sent the PLC (IP 10.1.0.102) replies to the different requests done by the client (10.8.0.101). It can be seen as well that a few seconds later some ping commands are sent and the PLC do not reply them:

Wireshark capture of the moment of the attack

The vulnerable UMAS packet is shown shadowed, but it can be seen that it requests the offset 0xF503 (while the top offset is 0xF500.

The following image shows the python script written for testing purposes. In the screen of the right the script is launched, in the screen of the right a ping command is running and suddenly it stops showing ping connection (when the exploit packet is finally sent).

Scripts used for the test

Current Situation

Schneider-Electric bundled a number of fixes in a new firmware version (V2.90) which was released on December 21st. However, 2 months after the public release, very few from the currently available Schneider-Electric M340 PLCs are currently protected.

The Schneider-Electric public disclosure of this vulnerability was published a few days ago.

A simple search in shodan for on of the affected devices (looking for "BMX P34 2020") show that more than 700 devices (of this type) are currently accessible to internet and have not been upgraded to the firmware version V2.8 or V2.9.

Shodan capture showing more than 700 vulnerable devices

This means that more than 700 potentially critical devices could be sent offline remotely in a question of seconds. I have an exploit script for metasploit ready, but clearly it will not be publicly released for a few months (or years!).

Acknowledges

The discovery of this vulnerability has been possible by the funding of INCIBE and thanks to the support of the colleagues of the CICLAB Laboratory at the University of León (Jacinto, Juanma and Borja) and the rest of the guys of the IAF.