Simple policy based routing in practice

Sometimes the network setup isn’t the one that you find in a textbook. Policy based routing is a mechanism to choose a route based on a policy (which can be anything, from the current time to the kind of packet/frame).

A little on the environment…

For a lab demo I installed OpenShift in a server under NAT, without the capability of port forwarding. In order to give the server an external access, I made a VPN with a VPS that we had in place for some projects.

Unfortunately the simple port forwarding wasn’t enough due to the fact that the VPS had some services running on the same port of OpenShift (ie. HTTPs). So, another IP address was prepared and assigned to the VPS.

All right. We’ll configure the VPS to forward the traffic for 203.0.113.5 to 192.0.2.2 (via the VPN). That’s it? No. In fact, the Router here will do an asymmetric routing: while an incoming packet will use the path (internet)->VPS->(VPN)->Router->OpenShift, the response will go directly OpenShift->Router->(internet) as the Router, by default, is looking to the routing table without “memory” (ie. the fact that the incoming packet was received by a different interface doesn’t have an impact in any future routing decision, by default). The asymmetric routing isn’t bad per-se: in this case, the NAT is interfering and it’s breaking any connection (as the reply public address is different than the VPS public address used for OpenShift).

The goal is to make a policy that says: ok, use a connection tracking module and, for each connection that pass through VPS, forward replies back through VPS. That is policy based routing (PBR).

Note: there might be an improvement on the load (and routing speed) by avoiding the connection tracking module. I configured the Router to use conntrack because this is a lab experiment, so speed is not a priority.

Let’s configure!

PRE: I assume that the VPS and the Router are linked with a VPN and the routing is correctly configured. How to setup the VPN is not the goal of this post, so I’ll not show how to do that.

We need to configure the Router and the VPS in order to make this works as expected. The Router is running Mikrotik RouterOS (but PBR can be configured in many mid- and high-end routers), and the VPS is using Debian GNU/Linux. In the VPS, a simple IP/port forwarding is sufficient:

# iptables -t nat -I PREROUTING -d 203.0.113.5 -j DNAT --to-destination 192.0.2.2
# iptables -t nat -I POSTROUTING -s 192.0.2.2 -o wan_interface0 -j SNAT --to-source 203.0.113.5
# iptables -I FORWARD -i wan_interface0 -d 192.0.2.2 -p tcp --dport 443 -j ACCEPT

This is sufficient in order to forward the traffic for the HTTPs port to OpenShift, and do the address translation back on replies.

Now, on the Mikrotik Router we need to create a new routing table (I named it “via-VPS“) with the link addresses for both the VPN and the LAN where OpenShift is, and the default route via the VPN address of VPS:

# /ip route
# add distance=1 gateway=192.0.0.2 routing-mark=via-VPS
# add distance=1 dst-address=192.0.0.0/24 gateway=vpn_interface0 pref-src=192.0.0.1 routing-mark=via-VPS scope=10
# add distance=1 dst-address=192.0.2.0/24 gateway=lan_interface0 pref-src=192.0.2.1 routing-mark=via-VPS scope=10

Then, configure the firewall and the conntrack in order to track each connection coming from the VPN and make next routing decisions (on replies) using the new routing table:

# /ip firewall
# add action=mark-connection chain=prerouting connection-state=new dst-address=192.0.2.2 dst-port=443 in-interface=vpn_interface0 new-connection-mark=via-VPS passthrough=yes protocol=tcp
# add action=mark-routing chain=prerouting connection-mark=via-VPS new-routing-mark=via-VPS passthrough=yes

Using fogproject to deploy Windows 10 images

Despite the web is full of pages about fogproject and Windows 10, there are many different things that you need to do in order to make fog to deploy a Windows 10 image in a fully automated way (without your physical intervention). This is my guide, just in case.

What is fogproject?

FOG Project is a set of tools made for automatic deployment of system images. For example, if you have a set of machines (a lab, an office, etc) that needs to be prepared with software, configuration, preloaded data, you can prepare a set of “master” machines (one for each kind of image you’ll need) and use them as a template, ready to be deployed in a hundred of PCs with a few clicks.

More info at: https://fogproject.org/

Requirements

  • A running fogserver instance (I’ll not cover the installation, it’s pretty trivial)
  • An hypervisor (even VirtualBox is good) that can boot from network
  • Windows 10 ISO (you can find it on Microsoft’s official website)
  • Windows 10 ADK (again, go on official Microsoft website)
  • A lot of time

Step 1: Master image

The first thing that we need to do is to install a master image. Let’s create a VM for Windows 10 and start it with the ISO loaded into the CD-ROM bay. After creating the VM, you need to change the boot order to have the network boot first (refer to your hypervisor manual). The NIC should be in the same network of the fog server (so you’ll probably use the bridge mode). Before installing Windows 10, boot from network and add the VM to fog inventory.

After the inventory, you can boot the Windows installer from the ISO. Note that you’ll need to stop installing when Windows 10 starts loading from disk (eg. after the first reboot) and press CTRL-SHIFT-F3 in order to enter into the “Audit mode”. In this mode, everything that you’ll customize will be saved as “Default User” profile (eg. the default setting for all new users).

You can dismiss the dialog about sysprep for now, and start personalizing it (eg. you can install applications, do changes, etc).

One thing that you’ll need to install is the Deployment Tools of the Windows ADK. They’ll be needed for the next step.

Also, in order to deploy printers, snapins and other things, you’ll need to install the FOG Client. After installing it, you’ll need to disable temporarly in order to work with sysprep (open a command prompt):

sc stop FOGService
sc config FOGService start= disabled

In order to start again the service, modify the following file:

C:\Windows\Setup\scripts\SetupComplete.cmd

and append/add the following:

sc config FOGService start= auto
shutdown -t 0 -r

When you have finished with customization, we need to automatize the OOBE part (Out Of the Box Environment; in other words, the wizard that you see the first time that you run Windows). That’s because I want that the PC is ready at the end of the procedure.

Open the WISM from here:

C:\Program Files (x86)\Windows Kits\10\Assessment and Deployment Kit\Deployment Tools\WSIM\imgmgr.exe

And customize the response file. Then, save the file in some local storage (such as c:\).

Make sure that you make all the updates to Windows. Then, shutdown the VM and take a snapshot. If you need to change something (such as update some software, or change some settings), you’ll need to revert back to this snapshot.

Step 2: ready to sysprep and image

Go on the fog web admin and schedule a capture for this host. Start the VM and skip the sysprep window, we’ll need to launch if from the command line in order to specify the answer file. Open a command prompt with admin rights and do the following:

cd c:\Windows\System32\sysprep
sysprep /oobe /generalize /reboot /unattend:C:\unattended.xml

At the end, you’ll have a generic Windows 10 image with your customization imaged into fog.

Step 3: deploy

You can deploy the image as usual, by using the web interface and/or when you’re inventorying new PCs.

NodeMCU/ESP8266 and SSL/TLS/HTTPS

Security is always good. But you need to pay attention to implement it well, otherwise it can be very harmful.

Let’s say that we want to secure the communication between our NodeMCU (ESP8266) and a server (cloud/LAN/whatever). Starting from NodeMCU devkit 2.4, you have a class named WiFiClientSecure, which implements a TLSv1.2 client with some recent (and secure) ciphers, such as aes_128_cbc_sha256.

Trust me, you want to use this class. Do not try to implement another TLS class if you don’t know what you’re doing.

Before seek for some examples, let’s clear some points:

  • WiFiClientSecure is a subclass of WiFiClient, so you can use it as a drop-in replacement, and you can use all methods from the parent class
  • connect() method of WiFiClientSecure will not return an error if the certificate is invalid – the check is up to you (later in this post you’ll see how do this)
  • the SDK doesn’t contains root certificates (such as a browser do)
  • you can check a certificate in two way: by checking its certification chain (as a browser), or you can check the certificate fingerprint itself (“certificate-pinning”)

Checking the certificate fingerprint

Each certificate has its own fingerprint. It’s hard to match with a (valid) private key, so checking the fingerprint is a valid method in order to secure a channel.

Pros:

  • Simple and effective: if the certificate fingerprint is wrong, then simply the certificate is not valid
  • There is no way to force a device to accept a certificate without altering its own code
  • Easy to deal with self-signed certificates

Cons:

  • You need to know the certificate fingerprint during compilation-time
  • The fingerprint cannot change during the lifetime of the code build/version, so you cannot change/renew the certificate

This method is useful on self-signed certificates or with a certificate that is valid far beyond the software version lifetime, because you cannot generate two certificates with the same fingerprint (this means, for example, that you cannot use Let’s encrypt certificates, as they expires after 90 days, unless you’re planning to release a new version every 90 days).

Code example:

WiFiClientSecure client;
int result = client.connect("tls.host", 443);
if (result != 1) {
  // Something bad happened on connection
} else {
  // Socket is connected BUT the certificate is not verified yet
  // Verifying the certificate by using fingerprint
  if(client.verify("D1:D0: ... :7E, "tls.host")) {
    // Certificate is OK
  } else {
    // Certificate is invalid
  }
}

Checking the certificate trust chain

This method is the most widely used because of its versatility: the client checks for a signature from a trusted certification authority (from a local storage of certificates). So, even if the certificate changes, as long as it’s signed by a trusted CA, the certificate is valid.

WiFiClientSecure doesn’t handle a CA bundle file (yet), so if you want to check multiple certification authorities, you need to load them manually and check each of them. In the example, we’ll see how to check only for one CA: it’s trivial to convert it in a loop for checking multiple certificates.

Pros:

  • The certificate can be changed (re-newed, re-issued, etc) without issues
  • Easy to have multiple certificates (for example, one for each host of a cluster)

Cons:

  • You need to embed the certification authority certificate (in the code or in a separate file/memory)
  • If the CA is a separate file, and there are external ways to access that memory (eg. an SD card), it’s trivial to tamper the CA store

Code example (in the example, ca_cert is the CA certificate binary loaded in memory, and ca_cert_len is the length of ca_cert memory area, in bytes):

WiFiClientSecure client;
int result = client.connect("tls.host", 443);
if (result != 1) {
  // Something bad happened on connection
} else {
  // Socket is connected BUT the certificate is not verified yet
  // Load CA certificate
  client.setCACert(ca_cert, ca_cert_len);
  // Verifying the certificate by using loaded certificate
  if(client.verifyCertChain("tls.host")) {
    // Certificate is OK
  } else {
    // Certificate is invalid
  }
}

IoT security vs hobbyists’ boards

So, you bought your brand new Arduino/Genuino Uno and some nice-but-useless sensor (such as a temperature sensor for your bedroom), and you feel ready to enter the Internet Of Things world. You want to build up a little “cloud” by yourself (by using Apache+PHP in some hosting, or perhaps an MQTT server like RabbitMQ), or maybe use some cloud-ready service.

Bad Things Happens

Even if you heard a lot of news about IoT-security, you’ll ask yourself: “Why me? There is no reason to hack my devices. I’m not working for CIA/NSA/FBI, there is no value in hacking me. Hackers won’t loose their time on me”. Then, you’re wrong. Very wrong.

Why? For two main reasons: first, many hacks are performed by automatic tools, not humans; and computers do not sleep, do not need to rest, so they have a lot of time to hack you. Second, they want to hack you because you always have some valuable things (at least): internet connection and local resources (such as disk space, computational power, etc.).

Let’s make some examples for the latter motivation: even if your device is a SoC (system-on-chip) with a CPU of 600 MHz and some MBs of RAM, if you sum up all the computational power of all IoT devices, this is something like an huge supercomputer, at zero cost. And if that won’t scare you, this example will: what if an hacker needs to hack into a bank to seize some money? What if they need to hack a military facility? They will use your device as a gateway (and then they wipe out all traces), so you will be identified as responsible (eg. by checking the IP address).

So, security is very important, even in IoT devices.

IoT platforms and security: overview

In IoT-enabled projects, two devices are common: microcontrollers (such as Arduino/Genuino) and System-On-Chip (such as Raspberry PI).

The former category (microcontrollers) is dominated by two processors: Atmel (expecially AVRs): pratically none of Atmel processors that you can find in hobbyist boards can support TLS v1.2 (two main reasons for that: first, the computational power is too low; second, the storage for certificates and memory for operations are insufficient); the other big is Expressif (that produces ESP8266, mounted over widely-known NodeMCU boards): these boards have limited capabilities in terms of memory, but they have sufficient computational power to handle TLS connection.

The System-on-Chip category, instead, is widely distributed (Intel, ARM, NVidia processors): pratically every boards is capable of doing TLS.

So, even if Arduino/Genuino is an Italian product (and I’m proud of it), I advise to choose Expressif microcontrollers or, better, SoC devices for Internet of Things. And, of course, I advise to use TLS.

In some future post I’ll describe into details the weakness (or strongness) of widely used devices and code.

LoRaWAN spreading factor allocation in a multiple-gateway environment

As this new trend of “Internet of Things” raise up, people and companies are proposing new protocols (at each ISO/OSI level) to manage the local/remote communication between an IoT device and the rest of the network. This is mainly because an “IoT-device” (such as a temperature sensor) may be off-grid (i.e. powered by batteries) and/or with limited resources.

For example, MQTT is proven to be very successful versus HTTP thanks to its specifications, which results in a lightweight implementation.

In this post I’ll talk about a Physical/Datalink protocol, named LoRaWAN.

A brief introdution to LoRaWAN

LoRaWAN is a datalink protocol, used to manage access and transmissions over LoRa (LPWAN) physical links. LoRa is a Low Power Wide Area Network medium (wireless). LoRaWAN is a MAC, Media Access Control, for LoRa links. LoRaWAN defines roles and duties on a LoRaWAN network, such as a node (eg. a temperature sensor), a gateway (the “base station” which receives all trasmissions from nodes and replies back) and the role of network server.

Everyone can use LoRa transreceivers in the free spectrum band (~434 MHz in Europe). Another frequency band (~800 MHz in Europe) is licensed for business use instead.

LoRa has many parameters; the one which the research is focusing on is the Spreading Factor. The Spreading Factor is a set of parameters that specify transmit power, subfrequency and air time. LoRa define spreading factors numbered from 6 to 12, where LoRaWAN is using from 7 to 12. The lower is the spreading factor, the higher is the throughput, and the lower is the distance covered. Also, lower spreading factor means lower power consumption.

At SF7, the distance is minimal, and the throughput is high. At SF12 (or SF11, depending on bandwidth – at 125 kHz, SF11 and SF12 are swapped), the distance is the max distance covered by LoRa standard, and the data rate is the lowest.

The spreading factor also changes the Air Time (or Time On Air): at SF7, the Air Time is minimal. At each higher spreading factor, the value is nearly doubled.

In many implementations, spreading factors were assigned manually. Unfortunately, this limits the scalability of this protocol. So, many algorithms were implemented in order to automatically assign spreading factors among nodes (as LoRaWAN supports changing S.F. remotely by a gateway). Some of these algorithms are using the RSSI (Receiver Signal Strength Indicator) value.

ADR (Adaptive Data Rate)

The ADR algorithm was the first algorithm proposed, and simulated in a single-gateway scenario. It’s straightforward: the spreading factor assigned to a node is the lowest s.f. possible (keep in mind that some nodes may have a lower bound for spreading factors due to the distance between the node and the nearest gateway).

A sample node disposition, with SF assigned by ADR

The pros are clear: it’s very simple and fast. The issue, however, is that when there are many nodes at the same spreading factor (because they’re near the receiver), collisions and air time for that spreading factor are getting worst. This means that the efficiency of this algorithm, measured by using DER (Data Extraction Rate) is very low: in some scenarios, nearly the half of the data is lost.

EXPLoRa

A team of students and professors from CNIT studied the “spreading factor issue”, and came up with a solution (it’s not the only one, but I think that’s the better one at the moment). Named “EXPLORA”, the algorithm moves some nodes between spreading factors, balancing them by considering the Air Time. In other words, it moves the nodes in order to have the same air time for each spreading factor (on the considered gateway).

The same sample node disposition as ADR, but with SF assigned by EXPLORA instead

This algorithm works well in a single-gateway environment, but it’s not applicable to multiple gateway environments, where a moved node can increase the “stress” over a spreading factor into a close gateway.

AD-MAIORA: Adaptive Mitigation of Air Time in LoRaWAN

On October, 2017, during Network Infrastructures class, two colleagues and I began to work on LoRaWAN as class-project, specifically on the (possible) optimization of spreading factor allocation on multiple gateway environments.

By studing how ADR and EXPLORA works, we built an algorithm capable of changing the spreading factor on some nodes in a way that the “stress” is more fair distributed between all gateways. By keeping in mind that spreading factors cannot be lowered if we start from an ADR-like scenario (where a node has the minimum spreading factor possible), the algorithm move nodes (even if this means that a node may be heard by more gateways than before due to overlaps) in order to minimize the load on all possible spreading factors on all nodes.

The improvement that we were able to achieve by using own algorithm, as opposed as the ADR applied on a multi-gateway environment, is ~10% of average increase for DER.

We’ll describe this algorithm in a paper, stay tuned 🙂

A sample 2-gateway topology, with node assigned by ADR
The same 2-gateway topology, with SF assigned by AD-MAIORA

IPv6 link-local and VPS-cloud services: an hidden threat?

As many IT folks, I have my VPS (for instance, this website is running on it). I use this virtual server mainly to host my blog and some other websites that I own. The main reason why I use a “server” (and not an “hosting solution”) is that, in this case, I have complete access to the machine. I like to be able to customize my services from top to bottom, even if it’s a simple blog.

The provider (Netsons) is not currently providing any IPv6 connectivity to VPSes. Even my home ISP is not providing IPv6 connectivity. Wonderful…

Some years ago I requested an account and a /48 subnet to Hurricane Electric IPv6 broker service for my house. It’s working very well (consider that it’s a VPN, only that it’s not “private“).

Then, last month I began to play with the network configuration of the VPS to check if everything is ok (I was wondering to request another tunnel for my server), and I discovered that, thanks to Link-Local IPv6 addresses, pretty much every host in my server network was reachable. No firewall, no ACLs.

If you don’t know what is a Link-Local address, let me explain you: there are some standard (one for IPv6, another for IPv6) which allows the host to autoconfigure a network interface with an IP address (IPv4 addresses are in 169.254.0.0/16, IPv6 addresses are in fe80::/64). This is optional in IPv4, but in IPv6 it’s needed for many network services. The way it works it’s quite simple: the host picks a random IP (in case of IPv6, it may use the MAC Address), then query the network if anyone has that IP address (in v4, a simple ARP request is sufficient). If there is a collision, another IP is chosen.

Link-local addresses were invented in order to satisfy the plug-and-play concept: you can plug two or more devices in an unconfigured network (no DHCP, no static IPs, …), yet these devices will be capable to talk each other (for example, a brand new PC and a printer).

Remember: even if your network is not running IPv6 (native or not), this doesn’t mean that your host cannot do IPv6 traffic. So, if you don’t use IPv6 at all (very bad move), you need to disable it completely, or you can configure a firewall rule to block all IPv6 packets.

Cold case: protocol reverse engineering (part 2)

If you missed the previous post, you can find it here.

I was determined to find out which protocol they were mimicking. If you know OSI model (often named ISO/OSI), you already know that each level may (in fact, they do) adds an header (sometimes also a footer) in every Protocol Data Unit (PDU) in a process named “encapsulation“.

So, I ignored the ethernet “type” field and I tried to manually match the protocol by looking at the raw PDU (printed in hexadecimal format in a piece of paper). After some trial-and-error, I found that the PDU was a real-complete IP packet! I haven’t tried the IP protocol as the first one because I thought that they built a custom protocol based on a simpler/older one than IP.

After that discovery, I forced Wireshark to decode it as IP, and also the TCP connection showed up. Counters, addresses and other things that I was supposing before (see previous post) were actually IP and TCP fields (IP addresses, TCP ports, sequence numbers, etc).

I was happy because this was a big step forward. Now we can assume that PDU size (at TCP level) is actually a proprietary protocol. So it comes the hardest part.

I made a number of traces (~200) and I come back to vbindiff to compare TCP data. Note that, as in the previous case, the trace is made with a passive bridge tap and by doing some action on the controller/PLC. Then, every trace is saved with raw capture and “metadata” (the complete environment, eg: what commands are sent by the controller, what actions is performed on PLC, which are the values displayed or set).

What I learned, by doing many comparison (which took me nearly two weeks), is that:

  • The “dual-reply” that we were seeing before the IP decoding were actually TCK ACKs: if the PLC was slow on reply (we’re speaking about milliseconds…), the networking component on PLC were sending ACK replies. Otherwise, if queries data were available immediately, it responds with only one ACK,PSH packet (as standard)
  • The connection is made by two kind of messages:
    • Periodic messages from PLC to the controller (not enabled by default)
    • Controller actions/queries (with relative replies)
  • Message type is an 8-bit field inside the TCP PDU
  • The second byte indicates if the PDU is a reply or a request
  • There are two counters, one used to distinguish between duplicates in dual-ethernet config, another is for duplicates in the same link
  • On value set or get, values are coded in TLV (type-length-value) at the end of the PDU, even the strings and the item ID (which, in this system, is an 8+3 static string, whitespace padded)
  • Floating point values are coded in 4 bytes IEEE 754
  • The endianess is big-endian
  • The FCS field on ethernet was used for a custom counter (maybe something that the controller board uses)

A side node: actually the first thing that I noticed was the floating point value. I still remember the IEEE 754 logic and structure (I’ve studied that in High school, nearly 7 years ago), that helped me because floating point values are stored in a particular (characteristic) format.

There was sufficient data to write a custom software to do tests. But one question remains: how do I simulate a “wrong” ethertype for IP? I was suspecting that the ethertype were modified by the industrial ethernet board (or its driver), so I build my “PLC”-client for Windows and I made it run over the controller. And I was right: every ethernet frame with an IP/TCP connection to the PLC was changed to adjust the ethertype.

After some adjustment and testing, my client was working well (as the original controller) for setting and getting values (not the PLC-programming part yet). But still, how to use another industrial ethernet card with Windows and make it change the ethertype?

With a kernel driver, of course.

The Windows Kernel Driver parabola

I must admit: that was a really impulsive decision. But still, it taught me a lot.

Unfortunately I do not remember a lot – I’m not a Windows fan. I downloaded the whole Windows Driver toolkit and I began to play with some examples. Then, I wrote a simple “network filter” that was doing the same as the board: checking if the connection was for the TCP port of the PLC and change the ethertype field.

The “development-environment” was a pain: two physical machines (why I didn’t use VMs? I tried, but something in Windows kernel didn’t cooperate…) linked with both ethernet and serial connection. There was an option to use ethernet-only, but for some reason it didn’t work at all. The “host” was a simple-plain Windows 7 PC with Visual Studio, the “guest” was a Windows 7 with debug enabled.

As you can expect, with a debugger attached to a kernel you can do pretty much everything: freeze the machine (and then continue the execution), inject drivers, trigger events, and, of course, trigger an Blue Screen Of Death.

The driver was working perfectly – except for only two bugs (which causes some BSODs in tests) that I fixed immediately after the crash. I checked with Wireshark: packets were indistinguishable (I mean, you cannot tell if it was sent by the original controller or my software).

Then, after the tests with my client over the “driver-equipped-machine”, I accidentally launched my PLC client even on the host machine (which didn’t have the driver). And it worked like a charm. WTF? Why?

It turns out that custom ethertype (and so the Windows driver) was not really necessary. Even standard IP ethertype was OK.

See why I say that was an impulsive decision?

Conclusion

After that, I moved all the code into the OPC toolkit to build an OPC driver, so I was able to get/set values by using OPC compliant software, such as the Kepware one.

It took me nearly three months to reverse engineer this protocol. And I’m still asking me how the hell I made it.

Appendix – tools

To summarize tools and usages:

  • Wireshark and tcpdump: used for capturing and decoding packets. Wireshark made this work possible.
  • A binary diff-like software (like vbindiff), if you have a binary protocol to decode, or the standard diff if your protocol is plain-text (like HTTP).
  • A Linux distribution (Debian in this case) with bridge utils: by using the brctl command you can setup a bridge between two (or more) interfaces in a PC.
  • In order to do the bridge, I usually have with me one or more USB-to-Ethernet converter.
  • You may want to disable IPv6, disable Stateless IPv4 config, Zeroconf/Avahi/Bonjour services, DHCP clients, Network managers and other things that immediately do something when you plug a cable in your computer. If you don’t know what will happen, you need to study more in order to know more about what’s running on your computer (eg. your operative system).
  • The previous point is why I use GNU/Linux: I have full control on what is running (and where). By disabling pretty much every automation, you still have a full control of your system without garbage. For example, with Windows you’ll have SMB broadcast packets and things like that.
  • Reverse engineering a protocol is a matter of knowledge, hard work and luck: you need to know your environment perfectly (eg. protocols at each levels, engineering solutions embedded in various standards, etc); you need to do comparison, tests, speculations, inference, hypothesis, [go back to 1] for an unknown time, a lot more that you might expect; and you need to be lucky.

A lot of things that helped me were not coming from networking study: programming skills (I’m a C developer, and I made some small software in assembly), operative system internals, RFC comments about some technical trick in standards (see, for example, RFCs about IEEE 754 and TCP Tahoe/Reno).

Cold case: protocol reverse engineering (part 1)

Some years ago I was asked to do a reverse engineering of an older protocol used in automation systems. The goal was to be able to communicate with some equipment already in-place in some industrial buildings nearby. Let me explain.

Note: at the end of the story I’ll put some commands and techniques that I applied in order to reverse engineer this protocol. If you’re not interested in this story, feel free to subscribe and wait for the last part you can find the second post here.

Some background

In industrial sector, automation is everything. Either long or short processes are automated nowadays: there are many mechanical devices controlled by one or more PLCs (Programmable Logic Controller). Today PLC is a sort of “dedicated-computer”. Each PLC is connected to an automation network via fieldbus-protocols, such as PROFINET (over Industrial Ethernet).

Usually PLCs have some logic (for example, open valve X if temperature Y is raising), but of course your building process may require many variables and on-the-fly changes, so they’re constantly connected to one (or more) server, named supervisor. This system is called DCS, or Distributed control system, or SCADA, Supervisory control and data acquisition. Many PLCs with logic, some control stations (for local operators) and some supervisors.

With the same system you can acquire, store and manipulate data regarding the build process: for example, quantities of chemical compound, temperatures, timing.

Some of these systems (mostly the older one, the DCS) uses proprietary protocols to talk between PLC and supervisors, and offers no APIs. SCADA systems often uses standards protocols (as alternative with proprietary ones) and/or APIs thru OPC drivers.

Times go by

Times go by, for everything. In this case, I had my hands on a 15-years-old system, with a spare PLC and a spare supervisor (as the real system is still in use, and producing). The PLC was equipped with some digital and analog I/O boards. It was working perfectly: the PLC logic was ok, the supervisor was supervis-ing, etc.

So, why I was there? Because the supervisor had one big issue: it runs on Windows 2000. And there was no way to make software work with newer systems nor accessing/interfacing with some driver. Of course, there was an easy way to solve this: dismantle everything and rebuild the building chain with newer hardware. In short, replace fully-functional hardware (and software) with a new product (which may have some issues, who knows).

You would agree with me by thinking that rebuild the chain was a big waste of time, money and hardware.

So my job was find a way to interface PLCs with a newer system.

Ground truth

I had little clues. All that I had was:

  • An OPC devkit to write a driver which talks OPC (an open standard SCADA-compatible)
  • The PLC and the supervisor in my hands

The first thing that I noticed is that the supervisor and the PLC were connected with two (distinct) cables that looked like industrial ethernet cables. Pin mapping and some tests revealed that I was right: interfaces were ethernet. That might be consistent with some sort of ancient PROFINET drafts. At least, I was hoping that.

So the first step was doing a man-in-the-middle “attack”: with a dual-ethernet workstation, I made a passive-tap-bridge and I put the workstation in the middle of supervisor-PLC connection. A passive-tap-bridge is an ethernet bridge with some features:

  • No change on packets (no firewall, etc)
  • No Spanning Tree Protocol, loop protections or something like that
  • Promiscuous interface set
  • No IP address or something like that
  • and Wireshark, of course

A side note: the PLC was connected with both cables (for redundancy purpose), so I took offline the second cable to force the supervisor and PLC to talk with me listening on the first link.

I wasn’t lucky enough. The data stream that I expected was something like a TCP/UDP connection of some sort. Instead, I had a bunch of Ethernet II frames with unknown data.

I made some hypothesis:

  • The protocol was too old for some sort of encryption (so, no encryption)
  • I was looking at the raw data of a proprietary protocol
  • OR, there was some sort of proprietary weak encryption for higher protocols than Ethernet

It’s not uncommon to see direct-to-ethernet C&C protocols. So I started to analyze some frames.

The analysis (part 1)

At first look, no clue. There were two strings of 3 chars (maybe some sort of “magic” for that protocol, or maybe some random coincidence) in many frames. Also, layer 2 protocol 0x8275 said nothing to me. So I collected some data streams between supervisor and PLC (by query the PLC from the supervisor’s software), extracted the ethernet payload and I started to analyze it with vbindiff, a diff-like tool that deals with binary files.

I noticed that there were some bytes incremented linearly, like a counter. Other bytes instead were swapped if the direction was the opposite (eg. on PLC response). That might indicate some sort of addressing (it makes sense). Then, the value that I was querying was probably at the end of the frame, because a change on the PLC side was reflected by a change (somehow) at the end of the ethernet payload. But it was not linear (consistent with real changes), so no idea how that value was calculated.

To recap, this analysis told me that:

  • The protocol was not encrypted – eg. if it had been encrypted I would not have seen the same bits, in the same place, between two messages.
  • There was some sort of counter: usually this is made to avoid duplicating packets at the destination, and to provide flow-control/retransmissions/etc. So at least one of these mechanism were in place.
  • There was some sort of addressing scheme (with at least 1 byte)
  • That two cables were meant to be used with two separated networks, but they were transmitting the same data (for redundancy purpose)
  • Each command (eg. set this value to X) requires at least 1 reply. Usually 2.
  • The Ethernet FCS (Frame Check sequence) was wrong. In each frame.

With 50-60 traces (with various commands and replies each), I started to look for correlations between these bytes in different frames.

One thing that I noticed after some hours of digging is that, when there were two replies, the first one was somehow really small. Then, analyzing more I discovered that there was another short (16bit) value that was swapped between two frames of the same transmission. Last conclusion collides with the addressing scheme assumption said before, so I was really unhappy.

I tried to do some interesting thing, such as byte conversions and endianess swapping. There was no clue until I found something caught my attention: swapping the cable results in a change on a byte before the “address-byte” we mentioned before. So it might indicate some sort of 2-byte addressing scheme or subnetting thing.

After a week of testing, I was about to surrender. There was something familiar with all these bits, but I didn’t know what.

Then, I had somehow a breakthrough: what if that is a subset (or custom version) of a wide-used protocol?

(go to part 2)

Mikrotik: how to use hostnames in firewall rules (instead of IP addresses)

Althrough the IP layer doesn’t carry the hostname used to send that packet (in fact, may not exists, because DNS is higher in ISO/OSI stack), some firewall allows DNS hostnames in “Source” or “Destination” fields. It might be useful in many situations: a fast TTL (for example, NTP Pool project hostnames, “pool.ntp.org”), a dynamic IP address associated to the hostname, a round-robin record with many IPs that might change often (CDN, load balancers, etc). So, how these firewalls can discriminate if there is no such info at IP layer?

A very ugly hack that works

In fact, they don’t. Instead, they resolve each hostname at regular interval (smarter firewall uses TTL DNS field) and use the result (IP addresses of course) to build the firewall rule. A very ugly hack.

If you wonder if there is any other solution for these requirements, the answer is no, AFAIK.

Doing this on Mikrotik routers

Mikrotik RouterOS firewall is an iptables-based firewall, so there is no embedded support to this trick (it might be supported on a fully-fledged OS with some iptables module, but this is not the case).

RouterOS has a “scripting language” to achieve some automations. With this simple script, we exploit the embedded resolver and cache to popolate firewall address-lists.

:local hosts {"it.pool.ntp.org"; "time.nist.gov"; "0.debian.pool.ntp.org"; "1.debian.pool.ntp.org"}
:foreach k,v in=$hosts do={
  :log info "Doing $v"
  :local listname $v
  :resolve $v
  :local iscname [/ip dns cache all find where name=$v and type="CNAME"]
  :if ($iscname != "") do={
    :local newname [/ip dns cache all get $iscname data]
    :log info "$v is CNAME to $newname"
    :set v $newname
  }
  :resolve $v
  /ip firewall address-list remove [/ip firewall address-list find where list=$listname]
  :foreach i in=[/ip dns cache all find where name=$v and type="A"] do={
    :local ipaddr [/ip dns cache all get $i data]
    /ip firewall address-list add list=$listname address=$ipaddr comment=$v
    :log info "IP address: $ipaddr"
  }
  /ipv6 firewall address-list remove [/ipv6 firewall address-list find where list=$listname]
  :foreach i in=[/ip dns cache all find where name=$v and type="AAAA"] do={
    :local ipaddr [/ip dns cache all get $i data]
    /ipv6 firewall address-list add list=$listname address=$ipaddr comment=$v
    :log info "IPv6 address: $ipaddr"
  }
}
:log info "end"

Let’s explain:

  • The first step is to define the host list (first row): a simple array of hostnames. This script will create the address-lists with these names (either in IPv4 and/or IPv6 firewall)
  • If the hostname is a CNAME, the script resolves the pointed hostname (just as expected from a DNS resolver)
  • All IPs (for every hostname) that are in DNS cache (due to “:resolve”) are copied into the relevant address-list (that is temporarly cleared before this step)

Many other Mikrotik scripts exists for that, but every script has some flaws: the main one is that everyone use “:resolve” to get IP address (but “:resolve” only returns one ip address, even if the hostname has multiple IPs). This script instead is capable to get all IPs (4 or 6).

How to use

Copy this script in RouterOS system scripts, modify the first line (for hostnames to be resolved) and schedule this script to run at regular intervals. After the first execution you’ll have the lists populated, and then you can modify firewall rules.

Please mind that “:resolve” on Mikrotik throws uncatchable errors: if the hostname doesn’t exists, this script will stop at some point, leaving some items not updated.

IoT party: installing InfluxDB (+Grafana) on a Raspberry PI for IoT sensors

Last year I upgraded my solar water panels with some temperature probes, and I scattered NodeMCU in my house (with some DHT22 and DHT11 sensors). At the time I had an old PC that was acting as a server: IoT devices were sending readings to RabbitMQ instance on that server, with a shell script that was bridging between MQTT and RRDTool database.

Of course, for me that was perfect. But I when I tried to share these tools with my family, quickly became clear that I had to make a nice interface. I ended up by sending the chart by Telegram with a bot.

Then, during last month, the old server died. I moved some things to the cloud, and some other things were archived. After some free-time during cleanup (I had two 1TB disks to cleanup…) I landed into Grafana home page. And I decided to give it a try.

Continue reading “IoT party: installing InfluxDB (+Grafana) on a Raspberry PI for IoT sensors”