MQTT Across Separate Home Networks

Status: Completed

Note: I’m not super au fait with Linux networking, and I’ve struggled to make progress with every learning resource for it that I’ve tried, so when I’m working on my home network in this way I’m mostly learning by bumbling about and trying stuff until I kind of get the result that I want. Consequently I’ll do my best with the language but no promises that it’ll all be correct. Get in touch about stuff I’ve got wrong, it’ll be useful for my own learning.

The Setup

In my home network, there are a few separate “subnetworks”. I’m using OpenWRT, and I see these as both interfaces in Network > Interfaces, and zones in Network > Zones.

The three key ones important to the content of this note are lan, wan and iot.

lan is the trusted local network for wifi and ethernet - all of our personal computers, phones, servers, etc… go on this network. It is associated with SSIDs for the 2G and 5G radios that the router has.

wan is obviously the rest of the internet outside of any of the local networks.

iot is the network onto which “smart home” devices go. Pretty much any smart device that I don’t trust. This is associated with a separate “IoT” SSID for the 2G radio.

I don’t trust these devices not to send stuff out to the internet that I don’t know about. I also don’t trust these devices not to try to get OTA updates that I don’t want them to have.

The iot interface/zone consequently gets quite a few firewall restrictions around the connections things in it can and cannot make to the other zones.

These are the default settings on interest for the zones

| Zones          | Input  | Output | Forward |
| lan -> wan     | accept | accept | accept  |
| lan -> iot     | accept | accept | accept  |
| iot -> NOTHING | reject | accept | reject  |

I always have to remind myself that these “directions” refer to the core router itself. So Input means packets into the router from the zone, Output means packets coming out of the router into the zone, and Forward means packets passing from the left zone, through the router, out to the right zone (where there is one).

So as I understand it this setup means that:

Devices in lan, as the trusted part of the network, can do whatever they want. Connect into the router, receive connections out from the router, and have connections forwarded through to the internet wan and to the smart home stuff iot.

Home Assistant lives in lan, is connected to from wan for software updates and remote access (yes I know I should use a VPN), and connects to devices in iot to do smart home things.

However the iot devices cannot make connections to the router (by default), cannot have their packets forwarded anywhere but can receive packets out from the router.

There are two holes I have found that I need to poke in the firewall from iot into the router (i.e. Accept input). DHCP (UDP ports 67 & 68) must be opened for smart home devices to get dynamic IP addresses. DNS (UDP port 53) must be opened for some reason that I cannot remember (perhaps to resolve *.lan names on the local network?). This is all that devices on the iot zone are allowed to do currently.

The Problem

Now I have MQTT devices.

These work in a different way to other devices I have on the network, which either have a Home Assistant driven integration (so it works with the existing rules) or use Zigbee.

Those MQTT devices need to connect from the iot zone to Home Assistant in the lan zone, and ideally only Home Assistant.

The challenge I have yet to solve is how to poke the right holes in the Firewall to make this all work.

TODO: find the Youtube video where I originally set all this up

The Solution

First Attempt

My first attempt at a firewall rule looked like this

| Name                | MQTT                                  |
| Protocol            | TCP, UDP                              |
| Source zone         | iot                                   |
| Source address      | any                                   |
| Source port         | 1883                                  |
| Destination zone    | lan                                   |
| Destination address | {{Home Assistant's local IP address}} |
| Destination port    | 1883                                  |
| Action              | accept                                |

I don’t think both TCP and UDP are needed, I think I added them both to have the best chance of it actually working. We can refine this later.

I use Source address | any because there will probably be more and more MQTT devices over time, and because their IP addresses will be dynamically assigned.

Because we have Source zone | iot and Destination zone | lan this defines a forwarding rule, i.e. to forward connections initiated from iot in to the lan zone.

I have set both Source port and Destination port to the standard HTTP MQTT port 1883.

Of course, this did not work. Anyone with a basic knowledge of how IP* networking works will see immediately why. I saw it immediately when I came back to look at this rule a few weeks after my first attempt to setting it up.

* if I remember well, ports aren’t a part of IP itself, but added “on top” by UDP/TCP, but since most common IP stuff uses one of those protocols anyway I won’t worry too much about these finer points

Testing

We’ll use mqttx to test, because it will make testing from the laptop which is being used to edit the OpenWRT configuration super easy.

To test this, I switch to the ‘IoT’ SSID on the laptop and run

export MQTT_USER=REDACTED
export MQTT_PASSWORD=REDACTED
mqttx sub -t '#' \
  -h homeassistant.lan -p 1883 \
  -u $MQTT_USER -P $MQTT_PASSWORD 

This is basically a command to subscribe to all topics on the MQTT broker.

With no hole poked in the firewall, i.e. no rule or an incorrect rule, we’ll get output like

✖ Error: connect ECONNREFUSED 10.10.10.133:1883
✖ Connection closed

Second Attempt

When an TCP or UDP connection is initiated from device A to device B the port to connect to on device B is specified as part of the URL.

Here it’s an MQTT smart home device (A) to Home Assistant (B), and for MQTT in this case the URL might look something like http://homeassistant.lan:1883/ where 1883 is the port number.

However device A also has a port used as part of the connection. This port is almost never specified in application layer software**. It’s almost always randomly chosen from the very high port numbers, like 30000+ (I’m not sure if there’s an actual specification for this).

** I’ve certainly never seen it specified, though no doubt there are use cases where this is needed?

In any case, this means the Source port that I input makes no sense. The Source port will be chosen randomly each time a connection is established from the smart home device to Home Assistant, and this will never be as low as 1883.

I think this should be our specification, with the change highlighted by an arrow

| Name                | MQTT                                  |
| Protocol            | TCP, UDP                              |
| Source zone         | iot                                   |
| Source address      | any                                   |
| Source port         | any   <----------------               |
| Destination zone    | lan                                   |
| Destination address | {{Home Assistant's local IP address}} |
| Destination port    | 1883                                  |
| Action              | accept                                |

When testing this corrected rule, we’ll see some response from the MQTT broker, showing that the rule gives the behaviour we want

✔ Connected
✔ Subscribed to #
topic: ... bunch of messages

Third Attempt

It seems that MQTT is supposed to use TCP on port 1883 not UDP, so it should be possible to reduce the allowed protocols down to just TCP.

This would be the new rule setup, change highlighted with an arrow.

| Name                | MQTT                                  |
| Protocol            | TCP   <----------------               |
| Source zone         | iot                                   |
| Source address      | any                                   |
| Source port         | any                                   |
| Destination zone    | lan                                   |
| Destination address | {{Home Assistant's local IP address}} |
| Destination port    | 1883                                  |
| Action              | accept                                |

Repeating the test - switching to the IoT SSID and attempting to subscribe to all topics with mqttx - produces the same results (a successful connection)

✔ Connected
✔ Subscribed to #
topic: ...

This is confirmed with a quick test of one of the smart home devices, which can still be controlled from Home Assistant.

So that’s it I think - this is how to allow MQTT devices in an isolated (restricted) firewall zone to talk to the MQTT broker running on Home Assistant in a more trusted firewall zone.