Using Reverse Tunnels to Save the Day

A while ago I set up a system for accessing and viewing security cameras remotely. Typically, these security cameras do not provide any form of encryption - even the password is in the clear - so rather than let them hang out on the Internet for everyone to poke at (https://www.shodan.io) I configured them to only be accessible from the LAN side. Therefore, to access them from the WAN side I would have to first have to find a way to get into the network from my remote location. VPN was too heavy - I didn't need that much  networking capability - so I decided to simply use SSH with tunneling. To accomplish this I set up a Raspberry PI running SSH which would allow me to tunnel into the remote location and view the various feeds (or configure the system) securely; the modem was also setup to Port Forward the SSH requests to that Pi.

 Breaking the System

This setup worked well for about 6 months - and then our network started acting up. For whatever reason the modem stopped talking on the WAN side - and because I wasn't checking the cameras it took me about 10 days before I realized I had no network access.  We had the techs come out; they rewired it and it worked for a day or two. So we had another tech come out and he laid out all new wiring and then attempted to change something in the modem. Seeing that the default password didn't work and not wanting to be at the location any longer than he had to be he did a HARD RESET on the modem; essentially blowing away all our configuration and leaving me unable to connect to the system.

It took a few days before I could get someone out there; I walked them through the process of setting up the Port Forwarding again (a great use of FaceTime to see what they were seeing on the screen and walking them through the various options and settings).  In the interim, I got a chance to investigate what could be done in the future to mitigate such a failure.

Reverse SSH Tunnels to the rescue

I was already aware of reverse tunneling, but hadn't come up with a good use case for it until now. In essence since I'm not physically present at the remote system system, I can't really control what's going on there.  It therefore makes sense that the remote systems should have the ability to, as long as it has an internet connection, establish a connection with my system just to say hello. So even if something glitched on the other end and I could no longer connect to it, as long as it could connect to me, it would provide me with a path with which I could configure the system and get full access. If setup successfully, it would provide me with enough of a beachhead from which I could gain additional access.

Security Concerns

     It is a very rare occurrence that I allow any remote system to connect to my home network so allowing the remote PI to call home would require some special considerations.

     The first thing I did was to disable root access and password authentication on the SSH local server. The only way to connect to it would be through a crypto key.

     Secondly, a new user was created on the SSH server specifically for this remote client.  This user was setup with very limited access.

     Lastly, when connecting to my system, the client would create the reverse tunnels but would not be provided with a terminal on the local system (so no commands could be executed from the remote device on the local SSH server).

     This last bit was required because the remote system did not have a password associated with the authentication key. This was necessary to ensure that the system could connect to my system even after a reboot when nobody would be around to add the key to the ssh-agent (which is particularly difficult as the remote Pi is running headless - no keyboard or monitor).

Regular Scheduled Communications

     The connection process had options;  I could set up the remote system to establish and keep a connection 24/7 or call in at regular intervals.   The second approach was chosen as it would not tie up my local system but it would also give me the benefit of a heads up when the WAN IP of the remote system would change.   This was a backup to using a Dynamic DNS service such as No-IP.

     Along with the daily connections,  the local system was updated to send me a text message every time the remote system connected.   After a few days of getting these messages (every night at 7pm) I quickly learned to expect them.   If a connection were to not happen at that time, I would immediately know that the network may be having issues or the remote system may not be able to connect to my system and quickly dispatch someone to investigate.

     So far this setup has worked really well and I should no longer be at the mercy of  a trigger happy tech.