detecting ssh tunnels

Forward SSH tunnels can be used to pivot around a network, aiding in data exfil, recon, and further compromise. Forward tunnels are also extremely useful for normal purposes though so blocking them outright would most likely be a hindrance to engineers. Using osquery, you can pretty quickly and generically detect most instantiations of SSH forward tunnels and then alert on them.

what to look for

A forward tunnel instantiation might look like this: 

ssh -L2022:devel-env01:22 devel-jmp01

For people not familiar with SSH tunnels this takes port 2022 on the local machine and forwards it to port 22 on devel-env01 though devel-jmp01. This would be a standard command for an engineer whose development environment is protected from corp via the use of jump hosts.

However, you probably have a good idea of which employees should be opening SSH tunnels to jump hosts; it would be strange for sales to directly access the development environment for example. Using osquery you can easily achieve pretty good detection for SSH tunnels with just a couple joins.

listening ports

When you open a forward tunnel, SSH starts listening on the port specified at the beginning of your -L statement, in this case 2022. osquery has a listening ports table we can use to try and find these.

+------+-------+----------+--------+-----------+
| pid  | port  | protocol | family | address   |
+------+-------+----------+--------+-----------+
.
.
.
| 9361 | 2022  | 6        | 10     | ::1       |
| 9361 | 2022  | 6        | 2      | 127.0.0.1 |
.
.
.
+------+-------+----------+--------+-----------+

So in the abbreviated output above we can see that the SSH tunnel port does show up, but so will a lot of other things and we won't know what port the user decides to forward so we can't filter on that. To fix this, we join against processes using pid, this gives a wealth of information on every process in here.

Note, this query is also useful for figuring out which process is using a particular port generally when you get the dreaded BIND Failed: Port in use error.

osquery> select pid, port, path, cmdline from listening_ports join processes using (pid);
+------+------+--------------+-----------------------------------------------+
| pid  | port | path         | cmdline                                       |
+------+------+--------------+-----------------------------------------------+
| 9361 | 2022 | /usr/bin/ssh | ssh -L2022:devel-env01:22 devel-jmp01 -p 8389 |
| 9361 | 2022 | /usr/bin/ssh | ssh -L2022:devel-env01:22 devel-jmp01 -p 8389 |
+------+------+--------------+-----------------------------------------------+

This is looking pretty good, we know the path of the executable now, the full command line, and the pid. The last thing to do is to only select command that look like SSH commands.

select pid, port, path, cmdline from listening_ports join processes using (pid) where path='/usr/bin/ssh';

Next steps

There many other fields you can get, username for example using processes' UID joining against users. Another useful column might be the remote address of an SSH tunnel. Using SQL string functions would be difficult as commands can have different options and orderings but we can solve this problem with another join, this time against process_open_sockets using PID. Here are a list of things which might or might not be useful to you depending on your use case.

  • Parent path
  • Parent cmdline
  • Username
  • Remote address
  • Local port (but there can be more than one per command. Be careful.)
  • Remote Address