The only port that I have open in my firewall is 22, SSH. Through one can access all other services by means of ssh tunnels, SFTP, scp etc.. I used to leave it at that, just disallowing root logins (from IP-addresses outside my LAN) as most of the time the attacker tried to use username root and I thought that at least that would void those brute force attempts.
But the SSH port gets hammered relentlessly around the clock and the log files grow to be humongus so I thought I’d try to do something at least try to limit the attacks.
So I started by using the iptables recent module to blacklist IP-addresses that tries to connect more than 4 times within 10 minutes to port 22 for 1 hour (i think). But there were still quite a lot of hammering going on as many different source IP’s were used, and, most annoyingly you could (and I did on multiple occasions) lock yourself out by starting more that 4 ssh sessions within 10mins (FileZilla for example quickly does this for you if you let it use multiple parallel connections (the default) when downloading files). But for a long time this worked well enough I thought.
The iptables commands to set this up is somewhat cryptic:
iptables -A INPUT -m recent --update --seconds 600 --hitcount 4 --name blacklist --mask 255.255.255.255 --rsource -j DROP iptables -A INPUT -m recent --set --name blacklist --mask 255.255.255.255 --rsource -j ACCEPT
Later, after having turned on TOTP two-factor-authentication on all of my online accounts with that option I wanted to enable the same for SSH logins to my machines and found that this was easily realized by utilizing google-authenticator-libpam.
All one had to do was to build and install the PAM-module and add one line to /etc/pam.d/system-remote-login, requiring pam_google_authenticator.so (nullok makes it so that if a user haven’t setup google-authenticator for his account the login is allowed).
auth required pam_tally2.so onerr=succeed auth required pam_shells.so auth required pam_nologin.so auth required pam_env.so auth required pam_unix.so try_first_pass auth required pam_google_authenticator.so echo_verification_code debug nullok auth optional pam_permit.so account include system-login password include system-login session include system-login
Then the user runs a google-authenticator program which generates a secret key, provides you with a number of backup keys (as per the norm), and stores information about this in a dotfile in your home directory that the PAM module will look for. The script even displays a QR-code in the terminal window so you easily can scan it into your phone’s TOTP app!
Then I asked myself if port knocking wouldn’t make a lot more sense that rate-limiting. Then the port would be completely hidden and practically no attack attempts would reach it. Port knocking is a technique whereby one make connections to a specific sequence of ports within a given time frame to open a hole in the firewall to the port you want to give access to to the knocking ip. I chose the iptables pknock module to enable this. It is not included in the main kernel tree but available as an external module. Using gentoo one can set XTABLES_ADDONS=”pknock” in make.conf and then emerge xtables. After setting this up i wondered why I had not just done this from the start, so easy and efficient.
A simple iptables rule that says that we have to make tcp connections to ports 1234, 5678 and 5555 in that order and within 10s to grant access to port 22 for 10min (access to setup connections, ssh sessions can last longer than that but that is handled by another conntrack rule that allows any established connections to continue.)
iptables -A INPUT -p tcp -m pknock --knockports 1234,5678,5555 --time 10 --autoclose 10 --name SSH --strict -m tcp --dport 22 -j ACCEPT
A simple knock-script that uses nc (netcat) utility to connect to the defined ports in sequence for 1s 1s apart.
#!/bin/sh for x in 1234 5678 5555 do nc -z -w 1 hades.eleusis.se $x &>/dev/null echo $x sleep 1 done
A rather annoying thing with this I notice is that if you build a new kernel and forgot to re-emerge (emerge @modules-rebuild) the xtables package, iptables-restore will fail with missing module error on startup and leave the firewall completely open when one boots one’s new kernel. This is not that easily spotted on a headless server machine. I would have thought it would have made more sense to leave it completely closed in such cases. It would be more secure and one would immediately notice that something was wrong and have the chance to timely remedy the situation. Hopefully I won’t forget again (or maybe I should look into the possibility to have iptables DENY instead of ALLOW by default and make the change).
Before reading that howto I had not thought kerberos was possible, or feasible, without a directory server. But afterwards I had realized that a setup like one I would have liked to have actually was not only possible, but pretty simply so, and without any questionable hacks.
I wanted to be able to ssh to any machine in the network without providing any passwords if a valid kerberos ticket was held and the kerberos ticket granting server accessible (which should be only from within my LAN). If trying to connect without ticket or access to the server (I’m actually not really sure about this, maybe access to server is not necessary if one only holds a currently valid ticket?) one would be asked for password and TOTP like before.
I solved this by emerging mit-krb5, setting up a most basic /etc/krb5.conf and a /etc/krb5.keytab with the host principal’s key) and emerging the pam_krb5 package and altering system-remote-login as so:
auth required pam_tally2.so onerr=succeed auth required pam_shells.so auth required pam_nologin.so auth required pam_env.so auth sufficient pam_krb5.so force_first_pass minimum_uid=1000 auth required pam_unix.so try_first_pass auth required pam_google_authenticator.so echo_verification_code debug nullok auth optional pam_krb5.so force_first_pass minimum_uid=1000 auth optional pam_permit.so account include system-login password include system-login session include system-login
The first pam_krb5.so row makes it so that if a user holds a valid kerberos ticket that is sufficient for the login to be allowed, force_first_pass makes it not ask for a password but only use one previously provided, if any, in this case that would be a ticket). No unix password nor OTP will be asked for in that case. The second almost identical row which is marked optional makes it so that if we don’t hold a ticket and move on to unix password and TOTP authentication it will try to use the entered unix password to automatically request a new ticket from the granting server, which potentially can save one from another kinit command with accompanying manual password entry, as long as the local unix and kerberos server passwords match. Otherwise one has to update the local password to match the servers (as we are using kerberos without any directory server or such, even if the pam module can keep the passwords synchronized when changing on one host the change of local password will not automatically propagate to other hosts.).
This works nicely with the Windows Subsystem for Linux. You can install the kerberos packages, copy the /etc/krb5.conf and use kinit to authenticate within your WSL session. Very neat.
So now my server support kerberos authentication within the LAN so we can ssh freely between the hosts without providing credentials other than the ticket that is granted at first login on any machine. A very convenient thing is that one can specify which user one would like to authenticate as. The root user has no kerberos principal (only local logins) and is not permitted to log in via SSH anyways. But by executing “kinit mignon” the root user can be granted a ticket which provides SSH access as mignon, kind of neat. So instead of “ssh mignon@hades” and having to give password and OTP one can use “kinit mignon && ssh mignon@hades” and only give the password and keep the phone in the pocket. Access from outside of the LAN requires port knocking before a connection can be made, and then password and OTP like before.