Hardening Linux for Production Use

Special Thanks: This article was made possible by support from Linux Professional Institute

A standard server or minimal install (CentOS) provides you with a secure server system, but its production usefulness is very limited. To make a server production ready, you’ll have to install software such as web services, DNS, DHCP, Samba/CIFS, print services, and databases. Because these services communicate over the network, security vulnerabilities are an inevitability. Exposing any service provides an attack vector. However, there are actions you can take to protect your systems and your data from attacks.

There are too many utilities available, both free and commercial, to help secure your systems. This article provides you a brief overview of the most common methods and utilities to help you maintain system security. Please remember that security is a best effort and that no system is 100 percent secure, but using these hardening suggestions will make it more difficult for malicious actors to compromise your systems.

Low-Hanging Fruit

At a minimum, you must prevent easy hacks on your system by updating it as soon as it comes online and then at least weekly thereafter. Developers release security and performance updates daily. Systems must be kept up to date, or they are vulnerable to attack. The other minimum barrier to attack is to enforce complex passwords. Don’t allow users to use simple, dictionary passwords on systems. Additionally, users should be forced to change passwords every 90 days and prevented from reusing the same password.

You can also minimize local (insider) hacks by logging off when you leave your system unattended. This rule applies to servers, workstations, and SSH sessions alike. This simple act will prevent any malicious user passing by your unlocked computer from stealing root access.

Don’t minimize the obvious protections such as regular backups and physical security. Performing backups is a security method that many administrators overlook. Destructive malware that might change critical files can be easily restored from a backup to their former states. Limiting physical access to systems is a good deterrent for those who want to steal directly from a system with a USB drive or who want to see if an administrator has remained logged in to a console. Only administrators need access to areas where servers are housed.

Multifactor Authentication

Multifactor authentication (MFA) is the best line of defense against password cracking by guessing, brute force, or dictionary attacks. Adding a second factor to logins is a very strong deterrent for account hacks. However, MFA isn’t 100 percent foolproof as some hackers have devised ways to phish your MFA tokens. Before logging in, you should always be aware of where you are entering your passwords and any codes or tokens for additional factors. MFA deters most low-level hackers. Professional hackers and advanced persistent threat (APT) groups will resort to phishing and other social engineering methods to acquire a token, code, or one-time password (OTP) from you.

The answer to these so-called soft tokens and their vulnerabilities is hardware tokens, such as YubiKeys, which are USB-based hardware tokens that require fingerprint verification as the second factor. These hardware tokens require something that you are (fingerprint) rather than something you know (password, passphrase, or picture recognition) or even something that you have (a hardware token that generates random numbers).

Pruning Unused Services

When it comes to system services, less is more – meaning that you should run as few system services as possible while still maintaining a productive environment for yourself and your users. I’ve always used the “one service per server” rule. Virtual machines make this practice a lot more economical than purchasing one physical system per service. One system per service is a good idea, because, if your Samba/CIFS server is compromised through some exploited vulnerability, your other servers that are not running Samba/CIFS will be protected – that is if you’ve implemented MFA.

You can begin with a minimal CentOS installation, which provides one external service: SSHD. An external service means one that is available to other hosts on the network as a service or listening port. Whichever distribution or installation you choose, you should perform a quick test to check which services are set to automatically start and which ones are listening for network connections.

There are two commands that provide you with the information required for making decisions about which services and protocols to trim and which ones need protection and monitoring. The first is chkconfig (Listing 1), which checks the runlevel information for system services.

$ chkconfig
netconsole   0:off   1:off   2:off   3:off   4:off   5:off   6:off
network      0:off   1:off   2:on    3:on    4:on    5:on    6:off

The second command, nmap, scans a system for open ports. The nmap command can be used locally or remotely. You should do both. Run an nmap scan locally to determine which services are network-facing and which ones are localhost only. Performing the same scans remotely and locally will help determine if there are any rogue services running on your systems. It will also help you to protect ones that are legitimate but may be overlooked by other methods.

The following nmap commands (Listing 2) work both locally and remotely. For remote scans, use the remote host’s IP address and localhost for local scans.

# nmap –sT IP_Address
Remote: 
# nmap -sT 192.168.1.59
Starting Nmap 6.40 ( http://nmap.org ) at 2019-05-29 16:08 CDT
Nmap scan report for 10.50.48.59
Host is up (0.74s latency).
Not shown: 999 filtered ports
PORT   STATE SERVICE
22/tcp open  ssh
Nmap done: 1 IP address (1 host up) scanned in 60.18 seconds

Local:
# nmap -sT localhost
Starting Nmap 6.40 ( http://nmap.org ) at 2019-05-29 16:07 CDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0016s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 998 closed ports
PORT   STATE SERVICE
22/tcp open  ssh
25/tcp open  smtp
Nmap done: 1 IP address (1 host up) scanned in 0.15 seconds

The nmap command in Listing 3 listsboth UDP and TCP ports for a host. You may replace localhost with the IP address of the remote host you want to scan.

# nmap -sTU localhost
Starting Nmap 6.40 ( http://nmap.org ) at 2019-05-29 16:25 CDT
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0014s latency).
Other addresses for localhost (not scanned): 127.0.0.1
Not shown: 1997 closed ports
PORT   STATE         SERVICE
22/tcp open          ssh
25/tcp open          smtp
68/udp open|filtered dhcpc
Nmap done: 1 IP address (1 host up) scanned in 1.42 seconds

The lsof (LiSt Open Files) command (Listing 4) displays a list of processes and to which ports they are bound.

# lsof -i
COMMAND   PID   USER  FD   TYPE DEVICE SIZE/OFF NODE NAME
chronyd  2429 chrony   1u  IPv4  20062      0t0 UDP localhost:323
chronyd  2429 chrony   2u  IPv6  20063      0t0 UDP localhost:323
dhclient 2766   root   6u  IPv4  22028      0t0 UDP *:bootpc
sshd     2959   root   3u  IPv4  22893      0t0 TCP *:ssh (LISTEN)
sshd     2959   root   4u  IPv6  22909      0t0 TCP *:ssh (LISTEN)
master   3207   root  13u  IPv4  23647      0t0 TCP localhost:smtp (LISTEN)
master   3207   root  14u  IPv6  23648      0t0 TCP localhost:smtp (LISTEN)

The netstat command (Listing 5) lists listening TCP and UDP ports for a system. Those listed as localhost can only be reached on the local system, but those listed as 0.0.0.0:<protocol> are accessible over the network. You can’t eliminate all listening ports, because doing so would render your system useless as a server, but these commands provide you with a snapshot view of potential entry vectors to your system by malicious actors.

# netstat -l 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address     Foreign Address    State
tcp        0      0 0.0.0.0:ssh       0.0.0.0:*          LISTEN
tcp        0      0 localhost:smtp    0.0.0.0:*          LISTEN
tcp6       0      0 [::]:ssh          [::]:*             LISTEN
tcp6       0      0 localhost:smtp    [::]:*             LISTEN
udp        0      0 0.0.0.0:bootpc    0.0.0.0:*
udp        0      0 localhost:323     0.0.0.0:*
udp6       0      0 localhost:323     [::]:*
raw6       0      0 [::]:ipv6-icmp    [::]:*             7

Controlling Root User Access

System administrators know the power of the root user account. And that power is absolute on a Linux system. There is nothing the root user cannot do, including removing or changing every file on the filesystem. But root access doesn’t simply apply to logging into a system as root – in fact that is a privilege that no one should have. Only a console root login should ever be allowed. In other words, no root access across the network via SSH or any other protocols. Root user access is typically controlled by a super secret password that only a few privileged individuals know. However, there is also root access that can be used via the sudo command. sudo is the standard method that system administrators use to issue commands as root but without the responsibility of being the root user.

The sudo command is generally limited to system administrators. It keeps everyone on a system honest in that sudo must be used before each command that normally would be issued by the root user account. For example, only the root user can stop and start network interfaces:

# ifdown eth0
# ifup eth0

If a user has sudo access, she can issue this command as:

$ sudo ifdown eth0
Password:

The sudo privileged user must enter her own password to prove she is indeed the owner of the account she is using and that she has sudo access. The value of sudo is that commands are entered purposefully using the sudo command and only one command at a time. Once the privileged user has issued a sudo command and prompted for her password, the authentication remains for five minutes only, which is another way of keeping everyone honest by having privileged users reauthenticate periodically.

Alternatively, if a system administrator must become the root user to perform a series of tasks that requires the root user’s environment, she can become root by using su -:

$ su –
Password:

This time the password that’s required is not the user’s password but the actual root user’s password. The advantage of becoming root is that the user then is root and has root’s environment, which is not available to users who use the sudo command. It is imperative that the actual root user’s password be kept secret and only shared with those who really require it. It should also be changed often, which translates to every 30 days for standard production environments.

If administrators have more than a handful of systems to manage, this process can be a bit tedious. The use of LDAP, Active Directory, or NIS+ is recommended in these cases to decrease the number of systems on which the root user’s password must be changed. Having a centralized management interface can also prevent mistyped passwords during a password change event.

Preventing Service Account Logins

Accounts that don’t require shell access should have no standard shell access in the /etc/passwd file. Only actual interactive users and the root user should have a real shell, such as /bin/bash, /bin/ksh, or /bin/csh, for example. All other accounts should either have /bin/false or /bin/nologin for their shell designations, which means that they cannot log in to the system interactively.

Certain accounts will have “pseudo” shells, such as shutdown, sync, and halt, which when used perform those functions. They’re still around for legacy reasons, which were used to allow non-root users the ability to halt or shutdown a system. Unix tradition is that you issue the sync command twice before reboot or halt (shutdown) to write unwritten data to disk. This isn’t necessary anymore, because the system does this itself prior to reboot or halt.

Denying Network Access to the Root Account

It’s easy enough to fix this one, because root logins are disabled by default. Some system administrators enable this feature, including myself, but it’s generally not a good idea. The one possible exception to this is if you have a hardware token and multifactor authentication setup in your environment. But for 99.99 percent of us, leaving this option disabled is exactly what you want to maintain a higher level of security.

The reason for denying network access to the root account might not be immediately obvious. Since most accounts are only protected by a password, root account compromise is only a guess or a brute force attack away. Because of this, you should make becoming root as difficult as possible without impeding productivity.

Auditing Security

There are many applications and application suites available to perform regular security audits and track changes to critical files. In addition, you have the power to perform a quick security audit on your system by using the find command. This is by no means a deep security audit but just a quick and dirty scan of your system to find files that have the SUID/SGID permission set that shouldn’t. Having these permissions bits set on files owned by root is a serious vulnerability, because they can allow users who do not have root access to execute commands as root and potentially initiate a root account compromise.

What you’re looking for is files that have permissions set similar to the following:

-rwSr-xr-x (SUID) or –rw-r-Sr-x (SGID)

The following find command locates all files with the SUID permission set for root:

# find / -type f -perm -u=s -ls

Here are some examples of files that match this permission:

12734848   44 -rwsr-xr-x  1 root root  44320 Mar 14 05:37 /usr/bin/mount
12734863   32 -rwsr-xr-x  1 root root  32208 Mar 14 05:37 /usr/bin/su
12734867   32 -rwsr-xr-x  1 root root  32048 Mar 14 05:37 /usr/bin/umount
13025626  144 ---s--x--x  1 root root  147392 Oct 30  2018 /usr/bin/sudo
12777317   60 -rwsr-xr-x  1 root root   57664 Nov 20  2018 /usr/bin/crontab
13037754   28 -rwsr-xr-x  1 root root   27832 Jun 10  2014 /usr/bin/passwd

Similarly, the find command to locate files that have the SGID permission set is as follows:

# find / -type f -perm -u=s -ls

And here are some examples of matching files:

12649555  16 -r-xr-sr-x 1 root tty     15344 Jun  9  2014 /usr/bin/wall
12734873  20 -rwxr-sr-x 1 root tty     19624 Mar 14 05:37 /usr/bin/write
13015059 376 ---x--s--x 1 root nobody 382240 Apr 10  2018 /usr/bin/ssh-agent

As you can see from you own listing, the number of files with SGID set is far fewer than those with SUID set. Some system files require SUID/SGID permission to be set, but there are very few of them. You need to perform a baseline audit of your systems upon initial installation and then track those changes periodically to be sure that no rogue programs or users have exploited this security flaw.

Conclusion

There are several other Linux hardening methods such as PAM, iptables, enabling SELinux, removing any X display managers, and regular port monitoring, but these are outside the scope of this article. I may revisit them in future installments individually. As stated previously, you can’t remove all network access to your servers, because that defeats the purpose of having a server. But now you have a small but powerful toolbox of utilities and techniques that will help you to keep your systems safer.

© Nikolay Okhitin, Fotolia.com