A study in detecting network intruders

Uninvited Guests

© Lead Image © Chrisharvey, Fotolia.com

© Lead Image © Chrisharvey, Fotolia.com

Article from Issue 224/2019

The nightmare of any admin is a user who can't resisting clicking on an unknown attachment labeled Application.exe. This article draws on a real-world example to show how you can use built-in Linux resources to detect unauthorized traffic that might have been invited in by a trigger-happy user.

If a network monitoring process detects malware, a system administrator needs to identify the affected systems and contain the damage. A customer from a Microsoft-heavy environment recently came to me with a problem. Many of the clients on his network were infected with malware. Because the malware was quite sophisticated, the virus scanner did not help detect it. He wanted my help with finding all the infected clients.

To make things even more exciting, several versions of the malware appeared on the network. The variations in form meant that it was not easy to detect the malware using simple pattern matching with a filesystem scan. Fortunately, although the attackers were good at infiltration, they were not very skilled at connecting back to their Command and Control (C&C) server. This article describes our investigation and offers some tips on how to respond to similar attacks.

The Malware

The first generation of the malware attempted to connect to Telnet servers in Asia via TCP port 23. Since the Telnet protocol is hardly ever used on today's networks, this attack was quickly noticed. A single tcpdump [1] command directed at the gateway returned some initial hits:

tcpdump -i eth0 tcp port 23

The packets had some peculiarities in the fields of the Telnet header, which were revealed by an analysis with Wireshark [2]. Headers are typically used to negotiate things like terminal emulation or even X forwarding. In the case of the malware, the headers contained nonsense, but we always found the same 4-byte sequence at the beginning of the packet.

The second generation of the malware tried to disguise itself as a DNS packet. This time it used UDP port 53, but it wanted to contact servers on the same Asian network. The desired DNS traffic on the network moved back and forth between clients and the Active Directory domain controller, because the domain controller machine was acting as a DNS server. Undesirable traffic was filtered out using tcpdump:

tcpdump -i eth0 port 53 and not host IP_address_of_AD_controller

tcpdump detected the traffic that went to port 53 but not to the Active Directory server. The analysis with Wireshark also showed that the sequence already known from the Telnet packets reappeared at the beginning of the packets. In addition, the packets were again not valid DNS packets.

The last version of the malware finally tried HTTP port 80, but it was caught due to invalid HTTP packets. In addition, starting in version 2 of the malware, DNS queries had already been noted against different hosts in three different subdomains. The victim of the attack had several sites, so there were several Internet gateways.

The Search

Although most of the network was Microsoft, the gateways and DNS servers fortunately ran on Linux. I was thus quickly able to launch a search for clues.

Many companies allow their employees outgoing access to the Internet via the company gateway and do not log the permitted traffic at the firewall. This approach has some disadvantages: If the logs had been run through a log management solution such as ELK [3] or Splunk [4], it would have been very easy to narrow down the computers because traffic via ports 23 or 53 is fairly atypical.

If web access is first routed through an internal enterprise proxy, computers that do not use this proxy but look for a direct connection to the outside through ports 80 (for HTTP) or 443 (for HTTPS) are at least suspicious.

If the firewall also blocks outgoing connections that do not originate from computers with known services (proxy, DNS server, mail server, and so on), the analyst can simply search for rejected packets that use an address outside the destination. A perfect world should have hardly any packets of this type.

Clients that do send these packets are either configured incorrectly (depending on destination and port) or should be inspected more closely by the admin.

Wireshark: The Forensic Scientist's Friend

Wireshark [2] is the industry's standard tool for investigating network incidents from logged packets. The large number of protocol modules available with Wireshark – for application protocols such as HTTP or SMTP – allows for more in-depth analysis.

At the same time, Wireshark massively supports the admin in finding a needle in a haystack. The software makes it possible to set filters for each protocol field. Figure 1 shows a Wireshark display filter for all HTTP packets that use the POST method and go to the target host Wireshark also offers statistical functions that can help with your analysis.

Figure 1: Wireshark with a display filter for HTTP packets that use the POST method.

Security experts advise against operating Wireshark with root privileges. Previous examples of errors in software's packet analysis modules have allowed attackers to inject malicious packets into the traffic. See the box entitled "Avoiding Root" for more on running Wireshark without root privileges.

Avoiding Root

The best way to make use of Wireshark without assigning it root privilege is to first create a .pcap file with tcpdump and then evaluate it with Wireshark using an unprivileged user ID or a named pipe. Create a pipe with the command:

mkfifo /tmp/wirepipe

Then call Wireshark using:

wireshark -k -i /tmp/wirepipe

Now you can now feed packets into this pipe using the root account and evaluate them using non-root Wireshark. Use the following command, to which you can add a filter for packet types if needed:

tcpdump -w /tmp/wirepipe

Alternatively, you can use the dumpcap command:

dumpcap -w /tmp/wirepipe

Dumpcap [5] understands some options. For instance, you can specify the interface from which the packets should come. The -f option lets you specify a filter in tcpdump syntax.

The Conversations menu entry is the first port of call when searching for infected systems. You may also be able to use this option to find suspicious endpoints outside your own network. If you know the IP address of a control server, in the ideal case, you can quickly find all systems that communicate with it.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Tshark

    The simple and practical Tshark packet analyzer gives precise information about the data streams on the network.

  • Capture File Filtering with Wireshark

    Wireshark doesn’t just work in real time. If you save a history of network activity in a pcap file using a tool such as tcpdump, you can filter the data with Wireshark to search for evidence.

  • Security Lessons

    Building a network flight recorder with Wireshark.

  • Wireshark

    If you know your way around network protocols, you can get to the source of a problem quickly with Wireshark.

  • An Essential Sys Admin and Security Tool

    Wireshark fills the gap between security and system administration for those who need to know more about what’s flowing through the wires or over the airwaves in the corporate network.

comments powered by Disqus

Direct Download

Read full article as PDF:

Price $2.95