Managing port scan results with Dr. Portscan

The Doctor Is In

Article from Issue 155/2013
Author(s): , Author(s): , Author(s): , Author(s):

Regularly scanning the ports on your own network prevents intruders from sneaking in, but if you have dozens or hundreds of servers, you'll need professional help: Dr. Portscan to the rescue.

One of the main problems with regular port scans on a large network is the overhead: Depending on the number of servers, the list of open ports can quickly scale to a point that is impossible to track manually. Watching the whole network requires careful, and thus very time consuming, scrutiny of the scan results, and in many cases, you will not have the staff to expend the necessary attention. Additionally, many organizations lack a precisely defined, centrally available target state with which to compare the results, especially if individual departments are allowed to set up their own servers and services.

If you use different scanner locations to analyze your networks (e.g., one location within the organization and one across the Internet), you also face the problem of interpreting different lists of results. Inconsistencies arise because of connections blocked by firewalls and scans occurring at the different times. Reconciling the results requires manual evaluation.

An open source tool known as Dr. Portscan can help you automate distributed port scans and their evaluation. Dr. Portscan is a delta reporting tool that can assist almost any port scanning tool in analyzing parallel or staggered scans of arbitrarily overlapping networks. The results of these scans are automatically aggregated and compared. Changes discovered in the process can be added to reports for various target groups or passed into scripts as parameters. This approach opens up advanced diagnostic options, including the ability to compare the results with a defined target state.

In this article, we examine the general structure of distributed port scan infrastructures, the architecture and functionality of Dr. Portscan, and the basic steps for commissioning Dr Portscan to keep watch over your network.

Distributed Port Scans

If you are looking for authoritative results from the port scanning process, you'll need to ask yourself the following questions:

  • What port scanning tools will you deploy? In addition to the classic Nmap tool, you will find numerous other free and commercial scanners. Security and network management tools can deliver information about open ports either explicitly or implicitly and thus serve as data sources.
  • How many systems will run port scans? Multiple port scanners at different locations let you differentiate between internal and external views of the networks you analyze.
  • How often will you launch scans from these systems? You need to take into consideration that some Internet-based port scanning services only envisage fixed intervals.
  • What is the scope of the scan? External port scans are typically slower than internal LAN scans. For this reason, you might want to restrict scans to individual subnets, or some other subset of all possible TCP/UDP ports.
  • How can you validate the ports detected as being open; that is, how can you compare this with a target state? Besides an initial scan with subsequent manual checks, you might be able to retrieve a target state documented in an asset database or a configuration management system.

Figure 1 shows a somewhat simplified view of a typical scenario for using Dr. Portscan.

Figure 1: Sample scenario for distributed port scans with Dr. Portscan.

The scripts on which Dr. Portscan is based let you compare an arbitrary number of port scan result lists created by different computers. (The popular Ndiff tool lets you compare exactly two Nmap scan results.)

Figure 2 shows the system architecture of Dr. Portscan with its most important components. Data sources (1) can be any port scanning tools. Agents (2) are responsible for converting the product-specific scan results into a uniform format. The input agent for Nmap, for example, contains parsers for XML and grepable Nmap output as of version 5. The overhead for implementing additional input agents depends on the port scanning tool you need to integrate, but it is typically reasonable because the code segments you need to modify simply consist of a list of the detected IP addresses and open ports. The delta reporter (3) transfers the results delivered by the input agents to a database (4), which stores and evaluates the results of all port scans. The information generated in the process indicates changes compared with recent scans.

Figure 2: Dr. Portscan system architecture.

Freely definable output agents (5) can provide an overview, create reports about changes on the individual subnets, mail information to the administrator in charge, or trigger arbitrary scripts to retrieve more information about the service hiding behind the newly opened port.

Dr. Portscan is designed to serve as a centralized relational database, initially to collect the results of the decentralized port scans in a uniform format. Because the input agents, the delta reporter, and the output agents are all implemented in Perl, you can use a wide spectrum of open source and commercial database products. The minimal sample configuration relies on SQLite. This sample configuration makes it easy to try out Dr. Portscan without having to set up a database server and a new database. For larger installations, it makes sense to use MariaDB, PostgreSQL, or some other relational database management system for which the Perl DBI modules are available.

The database stores all the information for every scanned machine and every port that has drawn attention to itself. The most important thing to watch is the changes compared with the last port scan run for the same entry. Table 1 shows the supported change types.

Table 1

Change Types

Type

Description

0

No change compared with last scan

1

New machine (IP address detected for the first time)

2

Machine no longer running

4

Changed DNS name

8

New open port, not previously open

16

Port now closed (port previously open)

32

Port opened again after being closed in the meantime

Data Sources and Input Agents

The port scanning tools (or data sources, as they are known in Dr. Portscan) are initially configured independently of the input agents. For example, you could automatically run nmap as a cronjob once a day on a Linux machine to scan all the ports in a network range equivalent to a legacy Class B network:

/usr/bin/nmap -v -v -oG - -p 1-65535 10.1.0.0/16 > /tmp/nmap-results.txt

Each data source is assigned to an input agent; its main task is to generate a list of all IP address and port pairs detected as open by the data source. The agent reads and parses the output from the scan tool. The preceding call to Nmap would return the following text output on a web server:

Host: 10.1.123.123 (www.example.com) \
Ports: 80/open/tcp//http///, 443/open/tcp//https///    Ignored State: filtered

The results from the input agent – a simple text file with an IP address, a port, the protocol details, and an optional timestamp in each line – are then either deposited for the delta reporter to pick up or pushed to another system (e.g., using scp). This intermediate step is often necessary because the SQLite database used by Dr. Portscan in the default configuration (without add-on software such as cubeSQL) is not accessible on the network and because other databases are often hardened against Internet access by a firewall, thus preventing direct entry of the results in a central database.

Delta Reporter, Output Agents, and Reports

The delta reporter first transfers the port scan results returned by the input agents to the central database. The reporter checks for each IP address and port pair, taking into consideration the Layer 4 protocol, whether an entry for the current scanner already exists, or whether a new entry needs to be created. When existing entries are updated, the appropriate change type is set.

After all the port scan results have been imported, the output agent can evaluate the central data repository. Because changes compared with the status quo are of primary interest, output agents typically only look for database entries with specific change types. To prevent an output agent from processing the same entry multiple times, the agent enters its own identifier in the corresponding column of the database table.

To suit your own needs, you should modify the number and type of output agents. Dr. Portscan offers a preconfigured output agent that gives you an overview of all changes since the last report, but you can restrict evaluation by the output agents in a targeted way. For example, you might just want to see the results for specific subnets or specific events (e.g., newly opened ports) to compare these results with a defined target state. The box titled "Cleanup Tips" describes some steps for ensuring a readable result.

Cleanup Tips

To avoid multiple notifications and false positives, take the following approach:

  • Ports are first reported to be open or closed when freely configurable scanner-specific thresholds are exceeded. These thresholds contribute toward stabilizing the port scan and help you avoid false alarms. After all, a scanned system can just happen to be unavailable when the scan runs, or scans off the web might not always reliably detect all open ports if more aggressive timing values are selected for the port scan tool.
  • For each report, a check is performed to see whether it has already been filed by other output agents with the same report target group or whether the same output agent has already reported based on the results of another scanner. In this case, you can completely suppress the repeat message or mark it as explicitly confirming a known state.
  • Discrepancies between the results of various scanners are typically legitimate and can be caused by different scan times, scan scopes, or scanner locations. These different views of the same system need to be processed adequately in the report, for example, by listing the current results for the different scanners each time the report is generated. The configuration assigns each scanner to a group, within which all sensors should have the same view to be able to detect inconsistencies reliably.

Changes to the scanner infrastructure can make it necessary to clean up the stored data from time to time. Optional scripts in the Dr. Portscan distribution can delete entries with very old timestamps, results from a specific scanner (if you are discontinuing its use), all entries for a specific network range, or all entries for an IP address (e.g., if you are replacing one machine with another that will use the same address).

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Intrusion Detection

    The Prelude security information management system receives both host- and network-based IDS messages and displays them in an easy web interface. We show you how to set it up.

  • Nmap Methods

    How does the popular Nmap scanner identify holes in network security? In this article, we examine some Nmap analysis techniques.

  • Nmap Workshop

    In "The Matrix Reloaded," Trinity uses Nmap to hack into the power grid to pave Neo's way to the architect of the virtual world. However, the port scanner is also ideal for more mundane purposes – such as discovering vulnerabilities in your domestic network.

  • Command Line: SANE

    Running your scanner from the command line offers greater control of tasks. We show you how to get started.

  • Nmap Scripting

    Nmap is rolling out a new scripting engine to automatically investigate vulnerabilities that turn up in a security scan. We’ll show you how to protect your network with Nmap and NSE.

comments powered by Disqus

Direct Download

Read full article as PDF:

Price $2.95

News

njobs Europe
What:
Where:
Country:
Njobs Netherlands Njobs Deutschland Njobs United Kingdom Njobs Italia Njobs France Njobs Espana Njobs Poland
Njobs Austria Njobs Denmark Njobs Belgium Njobs Czech Republic Njobs Mexico Njobs India Njobs Colombia