Advanced awk, sed, and grep

Awk, sed, and grep are essential tools for text processing, and their advanced features allow for complex data manipulation with minimal overhead. These utilities are indispensable for parsing logs, extracting configuration details, and automating repetitive tasks.

Consider a scenario where you need to analyze a web server log (/var/log/nginx/access.log) to identify the most frequent IP addresses accessing the server:

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr |head -10

In this command, awk extracts the first field (the IP address), sort organizes the addresses, and uniq -c counts occurrences. The final sort -nr ranks the results numerically in descending order, and head displays the top 10 IP addresses. This approach is both efficient and scalable, making it ideal for large logfiles.

Sed excels in stream editing, allowing you to modify text in-place without manual intervention. For example, you can replace all instances of http with https in a configuration file as follows:

sed -i 's/http/https/g' /etc/nginx/sites-available/default

The -i flag applies changes directly to the file, and the g flag ensures all occurrences on a line are replaced. This is particularly useful for bulk updates across multiple configuration files.

For targeted text searches, grep provides unmatched speed and precision. To extract only error lines from a system log while excluding debug messages, you can use:

grep -i "error" /var/log/syslog | grep -v "debug"

Here, the -i flag makes the search case-insensitive and grep -v excludes lines containing debug. Combined with other utilities, grep becomes a versatile tool for data filtering and extraction.

Scheduling

Task scheduling is vital for automation, ensuring that jobs like backups, updates, or log rotations run at specified intervals. The cron utility has been a traditional choice for scheduling, whereas systemd timers offer enhanced flexibility in modern Linux distributions.

To schedule a daily backup using cron, edit the crontab file:

crontab -e

Add the following line to schedule a backup script (/usr/local/bin/backup.sh) to run at 2:00am daily:

0 2 * * * /usr/local/bin/backup.sh

This format specifies the minute, hour, day of the month, month, and day of the week. You can verify scheduled jobs with:

crontab -l

Managing System Resources

Resource management is a cornerstone of system administration, ensuring optimal performance and quick resolution of issues. Commands like lsof, ps, and kill enable effective monitoring and control over system resources.

lsof (list open files) is invaluable for identifying processes using specific files or ports. For instance, to identify the process occupying port 80:

lsof -i :80

This command provides details about the process, including its PID, user, and associated files, which are critical for troubleshooting service conflicts.

The ps command provides detailed information about running processes. To display processes in a tree format, showing parent-child relationships, use:

ps -e --forest

This view is particularly useful for understanding dependencies or investigating rogue processes. To monitor resource usage, combine ps with sorting:

ps -eo pid,comm,%cpu,%mem --sort=-%cpu | head

This command lists processes by their CPU usage, making it easy to identify resource-hungry tasks. When processes become unresponsive, kill offers a straightforward way to terminate them. To gracefully stop a process:

kill -15 <PID>

The -15 signal requests termination, allowing the process to clean up before exiting. If the process ignores this signal, force termination with -9:

kill -9 <PID>

Combining these utilities into scripts allows for automated monitoring and intervention. For example, a script to restart a service if its memory usage exceeds a threshold might use ps to detect the condition, followed by kill and a systemctl restart command.