Linux Hack of the Week #12: Centralized Logging with Syslog

Always being aware of what’s happening on your organization’s systems is key to having a successful security strategy. One thing that helps immensely with this kind of visibility is using syslog. Syslog is supported by all enterprise hardware and all Linux OS’s. You can send messages from your firewall and servers to a centralized logging server for long term storage and analysis. SIM tools like Splunk and LogRhythm are great ways to visualize and organize the data. In this post, we’ll go a step further and demonstrate how to log all user commands in syslog for increased situational awareness.

In today’s example, I’ll use rsyslog on Fedora 28 and Raspbian. Lets dive in!

Syslog Server Setup

The first step is to see if everything is installed. Use rpm -q rsyslog to see if it is installed. Otherwise, install with dnf install rsyslog:

Next, edit /etc/rsyslog.conf and uncomment the two lines below:

# Provides UDP syslog reception

$ModLoad imudp
$UDPServerRun 514

You can also uncomment the TCP section right below those if you’d like to receive syslog over TCP:

Next, add configuration directives on where to store logs that are received to the bottom of the file. It’s best practice to store these on their own slice of disk to prevent filling up a partition and making the server unresponsive. To store these files, I use /data/logs:

$template TmplAuth, "/data/logs/HOSTS/%fromhost%/%PROGRAMNAME%.log"
$template TmplMsg, "/data/logs/HOSTS/%fromhost%/%PROGRAMNAME%.log"
authpriv.*   ?TmplAuth
*.info,mail.none,authpriv.none,cron.none   ?TmplMsg

Save your changes and exit. Be make sure the destination directory exists, mkdir -p /data/logs, and then restart rsyslog, systemctl restart rsyslog:

Before you start receiving messages from remote hosts, ensure your firewall rules allow in 514 on UDP/TCP:

Syslog Client Setup

Configuration on remote machines is much simpler. Simply edit /etc/rsyslog.conf and add this line to the bottom:

*.* @

Restart rsyslog with systemctl restart rsyslog.

At this point, logs should be going to your syslog server. However, we can do more. Say you have a production machine that users connect to, including support staff, and you need to track what your users are doing. You can update the bashrc so every command is logged to your remote syslog server.

Edit /etc/bashrc and add this line to the file:

export PROMPT_COMMAND='logger "$(whoami) : $(history 1 | cut -d" " -f 5-)"'

What exactly does this do? Bash has a built in environment variable called “PROMPT_COMMAND”. This gets executed before the prompt is displayed. Each time a user runs a command, a message is sent to the syslog process using logger which is stored in a file called username.log:

If we look on the syslog server, we’ll see that all of the commands the user has run have been recorded:

I hope this post helps you set up a centralized log server in your environment. Adding this increase situational awareness will greatly improve your overall security. If you’re researching an outage or a hack, this can be especially useful. As always, if you have any questions feel free to reach out:

About Automox

Facing growing threats and a rapidly expanding attack surface, understaffed and alert-fatigued organizations need more efficient ways to eliminate their exposure to vulnerabilities. Automox is a modern cyber hygiene platform that closes the aperture of attack by more than 80% with just half the effort of traditional solutions.

Cloud-native and globally available, Automox enforces OS & third-party patch management, security configurations, and custom scripting across Windows, Mac, and Linux from a single intuitive console. IT and SecOps can quickly gain control and share visibility of on-prem, remote and virtual endpoints without the need to deploy costly infrastructure.

Experience modern, cloud-native patch management today with a 15-day free trial of Automox and start recapturing more than half the time you're currently spending on managing your attack surface. Automox dramatically reduces corporate risk while raising operational efficiency to deliver best-in-class security outcomes, faster and with fewer resources.

Dive deeper into this topic