End-to-End Configuring a RHEL 8 Common Event Format Forwarder for Sentinel

Update 2021-01-13 9:15 AM CST: Added additional resources at the end of the article

Update 2021-03-23 2:31 PM CDT: Added additional firewall configuration for port 514 and additional SELinux configuration.

Update 2021-10-24 3:11 PM CDT: I have a GitHub repo with Azure Bicep templates to build a single or multiple syslog forwarders in high availability fashion: https://github.com/SvenAelterman/AzSentinel-syslogfwd-HA. Feel free to use these templates and submit pull requests for enhancements.

Azure Sentinel is Microsoft’s cloud-native SIEM/SOAR. While being cloud-native means that there is little to no configuration required to ingest a variety of log sources, such as Azure AD, Azure Activity Logs, and AWS logs, customers also have a need to ingest logs from more traditional sources, such as firewalls and Linux virtual machines. For this purpose, Sentinel supports ingesting syslog and Common Event Format (CEF) logs. In this post, I will describe end-to-end how to configure a Red Hat Enterprise (RHEL) 8 VM as a CEF (and potentially syslog) forwarder.

Conceptually, the CEF forwarder accepts events from a CEF-compatible source, either over TCP or TLS, and caches it locally. The Log Analytics agent, which will be installed on the VM as part of this walkthrough, picks up the events from the cache and forwards them to a Log Analytics workspace associated with Sentinel. More details are available in this Microsoft Doc and Tech Community article.

Set Up the RHEL 8 VM

Begin by creating a new virtual machine in Azure. Choose the RHEL 8.2 (or current) image. Deploy the VM to a virtual network and resource group of your choice.

As a best practice, create an availability set, even if you currently only plan to create a single forwarder.

The VM name can cause issues during the generation of a certificate for the Log Analytics agent. I recommend keeping the VM name short (15 characters or less) and not using hyphens.

After the VM deployment completes, connect to the VM. Depending on your network configuration, there are different ways of connecting. I am partial to Azure Bastion, a managed jump box.

As a good practice, we’ll update the VM first:

sudo yum update -y

When the update is finished, reboot the VM:

sudo reboot

Configuring CEF Forwarding

The steps in this section were learned the “hard way.” I will note that I am not a Linux expert and recognize that in some cases, an easier way might be available. Please share opportunities for improvement in the comments, and thanks!

RHEL 8.2 has Python3 installed. The provided scripts expect a python command, so we’ll first create an alias:

sudo alternatives --set python /usr/bin/python3

I have noticed that SELinux can interfere with the configuration of the Log Analytics agent and the rsyslog service. For now, we’ll turn off the enforcer:

sudo setenforce 0

Don’t worry, we’ll turn it back on soon!

Now, open the Azure Portal and Navigate to the Sentinel Data connectors pane. Click on the Common Event Format (CEF) connector and open the connector page. On the Instructions tab, you’ll find the command to run under heading 1.2 Install the CEF collector on the Linux machine. Use the handy copy icon to copy the command. It looks like this:

sudo wget -O cef_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_installer.py&&sudo python cef_installer.py [workspaceid] [workspacekey]

[workspaceid] and [workspacekey] will contain your actual Log Analytics Workspace values.

Breaking down this command, it retrieves a Python script from the Azure-Sentinel repo on GitHub and saves it as cef_installer.py. Using the && (logical AND) operator, the saved script is executed using the workspace ID and workspace key as arguments.

This script will generate quite a bit of output. Examine the output and verify that the configuration is successful. Two warnings are expected on most systems:

• If the system doesn’t have the Desired State Configuration (DSC) extension installed (which it won’t if you just deployed the VM), you’ll see an error indicating that the directory /opt/dsc/ doesn’t exist.
• If the system doesn’t have Docker installed (which it won’t if you just deployed the VM), you’ll see a notice that Docker was not found and the Docker agent won’t be installed.

Configuration

After the script runs and has successfully completed, you’ll need to make a few additional changes, which are required for correct operation of the forwarder. Keep the SELinux enforcer off for now.

Some of these additional commands take the workspace ID as an argument, so for ease of use, we’ll define the workspaceid environment variable:

workspaceid=[workspaceid]

Replace [workspaceid] with your actual Log Analytics workspace ID, as found in the command above, for example.

Firewall

The firewall needs a rule to accept internal connections on port 25226:

sudo firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 -p tcp --dport 25226  -j ACCEPT

The firewall needs a rule to accept connections from your sources on port 514:

sudo firewall-cmd --permanent --add-port=514/tcp
sudo firewall-cmd --permanent --add-port=514/udp

Reload the firewall to activate the new rules:

sudo firewall-cmd --reload

The last command simply verifies the presence of the rule:

sudo firewall-cmd --direct --get-rules ipv4 filter INPUT

Host Name Setting

When the Log Analytics agent forwards events, it should not use the host name of the syslog forwarder, but keep the host name as found in the event. This command configures that. Make sure you keep it as a single line.

sudo sed -i -e "/'Severity' => tags[tags.size - 1]/ a \ \t 'Host' => record['host']" -e "s/'Severity' => tags[tags.size - 1]/&,/" /opt/microsoft/omsagent/plugin/filter_syslog_security.rb && sudo /opt/microsoft/omsagent/bin/service_control restart $workspaceid

Keep Time Generated Original

When the agent forwards events, it should keep the original timestamp, not use the time the event was forwarded. This command ensures that:

sudo wget -O TimeGenerated.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/TimeGenerated.py && python TimeGenerated.py $workspaceid

Re-Enabling and Configuring SELinux Policy

Now that the configuration is finished, it’s time to re-enable the SELinux enforcement mode. After enabling it, you’ll restart the rsyslog.service and notice it will fail. The journal will have logged the cause and you can use audit2allow to create a new policy package.

sudo setenforce 1
sudo systemctl restart rsyslog.service
sudo ausearch -c 'rsyslogd' --raw | audit2allow -M my-rsyslogd
sudo semodule -X 300 -i my-rsyslogd.pp
sudo systemctl restart rsyslog.service
sudo semanage port -a -t syslogd_port_t -p tcp 25226

The last line is necessary to allow the syslog daemon to open a connection to the OMS agent’s listening port for CEF messages

After the second restart, there should be no output. If you want to verify that the rsyslog service is running again, execute this command:

systemctl status rsyslog.service

The output should show, in green, active (running).

Verifying the Configuration

There’s a verification script provided, which you can execute using the following command. It assumes that the workspaceid variable is still set (i.e. you haven’t logged off or rebooted yet).

sudo wget -O cef_troubleshoot.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/CEF/cef_troubleshoot.py&&sudo python cef_troubleshoot.py $workspaceid

Examine the output of the verification. Note that if you haven’t configured any sources to send events to the forwarder yet, you’ll see a few occasions where the script will report that no CEF data was received. That would be expected until you connect sources. You can run the verification script as needed, so feel free to run it both before and after you configure your source(s).

The verification script will send some mock data to the Log Analytics workspace. Notice that the forwarding and ingestion might take a little bit of time. After some time has passed, navigate to the Sentinel Logs pane, open the Azure Sentinel group of tables, and double-click on the CommonSecurityLog table. This will add the name of the table in the query pane. Click the Run button and examine the query results.

Additional Configuration

After setting this up, you should consider taking the following steps. I didn’t document these here, but please let me know if you’d like to see a blog post with details for these.

• Set up syslog rules and configure the VM to not duplicate CEF events as syslog events too.
• Expand the VM disk size to accommodate caching the logs for a period of time when connectivity to the Log Analytics Workspace might be down.
• Verify that the VM will not keep events indefinitely.
• Set up a second, identically configured VM and a load balancing solution for load balancing and high availability.
  There’s a walkthrough and a set of ARM templates available here: Scaling Up Syslog CEF Collection – Microsoft Tech Community

Additional Resources

Summary

With this walkthrough, you’ll now have a Linux RHEL 8 VM that’s configured to forward Common Event Format (CEF) events to Sentinel.