This article will explain how to send NetFilter/shorewall tracked connections logs made with Ulogd to the elk stack. You can then use kibana to search for specific activity from your network.

What we use:

  • Shorewall is a gateway/firewall configuration tool for GNU/Linux. It has a very good documentation and is very easy to configure.
  • Ulogd is a userspace logging daemon for netfilter/iptables related logging. It lets you log information in different file output format like CSV or JSON or even in databases.
  • The ELK stack: Elasticsearch + Logstash + Kibana + Filebeat.

Logging every packets going through Shorewall is not a very good idea since, depending on your use case, this could result on very large log files size. We can reduce the information using Connection Tracking and log only network sessions.

We will set up a JSON log file with Ulogd and read it with Filebeat.

Ulogd

You can find a good blog post on Ulogd and Json here.

Here is my ulogd.conf file made from this blog post:

/etc/ulogd.conf

...
[global]
...
#Use json plugin
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_JSON.so"
...
# Get netfilter conntrack data and use jsonnfct1 with JSON format
stack=ct2:NFCT,ip2str1:IP2STR,jsonnfct1:JSON
...
[ct2]
event_mask=0x00000004
hash_enable=0
...
[jsonnfct1]
sync=1
file="/var/log/ulogd_nfct.json"
...

Filebeat

We configure filebeat to read the json file:

/etc/filebeat

filebeat:
  prospectors:
    -
      paths:
        - /data/ulog/ulogd_nfct.json
      input_type: log
      document_type: ulog
...

On the other side we need a logstash filter configuration to specify the json configuration and fix a bug: we have to add a ruby code to replace all dot characters by another one because elasticsearch don’t like them on the key indices.

conf.d/10-ulog-filter.conf

filter {
  if [type] == "ulog" {
    json{
      source => "message"
    }
    ruby {
      code => "event.to_hash.keys.each { |k| event[ k.gsub('.','_') ] = event.remove(k) if k.include?'.' }"
    }
  }
}

Kibana

Here is the result:

shorewall-kibana

Thanks to Eric Leblond for his posts using-ulogd-and-json-output and logging-connection-tracking-event-with-ulogd.