Context Navigation

Ulogd2

Ulogd2 is a userspace logging daemon for netfilter/iptables related logging.

Ulogd-2.x is able to do:

Packet based logging (via libnetfilter_log or ULOG target)
Flow based logging (via libnetfilter_conntrack)

Logging can be done to file, syslog, pcap or a database.

Description

Global idea

Ulogd2 configuration is based on stack definition. A stack define the way to act and output information about an input source. To define a stack, you need to aggregate some elements:

A source
Filters
An output module

A stack is defined like that:

stack=input1:INPUTMODULE,filter1:FILTERMODULE,output1:OUTPUTMODULE

Basically, a stack is the ulogd2 mean to say: I want to log things coming from this INPUT module to this OUTPUT module.

For example, a simple working stack is the following:

stack=ct1:NFCT,ip2str1:IP2STR,print1:PRINTFLOW,log1:LOGEMU

This stack gets data from the NFCT module (conntrack), modify it via IP2STR which convert IP to string, parse it through PRINTFLOW module and output it via LOGEMU (file based logging).

The basic idea is to treat data as a list of key->value association. Input module defined a list of keys which is used by filter to generate other keys. Finally, the output module use the key obtained from input and filter module to build its output.

Since subversion revision 7415, the same module instance can be used twice or more. It is thus possible to have two working stacks with the following configuration:

# this is a stack for NFLOG packet-based logging to PCAP
stack=log1:NFLOG,base1:BASE,pcap1:PCAP
# this is a stack for logging packet to MySQL
stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2bin1:IP2BIN,mac2str1:MAC2STR,mysql1:MYSQL

Stack configuration

Each module is a plugin which is dynamically loaded during ulogd start. A module has:

Input keys: It defined the entry which are needed (mandatory or not) by the module to be able to work (ie output other keys)
Output keys: The module outputs key->value association with key in the output list.

Each key has a type and may have a default value.

To have a working stack, you will need to provide an input module and filter which will be able to modify the initial set of key->value provided by the input module in a set of key->value which will contain all the input keys which are mandatory for the output plugin.

Information about a given module can be obtained through the info command of ulogd:

# /opt/ulogd2/sbin/ulogd --info /opt/ulogd2/lib/ulogd/ulogd_filter_IP2STR.so
Name: IP2STR
Input keys:
        Key: oob.family (unsigned int 8)
        Key: oob.protocol (unsigned int 16)
        Key: ip.saddr (IP addr)
        Key: ip.daddr (IP addr)
        [...]
Output keys:
        Key: ip.saddr.str (string)
        Key: ip.daddr.str (string)
        [...]

From this output, we see that the IP2STR module output IP in string format and take in input IP in rw format and some additional informations (oob.family and oob.protocol).

Plugins informations

Input plugins

ULOG: get packet select via the iptables target ULOG
NFLOG: get packet from NFLOG target which is the successor of ULOG
NFCT: get flow information from Netfilter connection tracking via libnetfilter_conntrack

Output plugins

LOGEMU: log packet/flow into a file
OPRINT: log packet/flow to a file in multiline format
SYSLOG: log packet/flow to syslog system
MYSQL: log packet/flow to a MySQL database
PGSQL: log packet/flow to a PGSQL server
SQLITE3: log packet to a SQLITE3 file
PCAP: log packet to a Pcap file
IPFIX: log flow via IP Flow Information Export
NACCT: log flow to a nacct compatible format (accounting)

Compilation

For the moment, you will need to get source from Netfilter git:

git clone git://git.netfilter.org/ulogd2.git

The only thing to notice is that you will need to have libnetfilter_conntrack and libnetfilter_log library and development files on your system.

Compilation itself is straight-forward:

./configure
make
make install

If you want to use MySQL or PGSQL output, you will need corresponding developement headers as well.

Configuration

Ulogd configuration is in INI format and is split in different sections:

A global section
A section per plugin

A example configuration file is available in the sources (ulogd.conf).

You should use it as a start for your own configuration.

Global section

The global section define the behaviour of ulogd relatively memory usage and system.

[global]
# logfile for status messages
logfile="/var/log/ulogd.log"

# loglevel: debug(1), info(3), notice(5), error(7) or fatal(8)
loglevel=1

# socket receive buffer size (should be at least the size of the
# in-kernel buffer (ipt_ULOG.o 'nlbufsiz' parameter)
rmem=131071

# libipulog/ulogd receive buffer size, should be > rmem
bufsize=150000

Plugin loading

We have to configure and load all the plugins we want to use. Plugins have to be loaded first. They will be configured in their dedicated sections.

plugin="@libdir@/ulogd/ulogd_inppkt_NFLOG.so"
plugin="@libdir@/ulogd/ulogd_inpflow_NFCT.so"
plugin="@libdir@/ulogd/ulogd_filter_IFINDEX.so"
plugin="@libdir@/ulogd/ulogd_filter_IP2STR.so"

Stacks definitions

Next step, is to defined the used stacks. You need to have at least one stack enable to have ulogd running. A simple mean to have ulogd working is to used one of the stack defined in the configuration file example. To do so, you will need to uncomment one of the following stack= line:

# this is a stack for IPv4 packet-based logging via LOGEMU
#stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,print1:PRINTPKT,emu1:LOGEMU

# this is a stack for IPv6 packet-based logging via LOGEMU
#stack=log2:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,print1:PRINTPKT,emu1:LOGEMU

# this is a stack for ebtables packet-based logging via LOGEMU
#stack=log3:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,print1:PRINTPKT,emu1:LOGEMU

# this is a stack for ULOG packet-based logging via LOGEMU
#stack=ulog1:ULOG,base1:BASE,ip2str1:IP2STR,print1:PRINTPKT,emu1:LOGEMU

# this is a stack for IPv4 packet-based logging via LOGEMU with filtering on MARK
#stack=log1:NFLOG,mark1:MARK,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,print1:PRINTPKT,emu1:LOGEMU

# this is a stack for flow-based logging via LOGEMU
#stack=ct1:NFCT,ip2str1:IP2STR,print1:PRINTFLOW,emu1:LOGEMU

# this is a stack for flow-based logging via OPRINT
#stack=ct1:NFCT,op1:OPRINT

Simply uncomment the one you want to use or write your own stack.

Plugins configuration

Now, we have to setup the plugins. An empty section may be needed to avoid some problems:

[ct1]

Here are some example taken from default configuration:

# IPv4 logging through NFLOG
[log1]
# netlink multicast group (the same as the iptables --nflog-group param)
group=0

# IPv6 logging through NFLOG
[log2]
group=1 # Group has to be different from the one use in log1
addressfamily=10 # 10 is value of AF_INET6

# ebtables logging through NFLOG
[log3]
group=2 # Group has to be different from the one use in log1/log2
addressfamily=7 # 7 is value of AF_BRIDGE

[ulog1]
# netlink multicast group (the same as the iptables --ulog-nlgroup param)
nlgroup=1

[emu1]
file="/var/log/ulogd_syslogemu.log"
sync=1

[op1]
file="/var/log/ulogd_oprint.log"
sync=1

[sys2]
facility=LOG_LOCAL2

Remember you can use the --info flag to get the list of options for one module:

# /opt/ulogd2/sbin/ulogd --info /opt/ulogd2/lib/ulogd/ulogd_inpflow_NFCT.so
Name: NFCT
Config options:
	Var: pollinterval (Integer, Default: 0)
	Var: hash_enable (Integer, Default: 1)
	Var: hash_buckets (Integer, Default: 8192)
	Var: hash_max_entries (Integer, Default: 32768)
	Var: event_mask (Integer, Default: 5)
	Var: netlink_socket_buffer_size (Integer, Default: 0)
	Var: netlink_socket_buffer_maxsize (Integer, Default: 0)

NFCT logging

NFCT plugin usage is simple but there is a choice to made on the hash_enable value. With hash_enable set to 1 (which is the default), ulogd2 will maintain a internal copy of the connection table and log a connection once the DESTROY event occurs. With value set to 0, all received events will be logged and the output plugins will have to adapat their behaviour.

PostgreSQL logging

Ulogd2 has an PGSQL output module.

Database creation and setup

First create user and database:

# su - postgres
$ createuser -P ulog2
$ createdb -O ulog2 ulog2;

Stored procedures use PL/pgSQL langage, and you will need to add the support for this langage in the database:

$ createlang plpgsql ulog2

PostgreSQL connection

Here's a small reminder for connection procedure to a PostgreSQL database:

Edit the file /etc/postgresql/8.2/main/pg_hba.conf and verify that md5 method is available for localhost:

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
host    all         all         127.0.0.1/32          md5

Connect to localhost:
```
$ psql -U ulog2 -h localhost ulog2
```

Schema insertion

Now, we will insert the schema into the database:

$ psql -U ulog2 -h localhost ulog2 -f doc/pgsql-ulogd2.sql

Ulogd configuration

NFLOG logging

First thing is to have a working stack:

stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,mac2str1:MAC2STR,pgsql1:PGSQL

Last thing is to setup pgsql1 instance of the PGSQL plugin:

[pgsql1]
db="ulog2"
host="localhost"
user="ulog2"
table="ulog"
pass="nupik"
procedure="INSERT_PACKET_FULL"

You should not have to change the procedure line if you did not modify the SQL schema by yourself.

Now, you can launch ulogd. You should have a working PGSQL logging.

NFCT logging

Ulogd2 is able to listen to connection tracking event and log them. To log connection event in PGSQL, you can use the following stack:

stack=ct1:NFCT,ip2str1:IP2STR,pgsql2:PGSQL

pgsql2 configuration does not differ a lot from pgsql11 one:

[pgsql2]
db="nulog"
host="localhost"
user="nupik"
table="ulog2_ct"
pass="changeme"
procedure="INSERT_CT"

The main differences are the table ulogd2_ct instead of ulog and the usage of the procedure INSERT_CT.

This supposes ct1 is configured with hash_enable=1 which is the default. If you set hash_enable to 0, you will need to use INSERT_OR_REPLACE_CT procedure:

[pgsql3]
db="nulog"
host="localhost"
user="nupik"
table="ulog2_ct"
pass="changeme"
procedure="INSERT_OR_REPLACE_CT"

MySQL logging

Initialisation of the database

You will need to have at least MYSQL 5.0 because the schema use stored procedure and other SQL technique which were not available in older version.

You can create the database with:

mysqladmin create nulog

To populate the database:

cat doc/mysql-ulogd2.sql | mysql nulog

Ulogd configuration

Configuration is similar as PGSQL configuration with the difference that MySQl uses binary storage of IP addresses (instead of using a dedicated type as with PGSQL). Thus the stack uses the IP2BIN filter to convert IP addresses to a usable form.

stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2bin1:IP2BIN,mysql1:MYSQL

The configuration of the mysql1 instance of the plugin is simple:

[mysql1]
host="localhost"
db="nulog"
user="root"
table="ulog"
pass=""
procedure="INSERT_PACKET_FULL"

You should not have to change the procedure line if you did not modify the SQL schema by yourself.

IPv6 logging

With ulogd2 and a recent kernel, you will be able to do IPv6 logging.

It is mandatory to use NFLOG as input plugin for IPv6 packet-based logging:

The stack is not different for Ipv6 and IPv4:

stack=log2:NFLOG,base1:BASE,ip2str1:IP2STR,ifi1:IFINDEX,print1:PRINTPKT,emu1:LOGEMU

The only difference is in log2 configuration where you have to specify that you want to use AF_INET6 address family.

[log2]
# netlink multicast group (the same as the iptables --ulog-nlgroup param)
group=1
addressfamily=10

You've guess it: AF_INET6 is defined as 10 in linux headers.

After launching ulogd, you will be able to log by using iptables rules like:

ip6tables -A OUTPUT -p tcp --syn -j NFLOG --nflog-group 1

The important part in the previous command is nflog-group which has to be different from other defined group and in particular different from the one used for IPv4.

Download in other formats:

Plain Text