How to collect flows or how to install pmacct with kafka and nDPI support

You need a flow probe, a program that can create flows. The flows are sent to a flow collector, a program that – guess what – collects flows.

Here, I’ll be assuming a current Ubuntu distro. Here, we use Pmacct written by Paolo Lucente who published it as open source (thank you Paolo!). Its the name for a project consisting of several modules and also the name of a command line client. It contains pmacctd a probe, i.e. it collects packages and exports ipfix via a tool called nfprobe. We then send the flow to nfacctd a collector for IPFIX that is part of the pmacct suite and that conventiently not only collects the flows but can export them to our Kafka topic.

How to install pmacct with kafka support

pmacct is build with ./configure; make; make install, thus, first you need to install the autotools, if you haven’t done so yet

sudo apt install pkg-config autotools-dev build-essential

Furthermore, you need

sudo apt install libpcap-dev libtool librdkafka-dev

Get pmacct

git clone https://github.com/pmacct/pmacct.git

Since the configure script is usually not kept in version control you need to create it running autoreconf -fi in the pmacct directory.

cd pmacct
autoreconf -fi

You’ll also need to install jansson

cd ..
git clone https://github.com/akheron/jansson
cd jansson
autoreconf -fi
./configure
make
sudo make install


You’ll also need to install nDPI if you want the process to guess the used protocol for each flow.
You can read in pmacct's Quickstart guide about how to install this. The pmacct -version I am using here is 1.7.7. In the documentations it says pmacct version 1.7.6 is compatible with ndpi version 3.4 but I did use the latest stable version (ndpi 3.7) nonetheless. It did work.

git clone https://github.com/ntop/nDPI.git
cd nDPI
sudo apt install libjson-c-dev //that was missing
./autogen.sh
./configure
make
sudo make install
sudo ldconfig 

Now, that everything that is needed for pmacct is installed, we can start compiling the source:
./configure --enable-kafka --enable-jansson --enable-ndpi --enable-debug 

make
sudo make install

In order to create ipfix and to send it to a kafka topic, we need to configure pmacctd. We create a config file called pmacctd.conf and fill it as follows:

plugins: nfprobe
!nfprobe runs at 2100
nfprobe_receiver: 127.0.0.1:2100
nfprobe_version: 10
!set the number of bytes ndpi uses for protocol estimation
snaplen: 300

As said in the beginning, we want to use the nfacct daemon as collector and exporter to Kafka. So you need to configure that too. We create a file called nfacctd.conf with the following content:

kafka_topic: test !name of topic
kafka_broker_host: 127.0.0.1 !ip of your kafka broker
kafka_broker_port: 9092 !port of kafka broker
kafka_refresh_time: 1
daemonize: true
plugins: kafka
pcap_interface: eth0 # name of the interface to capture from
nfacctd_ip: 127.0.0.1
nfacctd_port: 2111
!define the features you want as output
aggregate: src_host, dst_host,in_iface, out_iface, timestamp_start, timestamp_end, src_port, dst_port, proto, tos, tcpflags, class 

You can then start both processes:

pmacctd -f pmacctd.conf
nfacctd -f nfacctd.conf

Was this helpful?

YesNo

7 / 1