You need a flow probe, a program that can create flows. The flows are sent to a flow collector, a program that – guess what – collects flows.
Here, I’ll be assuming a current Ubuntu distro. Here, we use Pmacct written by Paolo Lucente who published it as open source (thank you Paolo!). Its the name for a project consisting of several modules and also the name of a command line client. It contains pmacctd a probe, i.e. it collects packages and exports ipfix via a tool called nfprobe. We then send the flow to nfacctd a collector for IPFIX that is part of the pmacct suite and that conventiently not only collects the flows but can export them to our Kafka topic.
How to install pmacct with kafka support
pmacct is build with ./configure; make; make install, thus, first you need to install the autotools, if you haven’t done so yet
sudo apt install pkg-config autotools-dev build-essential
Furthermore, you need
sudo apt install libpcap-dev libtool librdkafka-dev
git clone https://github.com/pmacct/pmacct.git
Since the configure script is usually not kept in version control you need to create it running autoreconf -fi in the pmacct directory.
cd pmacct autoreconf -fi
You’ll also need to install jansson
cd .. git clone https://github.com/akheron/jansson cd jansson autoreconf -fi ./configure make sudo make install
You’ll also need to install nDPI if you want the process to guess the used protocol for each flow.
You can read in pmacct's Quickstart guide about how to install this. The pmacct -version I am using here is 1.7.7. In the documentations it says pmacct version 1.7.6 is compatible with ndpi version 3.4 but I did use the latest stable version (ndpi 3.7) nonetheless. It did work.
git clone https://github.com/ntop/nDPI.git
sudo apt install libjson-c-dev //that was missing
sudo make install
Now, that everything that is needed for pmacct is installed, we can start compiling the source: ./configure --enable-kafka --enable-jansson --enable-ndpi --enable-debug
make sudo make install
In order to create ipfix and to send it to a kafka topic, we need to configure pmacctd. We create a config file called pmacctd.conf and fill it as follows:
plugins: nfprobe !nfprobe runs at 2100 nfprobe_receiver: 127.0.0.1:2100 nfprobe_version: 10 !set the number of bytes ndpi uses for protocol estimation snaplen: 300
As said in the beginning, we want to use the nfacct daemon as collector and exporter to Kafka. So you need to configure that too. We create a file called nfacctd.conf with the following content:
kafka_topic: test !name of topic kafka_broker_host: 127.0.0.1 !ip of your kafka broker kafka_broker_port: 9092 !port of kafka broker kafka_refresh_time: 1 daemonize: true plugins: kafka pcap_interface: eth0 # name of the interface to capture from nfacctd_ip: 127.0.0.1 nfacctd_port: 2111 !define the features you want as output aggregate: src_host, dst_host,in_iface, out_iface, timestamp_start, timestamp_end, src_port, dst_port, proto, tos, tcpflags, class
You can then start both processes:
pmacctd -f pmacctd.conf nfacctd -f nfacctd.conf
Was this helpful?
6 / 0