默认安装的Syslog-ng是不支持kafka模块的。如果我们要使用该模块,就需要手动安装Syslog-ng的3.6版本,并编译安装它的一个拓展模块管理插件 — syslog-ng-incubator?。
主要步骤:
1、安装依赖环境(注意:所需依赖是否版本符合要求)

#安装相关yum源
wget -O /etc/yum.repos.d/syslog-ng36epel6-epel-6.repo http://copr.fedoraproject.org/coprs/czanik/syslog-ng36epel6/repo/epel-6/czanik-syslog-ng36epel6-epel-6.repo
rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
#基本的gcc zlib-devel git等
yum install gcc git eventlog-devel zlib-devel gcc-c++ -y
#升级或安装编译工具autoconf automake libtool等
yum remove autoconf automake -y
cd /tmp
wget http://files.directadmin.com/services/custombuild/autoconf-2.68.tar.gz
wget http://files.directadmin.com/services/custombuild/automake-1.15.tar.gz
wget http://mirrors.kernel.org/gnu/libtool/libtool-2.4.6.tar.gz
tar zxvf autoconf-2.68.tar.gz && tar zxvf automake-1.15.tar.gz && tar zxvf libtool-2.4.6.tar.gz
cd autoconf-2.68
./configure --prefix=/usr
make && make install
cd ../automake-1.15
./configure --prefix=/usr
make && make install
cd ../libtool-2.4.6
./configure --prefix=/usr
make && make install
#安装bison-3.0
cd /tmp
wget http://ftp.gnu.org/gnu/bison/bison-3.0.tar.gz
tar zxvf bison-3.0.tar.gz
cd bison-3.0
./configure && make && make install

2、安装syslog-ng和syslog-ng-devel

yum install syslog-ng syslog-ng-devel syslog-ng-json -y

3、安装rdkafka

cd /tmp
wget https://github.com/edenhill/librdkafka/archive/0.8.5.tar.gz
tar zxvf 0.8.5.tar.gz
cd librdkafka-0.8.5/
./configure && make && make install

4、安装syslog-ng-incubator,并开启kafka模块

cd /tmp
git clone git://github.com/balabit/syslog-ng-incubator.git
cd syslog-ng-incubator
autoreconf -i
./configure --with-librdkafka=/usr/local/include/librdkafka
make && make install

5、编辑syslog-ng配置文件

vi /etc/syslog-ng/syslog-ng.conf
#在文件末尾添加下面代码
source s_system {
     system();
};

destination d_kafka {
    kafka(properties(metadata.broker.list("51idc.biglog.cn:9092"))
        topic("syslogng")
        payload("$(format-json --scope all-nv-pairs --scope core)")
        partition("$PROGRAM")
    );
};

log {
    source(s_system);
    destination(d_kafka);
};

6、启动syslog-ng进行验证

/etc/init.d/syslog-ng restart

常见错误:
1、启动syslog-ng时,找不到librdkafka.so.1文件,报错如下:

Error opening plugin module; module='kafka', error='librdkafka.so.1: cannot open shared object file: No such file or directory'

解决方法:
(1).查找librdkafka.so.1文件路径

[root@syslog-ng ~]# find  / -name librdkafka.so.1
/usr/local/lib/librdkafka.so.1
/tmp/librdkafka-0.8.5/src/librdkafka.so.1

(2).添加环境变量

vi /etc/profile
#在末尾添加一行:
export LD_LIBRARY_PATH=/usr/local/lib

2、报错:Error parsing kafka, Error compiling template (Unknown template function “format-json”)

[root@syslog-ng librdkafka-0.8.5]# /etc/init.d/syslog-ng restart
Stopping syslog-ng:                                        [FAILED]
Error parsing kafka, Error compiling template (Unknown template function "format-json") in /etc/syslog-ng/syslog-ng.conf at line 76, column 17:

        payload("$(format-json --scope all-nv-pairs --scope core)")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng

解决方法:

yum install syslog-ng-json