Flume NG集群+Kafka集群整合：
修改Flume配置文件（flume-kafka-server.conf），让Sink连上Kafka
hadoop1:
#set Agent namea1.sources = r1a1.channels = c1a1.sinks = k1#set channela1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# other node,nna to nnsa1.sources.r1.type = avroa1.sources.r1.bind = hadoop1a1.sources.r1.port = 52020a1.sources.r1.interceptors = i1a1.sources.r1.interceptors.i1.type = statica1.sources.r1.interceptors.i1.key = Collectora1.sources.r1.interceptors.i1.value = hadoop1a1.sources.r1.channels = c1#set sink to hdfsa1.sinks.k1.type=org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.topic = ScalaTopica1.sinks.k1.brokerList = hadoop1:9092a1.sinks.k1.requiredAcks = 1a1.sinks.k1.batchSize = 20a1.sinks.k1.channel=c1
hadoop2:
#set Agent namea1.sources = r1a1.channels = c1a1.sinks = k1#set channela1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# other node,nna to nnsa1.sources.r1.type = avroa1.sources.r1.bind = hadoop2a1.sources.r1.port = 52020a1.sources.r1.interceptors = i1a1.sources.r1.interceptors.i1.type = statica1.sources.r1.interceptors.i1.key = Collectora1.sources.r1.interceptors.i1.value = hadoop2a1.sources.r1.channels = c1#set sink to hdfsa1.sinks.k1.type=org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.topic = ScalaTopica1.sinks.k1.brokerList = hadoop2:9092a1.sinks.k1.requiredAcks = 1a1.sinks.k1.batchSize = 20a1.sinks.k1.channel=c1
集群测试：
- 启动zookeeper（hadoop1，hadoop2，hadoop3）
- 启动kafka server和consumer（hadoop1，hadoop2）
- 启动Flume server（hadoop1，hadoop2）：flume-ng agent --conf conf --conf-file /usr/local/flume/conf/flume-kafka-server.conf --name a1 -Dflume.root.logger=INFO,console
- 启动Flume client（hadoop3）：flume-ng agent --conf conf --conf-file /usr/local/flume/conf/flume-client.conf --name agent1 -Dflume.root.logger=INFO,console
- 在hadoop3上追加一条日志记录
- kafka consumer收到记录，从则测试完毕。
hadoop3：
hadoop1：
测试完毕，这样Flume+kafka就整合起来了，即Flume+Kafka+Spark Streaming的实时日志分析系统就孕育而生了。
Flume 学习笔记之 Flume NG+Kafka整合
原文地址：http://www.cnblogs.com/AK47Sonic/p/7440197.html