一、前置准备：
Linux命令基础
Scala、Python其中一门
Hadoop、Spark、Flume、Kafka、Hbase基础知识
二、分布式日志收集框架Flume
业务现状分析：服务器、web服务产生的大量日志，怎么使用，怎么将大量日志导入到集群
1、shell脚本批量，再传到Hdfs：实效性不高，容错率低，网络/磁盘IO，监控
2、Flume：
Flume：关键在于写配置文件
1）配置 agent
2）配置 Source
3）配置 Channel
4）配置 Sink
1-netcat-mem-logger.conf ：监听端口数据
#example for source=netcat, channel=memory, sink=logger# Name the components on this agenta1.sources = r1a1.channels = c1a1.sinks = k1# configure for sourcesa1.sources.r1.type = netcata1.sources.r1.bind = localhosta1.sources.r1.port = 44444# configure for channelsa1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# configure for sinksa1.sinks.k1.type = logger# configure a1.sinks.k1.channel = c1a1.sources.r1.channels = c1
启动 flume-ng agent \
-n a1 \
-c conf -f ./1-netcat-mem-logger.conf \
-Dflume.root.logger=INFO,console
exec-mem-logger.conf ：监控文件
# Name the components on this agenta1.sources = r1a1.channels = c1a1.sinks = k1# configure for sourcesa1.sources.r1.type = execa1.sources.r1.command = tail -F /opt/datas/flume_data/exec_tail.log# configure for channelsa1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# configure for sinksa1.sinks.k1.type = loggera1.sinks.k1.channel = c1a1.sources.r1.channels = c1
flume-ng agent \
-n a1 \
-c conf -f ./4-exec-mem-logger.conf \
-Dflume.root.logger=INFO,console
日志收集过程：
1. 日志服务器，启动agent，exec-source, memory-channel,avro-sink（数据服务器）, 将收集到的日志数据，写到数据服务器
2. 数据服务器，启动agent，avro-aource，memory-channel，logger-sink/kafka-sink
conf1：exec-mem-avro.conf
# Name the components on this agenta1.sources = exec-sourcea1.channels = memory-channela1.sinks = avro-sink# configure for sourcesa1.sources.exec-source.type = execa1.sources.exec-source.command = tail -F /opt/datas/log-collect-system/log_server.log# configure for channelsa1.channels.memory-channel.type = memorya1.channels.memory-channel.capacity = 1000a1.channels.memory-channel.transactionCapacity = 100# configure for sinksa1.sinks.avro-sink.type = avroa1.sinks.avro-sink.hostname = localhosta1.sinks.avro-sink.port = 44444# configure a1.sinks.avro-sink.channel = memory-channela1.sources.exec-source.channels = memory-channel
conf2：avro-mem-logger.conf
# Name the components on this agenta1.sources = avro-sourcea1.channels = memory-channela1.sinks = logger-sink# configure for sourcesa1.sources.avro-source.type = avroa1.sources.avro-source.bind = localhosta1.sources.avro-source.port = 44444# configure for channelsa1.channels.memory-channel.type = memorya1.channels.memory-channel.capacity = 1000a1.channels.memory-channel.transactionCapacity = 100# configure for sinksa1.sinks.logger-sink.type = logger# configure a1.sinks.logger-sink.channel = memory-channela1.sources.avro-source.channels = memory-channel
（非常重要！！！）启动顺序：先启动exec-mem-avro.conf再启动exec-mem-avro.conf
Flume+Kafka+SparkStreaming+Hbase+可视化（一）
原文地址：https://www.cnblogs.com/mlxx9527/p/9367495.html