查看: 72|回复: 0

flume连接mysql遇到的坑

[复制链接]
论坛徽章:
0
发表于 2019-6-4 13:32 | 显示全部楼层 |阅读模式

SQL MySQL

配置文件内容如下:
agent.channels.ch1.type = memory
agent.sources.SQL-source.channels = ch1
agent.channels = ch1
agent.sinks = HDFS
agent.sources = sql-source
agent.sources.sql-source.type = org.keedio.flume.source.SQLSource
agent.sources.sql-source.connection.url = jdbc:mysql://172.16.1.127:3306/test
agent.sources.sql-source.user = root
agent.sources.sql-source.password = 123456
agent.sources.sql-source.table = wlslog
agent.sources.sql-source.columns.to.select = *
agent.sources.sql-source.incremental.column.name = id
agent.sources.sql-source.incremental.value = 0
agent.sources.sql-source.run.query.delay=5000
agent.sources.sql-source.status.file.path = /var/lib/flume
agent.sources.sql-source.status.file.name = sql-source.status
agent.sinks.HDFS.channel = ch1
agent.sinks.HDFS.type = hdfs
agent.sinks.HDFS.hdfs.path = hdfs://mycluster/flume/mysql
agent.sinks.HDFS.hdfs.fileType = DataStream
agent.sinks.HDFS.hdfs.writeFormat = Text
agent.sinks.HDFS.hdfs.rollSize = 268435456
agent.sinks.HDFS.hdfs.rollInterval = 0
agent.sinks.HDFS.hdfs.rollCount = 0

同步过程往mysql插入数据,发现在hive中查不到数据,后来发现
agent.sinks.HDFS.hdfs.rollSize = 268435456
agent.sinks.HDFS.hdfs.rollInterval = 0
agent.sinks.HDFS.hdfs.rollCount = 0这三个参数是使HDFS上面的临时文件变为最终文件的trigger,当满足任意条件时,HDFS上面的临时文件才可以转换为最终文件,只有转换为最终文件hive中才可以查到数据



回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册 新浪微博登陆

本版积分规则

 

GMT+8, 2019-6-17 10:36 , Processed in 0.136025 second(s), 29 queries .

关闭

扫一扫加入
本版微信群