Excellent, and I am going to try it. But I still think using something like Flum... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		abhishektwr on Nov 20, 2011 \| parent \| context \| favorite \| on: Fluentd: the missing log collector Excellent, and I am going to try it. But I still think using something like Flume has some advantages mainly because of Hadoop eco-system. For instance you can plug-in the log data to HBase and use Hive to write to high level abstracted queries and run on Hadoop. I am guessing but seems like there are plugins on the way for various systems but not Hadoop. Update: Also Flume can use any data stream, for instance Twitter stream so not limited to log Analysis only.

kzk_mover on Nov 20, 2011 [–]

Yep, we agree with your opinion CURRENTLY.

We're developing Hoop (REST API for HDFS) plugin, and also Stargate (REST API for HBase). It requires additional setup at the infra-side, but still very useful and performs well.

You can feed Twitter stream by writing in_twitter plugin. That will 30 lines of Ruby code :-)

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact