Flume Real Time Projects


List of Flume Real Time Projects

















Kalyan Big Data Project 1
Project Name
How To Stream Twitter Data Into Hadoop in AVRO format Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project1-twitter-hadoop-avro


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Twitter Source)
2. Channel (Memory Channel)
3. Sink (Hdfs Sink)
Major project in Real Time `Social Media (Twitter) Sentiment Analysis`
1. We are extracting the data from twitter using twitter api credentials
2. This data will be useful to do setiment analysis on twitter tweets
3. Avro is the output format
We can use hive / pig / mapreduce to analyze this data
1. explore hive query to analysis
2. explore pig scripts to analysis
3. explore mapreduce to analysis



Kalyan Big Data Project 2
Project Name
How To Stream Twitter Data Into Hadoop in JSON format
Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project2-twitter-hadoop-json


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Twitter Source)
2. Channel (Memory Channel)
3. Sink (Hdfs Sink)
Major project in Real Time `Social Media (Twitter) Sentiment Analysis`
1. We are extracting the data from twitter using twitter api credentials
2. This data will be useful to do setiment analysis on twitter tweets
3. JSON is the output format
We can use hive / pig / mapreduce to analyze this data
1. explore hive query to analysis
2. explore pig scripts to analysis
3. explore mapreduce to analysis




Kalyan Big Data Project 3
Project Name
How To Stream Twitter Data Into MongoDB in JSON format
Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project3-twitter-mongodb-json


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Twitter Source)
2. Channel (Memory Channel)
3. Sink (MongoDB Sink)
Major project in Real Time `Social Media (Twitter) Sentiment Analysis`
1. We are extracting the data from twitter using twitter api credentials
2. This data will be useful to do setiment analysis on twitter tweets
3. JSON is the output format
We can use mongodb / hive / pig / mapreduce to analyze this data
1. explore mongodb to analysis
2. explore hive query to analysis
3. explore pig scripts to analysis
4. explore mapreduce to analysis




Kalyan Big Data Project 4
Project Name
How To Stream Twitter Data Into Hadoop and MongoDB in
JSON format Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project4-twitter-hadoop-mongodb-json


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Twitter Source)
2. Channel (Memory Channel)
3. Sink (MongoDB Sink)
Major project in Real Time `Social Media (Twitter) Sentiment Analysis`
1. We are extracting the data from twitter using twitter api credentials
2. This data will be useful to do setiment analysis on twitter tweets
3. JSON is the output format
We can use mongodb / hive / pig / mapreduce to analyze this data
1. explore mongodb to analysis
2. explore hive query to analysis
3. explore pig scripts to analysis
4. explore mapreduce to analysis




Kalyan Big Data Project 5
Project Name
How To Stream CSV Data Into Phoenix Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project5-phoenix-csv


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Exec Source)
2. Channel (Memory Channel)
3. Sink (Phoenix Sink)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. CSV is the output format
We can use phoenix to analyze this data




Kalyan Big Data Project 6
Project Name
How To Stream JSON Data Into Phoenix Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project6-phoenix-json


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Exec Source)
2. Channel (Memory Channel)
3. Sink (Phoenix Sink)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. JSON is the output format
We can use Phoenix to analyze this data




Kalyan Big Data Project 7
Project Name
How To Stream REGEX Data Into Phoenix Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project7-phoenix-regex


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Exec Source)
2. Channel (Memory Channel)
3. Sink (Phoenix Sink)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. Complex Data is the output format then REGEX is best solution
We can use Phoenix to analyze this data




Kalyan Big Data Project 8
Project Name
How To Stream CSV Data Into Hive Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project8-hive-csv


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Exec Source)
2. Channel (Memory Channel)
3. Sink (Hive Sink)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. CSV is the output format
We can use hive to analyze this data






Kalyan Big Data Project 9
Project Name
How To Stream JSON Data Into Hive Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project9-hive-json


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Exec Source)
2. Channel (Memory Channel)
3. Sink (Hive Sink)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. JSON is the output format
We can use Hive to analyze this data





Kalyan Big Data Project 10
Project Name
How To Stream REGEX Data Into Hive Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project10-hive-regex


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Exec Source)
2. Channel (Memory Channel)
3. Sink (Hive Sink)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. Complex Data is the output format then REGEX is best solution
We can use Hive to analyze this data




Kalyan Big Data Project 11
Project Name
How To Stream CSV Data Into Hbase Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project11-hbase-csv


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Exec Source)
2. Channel (Memory Channel)
3. Sink (Hbase Sink)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. CSV is the output format
We can use hbase to analyze this data




Kalyan Big Data Project 12
Project Name
How To Stream JSON Data Into Hbase Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project12-hbase-json


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Exec Source)
2. Channel (Memory Channel)
3. Sink (Hbase Sink)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. JSON is the output format
We can use Hbase to analyze this data




Kalyan Big Data Project 13
Project Name
How To Stream REGEX Data Into Hbase Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project13-hbase-regex


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Exec Source)
2. Channel (Memory Channel)
3. Sink (Hbase Sink)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. Complex Data is the output format then REGEX is best solution
We can use Hbase to analyze this data




Kalyan Big Data Project 14
Project Name
How To Stream Text Data Into Hbase Using Apache Flume
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project14-hbase-text


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Netcat Source)
2. Channel (Memory Channel)
3. Sink (Hbase Sink)
Major project in Real Time `Chat Applications`
1. We are extracting the data from Chat Applications
2. This data will be useful to do analysis on Sentiment on Tweets
3. Complex Data is the output format then REGEX is best solution
We can use Hbase to analyze this data




1 comment :

Related Posts Plugin for WordPress, Blogger...