Flume Real Time Projects

List of Flume Real Time Projects

Kalyan Big Data Project 1
Project Name	How To Stream Twitter Data Into Hadoop in AVRO format Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project1-twitter-hadoop-avro
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Twitter Source) 2. Channel (Memory Channel) 3. Sink (Hdfs Sink) ➢ Major project in Real Time `Social Media (Twitter) Sentiment Analysis` 1. We are extracting the data from twitter using twitter api credentials 2. This data will be useful to do setiment analysis on twitter tweets 3. Avro is the output format ➢ We can use hive / pig / mapreduce to analyze this data 1. explore hive query to analysis 2. explore pig scripts to analysis 3. explore mapreduce to analysis

Kalyan Big Data Project 2
Project Name	How To Stream Twitter Data Into Hadoop in JSON format Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project2-twitter-hadoop-json
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Twitter Source) 2. Channel (Memory Channel) 3. Sink (Hdfs Sink) ➢ Major project in Real Time `Social Media (Twitter) Sentiment Analysis` 1. We are extracting the data from twitter using twitter api credentials 2. This data will be useful to do setiment analysis on twitter tweets 3. JSON is the output format ➢ We can use hive / pig / mapreduce to analyze this data 1. explore hive query to analysis 2. explore pig scripts to analysis 3. explore mapreduce to analysis

Kalyan Big Data Project 3
Project Name	How To Stream Twitter Data Into MongoDB in JSON format Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project3-twitter-mongodb-json
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Twitter Source) 2. Channel (Memory Channel) 3. Sink (MongoDB Sink) ➢ Major project in Real Time `Social Media (Twitter) Sentiment Analysis` 1. We are extracting the data from twitter using twitter api credentials 2. This data will be useful to do setiment analysis on twitter tweets 3. JSON is the output format ➢ We can use mongodb / hive / pig / mapreduce to analyze this data 1. explore mongodb to analysis 2. explore hive query to analysis 3. explore pig scripts to analysis 4. explore mapreduce to analysis

Kalyan Big Data Project 4
Project Name	How To Stream Twitter Data Into Hadoop and MongoDB in JSON format Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project4-twitter-hadoop-mongodb-json
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Twitter Source) 2. Channel (Memory Channel) 3. Sink (MongoDB Sink) ➢ Major project in Real Time `Social Media (Twitter) Sentiment Analysis` 1. We are extracting the data from twitter using twitter api credentials 2. This data will be useful to do setiment analysis on twitter tweets 3. JSON is the output format ➢ We can use mongodb / hive / pig / mapreduce to analyze this data 1. explore mongodb to analysis 2. explore hive query to analysis 3. explore pig scripts to analysis 4. explore mapreduce to analysis

Kalyan Big Data Project 5
Project Name	How To Stream CSV Data Into Phoenix Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project5-phoenix-csv
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Exec Source) 2. Channel (Memory Channel) 3. Sink (Phoenix Sink) ➢ Major project in Real Time `Product Log Analysis` 1. We are extracting the data from server logs 2. This data will be useful to do analysis on product views 3. CSV is the output format ➢ We can use phoenix to analyze this data

Kalyan Big Data Project 6
Project Name	How To Stream JSON Data Into Phoenix Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project6-phoenix-json
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Exec Source) 2. Channel (Memory Channel) 3. Sink (Phoenix Sink) ➢ Major project in Real Time `Product Log Analysis` 1. We are extracting the data from server logs 2. This data will be useful to do analysis on product views 3. JSON is the output format ➢ We can use Phoenix to analyze this data

Kalyan Big Data Project 7
Project Name	How To Stream REGEX Data Into Phoenix Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project7-phoenix-regex
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Exec Source) 2. Channel (Memory Channel) 3. Sink (Phoenix Sink) ➢ Major project in Real Time `Product Log Analysis` 1. We are extracting the data from server logs 2. This data will be useful to do analysis on product views 3. Complex Data is the output format then REGEX is best solution ➢ We can use Phoenix to analyze this data

Kalyan Big Data Project 8
Project Name	How To Stream CSV Data Into Hive Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project8-hive-csv
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Exec Source) 2. Channel (Memory Channel) 3. Sink (Hive Sink) ➢ Major project in Real Time `Product Log Analysis` 1. We are extracting the data from server logs 2. This data will be useful to do analysis on product views 3. CSV is the output format ➢ We can use hive to analyze this data

Kalyan Big Data Project 9
Project Name	How To Stream JSON Data Into Hive Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project9-hive-json
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Exec Source) 2. Channel (Memory Channel) 3. Sink (Hive Sink) ➢ Major project in Real Time `Product Log Analysis` 1. We are extracting the data from server logs 2. This data will be useful to do analysis on product views 3. JSON is the output format ➢ We can use Hive to analyze this data

Kalyan Big Data Project 10
Project Name	How To Stream REGEX Data Into Hive Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project10-hive-regex
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Exec Source) 2. Channel (Memory Channel) 3. Sink (Hive Sink) ➢ Major project in Real Time `Product Log Analysis` 1. We are extracting the data from server logs 2. This data will be useful to do analysis on product views 3. Complex Data is the output format then REGEX is best solution ➢ We can use Hive to analyze this data

Kalyan Big Data Project 11
Project Name	How To Stream CSV Data Into Hbase Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project11-hbase-csv
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Exec Source) 2. Channel (Memory Channel) 3. Sink (Hbase Sink) ➢ Major project in Real Time `Product Log Analysis` 1. We are extracting the data from server logs 2. This data will be useful to do analysis on product views 3. CSV is the output format ➢ We can use hbase to analyze this data

Kalyan Big Data Project 12
Project Name	How To Stream JSON Data Into Hbase Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project12-hbase-json
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Exec Source) 2. Channel (Memory Channel) 3. Sink (Hbase Sink) ➢ Major project in Real Time `Product Log Analysis` 1. We are extracting the data from server logs 2. This data will be useful to do analysis on product views 3. JSON is the output format ➢ We can use Hbase to analyze this data

Kalyan Big Data Project 13
Project Name	How To Stream REGEX Data Into Hbase Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project13-hbase-regex
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Exec Source) 2. Channel (Memory Channel) 3. Sink (Hbase Sink) ➢ Major project in Real Time `Product Log Analysis` 1. We are extracting the data from server logs 2. This data will be useful to do analysis on product views 3. Complex Data is the output format then REGEX is best solution ➢ We can use Hbase to analyze this data

Kalyan Big Data Project 14
Project Name	How To Stream Text Data Into Hbase Using Apache Flume
Project Code	https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/flume/project14-hbase-text
Learnings of this Project	➢ We will learn Flume Configurations and Commands ➢ Flume Agent 1. Source (Netcat Source) 2. Channel (Memory Channel) 3. Sink (Hbase Sink) ➢ Major project in Real Time `Chat Applications` 1. We are extracting the data from Chat Applications 2. This data will be useful to do analysis on Sentiment on Tweets 3. Complex Data is the output format then REGEX is best solution ➢ We can use Hbase to analyze this data