Kafka Real Time Projects




List of Kafka Real Time Projects







Kalyan Big Data Project 1
Project Name
How To Stream CSV Data Into Phoenix Using Apache Kafka
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/kafka/project1-phoenix-csv


Learnings of this Project
We will learn Kafka Configurations and Commands
Kafka Information
1. Kalyan Util (CSV data generator)
2. Kafka Producer (Listen on CSV data)
3. Kafka Consumer (Recieves the data from Kafka Producer)
4. Phoenix Consumer (Write the data into Phoenix Table)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. CSV is the output format
We can use phoenix to analyze this data





Kalyan Big Data Project 2
Project Name
How To Stream JSON Data Into Phoenix Using Apache Kafka
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/kafka/project2-phoenix-json


Learnings of this Project
We will learn Kafka Configurations and Commands
Kafka Information
1. Kalyan Util (JSON data generator)
2. Kafka Producer (Listen on JSON data)
3. Kafka Consumer (Recieves the data from Kafka Producer)
4. Phoenix Consumer (Write the data into Phoenix Table)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. JSON is the output format
We can use phoenix to analyze this data





Kalyan Big Data Project 3
Project Name
How To Stream REGEX Data Into Phoenix Using Apache Kafka
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/kafka/project3-phoenix-regex


Learnings of this Project
We will learn Kafka Configurations and Commands
Kafka Information
1. Kalyan Util (Complex data generator)
2. Kafka Producer (Listen on Complex data)
3. Kafka Consumer (Recieves the data from Kafka Producer)
4. Phoenix Consumer (Write the data into Phoenix Table)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. Complex Data is the output format then REGEX is best solution
We can use phoenix to analyze this data




Kalyan Big Data Project 4
Project Name
How To Stream CSV Data Into Hadoop Using Apache Flume - Kafka Source
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/kafka/project4-flume-kafka-source


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Kafka Source)
2. Channel (Memory Channel)
3. Sink (Hdfs Sink)
We will learn Kafka Configurations and Commands
Kafka Information
1. Kalyan Util (CSV data generator)
2. Kafka Producer (Listen on CSV data)
3. Kafka Consumer (Recieves the data from Kafka Producer)
4. Flume Kafka Source (Will Send the Kafka Producer data to Flume Channel)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. CSV is the output format
We can use hive / pig / mapreduce to analyze this data
1. explore hive query to analysis
2. explore pig scripts to analysis
3. explore mapreduce to analysis




Kalyan Big Data Project 5
Project Name
How To Stream CSV Data Into Hadoop Using Apache Flume – Kafka Sink
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/kafka/project5-flume-kafka-sink


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Exec Source)
2. Channel (Memory Channel)
3. Sink (Kafka Sink)
We will learn Kafka Configurations and Commands
Kafka Information
1. Kalyan Util (CSV data generator)
2. Kafka Producer (Listen on CSV data)
3. Kafka Consumer (Recieves the data from Kafka Producer)
4. Flume Kafka Sink (Will Recieves the Kafka Sink data from Flume Channel)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. CSV is the output format
We can use hive / pig / mapreduce to analyze this data
1. explore hive query to analysis
2. explore pig scripts to analysis
3. explore mapreduce to analysis



Kalyan Big Data Project 6
Project Name
How To Stream CSV Data Into Hadoop Using Apache Flume - Kafka Channel
Project Code


https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/tree/master/kafka/project6-flume-kafka-channel


Learnings of this Project
We will learn Flume Configurations and Commands
Flume Agent
1. Source (Exec Source)
2. Channel (Kafka Channel)
3. Sink (Hdfs Sink)
We will learn Kafka Configurations and Commands
Kafka Information
1. Kalyan Util (CSV data generator)
2. Kafka Producer (Listen on CSV data)
3. Kafka Consumer (Recieves the data from Kafka Producer)
4. Flume Kafka Channel (Will Recieves the Kafka Channel data from Flume Source)
Major project in Real Time `Product Log Analysis`
1. We are extracting the data from server logs
2. This data will be useful to do analysis on product views
3. CSV is the output format
We can use hive / pig / mapreduce to analyze this data
1. explore hive query to analysis
2. explore pig scripts to analysis
3. explore mapreduce to analysis











1 comment :

Related Posts Plugin for WordPress, Blogger...