How to generate large amount of sample data with simple techniques for Big Data Projects

Kalyan Big Data Projects

How to generate large amount of sample data with simple techniques for Big Data Projects

Follow the below commands to generate large amount of sample data.

Create 'kalyan_bigdata_projects' folder in user home (i.e /home/orienit)

Command: mkdir /home/orienit/kalyan_bigdata_projects

Download 'kalyan-bigdata-examples.jar' jar file from this link.
(https://github.com/kalyanhadooptraining/kalyan-bigdata-realtime-projects/blob/master/kalyan/kalyan-bigdata-examples.jar)


Copy 'kalyan-bigdata-examples.jar' jar file into '/home/orienit/kalyan_bigdata_projects' folder

We are going to learn below Use Cases

Use Case1: Generating Sample Server Logs with simple command
Use Case2: Generating Sample Users in JSON format with simple command
Use Case3: Generating Sample Users in CSV format with simple command
Use Case4: Generating Sample Users in TSV format with simple command
Use Case5: Generating Sample Users in DELIMITED format with simple command
Use Case6: Generating Sample Product Log in JSON format with simple command
Use Case7: Generating Sample Product Log in CSV format with simple command
Use Case8: Generating Sample Product Log in TSV format with simple command
Use Case9: Generating Sample Product Log in DELIMITED format with simple command


Use Case1: Generating Sample Server Logs with simple command

java -cp /home/orienit/kalyan_bigdata_projects/kalyan-bigdata-examples.jar \
com.orienit.kalyan.examples.GenerateServerLog \
-f /tmp/serverlog.txt \
-n 100 \
-s 10 \
-d 2016/01/01 \
-w 5

Read SERVER LOG data

Use Case: Generating Sample Users with simple command

java -cp /home/orienit/kalyan_bigdata_projects/kalyan-bigdata-examples.jar \
com.orienit.kalyan.examples.GenerateUsers

We can pass different arguments for above command

-d => field delimiter like (tab, comma, semicolon, etc )
-f => output file path
-n => number of users, maximum number is 10000
-s => starting number of user id, bydefault is 1
-w => waiting time in milli sec, bydefault is 100 millisec


Use Case2: Generating Sample Users in JSON format with simple command

java -cp /home/orienit/kalyan_bigdata_projects/kalyan-bigdata-examples.jar \
com.orienit.kalyan.examples.GenerateUsers \
-f /tmp/users.json \
-n 10 \
-s 1

Read JSON Data

Use Case3: Generating Sample Users in CSV format with simple command

java -cp /home/orienit/kalyan_bigdata_projects/kalyan-bigdata-examples.jar \
com.orienit.kalyan.examples.GenerateUsers \
-f /tmp/users.csv \
-d ',' \
-n 10 \
-s 1

Read CSV data

Use Case4: Generating Sample Users in TSV format with simple command

java -cp /home/orienit/kalyan_bigdata_projects/kalyan-bigdata-examples.jar \
com.orienit.kalyan.examples.GenerateUsers \
-f /tmp/users.tsv \
-d '\t' \
-n 10 \
-s 1

Read TSV data

Use Case5: Generating Sample Users in DELIMITED format with simple command

java -cp /home/orienit/kalyan_bigdata_projects/kalyan-bigdata-examples.jar \
com.orienit.kalyan.examples.GenerateUsers \
-f /tmp/users.txt \
-d '#' \
-n 10 \
-s 1

Read Any DELIMITED Data

Use Case: Generating Sample Product Log with simple command

java -cp /home/orienit/kalyan_bigdata_projects/kalyan-bigdata-examples.jar \
com.orienit.kalyan.examples.GenerateProductLog

We can pass different arguments for above command

-d => field delimiter like (tab, comma, semicolon, etc )
-f => output file path
-l => number of logs, maximum number is 100000
-n => number of users, maximum number is 10000
-w => waiting time in milli sec, bydefault is 100 millisec


Use Case6: Generating Sample Product Log in JSON format with simple command

java -cp /home/orienit/kalyan_bigdata_projects/kalyan-bigdata-examples.jar \
com.orienit.kalyan.examples.GenerateProductLog \
-f /tmp/productlog.json \
-n 10 \
-l 20

Read JSON data

Use Case7: Generating Sample Product Log in CSV format with simple command

java -cp /home/orienit/kalyan_bigdata_projects/kalyan-bigdata-examples.jar \
com.orienit.kalyan.examples.GenerateProductLog \
-f /tmp/productlog.csv \
-d ',' \
-n 10 \
-l 20

Read CSV data

Use Case8: Generating Sample Product Log in TSV format with simple command

java -cp /home/orienit/kalyan_bigdata_projects/kalyan-bigdata-examples.jar \
com.orienit.kalyan.examples.GenerateProductLog \
-f /tmp/productlog.tsv \
-d '\t' \
-n 10 \
-l 20

Read TSV data

Use Case9: Generating Sample Product Log in DELIMITED format with simple command

java -cp /home/orienit/kalyan_bigdata_projects/kalyan-bigdata-examples.jar \
com.orienit.kalyan.examples.GenerateProductLog \
-f /tmp/productlog.txt \
-d '#' \
-n 10 \
-l 20

Read Any DELIMITED data

How to generate large amount of sample data with simple techniques

Download the Required jar file from this link.

Generate Sample Users using Below Command

java -cp bigdata-examples-0.0.1-SNAPSHOT-dependency-jars.jar \
com.orienit.kalyan.examples.GenerateUsers

usage: help
-d,--delimiter Field Delimiter, bydefault is json format
-f,--file output file path
-h,--help Show this help and quit
-n,--numberOfUsers number of users
-s,--startNumber starting number of userid, bydefault is 1

Example1 : To Generate json data, use below command

java -cp bigdata-examples-0.0.1-SNAPSHOT-dependency-jars.jar \
com.orienit.kalyan.examples.GenerateUsers \
-f /tmp/users.json \
-n 10 \
-s 1

Example2 : To Generate csv data, use below command

java -cp bigdata-examples-0.0.1-SNAPSHOT-dependency-jars.jar \
com.orienit.kalyan.examples.GenerateUsers \
-f /tmp/users.csv \
-d ',' \
-n 10 \
-s 1

Example3 : To Generate tsv data, use below command

java -cp bigdata-examples-0.0.1-SNAPSHOT-dependency-jars.jar \
com.orienit.kalyan.examples.GenerateUsers \
-f /tmp/users.tsv \
-d '\t' \
-n 10 \
-s 1

Example4 : To Generate any delimiter data, use below command

java -cp bigdata-examples-0.0.1-SNAPSHOT-dependency-jars.jar \
com.orienit.kalyan.examples.GenerateUsers \
-f /tmp/users.txt \
-d '#' \
-n 10 \
-s 1

Generate Sample Product Log using Below Command

java -cp bigdata-examples-0.0.1-SNAPSHOT-dependency-jars.jar \
com.orienit.kalyan.examples.GenerateProductLog

usage: help
-d,--delimiter Field Delimiter, bydefault is json format
-f,--file output file path
-h,--help Show this help and quit
-l,--numberOfLogs number of logs
-n,--numberOfUsers number of users

Example1 : To Generate json data, use below command

java -cp bigdata-examples-0.0.1-SNAPSHOT-dependency-jars.jar \
com.orienit.kalyan.examples.GenerateProductLog \
-f /tmp/productlog.json \
-n 10 \
-l 20

Example2 : To Generate csv data, use below command

java -cp bigdata-examples-0.0.1-SNAPSHOT-dependency-jars.jar \
com.orienit.kalyan.examples.GenerateProductLog \
-f /tmp/productlog.csv \
-d ',' \
-n 10 \
-l 20

Example3 : To Generate tsv data, use below command

java -cp bigdata-examples-0.0.1-SNAPSHOT-dependency-jars.jar \
com.orienit.kalyan.examples.GenerateProductLog \
-f /tmp/productlog.tsv \
-d '\t' \
-n 10 \
-l 20

Example4 : To Generate any delimiter data, use below command

java -cp bigdata-examples-0.0.1-SNAPSHOT-dependency-jars.jar \
com.orienit.kalyan.examples.GenerateProductLog \
-f /tmp/productlog.txt \
-d '#' \
-n 10 \
-l 20