Twitter Data Sentiment Analysis Using Hive

Wednesday, 26 October 2016

Twitter Data Sentiment Analysis Using Hive

Pre-Requisites of Twitter Data + Hive + Sentiment Analysis Project:
hadoop-2.6.0
hive-1.2.1
java-1.7

NOTE: Make sure that install all the above components

Twitter Data + Hive + Sentiment Analysis Project Download Links:

`hadoop-2.6.0.tar.gz` ==> link
`apache-hive-1.2.1-src.tar.gz` ==> link
`sentimentanalysis-hive.jar` ==> link
`tweets` ==> link

-----------------------------------------------------------------------------

1. Create `sentimentanalysis` folder in your machine

command: mkdir ~/sentimentanalysis

2. Download sample tweets or Download twitter data using flume to do Sentiment Analysis and copy to '~/sentimentanalysis' folder

Note: Download sample tweets link

Example: Sample Tweets

i am learning hadoop course
i am good in hadoop
i am learning hadoop
i am not feeling well
why we need bigdata
i am not happy with rdbms
ravi is not working today
india got the world cup
learn hadoop from kalyan blog
learn spark from kalyan blog

3. verify using cat command

command: cat ~/sentimentanalysis/tweets

4. start the hadoop using below command

command: start-all.sh

5. verify is running or not using "jps" command

6. Open browser using below url

http://localhost:50070/dfshealth.jsp

7. Load the sample tweets into HDFS

hadoop fs -mkdir -p /kalyan/sentimentanalysis/hive/input

hadoop fs -put ~/sentimentanalysis/tweets /kalyan/sentimentanalysis/hive/input

8. Create kalyan database in hive using below command

CREATE DATABASE IF NOT EXISTS kalyan;

USE kalyan;

9. Create tweets table in hive with sample tweets

CREATE EXTERNAL TABLE IF NOT EXISTS kalyan.tweets (tweet string) LOCATION '/kalyan/sentimentanalysis/hive/input';

10. Display the tweets table data in hive using select query

SELECT * FROM kalyan.tweets;

11. Download `sentimentanalysis-hive.jar` file and copy to '~/sentimentanalysis' folder

Note: Download sentimentanalysis-hive.jar link

12. Load the `sentimentanalysis-hive.jar` into HDFS

hadoop fs -put ~/sentimentanalysis/sentimentanalysis-hive.jar /kalyan/sentimentanalysis/hive

13. Add jar file into hive class path using below command

ADD JAR <PATH OF THE JAR FILE>;

ADD JAR hdfs://localhost:8020/kalyan/sentimentanalysis/hive/sentimentanalysis-hive.jar;

14. Define the sentiment function in hive

Hive supports Temporary function and Permanent function:

i. Create Temporary function using below command

CREATE TEMPORARY FUNCTION <function name> AS 'UDF CLASS NAME WITH PACKAGE';

CREATE TEMPORARY FUNCTION sentiment AS 'com.orienit.kalyan.sentimentanalysis.hive.udf.SentimentUdf';

ii. Create Permanent function using below command

CREATE FUNCTION <db name>.<function name> AS 'UDF CLASS NAME WITH PACKAGE' USING JAR '<PATH OF THE JAR FILE>';

CREATE FUNCTION kalyan.sentiment AS 'com.orienit.kalyan.sentimentanalysis.hive.udf.SentimentUdf' USING JAR 'hdfs://localhost:8020/kalyan/sentimentanalysis/hive/sentimentanalysis-hive.jar';

15. Verify the function in hive using below command

SHOW FUNCTIONS;

16. Describe the function in hive using below command

DESCRIBE FUNCTION EXTENDED <function name>;

DESCRIBE FUNCTION EXTENDED sentiment;

DESCRIBE FUNCTION EXTENDED kalyan.sentiment;

17. Analyse the tweets using sentiment function using below commands

SELECT tweet, sentiment(tweet) FROM kalyan.tweets;

18. Create 'sentimenttweets' table in hive using below command

CREATE TABLE IF NOT EXISTS kalyan.sentimenttweets (tweet string, sentiment int) LOCATION '/kalyan/sentimentanalysis/hive/output';

19. Insert Sentiment tweets data into `sentimenttweets` table

INSERT OVERWRITE TABLE kalyan.sentimenttweets SELECT tweet, sentiment(tweet) FROM kalyan.tweets;

20. Retrieve sentiment tweets data from `sentimenttweets` table

SELECT tweet, sentiment FROM kalyan.sentimenttweets;

21. Retrieve sentiment tweets data from `sentimenttweets` table using case statement

SELECT tweet,
case
when sentiment = 1 then "positive"
when sentiment = 0 then "neutral"
when sentiment = -1 then "negative"
end
FROM kalyan.sentimenttweets;