Kalyan Hadoop and Spark Training in Hyderabad Learn Big Data From Basics... @ Kalyan @

Mr.Kalyan, Apache Contributor, Cloudera CCA175 Certified Consultant, 8+ years of Big Data exp, IIT Kharagpur, Gold Medalist.

This blog is mainly meant for Learn Big Data From Basics
1. Development practices
2. Administration practices
3. Interview Questions
4. Big Data integrations
5. Advanced Technologies in Big Data
6. Become more strong on Big Data

Call for Spark & Hadoop Training in Hyderabad, ORIENIT @ 040 65142345 , 9703202345

Friday, 7 April 2017

SPARK BASICS Practice on 02 Apr 2017

`Spark` is meant for `In-Memory Distributed Computing`

`Spark` provides 4 libraries:
1. Spark SQL
2. Spark Streaming
3. Spark MLLib
4. Spark GraphX

`Spark Context` is the Entry point for any `Spark Operations`

`Resilient Distributed DataSets` => RDD

RDD features:
-------------------
1. immutability
2. lazy evaluation
3. cacheable
4. type infer

RDD operations:
-----------------
1. Transformations
<old rdd> ----> <new rdd>

2. Actions
<rdd> ---> <result>

Examples on RDD:
-------------------------
list <- {1,2,3,4,5}

1. Transformations:
---------------------
Ex1:
-----
f(x) <- {x + 1}

f(list) <- {2,3,4,5,6}

Ex2:
-----
f(x) <- {x * x}

f(list) <- {1,4,9,16,25}

2. Actions:
---------------------

sum(list) -> 15
min(list) -> 1
max(list) -> 5

How to Start the Spark:
-----------------------------
scala => spark-shell
python => pyspark
R => sparkR

Spark-1.x:
--------------------------------------
Spark context available as 'sc'
Spark Sql Context available as 'sqlContext'

Spark-2.x:
--------------------------------------
Spark context available as 'sc'
Spark session available as 'spark'

How to Create a RDD:
---------------------------------------
We can create RDD 2 ways
1. from collections (List, Seq, Set, ....)
2. from data sets (text, csv, tsv, json, ...)

1. from collections:
---------------------------------------
val list = List(1,2,3,4,5)

val rdd = sc.parallelize(list)

val rdd = sc.parallelize(list, 2)

Syntax:
-----------------------
val rdd = sc.parallelize(<collection object>, <no.of partitions>)

2. from datasets:
---------------------------------------
val file = "file:///home/orienit/work/input/demoinput"

val rdd = sc.textFile(file)

val rdd = sc.textFile(file, 1)

Syntax:
-----------------------
val rdd = sc.textFile(<file path>, <no.of partitions>)

------------------------------------------------------

scala> val list = List(1,2,3,4,5)
list: List[Int] = List(1, 2, 3, 4, 5)

scala> val rdd = sc.parallelize(list)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:26

scala> rdd.getNumPartitions
res0: Int = 4

scala> val rdd = sc.parallelize(list, 2)
rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[1] at parallelize at <console>:26

scala> rdd.getNumPartitions
res1: Int = 2

------------------------------------------------------

scala> val file = "file:///home/orienit/work/input/demoinput"
file: String = file:///home/orienit/work/input/demoinput

scala> val rdd = sc.textFile(file)
rdd: org.apache.spark.rdd.RDD[String] = file:///home/orienit/work/input/demoinput MapPartitionsRDD[3] at textFile at <console>:26

scala> rdd.getNumPartitions
res2: Int = 2

scala> val rdd = sc.textFile(file, 1)
rdd: org.apache.spark.rdd.RDD[String] = file:///home/orienit/work/input/demoinput MapPartitionsRDD[5] at textFile at <console>:26

scala> rdd.getNumPartitions
res3: Int = 1

------------------------------------------------------
Examples on RDD
------------------------------------------------------
val list = List(1,2,3,4,5)

val rdd1 = sc.parallelize(list, 2)

------------------------------------------------------

val rdd2 = rdd1.map(x => x + 1)

val rdd3 = rdd1.map(x => x +* x)

------------------------------------------------------

scala> val list = List(1,2,3,4,5)
list: List[Int] = List(1, 2, 3, 4, 5)

scala> val rdd1 = sc.parallelize(list, 2)
rdd1: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[6] at parallelize at <console>:26

------------------------------------------------------

scala> val rdd2 = rdd1.map(x => x + 1)
rdd2: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[7] at map at <console>:28

scala> rdd1.collect
res4: Array[Int] = Array(1, 2, 3, 4, 5)

scala> rdd2.collect
res5: Array[Int] = Array(2, 3, 4, 5, 6)

scala> val rdd3 = rdd1.map(x => x * x)
rdd3: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[8] at map at <console>:28

scala> rdd3.collect
res6: Array[Int] = Array(1, 4, 9, 16, 25)

------------------------------------------------------

rdd1.min

rdd1.max

rdd1.sum

------------------------------------------------------
scala> rdd1.min
res7: Int = 1

scala> rdd1.max
res8: Int = 5

scala> rdd1.sum
res9: Double = 15.0

scala> rdd1.collect
res10: Array[Int] = Array(1, 2, 3, 4, 5)

scala> rdd1.count
res11: Long = 5

------------------------------------------------------
Word Count in Spark using Scala
------------------------------------------------------

val input = "file:///home/orienit/work/input/demoinput"

val output = "file:///home/orienit/work/output/spark-op"

val fileRdd = sc.textFile(input, 1)

val wordsRdd = fileRdd.flatMap(line => line.split(" "))

val tuplesRdd = wordsRdd.map(word => (word, 1))

val wordCountRdd = tuplesRdd.reduceByKey((a,b) => a + b)

wordCountRdd.saveAsTextFile(output)

------------------------------------------------------
Optimize the Code :
------------------------------------------------------

val input = "file:///home/orienit/work/input/demoinput"

val output = "file:///home/orienit/work/output/spark-op"

val fileRdd = sc.textFile(input, 1)

val wordCountRdd = fileRdd.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey((a,b) => a + b)

wordCountRdd.saveAsTextFile(output)

------------------------------------------------------

scala> val input = "file:///home/orienit/work/input/demoinput"
input: String = file:///home/orienit/work/input/demoinput

scala> val output = "file:///home/orienit/work/output/spark-op"
output: String = file:///home/orienit/work/output/spark-op

scala> val fileRdd = sc.textFile(input, 1)
fileRdd: org.apache.spark.rdd.RDD[String] = file:///home/orienit/work/input/demoinput MapPartitionsRDD[11] at textFile at <console>:26

scala> fileRdd.collect
res12: Array[String] = Array(I am going, to hyd, I am learning, hadoop course)

scala> val wordsRdd = fileRdd.flatMap(line => line.split(" "))
wordsRdd: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[12] at flatMap at <console>:28

scala> wordsRdd.collect
res13: Array[String] = Array(I, am, going, to, hyd, I, am, learning, hadoop, course)

scala> val tuplesRdd = wordsRdd.map(word => (word, 1))
tuplesRdd: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[13] at map at <console>:30

scala> tuplesRdd.collect
res14: Array[(String, Int)] = Array((I,1), (am,1), (going,1), (to,1), (hyd,1), (I,1), (am,1), (learning,1), (hadoop,1), (course,1))

scala> val wordCountRdd = tuplesRdd.reduceByKey((a,b) => a + b)
wordCountRdd: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[14] at reduceByKey at <console>:32

scala> wordCountRdd.collect
res15: Array[(String, Int)] = Array((learning,1), (hadoop,1), (am,2), (hyd,1), (I,2), (to,1), (going,1), (course,1))

scala> wordCountRdd.saveAsTextFile(output)

------------------------------------------------------
Grep Job in Spark using Scala:
------------------------------------------------------

val input = "file:///home/orienit/work/input/demoinput"

val output = "file:///home/orienit/work/output/spark-grep-op"

val fileRdd = sc.textFile(input, 1)

val grepRdd = fileRdd.filter(line => line.contains("am"))

grepRdd.saveAsTextFile(output)

------------------------------------------------------
Sed Job in Spark using Scala:
------------------------------------------------------

val input = "file:///home/orienit/work/input/demoinput"

val output = "file:///home/orienit/work/output/spark-sed-op"

val fileRdd = sc.textFile(input, 1)

val sedRdd = fileRdd.map(line => line.replaceAll("am", "xyz"))

sedRdd.saveAsTextFile(output)

------------------------------------------------------
Spark SQL:
------------------------------------------------------

hive> select * from kalyan.student;
scala> spark.sql("select * from kalyan.student").show

------------------------------------------------------

hive> select year, count(*) from kalyan.student group by year;
scala> spark.sql("select year, count(*) from kalyan.student group by year").show

------------------------------------------------------
Data Frames in Spark
------------------------------------------------------
val hiveDf = spark.sql("select * from kalyan.student")

hiveDf.show

hiveDf.registerTempTable("hivetbl")

val prop = new java.util.Properties
prop.setProperty("driver","com.mysql.jdbc.Driver")
prop.setProperty("user","root")
prop.setProperty("password","hadoop")

val jdbcDf = spark.read.jdbc("jdbc:mysql://localhost:3306/kalyan", "student", prop)

jdbcDf.show

jdbcDf.registerTempTable("jdbctbl")

------------------------------------------------------
scala> val hiveDf = spark.sql("select * from kalyan.student")
hiveDf: org.apache.spark.sql.DataFrame = [name: string, id: int ... 2 more fields]

scala> hiveDf.show
+------+---+------+----+
| name| id|course|year|
+------+---+------+----+
| arun| 1| cse| 1|
| sunil| 2| cse| 1|
| raj| 3| cse| 1|
|naveen| 4| cse| 1|
| venki| 5| cse| 2|
|prasad| 6| cse| 2|
| sudha| 7| cse| 2|
| ravi| 1| mech| 1|
| raju| 2| mech| 1|
| roja| 3| mech| 1|
| anil| 4| mech| 2|
| rani| 5| mech| 2|
|anvith| 6| mech| 2|
| madhu| 7| mech| 2|
| arun| 1| it| 3|
| sunil| 2| it| 3|
| raj| 3| it| 3|
|naveen| 4| it| 3|
| venki| 5| it| 4|
|prasad| 6| it| 4|
+------+---+------+----+
only showing top 20 rows

------------------------------------------------------

scala> val jdbcDf = spark.read.jdbc("jdbc:mysql://localhost:3306/kalyan", "student", prop)
jdbcDf: org.apache.spark.sql.DataFrame = [name: string, id: int ... 2 more fields]

scala> jdbcDf.show
+------+---+------+----+
| name| id|course|year|
+------+---+------+----+
| anil| 1| spark|2016|
|anvith| 5|hadoop|2015|
| dev| 6|hadoop|2015|
| raj| 3| spark|2016|
| sunil| 4|hadoop|2015|
|venkat| 2| spark|2016|
+------+---+------+----+

------------------------------------------------------

scala> hiveDf.registerTempTable("hivetbl")
warning: there was one deprecation warning; re-run with -deprecation for details

scala> jdbcDf.registerTempTable("jdbctbl")
warning: there was one deprecation warning; re-run with -deprecation for details

------------------------------------------------------

spark.sql("select hivetbl.*, jdbctbl.* from hivetbl join jdbctbl on hivetbl.name = jdbctbl.name").show

------------------------------------------------------

scala> spark.sql("select hivetbl.*, jdbctbl.* from hivetbl join jdbctbl on hivetbl.name = jdbctbl.name").show
+------+---+------+----+------+---+------+----+
| name| id|course|year| name| id|course|year|
+------+---+------+----+------+---+------+----+
| anil| 4| ece| 4| anil| 1| spark|2016|
| anil| 4| mech| 2| anil| 1| spark|2016|
|anvith| 6| ece| 4|anvith| 5|hadoop|2015|
|anvith| 6| mech| 2|anvith| 5|hadoop|2015|
| raj| 3| it| 3| raj| 3| spark|2016|
| raj| 3| cse| 1| raj| 3| spark|2016|
| sunil| 2| it| 3| sunil| 4|hadoop|2015|
| sunil| 2| cse| 1| sunil| 4|hadoop|2015|
+------+---+------+----+------+---+------+----+

------------------------------------------------------

val casDf = sqlContext.read.cassandraFormat("student", "kalyan").load()

casDf.show

------------------------------------------------------

scala> val casDf = sqlContext.read.cassandraFormat("student", "kalyan").load()
casDf: org.apache.spark.sql.DataFrame = [name: string, course: string ... 2 more fields]

scala> casDf.show
+------+------+---+----+
| name|course| id|year|
+------+------+---+----+
| anil| spark| 1|2016|
| raj| spark| 3|2016|
|anvith|hadoop| 5|2015|
| dev|hadoop| 6|2015|
| sunil|hadoop| 4|2015|
|venkat| spark| 2|2016|
| kk|hadoop| 7|2015|
+------+------+---+----+

------------------------------------------------------
INSERT INTO kalyan.student(name, id, course, year) VALUES ('rajesh', 8, 'hadoop', 2017);

------------------------------------------------------

scala> casDf.show
+------+------+---+----+
| name|course| id|year|
+------+------+---+----+
| anil| spark| 1|2016|
| raj| spark| 3|2016|
|anvith|hadoop| 5|2015|
| dev|hadoop| 6|2015|
| sunil|hadoop| 4|2015|
|venkat| spark| 2|2016|
| kk|hadoop| 7|2015|
|rajesh|hadoop| 8|2017|
+------+------+---+----+

------------------------------------------------------

cqlsh:kalyan> select year, count(*) from kalyan.student group by year;
SyntaxException: line 1:42 missing EOF at 'group' (...(*) from kalyan.student [group] by...)

------------------------------------------------------

casDf.registerTempTable("castbl")

spark.sql("select year, count(*) from castbl group by year").show

------------------------------------------------------

scala> casDf.registerTempTable("castbl")
warning: there was one deprecation warning; re-run with -deprecation for details

scala> spark.sql("select year, count(*) from castbl group by year").show
+----+--------+
|year|count(1)|
+----+--------+
|2015| 4|
|2016| 3|
|2017| 1|
+----+--------+

------------------------------------------------------
spark.sql("select castbl.*, jdbctbl.* from castbl join jdbctbl on castbl.name = jdbctbl.name").show
------------------------------------------------------

scala> spark.sql("select castbl.*, jdbctbl.* from castbl join jdbctbl on castbl.name = jdbctbl.name").show
+------+------+---+----+------+---+------+----+
| name|course| id|year| name| id|course|year|
+------+------+---+----+------+---+------+----+
| anil| spark| 1|2016| anil| 1| spark|2016|
|anvith|hadoop| 5|2015|anvith| 5|hadoop|2015|
| dev|hadoop| 6|2015| dev| 6|hadoop|2015|
| raj| spark| 3|2016| raj| 3| spark|2016|
| sunil|hadoop| 4|2015| sunil| 4|hadoop|2015|
|venkat| spark| 2|2016|venkat| 2| spark|2016|
+------+------+---+----+------+---+------+----+

------------------------------------------------------
scala> casDf.toJSON.collect.foreach(println)
{"name":"anil","course":"spark","id":1,"year":2016}
{"name":"raj","course":"spark","id":3,"year":2016}
{"name":"anvith","course":"hadoop","id":5,"year":2015}
{"name":"dev","course":"hadoop","id":6,"year":2015}
{"name":"sunil","course":"hadoop","id":4,"year":2015}
{"name":"venkat","course":"spark","id":2,"year":2016}
{"name":"kk","course":"hadoop","id":7,"year":2015}
{"name":"rajesh","course":"hadoop","id":8,"year":2017}

scala> hiveDf.toJSON.collect.foreach(println)
{"name":"arun","id":1,"course":"cse","year":1}
{"name":"sunil","id":2,"course":"cse","year":1}
{"name":"raj","id":3,"course":"cse","year":1}
{"name":"naveen","id":4,"course":"cse","year":1}
{"name":"venki","id":5,"course":"cse","year":2}
{"name":"prasad","id":6,"course":"cse","year":2}
{"name":"sudha","id":7,"course":"cse","year":2}
{"name":"ravi","id":1,"course":"mech","year":1}
{"name":"raju","id":2,"course":"mech","year":1}
{"name":"roja","id":3,"course":"mech","year":1}
{"name":"anil","id":4,"course":"mech","year":2}
{"name":"rani","id":5,"course":"mech","year":2}
{"name":"anvith","id":6,"course":"mech","year":2}
{"name":"madhu","id":7,"course":"mech","year":2}

scala> jdbcDf.toJSON.collect.foreach(println)
{"name":"anil","id":1,"course":"spark","year":2016}
{"name":"anvith","id":5,"course":"hadoop","year":2015}
{"name":"dev","id":6,"course":"hadoop","year":2015}
{"name":"raj","id":3,"course":"spark","year":2016}
{"name":"sunil","id":4,"course":"hadoop","year":2015}
{"name":"venkat","id":2,"course":"spark","year":2016}

------------------------------------------------------

SCALA BASICS Practice on 01 Apr 2017

`Scala` means `Scalable Language`

`Scala`is Functional + Object Oriented Programming Language

In Java:
------------
1. Primitive data types (int, float, double, long ...)
2. Wrapper Classes (integer, Float, Double, Long ..)

Wrapper Classes are possible to do `Serialization and DeSerialization`

Java Syntax:
-----------------
<data type> <variable name> = <data> ;

In Scala:
-------------------
Everything is `Object`

Scala Syntax:
-----------------
val <variable name> : <data type> = <data>

var <variable name> : <data type> = <data>

val => value => it is immutable (we can't change the reference)

var => variable => it is mutable (we can change the reference)

orienit@kalyan:~$ scala
Welcome to Scala 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_66-internal).
Type in expressions for evaluation. Or try :help.

scala>

`Scala` Provides `REPL` functionality.

`REPL` means Read Evaluate Print Loop

`REPL` functionality already there in `Python` and `R` programming languages

---------------------------------------------------
scala> val name : String = "kalyan"
name: String = kalyan

scala> name = "xyz"
<console>:12: error: reassignment to val
name = "xyz"
^

scala> var name : String = "kalyan"
name: String = kalyan

scala> name = "xyz"
name: String = xyz

---------------------------------------------------
`Scala` provides `Type Infer`
---------------------------------------------------
Based on the `data` it will find the `data type`

scala> val name : String = "kalyan"
name: String = kalyan

scala> val name = "kalyan"
name: String = kalyan

scala> val id = 1
id: Int = 1

scala> val id = 1.5
id: Double = 1.5

scala> val id = 1l
id: Long = 1

scala> val id = 1f
id: Float = 1.0

---------------------------------------------------
scala> val id = 1
id: Int = 1

scala> val id = 1l
id: Long = 1

scala> val id = 1f
id: Float = 1.0

scala> val id = 1d
id: Double = 1.0

scala> val id : Long = 1
id: Long = 1

scala> val id : Float = 1
id: Float = 1.0

scala> val id : Double = 1
id: Double = 1.0

---------------------------------------------------

scala> a.toInt
res4: Int = 1

scala> a.toDouble
res5: Double = 1.0

scala> a.toFloat
res6: Float = 1.0

scala> a.toLong
res7: Long = 1

scala> a.toChar
res8: Char = ?

---------------------------------------------------
`Scala` Provides `Operator Overloading` simillar to `C++`
---------------------------------------------------
scala> val a = 1
a: Int = 1

scala> val b = 2
b: Int = 2

scala> val c = a + b
c: Int = 3

scala> val c = a.+(b)
c: Int = 3

scala> val c = a.-(b)
c: Int = -1

scala> val c = a.*(b)
c: Int = 2

---------------------------------------------------

a + b <===> a.+(b)

a - b <===> a.-(b)

a * b <===> a.*(b)

---------------------------------------------------

scala> a < b
res0: Boolean = true

scala> a <= b
res1: Boolean = true

scala> a >= b
res2: Boolean = false

scala> a > b
res3: Boolean = false

---------------------------------------------------

scala> a.
!= >> isInfinite min toInt
% >>> isInfinity round toLong
& ^ isNaN self toOctalString
* abs isNegInfinity shortValue toRadians
+ byteValue isPosInfinity signum toShort
- ceil isValidByte to unary_+
/ compare isValidChar toBinaryString unary_-
< compareTo isValidInt toByte unary_~
<< doubleValue isValidLong toChar underlying
<= floatValue isValidShort toDegrees until
== floor isWhole toDouble |
> getClass longValue toFloat
>= intValue max toHexString

---------------------------------------------------
if, if else, if else if expressions in Scala
---------------------------------------------------

if(exp1) { body1 }

if(exp1) {
body1
}

---------------------------------------------------

if(exp1) { body1 } else { body2 }

if(exp1) {
body1
} else {
body2
}

Wrong in `Scala Prompt`
---------------------------------------------------
if(exp1) {
body1
}
else {
body2
}

if(exp1) {
body1
} else
{
body2
}

---------------------------------------------------
if(exp1) { body1 } else if(exp2) { body2 }

if(exp1) {
body1
} else if(exp2) {
body2
}

if(exp1) {
body1
} else if(exp2) {
body2
} else if(exp3) {
body3
}

---------------------------------------------------
val a = 10
val b = 20
val c = 30

---------------------------------------------------

scala> val a = 10
a: Int = 10

scala> val b = 20
b: Int = 20

scala> val c = 30
c: Int = 30

scala> if(a > b) println("nothing")

scala> if(a < b) println("i am there")
i am there

scala> if(a < b) println("i am there") else println("nothing")
i am there

scala> if(a > b) println("i am there") else println("nothing")
nothing

scala> if(a > b) println("i am there") else if(a < b) println("nothing")
nothing

---------------------------------------------------
Arrays in Java:
---------------------------------------------------
<data type>[] <variable name> = {};

<data type>[] <variable name> = new <data type>[size];

String[] names = {"anil", "raj", "venkat"}

or

String[] names = new String[3];
names[0] = "anil";
names[1] = "raj";
names[2] = "venkat";

---------------------------------------------------
Arrays in Scala:
---------------------------------------------------
val <variable name> : Array[<data type] = Array[<data type](...)

val <variable name> : Array[<data type] = new Array[<data type](size)

val names : Array[String] = Array[String]("anil", "raj", "venkat")

or

val names : Array[String] = new Array[String](3)
names(0) = "anil"
names(1) = "raj"
names(2) = "venkat"

---------------------------------------------------

scala> val names : Array[String] = Array[String]("anil", "raj", "venkat")
names: Array[String] = Array(anil, raj, venkat)

scala> names(0)
res14: String = anil

scala> names(1)
res15: String = raj

scala> names(2)
res16: String = venkat

---------------------------------------------------

scala> val names : Array[String] = new Array[String](3)
names: Array[String] = Array(null, null, null)

scala> names(0) = "anil"

scala> names(1) = "raj"

scala> names(2) = "venkat"

scala> names
res20: Array[String] = Array(anil, raj, venkat)

scala> names(0)
res21: String = anil

scala> names(1)
res22: String = raj

scala> names(2)
res23: String = venkat

---------------------------------------------------

scala> val names = Array[String]("anil", "raj", "venkat")
names: Array[String] = Array(anil, raj, venkat)

scala> val names = Array("anil", "raj", "venkat")
names: Array[String] = Array(anil, raj, venkat)

---------------------------------------------------

val nums = Array(1,2,3,4,5,6,7,8,9,10)

---------------------------------------------------
scala> val nums = Array(1,2,3,4,5,6,7,8,9,10)
nums: Array[Int] = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

scala> for(x <- nums) println(x)
1
2
3
4
5
6
7
8
9
10

scala> for(x <- nums) if(x %2 == 0) println(x)
2
4
6
8
10

scala> for(x <- nums if(x %2 == 0) ) println(x)
2
4
6
8
10

scala> for(x <- nums) if(x %2 == 0) println("Even Number: " + x) else println("Odd Number: " + x)
Odd Number: 1
Even Number: 2
Odd Number: 3
Even Number: 4
Odd Number: 5
Even Number: 6
Odd Number: 7
Even Number: 8
Odd Number: 9
Even Number: 10

---------------------------------------------------
for(x <- nums)
if(x %2 == 0) {
println("Even Number: " + x)
} else {
println("Odd Number: " + x)
}
---------------------------------------------------

scala> for(x <- nums)
| if(x %2 == 0) {
| println("Even Number: " + x)
| } else {
| println("Odd Number: " + x)
| }
Odd Number: 1
Even Number: 2
Odd Number: 3
Even Number: 4
Odd Number: 5
Even Number: 6
Odd Number: 7
Even Number: 8
Odd Number: 9
Even Number: 10

---------------------------------------------------
String Interpolation:
---------------------------------------------------
val name = "kalyan"
val course = "spark"
val percentage = 80.5
val count = 100
---------------------------------------------------

val exp1 = "name is " + name + ", course is " + course

val exp2 = "name is $name, course is $course"

val exp3 = s"name is $name, course is $course"

val exp4 = s"name is $name, percentage is $percentage"

val exp5 = s"name is $name, percentage is $percentage%.3f"

val exp6 = f"name is $name, percentage is $percentage%.3f"

val exp7 = s"name is $name\ncourse is $course"

val exp8 = raw"name is $name\ncourse is $course"

---------------------------------------------------
---------------------------------------------------
scala> val name = "kalyan"
name: String = kalyan

scala> val course = "spark"
course: String = spark

scala> val percentage = 80.5
percentage: Double = 80.5

scala> val count = 100
count: Int = 100

---------------------------------------------------

scala> val exp1 = "name is " + name + ", course is " + course
exp1: String = name is kalyan, course is spark

scala> val exp2 = "name is $name, course is $course"
exp2: String = name is $name, course is $course

scala> val exp3 = s"name is $name, course is $course"
exp3: String = name is kalyan, course is spark

scala> val exp4 = s"name is $name, percentage is $percentage"
exp4: String = name is kalyan, percentage is 80.5

scala> val exp5 = s"name is $name, percentage is $percentage%.3f"
exp5: String = name is kalyan, percentage is 80.5%.3f

scala> val exp6 = f"name is $name, percentage is $percentage%.3f"
exp6: String = name is kalyan, percentage is 80.500

scala> val exp7 = s"name is $name\ncourse is $course"
exp7: String =
name is kalyan
course is spark

scala> val exp8 = raw"name is $name\ncourse is $course"
exp8: String = name is kalyan\ncourse is spark

---------------------------------------------------
Collections in `Scala`
---------------------------------------------------
In scala we have 2 types of collections:

1. immutable collections (scala.collection.immutable)

2. mutable collections (scala.collection.mutable)

---------------------------------------------------

scala> import scala.collection.immutable.
:: LongMap SortedMap
AbstractMap LongMapEntryIterator SortedSet
BitSet LongMapIterator Stack
DefaultMap LongMapKeyIterator Stream
HashMap LongMapUtils StreamIterator
HashSet LongMapValueIterator StreamView
IndexedSeq Map StreamViewLike
IntMap MapLike StringLike
IntMapEntryIterator MapProxy StringOps
IntMapIterator Nil Traversable
IntMapKeyIterator NumericRange TreeMap
IntMapUtils Page TreeSet
IntMapValueIterator PagedSeq TrieIterator
Iterable Queue Vector
LinearSeq Range VectorBuilder
List RedBlackTree VectorIterator
ListMap Seq VectorPointer
ListSerializeEnd Set WrappedString
ListSet SetProxy

---------------------------------------------------

scala> import scala.collection.mutable.
AVLIterator LinkedListLike
AVLTree ListBuffer
AbstractBuffer ListMap
AbstractIterable LongMap
AbstractMap Map
AbstractSeq MapBuilder
AbstractSet MapLike
AnyRefMap MapProxy
ArrayBuffer MultiMap
ArrayBuilder MutableList
ArrayLike ObservableBuffer
ArrayOps ObservableMap
ArraySeq ObservableSet
ArrayStack OpenHashMap
BitSet PriorityQueue
Buffer PriorityQueueProxy
BufferLike Publisher
BufferProxy Queue
Builder QueueProxy
Cloneable ResizableArray
DefaultEntry RevertibleHistory
DefaultMapModel Seq
DoubleLinkedList SeqLike
DoubleLinkedListLike Set
FlatHashTable SetBuilder
GrowingBuilder SetLike
HashEntry SetProxy
HashMap SortedSet
HashSet Stack
HashTable StackProxy
History StringBuilder
ImmutableMapAdaptor Subscriber
ImmutableSetAdaptor SynchronizedBuffer
IndexedSeq SynchronizedMap
IndexedSeqLike SynchronizedPriorityQueue
IndexedSeqOptimized SynchronizedQueue
IndexedSeqView SynchronizedSet
Iterable SynchronizedStack
LazyBuilder Traversable
Leaf TreeSet
LinearSeq Undoable
LinkedEntry UnrolledBuffer
LinkedHashMap WeakHashMap
LinkedHashSet WrappedArray
LinkedList WrappedArrayBuilder

---------------------------------------------------

---------------------------------------------------
val list = List(1,2,3,4,5)

val list = List[Int](1,2,3,4,5)

val seq = Seq[Int](1,2,3,4,5)

val set = Set[Int](1,2,3,4,5)

val vec = Vector[Int](1,2,3,4,5)

val str = Stream[Int](1,2,3,4,5)

---------------------------------------------------

val st = Stack[Int](1,2,3,4,5)

val qu = Queue[Int](1,2,3,4,5)

---------------------------------------------------
scala> val st = Stack[Int](1,2,3,4,5)
<console>:11: error: not found: value Stack
val st = Stack[Int](1,2,3,4,5)
^
scala> val qu = Queue[Int](1,2,3,4,5)
<console>:11: error: not found: value Queue
val qu = Queue[Int](1,2,3,4,5)
^

---------------------------------------------------
val st1 = scala.collection.immutable.Stack[Int](1,2,3,4,5)

val qu1 = scala.collection.immutable.Queue[Int](1,2,3,4,5)

val st2 = scala.collection.mutable.Stack[Int](1,2,3,4,5)

val qu2 = scala.collection.mutable.Queue[Int](1,2,3,4,5)

---------------------------------------------------

scala> val st1 = scala.collection.immutable.Stack[Int](1,2,3,4,5)
st1: scala.collection.immutable.Stack[Int] = Stack(1, 2, 3, 4, 5)

scala> val qu1 = scala.collection.immutable.Queue[Int](1,2,3,4,5)
qu1: scala.collection.immutable.Queue[Int] = Queue(1, 2, 3, 4, 5)

scala> val st2 = scala.collection.mutable.Stack[Int](1,2,3,4,5)
st2: scala.collection.mutable.Stack[Int] = Stack(1, 2, 3, 4, 5)

scala> val qu2 = scala.collection.mutable.Queue[Int](1,2,3,4,5)
qu2: scala.collection.mutable.Queue[Int] = Queue(1, 2, 3, 4, 5)

---------------------------------------------------

scala> val list = List(1,2,3,4,5)
list: List[Int] = List(1, 2, 3, 4, 5)

scala> val list = List[Int](1,2,3,4,5)
list: List[Int] = List(1, 2, 3, 4, 5)

scala> val seq = Seq[Int](1,2,3,4,5)
seq: Seq[Int] = List(1, 2, 3, 4, 5)

scala> val set = Set[Int](1,2,3,4,5)
set: scala.collection.immutable.Set[Int] = Set(5, 1, 2, 3, 4)

scala> val vec = Vector[Int](1,2,3,4,5)
vec: scala.collection.immutable.Vector[Int] = Vector(1, 2, 3, 4, 5)

scala> val str = Stream[Int](1,2,3,4,5)
str: scala.collection.immutable.Stream[Int] = Stream(1, ?)

---------------------------------------------------

val list = List(1,2,3,4,5)

or

val list = 1 :: 2 :: 3 :: 4 :: 5 :: Nil

or

val list = 1 :: (2 :: (3 :: (4 :: (5 :: Nil))))

---------------------------------------------------

scala> val list = List(1,2,3,4,5)
list: List[Int] = List(1, 2, 3, 4, 5)

scala> val list = 1 :: 2 :: 3 :: 4 :: 5 :: Nil
list: List[Int] = List(1, 2, 3, 4, 5)

scala> val list = 1 :: (2 :: (3 :: (4 :: (5 :: Nil))))
list: List[Int] = List(1, 2, 3, 4, 5)

---------------------------------------------------

scala> var list = List(4,5,6)
list: List[Int] = List(4, 5, 6)

scala> list :+ 7
res0: List[Int] = List(4, 5, 6, 7)

scala> 3 +: list
res1: List[Int] = List(3, 4, 5, 6)

---------------------------------------------------

var list = List(4,5,6)

list = 3 +: list

list = list :+ 7

list = 2 +: list :+ 8

---------------------------------------------------
scala> var list = List(4,5,6)
list: List[Int] = List(4, 5, 6)

scala> list = 3 +: list
list: List[Int] = List(3, 4, 5, 6)

scala> list = list :+ 7
list: List[Int] = List(3, 4, 5, 6, 7)

scala> list = 2 +: list :+ 8
list: List[Int] = List(2, 3, 4, 5, 6, 7, 8)

---------------------------------------------------

val list1 = List(1,2,3,4,5)

val list2 = List(6,7,8,9,10)

---------------------------------------------------

scala> val list1 = List(1,2,3,4,5)
list1: List[Int] = List(1, 2, 3, 4, 5)

scala> val list2 = List(6,7,8,9,10)
list2: List[Int] = List(6, 7, 8, 9, 10)

scala> val list3 = list1 +: list2
list3: List[Any] = List(List(1, 2, 3, 4, 5), 6, 7, 8, 9, 10)

scala> val list3 = list2 +: list1
list3: List[Any] = List(List(6, 7, 8, 9, 10), 1, 2, 3, 4, 5)

scala> val list3 = list1 :+ list2
list3: List[Any] = List(1, 2, 3, 4, 5, List(6, 7, 8, 9, 10))

scala> val list3 = list1 ::: list2
list3: List[Int] = List(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

---------------------------------------------------
Functions in Scala:
---------------------------------------------------
1. anaonymus functions
2. named functions
3. curried functions
---------------------------------------------------

1. anaonymus functions
---------------------------------------------------
(x : Int, y : Int) => x + y

val add = (x : Int, y : Int) => x + y

add(1,2)

---------------------------------------------------

scala> (x : Int, y : Int) => x + y
res2: (Int, Int) => Int = <function2>

scala> val add = (x : Int, y : Int) => x + y
add: (Int, Int) => Int = <function2>

scala> add(1,2)
res3: Int = 3

2. named functions
---------------------------------------------------

def add(x : Int, y : Int) = x + y

add(1,2)

---------------------------------------------------

scala> def add(x : Int, y : Int) = x + y
add: (x: Int, y: Int)Int

scala> add(1,2)
res4: Int = 3

3. curried functions
---------------------------------------------------

def add(x : Int)(y : Int) = x + y

add(1)(2)

---------------------------------------------------

scala> def add(x : Int)(y : Int) = x + y
add: (x: Int)(y: Int)Int

scala> add(1)(2)
res5: Int = 3

---------------------------------------------------

`named functions` importance
---------------------------------------------------

def add(x : Int, y : Int) = x + y

def add(x : Int = 1, y : Int = 2) = x + y

---------------------------------------------------

scala> def add(x : Int = 1, y : Int = 2) = x + y
add: (x: Int, y: Int)Int

scala> add()
res6: Int = 3

scala> add(10)
res7: Int = 12

scala> add(10,20)
res8: Int = 30

scala> add(x = 10, y = 20)
res9: Int = 30

scala> add(y = 20, x = 10)
res10: Int = 30

---------------------------------------------------
factorial of n:
---------------------------------------------------

def factorial(n : Int) = {
if(n == 1) 1
else n * factorial(n-1)
}

scala> def factorial(n : Int) = {
| if(n == 1) 1
| else n * factorial(n-1)
| }
<console>:13: error: recursive method factorial needs result type
else n * factorial(n-1)
^

---------------------------------------------------

def factorial(n : Int) : Int = {
if(n == 1) 1
else n * factorial(n-1)
}

---------------------------------------------------
scala> def factorial(n : Int) : Int = {
| if(n == 1) 1
| else n * factorial(n-1)
| }
factorial: (n: Int)Int

scala> factorial(5)
res11: Int = 120

scala> factorial(4)
res12: Int = 24

---------------------------------------------------
Object Oriented Programming in Scala
---------------------------------------------------
trait
abstract class
class
object
case class
---------------------------------------------------

case class A (a :Int, b : String) {
override def toString() : String = s"a is $a, b is $b"
}
---------------------------------------------------
scala> case class A (a :Int, b : String) {
| override def toString() : String = s"a is $a, b is $b"
| }
defined class A

scala> A(1, "kalyan")
res13: A = a is 1, b is kalyan

scala> new A(1, "kalyan")
res14: A = a is 1, b is kalyan

scala> val a1 = A(1, "kalyan")
a1: A = a is 1, b is kalyan

scala> val a2 = new A(1, "kalyan")
a2: A = a is 1, b is kalyan

--------------------------------------------------
class B (a :Int, b : String) {
override def toString() : String = s"a is $a, b is $b"
}

---------------------------------------------------

scala> class B (a :Int, b : String) {
| override def toString() : String = s"a is $a, b is $b"
| }
defined class B

scala> B(1, "kalyan")
<console>:12: error: not found: value B
B(1, "kalyan")
^

scala> new B(1, "kalyan")
res16: B = a is 1, b is kalyan

scala> val b1 = new B(1, "kalyan")
b1: B = a is 1, b is kalyan

---------------------------------------------------

Subscribe to: Posts ( Atom )