Spark 2.0

Difference between var and val in spark- Interview question

Var keyword is just similar to variable declaration in Java whereas Val is little different. Once a variable is declared using Val the reference cannot be changed to point to another reference. This functionality of Val keyword in Scala can be related to the functionality of java final keyword. Val refers to immutable declaration of …

Difference between var and val in spark- Interview question Read More »

Read CSV and JSON file format in spark 2.0

Read CSV with spark 2.0 STEP 1. Open the spark-shell and fire the following command. scala> spark.read.format(“csv”).option(“header”,”true”).load(“F:/Hadoop Youtube/customer.csv”) STEP 2. Display the result with show command scala> .show +—–+——+———–+——-+———-+—-+——+ |empno| ename|designation|manager| hire_date| sal|deptno| +—–+——+———–+——-+———-+—-+——+ | 7369| SMITH| CLERK| 7902|12/17/1980| 800| 20| | 7499| ALLEN| SALESMAN| 7698| 2/20/1981|1600| 30| | 7521| WARD| SALESMAN| 7698| 2/22/1981|1250| 30| …

Read CSV and JSON file format in spark 2.0 Read More »