Hadoop

Difference between var and val in spark- Interview question

Var keyword is just similar to variable declaration in Java whereas Val is little different. Once a variable is declared using Val the reference cannot be changed to point to another reference. This functionality of Val keyword in Scala can be related to the functionality of java final keyword. Val refers to immutable declaration of …

Difference between var and val in spark- Interview question Read More »

Read CSV and JSON file format in spark 2.0

Read CSV with spark 2.0 STEP 1. Open the spark-shell and fire the following command. scala> spark.read.format(“csv”).option(“header”,”true”).load(“F:/Hadoop Youtube/customer.csv”) STEP 2. Display the result with show command scala> .show +—–+——+———–+——-+———-+—-+——+ |empno| ename|designation|manager| hire_date| sal|deptno| +—–+——+———–+——-+———-+—-+——+ | 7369| SMITH| CLERK| 7902|12/17/1980| 800| 20| | 7499| ALLEN| SALESMAN| 7698| 2/20/1981|1600| 30| | 7521| WARD| SALESMAN| 7698| 2/22/1981|1250| 30| …

Read CSV and JSON file format in spark 2.0 Read More »

Default Number of mapper and reducer in SQOOP job

Updated: Dec 12, 2018 #hadoop #sqoop #defaultmapper #defaultreducer #hadoopinterviewquestion In this post we are going to focus the default number of mappers and reducers in the sqoop. scope is the part of Hadoop ecosystem which is mainly useful to move the data from the RDBMS database to hdfs file system or to directly hive tables and …

Default Number of mapper and reducer in SQOOP job Read More »