Hadoop Sqoop Interview Questions

hadoop jira, hadoop command, hadoop icon, hadoop with python, ubuntu install hadoop, hadoop in the cloud, hadoop fs, hadoop hdfs commands, hadoop data, hadoop capacity scheduler

1. Where & why is Sqoop is used? Explain with syntax
Ans: Sqoop is used for importing data from any rdbms using jdbc connector to hdfs or export an HDFS directory to a database table
Can I directly import data to Hive tables using Sqoop?
Ans: Yes

2. How do you perform a incremental data load?
Ans: By importing data using sqoop command or loading data to hive without overwrite.

--check-column (col) Specifies the column to be examined when determining which rows to import.
--incremental
(mode) Specifies how Sqoop determines which rows are new. Legal values for mode include append and lastmodified.
--last-value
(value) Specifies the maximum value of the check column from the previous import.

Sqoop supports two types of incremental imports: append and lastmodified. You can use the --incremental argument to specify the type of incremental import to perform.
You should specify append mode when importing a table where new rows are continually being added with increasing row id values. You specify the column containing the row’s id with --check-column. Sqoop imports rows where the check column has a value greater than the one specified with --last-value.


3. Can you do incremental import using sqoop? – didn’t answer the exact syntax but, told the underlying concept . answered above

4. You have a data in HDFS sytem, if you want to put some more data to into the same table, will it append the data or overwrite?

Ans:No it can’t overwrite, one way to do is copy the new file in HDFS.
Then concatenate files to a new file as shown below.

bin/hadoop fs -cat /user/username/folder/csv1.csv /user/username/folder/csv2.csv | hadoop fs -put - /user/username/folder/output.csv

hadoop fs -cat myfolder/* | hadoop fs -put - myfolder/output.txt

for multiple folders use:
for i in `hadoop fs -ls my-job-folder | cut -d' ' -f19` ;do `hadoop fs -cat $i/* |hadoop fs -put - $i/output.csv`; done


A table has 1000 records, but only 100 records are needed. How to achieve that?
Ans: Assign sequence number to each row and select sequence id < 101.
Else for random 100 records select top function or sample 100

Read more

Hadoop SQL Interview Questions

1. There are some duplicate records in a table. How do you delete those duplicate rows(without Java)
Ans: using rowid by selecting min or max row id of duplicate rows.
Delete from table1 where (col1,rowid) in (sel col1,min(rowid),count(rowid) from table1 group by col1 having count(rowid) > 1)

Or
Select distinct rows from the table and put it to a temporary table.
Drop the original table and rename the temporary table to original table.


2. How to fetch the top 5th row from a table containing 100 records?
Ans: Using top function or using qualify row_number() over (partition by order by) <=6,
or using rank() function.

3. A table contains various departments, employees & their salaries. How do you fetch the maximum salary of each department and the salary must be greater than 5000
Ans: select dept,max(sal) from department group by dept having max(sal) > 5000.

hadoop jira, hadoop command, hadoop icon, hadoop with python, ubuntu install hadoop, hadoop in the cloud, hadoop fs, hadoop hdfs commands, hadoop data, hadoop capacity scheduler

Read more