Hadoop The Definitive Guide

Front 1

Distributed filesystems

Back 1

Filesystems that store files on multiple machines

Front 2

HDFS

Back 2

Hadoop's main distributed filesystem, designed for storing very large files, with streaming access data patterns, running on clusters of commodity hardware

Front 3

Areas where HDFS is not a good fit

Back 3

Low-latency data access, lots of small files, multiple writers & arbitrary file modifications

Front 4

HDFS default block size

Back 4

64 MB

Front 5

Unit of abstraction for HDFS

Back 5

Blocks, not the files themselves

Front 6

namenode (master node)

Back 6

Manages the filesystem namespace, maintains the filesystem tree and metadata for all files in the tree, knows which datanodes a block is stored on. This info is stored persistently on local disk in both the namespace image and edit log.

Front 7

datanode (worker/slave node)

Back 7

store and retrieve blocks of data when they are told by the namenode or client, report back to namenode periodically with the blocks they are storing

Front 8

Secondary namenode

Back 8

a namenode stored on a separate physical machine, which keeps a copy of the namespace image, and is used in event of first namenode failure

Front 9

HDFS Federation

Back 9

A multiple namenode system where each namenode manages: a namespace volume and a block pool

Front 10

namespace volume

Back 10

a subset of all files in a filesystem, they are independent of one another

Front 11

block pool

Back 11

contain all the blocks for the files in a namespace. Storage in a block pool is not partitioned, a file will register with all namenodes in the cluster and store on multiple block pools

Front 12

HDFS High Availability

Back 12

A pair of namenodes in standby mode, ready to act as the namenode in case of failure

Front 13

Failover controller

Back 13

manages the transition from an active namenode to a standby, typically uses Zookeeper

Front 14

Fencing

Back 14

the process of ensuring that a previously active namenode that was seen as failed, but had not failed, cannot do damage to the system...done by the High-Availability system

Front 15

Methods of fencing

Back 15

killing the namenode's process, revoking access, STONITH

Front 16

Hadoop fs

Back 16

the command line prefix to most Hadoop commands (stands for Hadoop filesystem)

Front 17

Path object

Back 17

represents a file in a Hadoop filesystem

Front 18

Configuration object

Back 18

stores the client or server's configuration, which is set using configuration files read from the classpath

Front 19

FileStatus class

Back 19

allows you to view metadata about files

Front 20

globbing

Back 20

using wildcards to match multiple files with one expression, often used with globStatus function

Front 21

Steps of a File Read

Back 21

1. Client opens file calling open() on the Filesystem object, an instance of DistributedFileSystem
2. DistributedFileSystem calls the namenode using RPC to get locations of the first few blocks in the file
3. DFSInputStream connects to the closest datanode for the first block in the file
4. Data is streamed from datanode back to client, which calls read() repeatedly on the stream
5. This continues to happen without interruption to client
6. When client has finished reading, calls close() on the FSDataInputStream

Front 22

Steps of a File Write

Back 22

1. Client calls create() on DistributedFileSystem
2. Namenode checks to make sure file doesn't already exists & client has permissions to create
3. If a pass, namenode makes record of new file
4. DistributedFileSystem returns an FSDataOutputStream for client to start writing data to
5. FSDataOutputStream wraps a DFSOutputStream, which handles communication with the datanodes and the namenode
6. Data queue is consumed by the Data Streamer, which asks the namenode to allocate new blocks by picking a list of suitable datanodes to store replicas
7. List of datanodes forms a pipeline, typically three in each pipeline (depends on replication factor
8. DataStreamer streams packets to first datanode in pipeline, which stores packet and forwards it to second datanode, and onward to each datanode in pipeline
9. Packet is only removed from this queue only when it is acknowledged by all datanodes in this pipeline
10. Namenode notices under-replication, and arranges for a further replica to be created on another node
11. Failed datanode is removed from pipeline and remainder of block's data is written to the 2 good datanodes in pipeline
12. Current block on good datanodes is given a new identity, which is communicated to the datanode so the partial block on failed datanode will be deleted if failed datanode later recovers
13. Pipeline is closed, any packets in ack queue are added to front of data queue, so that datanodes downstream from failed node will not miss any packets
14. When client finishes writing data, it calls close() on the stream
15. Namenode already knows which blocks the file is made of, so it only has to wait for blocks to be minimally replicated before returning successfully

Front 23

data queue

Back 23

An internal queue populated by client writing data, when the DFSOutputStream splits data into packets

Front 24

ack queue

Back 24

an internal queue of packets waiting to be acknowledged by datanodes by DFSOutputStream

Front 25

Hadoop's default replica strategy

Back 25

Three replicas: 1) same node as client, 2) a different rack chosen at random, 3) the same rack on a different node

Front 26

Coherency model for a filesystem

Back 26

describes the data visibility of reads and writes for a file

Front 27

Distcp

Back 27

Copies large amounts of data to and from Hadoop filesystems in parallel. Implemented as a MapReduce job

Front 28

Hadoop Archive (HAR) files

Back 28

pack files into HDFS blocks more efficiently, reducing namenode memory usage while still allowing access to files

Front 29

How does Hadoop detect errors in data read/write?

Back 29

Using a checksum, datanodes are responsible for verification. The last datanode in the pipeline will verify the checksum

Front 30

Types of compression compatible with Hadoop

Back 30

DEFLATE, gzip, bzip2, LZO, Snappy

Front 31

Hadoop's codec implementation

Back 31

the CompressionCodec interface

Front 32

Serialization

Back 32

the process of turning structured objects into a byte stream for transmission over a network or for writing to persistent storage

Front 33

Deserialization

Back 33

byte stream -> series of structured objects

Front 34

Remote Procedure Calls (RPC)

Back 34

* Handles interprocess communication between nodes in the system
* Uses serialization to render message into binary stream to be sent to remote node

Front 35

Hadoop's serialization format

Back 35

Writables, which have two methods: write, and readFields

Front 36

Hadoop equivalents to Long, Double, Int, String

Back 36

LongWritable, DoubleWritable, IntWritable, Text

Front 37

Apache Avro

Back 37

an IDL-based serialization framework, usually written in JSON

Front 38

Interface description language (IDL)

Back 38

a definition of types in a language-neutral, declarative fashion

Front 39

Sequence Files

Back 39

Provides a persistent data structure for binary key-value pairs. Temporary map outputs are stored as Sequence Files.

Front 40

Sync Points

Back 40

used with Sequence Files, a point in the stream that can be used to resynchronize with a record boundary if the reader is lost

Front 41

Format of a Sequence File

Back 41

1) First 3 bytes are the bytes SEQ, a magic #

2) 1 byte representing the version number

3) remainder of the header, which contains fields like key/value classes, compression details, sync marker

4) the records (which could be compressed)

Front 42

Map File

Back 42

a sorted Sequence File with an index to permit lookups by key

Front 43

the Configuration class

Back 43

a collection of configuration properties and their values, configurations read their properties from resources-XML files

Front 44

Example of a resources- XML file

Back 44

<?xml version="1.0"?>

color

yellow

Color

size-weight

${size},${weight}

Size and weight

Front 45

Unit Testing class

Back 45

Mockito's mock() method, passing the class of the type we want to mock

Front 46

Optimal # of reducers

Back 46

Approx. slightly less than the # of reduce slots in the cluster

Front 47

HPROF

Back 47

a profiling tool that comes with the JDK that gives information about a program's CPU and heap usage

Front 48

What are a few things to check when tuning a MapReduce job?

Back 48

# mappers, # reducers, combiners, intermediate compression, custom serialization, shuffle tweaks

Front 49

JobControl class

Back 49

represents a graph of MR jobs to be run, can add the job configurations and tell the instance of JobControl the dependencies between jobs

Front 50

Apache Oozie

Back 50

designed to manage the executions of thousands of dependent workflows, that could be made up of thousands of actions at the level of an independent MR job; composed of a workflow engine and coordinator engine

Front 51

workflow engine

Back 51

part of Oozie, stores and runs workflows composed of Hadoop jobs

Front 52

coordinator engine

Back 52

part of Oozie, runs workflow jobs based on predefined schedules and data availability

Front 53

workflow

Back 53

a group of action nodes and control-flow nodes

Front 54

action nodes

Back 54

performs a workflow task

Front 55

control-flow nodes

Back 55

governs the workflow execution between actions by allowing conditional logic, parallel execution, and other constructs

Front 56

jobtracker

Back 56

part of MapReduce 1, coordinates the job run, its main class is JobTracker

Front 57

tasktrackers

Back 57

part of MapReduce 1, run the tasks that the job has been split into, its main class is TaskTracker

Front 58

YARN

Back 58

Yet Another Resource Negotiator, aka MapReduce 2

Front 59

resource manager

Back 59

part of YARN, manages the use of resources across the cluster

Front 60

application master

Back 60

manages the lifecycle of applications running on the cluster

Front 61

node managers

Back 61

launch and monitor the computer containers on machines in the cluster

Front 62

3 failure modes to consider in Classic MapReduce

Back 62

failure of the running task, the tasktracker, and the jobtracker

Front 63

How does a tasktracker get blacklisted by a jobtracker? What happens?

Back 63

If 4 or more tasks from the same job fail, the jobtracker records this as a fault. IF the tasktracker goes over a certain # of faults, the tasktracker will not be assigned tasks, but will continue to communicate with the jobtracker...these faults expire over time, and a tasktracker can make it off the blacklist

Front 64

What can be done to mitigate a jobtracker failure?

Back 64

There is no mechanism for dealing with this type of failure, the job simply fails and must be restarted

Front 65

4 failure modes to consider in YARN

Back 65

failure of the task, the application master, the node manager, and the resource manager

Front 66

What occurs during an application master failure?

Back 66

The resource manager will detect failure and start a new instance of the master running in a new container...can recover the state of tasks being run

Front 67

What occurs during a node manager failure?

Back 67

it will be removed from the resource manager's pool of available nodes, can also be blacklisted

Front 68

What occurs during a resource manager failure?

Back 68

The most serious YARN failure, neither jobs nor task containers can be launched. The resource manager uses checkpoints to save its state. A new resource manager instance must be brought up, and can recover from the saved state

Front 69

Fair Scheduler

Back 69

a type of Hadoop scheduler, it aims to give every user a fair share of the cluster capacity over time

Front 70

Capacity Scheduler

Back 70

a type of Hadoop scheduler, splits the cluster into queues, which may be hierarchical, and each queue has an allocated capacity. Within each queue jobs are scheduled using FIFO

Front 71

Shuffle

Back 71

the process by which the system sorts map output keys and transfers the map outputs to reducers as inputs

Front 72

copy phase

Back 72

When the reduce task begins to copy map outputs as soon as the outputs are available. Doesn't wait until all map tasks are complete.

Front 73

sort phase

Back 73

part of the reduce task, map outputs are combined to feed the reduce tasks. done once all map outputs have been copied. Also known as the merge.

Front 74

speculative execution

Back 74

when Hadoop tries to detect when a task is running slower than expected, it will launch another equivalent task as a backup. Only launched if all tasks for a job have been launched

Front 75

Output committer

Back 75

a commit protocol used by Hadoop MapReduce to ensure jobs and tasks either succeed or fail cleanly, implemented by the Java class OutputCommitter

Front 76

Hadoop skipping mode

Back 76

Forces tasks to report the records being processed to the tasktracker, when the task fails the tasktracker will retry the task then skip the record(s) that cause failure

Front 77

General form of MapReduce

Back 77

Map: (K1, V1) -> (K2, V2)

(if exists) Combiner: (K2, list(V2)) -> list(K2, V2)

Reduce: (K2, list(V2)) -> list(K3, V3)

**Note: Often, K2=K3 and V2=V3, but not always

Front 78

What is the default input format for MapReduce?

Back 78

TextInputFormat, which produces keys of type LongWritable and values of type Text

Front 79

What is the default output format for MapReduce?

Back 79

TextOutputFormat, which writes out records one per line, by converting keys/values to strings and tab-separating them

Front 80

input split

Back 80

chunk of input that is processed by a single map, divided into records, which are key-value pairs

Front 81

which Java class is useful for processing small files?

Back 81

CombineFileInputFormat, which packs many files into each split, so that each mapper has more to process; and takes node/rack locality into consideration when packing together

Front 82

Which method can be overridden to prevent file splitting?

Back 82

isSplitable() on FileInputFormat

Front 83

NLineInputFormat

Back 83

the Java class that allows for a fixed number of lines, where the key/value is still the same as TextInputFormat (offset, lines). Controls the # of lines of input that each mapper receives

Front 84

What are the types of built-in counters in Hadoop?

Back 84

Task counters, Job counters, User-defined Java counters, Dynamic counters

Front 85

how can a counter be defined in Java?

Back 85

using the Java enum

Front 86

Total Sort

Back 86

sorting each file so that they can be concatenated to form a globally sorted file

Front 87

secondary sort

Back 87

Structuring key-value pairs to allow for sorting by values as well as the keys (keys only are the default)

Front 88

map-side join

Back 88

a join performed by the mapper. The data is joined before data reaches the map function, and requires the inputs to each map to be sorted by the join key and partitioned by that key as well, can be run through the CompositeInputFormat class

Front 89

reduce-side join

Back 89

a join performed by the reducer. More general than map-side joins, don't require the input data set to be structured a certain way. Less efficient since both datasets must go through the Shuffle. It DOES require that the data is sorted by source, one source before the other

Front 90

side data

Back 90

extra read-only data needed by a job to process the main dataset

Front 91

Pig

Back 91

a client-side application comprised of the language (Pig Latin) and the execution environment to run Pig Latin programs. It raises the level of abstraction for processing large datasets

Front 92

Grunt

Back 92

Pig's interactive shell, started when no file is specified on the command line for Pig to run, and the -e option is not used

Front 93

Three ways to run Pig programs

Back 93

Scripts (pig script.pig), Grunt, and Embedded in Java using the PigServer class

Front 94

bag

Back 94

in Pig, an unordered collection of tuples, represented in Pig Latin by curly braces

Front 95

Differences between SQL and Pig

Back 95

1) Pig Latin is a data flow programming language, while SQL is a declarative programming language

2) Pig Latin is a step-by-step set of single transformation operations on an input relation, SQL is a set of constraints to find a result set

3) RDBMS- set of predefined schemas, Pig is more relaxed about the data it processes

4) Pig supports complex, nested data structures, SQL is more flat

5) Pig doesn't support random reads/writes

Front 96

Pig Latin statement

Back 96

an operation or command in Pig Latin, which must end with a semicolon (ex: GROUP)

Front 97

Pig Latin commands

Back 97

not added to the Pig logical plan, they are executed immediately. Useful for moving data around before/after Pig processing

Front 98

Pig Latin expressions

Back 98

Something evaluated to yield a value, can be used as part of a statement containing a relational operator (ex: $n- field in position n)

Front 99

What are the four numeric types in Pig?

Back 99

int, long, float, double

Front 100

Which Pig types are analogous to Java string?

Back 100

bytearray, chararray

Front 101

What are the three complex Pig types?

Back 101

tuple, bag, map

Front 102

Read the schema from the following relation:

{(year: chararray), (amount: int), (name)}

Back 102

Year variable has a type of chararray, the amount variable is an int, and the name variable is a bytearray (the default, if no type is specified)

Front 103

When a value being inserted cannot be cast to the type declared in a schema, what occurs in Pig?

Back 103

Pig inserts null in its place, instead of erroring...because of the volume of data usually being processed, it is impractical to expect to be able to correct every incorrect record

Front 104

Macro in Pig

Back 104

Similar to a Java method, allows you to package reusable pieces of Pig Latin, done using the DEFINE keyword

Front 105

User Defined Function (UDF) in Pig

Back 105

a function written in Java (or Javascript, Python, other scripting languages) that house logic used in Pig scripts

Front 106

dynamic invoker

Back 106

allows you to use a function that is provided by a Java library without using a UDF. Is less efficient, since it uses reflection

Front 107

Pig's STREAM operator

Back 107

allows you to transform data in a relation using an external program or script

Front 108

Pig's COGROUP operator

Back 108

Similar to a JOIN, but it creates a nested set of output tuples, nice for exploiting the structure in subsequent statements. JOIN creates a flat structure, just a set of tuples

Front 109

Hive

Back 109

Uses the HiveQL language, similar to SQL, to convert queries into MapReduce jobs for execution on a Hadoop cluster

Front 110

the Hive metastore

Back 110

a database created by Hive to store metadata; it is normally stored in a central location and is shared across the cluster

Front 111

schema on write

Back 111

used in traditional RDBMS, schema is enforced at data load time; if data doesn't conform, it is rejected. This makes query performance much faster, but takes longer to load the data

Front 112

schema on read

Back 112

used in Hive, data isn't verified when it is loaded, rather when a query is issued. Makes for a very fast initial load, allows for multiple schemas in the same database, but query performance is slower

Front 113

How does Hive manage tables and the data in them?

Back 113

When a table is created using CREATE TABLE, and data is loaded using LOAD, the data is physically moved from its location into Hive, so when the table is dropped using DROP TABLE, the data is deleted and no longer exists (unless it was previously backed up)

Front 114

Hive external table

Back 114

specified using the EXTERNAL keyword, (e.g. CREATE EXTERNAL TABLE), tells Hive it is not managing the data, which avoids the data deletion situation

Front 115

HBase

Back 115

a distributed, column-oriented database built on top of HDFS. Used for real-time read/write random-access to very large datasets

Front 116

Zookeeper

Back 116

Hadoop's distributed coordination service that builds general distributed applications

Front 117

Sqoop

Back 117

an open-source tool that allows users to extract data from a relational database into Hadoop for further processing

Front 118

Partitions

Back 118

In Hive, specified at table creation and subsequently at data load, allows you to store the data physically in different places based on a partitioning column, makes for more efficient queries (similar to SQL partitions)

Front 119

Buckets

Back 119

In Hive, imposes extra structure to a table, specified on table creation. Useful for sampling, which can be done with the TABLESAMPLE clause. Also stored physically within a table/partition directory.

Front 120

Two formats that govern storage in Hive

Back 120

Row format (dictates how rows and their fields are stored, defined by a SerDe, a Serializer-Deserializer) and file format (dictates the container format for fields in a row, such as text file, row-oriented binary, etc.)

Front 121

What is the default storage format in Hive?

Back 121

CTRL-A delimited text, with a row per line

Front 122

RCFiles

Back 122

Record Columnar File, similar to SequenceFiles except they store all values in a column as a record, can be specified in a CREATE TABLE clause by using STORED AS RCFILE. These work well when you only need access to a few columns in a table

Front 123

What is the difference between ORDER BY and SORT BY in Hive?

Back 123

ORDER BY produces a globally sorted file, but only uses one reducer, which is highly inefficient. SORT BY produces a sorted file per reducer, but is not globally sorted

Front 124

Types of UDFs in Hive

Back 124

Regular UDFs- operate on a single row, produce a single row

UDAFs- operate on multiple rows, and produce a single row

UDTFs- operate on a single row, produce multiple rows (a table) as output

Front 125

What 2 properties must a regular UDF satisfy?

Back 125

Must be a subclass of org.apache.hadoop.hive.ql.exec.UDF, and must implement at least one evaluate() method

Front 126

splitting column

Back 126

in Sqoop, the column that is identified as having a close-to uniform distribution, and is used to divide data coming from a relational db to a MapReduce job for import into Hadoop

Front 127

LobFile

Back 127

Sqoop's way of storing imported large objects

Front 128

the DBWritable interface

Back 128

used in Sqoop, a Java class that provides read & write method for serialization, which allows the class to interact with JDBC

Front 129

Sqoop's direct mode

Back 129

the quickest, most efficient way to import data using Sqoop. Even when it is used, the metadata is queried through JDBC

Front 130

--hive-import

Back 130

When specified in Sqoop, this option tells Sqoop to create a Hive table with the data being moved. If the table already exists, can use --hive-overwrite

Front 131

--hive-table

Back 131

When specified in Sqoop, names the output Hive table

Front 132

mysqldump

Back 132

a Sqoop tool that allows for direct access to MySQL databases, faster than the default JDBC connection. This is enabled by using the --direct option

Front 133

PigStorage

Back 133

a Java class that constructs a Pig loader to load delimited text files

Front 134

Three types of SequenceFiles

Back 134

1. Uncompressed key/value records

2. Record compressed key/value records- only values compressed

3. Block compressed- both keys/values are collected in blocks separately and compressed