Hadoop CheaSheet

2 minute read

Introduction

We have decided to aggregate in a single post the most important things to know about hadoop in a concise way. Let’s us know if you have any comments!

Hadoop

   
Sequence File Java writable binary format not language independant
Avro Language independent
   
##########
## HDFS ##
##########

NameNode # => Managing filesystem namespace, if you loose it you have no pointers to your data, you practially lost your data.

DataNode # => You know it holds data, installed on each worker.

Block # => Each file split to B1,B2,.. where each block size 128MB replication is on blocks.  Name node knows that File X is split to B1,B2 and where.

##########
## YARN ##
##########

ResourceManager # => Like `NameNode` for computing, tracks NodeManagers and how available they are for work.

NodeManager # => Like `Datanode` for computing, offer computational resources run applications tasks in containers.

ApplicationMaster # => Each application has `ApplicationMaster` process which negotiates resources with `ResourceManager` which delivers a `container` descriptor back to `ApplicationMaster` processa and asks `NodeManager` to launch the `container.`
 
 ################
 ## Map Reduce ##
 ################
 
 Map(k1, v1) --> list(k2, v2) # => extract something you care about from each record. # map takes keyvalue pair and produces zero or more intermediate keyvalue pairs # then outside of map shuffle  (distribute to reduce) and sort.
 
 shuffle # => combined everything by same keys 
 
 Recduce(k2, list(v2)) --> list(k3, v3) # => aggregate, summarize, filter, or transform.  # Reduce take a single key and list of values and produces zero or more keyvalue, usually aggregation.
 
 input --> map --> shuffle --> reduce --> output

Spark Zeppelin with EMR (Amazon)

git clone https://resources.oreilly.com/examples/0636920077367 sparkemr-example
cd sparkemr-example
mkdir -p src/main/scala
tomers-mbp:sparkemr-example tomer.bendavid$ cp Analyzing\ Big\ Data\ with\ Spark\ and\ Amazon\ EMR\ -\ Working\ Files/Chapter\ 3/ALSMovieRecs1M.scala src/main/scala/
tomers-mbp:sparkemr-example tomer.bendavid$ cp Analyzing\ Big\ Data\ with\ Spark\ and\ Amazon\ EMR\ -\ Working\ Files/Chapter\ 3/ALSMovieRecs1M.sbt .
sbt package

https://us-east-2.console.aws.amazon.com/console/home?region=us-east-2# # => we wnat to create s3 bucket
// search for s3
// create a bucket i named it "tomer-global-bucket"
// mkdir in ui ml-1m
aws s3 cp movies.csv s3://tomer-global-bucket/ml-1m/

Start hadoop spark hive locally

git clone https://github.com/big-data-europe/docker-hadoop-spark-workbench
cd docker-hadoop-spark-workbench/
docker-compose up

# Download hadoop binary
# extract it
update zeppelin-env.sh and set HADOOP_HOME and HADOOP_CONF_DIR

udpate your credentials

$ aws s3 cp target/scala-2.11/alsmovierecs1m_2.11-1.0.jar s3://tomer-global-bucket/
upload: target/scala-2.11/alsmovierecs1m_2.11-1.0.jar to s3://tomer-global-bucket/alsmovierecs1m_2.11-1.0.jar
$ aws s3 ls s3://tomer-global-bucket/
2017-12-17 17:52:10      19702 alsmovierecs1m_2.11-1.0.jar

Start zeppelin, from zepplin home, you can view the hdfs in zeppelin with:

%file

ls /

zeppelin collides with spark master on port 8080 we change it to 8180

vi conf/zeppelin-env.sh export ZEPPELIN_PORT=8180 # http://localhost:8180

in zeppelin ui search for interpreter spark and update

master from local[*] to spark://localhost:8080

https://hortonworks.com/tutorial/hands-on-tour-of-apache-spark-in-5-minutes/

Summary

We kept that post small so you could rest :), but in general we went through what NameNode, DataNode, Block, ResourceManager, NodeManager, ApplicationMaster, Task are in a very short and concise way, isn’t that just great :) If you liked it please hit the share button below and leave a comment for any comment! :)

notes:

//import org.apache.hadoop.fs.{FileSystem,Path}

//FileSystem.get( sc.hadoopConfiguration ).listStatus( new Path(“hdfs:///”)).foreach( x => println(x.getPath ))

// import sys.process._

// “wget http://archive.ics.uci.edu/ml/machine-learning-databases/00222/bank.zip” ! // “mkdir data” ! // “unzip bank.zip -d data” ! // “rm bank.zip” !

Leave a Comment