Enable PRE- and POST-execution processing feature of Spectrum LSF in large scale Cluster

An HPC cluster consists of hundreds or thousands of compute servers that are networked together. InfiniBand is pervasively used in high-performance computing (HPC) to remove data exchange bottlenecks, delivering very high throughput and very low latency. As HPC becomes more mainstream and embraced by enterprise users, there is a need for assurances that performance is optimized

Users schedule their jobs to run on HPC cluster by submitting them through Spectrum LSF.  IBM® Spectrum LSF (formerly IBM® Platform™ LSF®) is a complete workload management solution for demanding HPC environments.

One of the important task before  execution of application is preparing jobs inorder to get best performance on the given HPC cluster. Select the most appropriate queue for each job and provide accurate wall-clock times in your job script. This will help us fit your job into the earliest possible run opportunity. Note the system's usable memory and configure your job script to maximize performance. Next, prepare/tune  the nodes (or servers)  to desired values. This can be done by pre-execution processing. For example , Set the CPU statically to highest frequency ( or any requirement for that matter). After , execution of LSF job , you could set it back to previously set values by post-execution processing.

source

Configuration to enable pre- and post-execution processing :

The pre- and post-execution processing feature is enabled by defining at least one of the parameters in the list below at the application or queue level, or by using the -E option of the bsub command to specify a pre-execution command. In some situations, specifying a queue-level or application-level pre-execution command can have advantages over requiring users to use bsub -E. For example, license checking can be set up at the queue or application level so that users do not have to enter a pre-execution command every time they submit a job.

The following example illustrates how job-based pre- and post-execution processing works at the queue or application level for setting the environment prior to job execution and for transferring resulting files after the job runs.

source

Host-based pre- and post-execution processing is different from job-based pre- and post-execution processing in that it is intended for parallel jobs (you can also use this feature for sequential jobs) and is executed on all execution hosts, as opposed to only the first execution host. The purpose of this is to set up the execution hosts(or servers) before all job-based pre-execution and other pre-processing which depend on host-based preparation, and clean up execution hosts after job-based post execution and other post-processing.

There are two ways to enable host-based pre- and post-execution processing for a job:

• Configure HOST_PRE_EXEC and HOST_POST_EXEC in lsb.queues.
  
• Configure HOST_PRE_EXEC and HOST_POST_EXEC in lsb.applications.
   
  

Lets take the example of queue level configurations with HOST_PRE_EXEC/HOST_POST_EXEC :

LSF queue (QUEUE_NAM=Queue_pre_post) set with HOST_PRE_EXEC and HOST_POST_EXEC .

where HOST_PRE_EXEC points to the " pre_setup_perf.sh" .Similarly, HOST_POST_EXEC points to "post_setup_perf.sh" and set previous values back on the server after Performance testing.

Modify the configuration file "lsb.queues" . For example
------------------------------------------------------------------------------

Begin Queue
QUEUE_NAME   = Queue_pre_post
PRIORITY     = 40
INTERACTIVE  = NO
FAIRSHARE    = USER_SHARES[[default,1]]
HOSTS        = server1 server2          # hosts on which jobs in this queue can run
EXCLUSIVE    = Y
HOST_PRE_EXEC     = /home/sachinpb/pre_setup_perf.sh >> /tmp/pre.out
HOST_POST_EXEC    = /home/sachinpb/post_setup_perf.sh >> /tmp/post.out
DESCRIPTION  = For P8 performance jobs, running only if hosts are lightly loaded.
End Queue

----------------------------------------------------------------------------------

After modification, please run the command.


badmin reconfigure
-------------------------------------------------------------------------------


Check the queue status:


[sachinpb@server1 ~]$ bqueues -l Queue_pre_post
QUEUE: Queue_pre_post
  -- For P8 performance jobs, running only if hosts are lightly loaded.
PARAMETERS/STATISTICS
PRIO NICE STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN SSUSP USUSP  RSV PJOBS
 40    0  Open:Active       -    -    -    -     0     0     0     0     0    0     0
Interval for a host to accept two jobs is 0 seconds
SCHEDULING PARAMETERS
           r15s   r1m  r15m   ut      pg    io   ls    it    tmp    swp    mem
 loadSched   -     -     -     -       -     -    -     -     -      -      -
 loadStop    -     -     -     -       -     -    -     -     -      -      -
SCHEDULING POLICIES:  FAIRSHARE  EXCLUSIVE  NO_INTERACTIVE
USER_SHARES:  [default, 1]
SHARE_INFO_FOR: Queue_pre_post/

USER/GROUP   SHARES  PRIORITY  STARTED  RESERVED  CPU_TIME  RUN_TIME   ADJUST
sachinpb          1       0.326      0        0       350.7        0       0.000
USERS: all
HOSTS:  server1 server2
HOST_PRE_EXEC:  /home/sachinpb/sachin/pre_setup_perf.sh >> /tmp/pre.out
HOST_POST_EXEC:  /home/sachinpb/sachin/post_setup_perf.sh >> /tmp/post.out
[sachinpb@server1 ~]$
---------------------------------------------------------------------------------

HOST_PRE_EXEC=command (in lsb.queues):

•     Enables host-based pre-execution processing at the queue level.
•     The pre-execution command runs on all execution hosts before the job starts.
•     If the HOST_PRE_EXEC command exits with a non-zero exit code, LSF requeues the job to the front of the queue.
•     The HOST_PRE_EXEC command uses the same environment variable values as the job.
•     The HOST_PRE_EXEC command can only be used for host-based pre- and post-execution processing.

HOST_POST_EXEC=command (in lsb.queues):

•     Enables host-based post-execution processing at the queue level.
•     The HOST_POST_EXEC command uses the same environment variable values as the job.
•     The post-execution command for the queue remains associated with the job. The original post-execution command runs even if the job is requeued or if the post-execution command for the queue is changed after job submission.
•     Before the post-execution command runs, LSB_JOBEXIT_STAT is set to the exit status of the job. The success or failure of the post-execution command has no effect on LSB_JOBEXIT_STAT.
•     The post-execution command runs after the job finishes, even if the job fails.
•     Specify the environment variable $USER_POSTEXEC to allow UNIX users to define their own post-execution commands.
•     The HOST_POST_EXEC command can only be used for host-based pre- and post-execution processing.

-------------------------------------------------------
Now submit the LSF job  as shown :

[sachinpb@server1 sachin]$ bsub  -q Queue_pre_post -n 8 -R "span[ptile=4]" < myjob.script
bsub> Job <19940> is submitted to queue <Queue_pre_post>.
[sachinpb@server1 ]$ bjobs
JOBID   USER    STAT  QUEUE          FROM_HOST   EXEC_HOST    JOB_NAME     SUBMIT_TIME
19940 sachinpb  RUN   Queue_pre_post  server1   server2     myjob.script  Jun 28 05:16
                                                                                  server2
                                                                                  server2
                                                                                  server2
                                                                                  server1
                                                                                  server1
                                                                                  server1
                                                                                  server1
[sachinpb@server1 ]$
---------------------------------------------------------------------------

Logs from each server available at " $SACHIN_HOME/logs" for both pre- and post-execute processing as shown below. Check for log files after completion of LSF jobID 19940:

There should be 4  log files -:

• There are 2 pre-check-logs  executed on server1 and server2
• Similarly, there are  2 post-check-logs  executed on server1 and server2

Example:

[sachinpb@server1 logs]$ ls -alsrt
 8 -rw-rw-r-- 1  sachinpb sachinpb  4657 Jun 28 05:16 preScript_28-Jun-05_16_server2.out
 8 -rw-r--r-- 1    sachinpb sachinpb  4657 Jun 28 05:16 preScript_28-Jun-05_16_server1.out
 8 -rw-r--r-- 1   sachinpb sachinpb  4653 Jun 28 05:17 postScript_28-Jun-05_17_server1.out
 8 -rw-rw-r-- 1  sachinpb sachinpb  4653 Jun 28 05:17 postScript_28-Jun-05_17_server2.out
[sachinpb@server1 logs]$


---------------------------------------------------------------------------

pre-exec script:

[sachinpb@server1]$ cat pre_setup_perf.sh
#!/bin/bash
echo "Start Pre-execution script on $(hostname)"
HOST=`hostname`
DATE=$(date +%d-%b-%H_%M)
sudo /home/sachinpb/sachin/tune_this_server.sh pre_check | tee $SACHIN_HOME/logs/preScript_${DATE}_${HOST}.out

echo "End of Pre-execution script on $(hostname)"
[sachinpb@server1 ]$

----------------------------------------------------------------------------

post-exec script:

[sachinpb@server1 ]$ cat post_setup_perf.sh
#!/bin/bash
echo "Start Post-execution script on $(hostname)"
HOST=`hostname`
DATE=$(date +%d-%b-%H_%M)
sudo /home/sachinpb/sachin/tune_this_server.sh post_check | tee /$SACHIN_HOME/logs/postScript_${DATE}_${HOST}.out
echo "End of Post-execution script on $(hostname)"
[sachinpb@server1 ]$

-----------------------------------------------------------------------------

NOTE: Similarly,  you could do this in PRE_EXEC/POST_EXEC=command (in lsb.applications, lsb.queues) & HOST_PRE_EXEC/HOST-POST-EXEC=command (in lsb.applications) as per the application requirements.

I hope this blog helped in understanding how to configure pre- and post processing feature of Spectrum LSF. 

Hewlett Packard Enterprise to Acquire Supercomputer Pioneer Cray

Supercomputers have long been a mainstay of military and intelligence agencies, used for chores ranging from cracking codes to designing nuclear weapons. They have many civilian uses as well, like predicting weather, creating new drugs and simulating the effect of crashes on auto designs.Computing firms are racing to reach exascale performance, or a quintillion operations per second.

Hewlett Packard Enterprise will pay about $1.3 billion to acquire Cray, which has designed some of the most powerful computer systems in use. It will also enable HPE to start selling supercomputer components to corporate clients and others. HPE will integrate Cray’s supercomputer technology into its product portfolio and build an HPC-as-a-service and AI cloud services on HPE GreenLake. It’s synergistic. Because of this acquisition, HPE has access to one of the only differentiated stacks in the market and can now compete more aggressively in those five vertical markets. The combination of Cray and HPE creates an industry leader in the fast-growing High-Performance Computing (HPC) and AI markets and creates a number of opportunities that neither company would likely be able to capture on their own

Interestingly, Cray is actually the second supercomputer manufacturer picked up by HPE over its lifetime; the company also picked up the remaining assets of Silicon Graphics back in 2016. On November 1, 2016, Hewlett Packard Enterprise completed its acquisition of SGI (formerly Silicon Graphics ) for $275 million.

HPE is already the HPC server market leader, but by adding Cray into the fold, HPE strengthens its position against its two chief rivals, Dell EMC and IBM. Currently, IBM is the maker of the world’s two fastest supercomputers, Summit and Sierra. In 2018, HPE ranked first in HPC server sales with 34.8% market share, followed by Dell EMC with 20.8% and IBM with 7.1% of the market. All other HPC players, including Lenovo, Atos, Inspur, and Sugon, have single-digit market share, including Cray with 2.3%.

Cray, based in Seattle, traces its lineage to a company founded in 1972 in Minnesota by the computer designer Seymour Cray (“The father of supercomputing,”). That company was bought in 1996 by Silicon Graphics; it was sold in 2000 to Tera Computer, which adopted the Cray name.  Cray was impressively successful for a small company, but it’s more difficult as a standalone company against the bigger companies. Cray has morphed into an integrator and scale-out specialist, combining processors from the likes of Intel, AMD, and NVIDIA into supercomputers, and applying their own software, I/O, and interconnect technologies. The company is currently working with AMD on a US government-backed project to build the world's first exascale supercomputer at Oak Ridge National Laboratory. The system will be capable of a quintillion calculations per second.

Frontier Supercomputer at ORNL , partnership with AMD

Cray is currently contracted to build two of the world’s fastest supercomputers for two US Department of Energy Labs: the Oak Ridge National Laboratory and the Argonne National Laboratory. Both systems, one called Frontier being built in partnership with AMD and one called Aurora with Intel, are promised to bring so-called “exascale” performance, with raw performance power of the excess of 1.5 exfaflops, or a quintillion calculations per second. Both systems will support the converged use of analytics, AI, and HPC at extreme scale, using Cray’s new Shasta system architecture, software stack and Slingshot interconnect.

Aurora Supercomputer at ANL , partnership with with Intel

What worries some officials in the United States is the rapid rise of suppliers based in China. One of them, Lenovo, which bought former IBM hardware operations, led the rankings with 140 supercomputers installed. Two others, Inspur and Sugon, were second with 84 and third with 54, respectively. With the acquisition, HPE will get Cray’s current generation XC and CS supercomputers and the next-generation Shasta supercomputer that features a new high-speed interconnect, code-named Slingshot. Cray’s products also include high-performance storage, a full HPC system software stack, and data analytics and AI solutions that are fully integrated.

source

Most HPC hardware vendors today sell commoditized cluster stacks comprising of Infiniband or Ethernet-based networking. But IBM and HPE are the only HPC vendors that can differentiate themselves: IBM with its Power9 processors, and now HPE with Cray’s Slingshot and ClusterStor HPC storage hardware. Cray gives HPE a unique, technological edge,” he said. “Slingshot provides HPE with an extremely high-bandwidth network for HPC and AI workloads. It enables new network topologies that are required for extreme-scale HPC and AI. Supercomputers of this scale can be massively beneficial to data-intensive industries like astronomy, climate science, medicine, neuroscience, and physics. Increasingly, these systems can be used in artificial intelligence research, which, in turn, can help accelerate many other areas of scientific inquiry. That said, a supercomputer like Aurora or Frontier tends only to be built and financed by the government for, at least initially, military applications.

Thankfully, over time, these upcoming exascale supercomputers will likely be freed from the military apparatus and put to work divining new insights from data. According to HPE, the acquisition of Cray is primarily to help it gain an edge in AI research and the hardware required to train ever-larger neural nets. The company was also trying to make further inroads into the enterprise market, and HPE has experience and enterprise customers at a time when many businesses are recognizing the need to invest in HPC for AI, data analytics, and other business operations, such as marketing and ERP. The requirements in the commercial markets are pushing up into the HPC space. Cray was trying to get into the commercial market too, but they didn’t have the experience to do that, and now they are part of a company that has a big position in the commercial markets. For a lot of enterprises this is a new type of compute, but they have to start embracing it because workloads increasingly require MPC (Massively Parallel Computing). You can’t compete anymore without doing AI, big data analytics, and increasingly, simulation and modeling. They don’t necessarily want to purchase the hardware and put all that together. It’s complex. So if you can get it with a cloud-like consumption model as HPE GreenLake offers, then you turn it into an OpEx investment.

Broadly speaking, major acquisitions and mergers in the supercomputing space are rare events. Due to their ever-increasing price tag, only a small number of world-class supercomputers are sold each year. And due to these prices the buyers are often governments, which inevitably gives supercomputer construction a nationalistic element to it. None the less, because costs are increasing – Frontier is the US’s most expensive system yet at over $500M for the system alone. HPE’s plans include using Cray’s technologies to improve HPE GreenLake, the company’s HPC-as-a-Service offering

Apache Hadoop 3.x installation on multinode cluster RHEL7 (ppc64le)

Hadoop is an open-source Apache project that allows creation of parallel processing applications on large data sets, distributed across networked nodes. It’s composed of the Hadoop Distributed File System (HDFS™) that handles scalability and redundancy of data across nodes, and Hadoop YARN: a framework for job scheduling that executes data processing tasks on all nodes.

Hadoop allows for the storage and processing of large data sets across clusters of computers. Hadoop was developed by the Apache Software Foundation and has become a popular tool for big data processing and analysis.

Some of the key use cases of Hadoop are:

Data storage: Hadoop Distributed File System (HDFS) is a highly scalable and fault-tolerant distributed file system that is used to store large amounts of data across multiple nodes in a cluster. Hadoop is often used to store and manage large amounts of unstructured and semi-structured data, such as log files, sensor data, and social media data.

Batch processing: Hadoop provides a powerful framework for batch processing of large data sets. This is typically done using the MapReduce programming model, which allows for the parallel processing of data across multiple nodes in a cluster. Hadoop is often used for tasks such as data cleansing, data transformation, and data aggregation.

Data analysis: Hadoop is often used for data analysis and machine learning tasks. Hadoop provides a number of tools for processing and analyzing large data sets, including Apache Pig and Apache Hive. These tools allow users to query and analyze large data sets using SQL-like commands.

Real-time processing: Hadoop can also be used for real-time processing of data, using tools such as Apache Spark and Apache Flink. These tools allow for the processing of data streams in real-time, enabling applications such as fraud detection, real-time recommendations, and IoT data processing.

Apache Hadoop 3.x Benefits

•     Support multiple standby NameNodes.
•     Supports multiple NameNodes for multiple namespaces.
•     Storage overhead reduced from 200% to 50%.
•     Support GPUs.
•     Intra-node disk balancing.
•     Support for Opportunistic Containers and Distributed Scheduling.
•     Support for Microsoft Azure Data Lake and Aliyun Object Storage System file-system

Architecture of Hadoop Cluster: 

Apache hadoop has 2 core components .

1) HDFS - its for storage

2) YARN - its for computation 

You could see the HDFS and YARN architecture as shown below:

HDFS ARCHITECTURE

 

YARN ARCHITECTURE

Before configuring the master and worker nodes, it’s good to understand the different components of a Hadoop cluster. A master node keeps knowledge about the distributed file system, like the inode table on an ext3 filesystem, and schedules resources allocation. node-master will handle this role in this guide, and host two daemons:

•    The NameNode: manages the distributed file system and knows where stored data blocks inside the cluster are.

•    The ResourceManager: manages the YARN jobs and takes care of scheduling and executing processes on worker nodes.

Worker nodes store the actual data and provide processing power to run the jobs and will host two daemons:

•    The DataNode manages the actual data physically stored on the node;

•    The NodeManager manages execution of tasks on the node.

Prerequisites for Implementing Hadoop

•     Operating system –  RHEL 7.6
•     Hadoop – You require Hadoop 3.x package
•     Passwordless SSH connections between nodes inthe cluster
•     Firewall settings on machines in the cluster
•     Machine details :

    Master node  : hadoopNode1 (Power8 server with K80 GPUs running RHEL7)
    Worker nodes: hadoopNode1 &  hadoopNode2 (Power8 server with K80 GPUs running RHEL7)

-------------------------------------  ------------------------------------------------------------------

Hadoop Installation Steps  :

Step 1: Download the Java 8 Package and Save this file in your home directory.

Java is the primary requirement for running Hadoop on any system.  They have compiled all the Hadoop jar files using Java 8 run time version. The user now has to install Java 8 to use Hadoop 3.0. And user having JDK 7 has to upgrade it to JDK 8.

If   your machine is IBM power architecture (ppc64le) , then you need to get IBM java package from this link copied below:

Download Link : https://developer.ibm.com/javasdk/downloads/sdk8/

Step 2: Extract the Java TarFile.

Step 3: Download the Hadoop 3.x Package.

Download stable version of Hadoop
wget http://apache.spinellicreations.com/hadoop/common/hadoop-3.2.0/hadoop-3.2.0.tar.gz

Step 4: Extract the Hadoop tar File.
Extract the files @ /home/users/sachinpb/sachinPB/

 tar xzvf hadoop-3.2.0.tar.gz

At the high level "/home/users/sachinpb/sachinPB/hadoop-3.2.0, you will see the following directories:

├── bin
│   ├── container-executor
│   ├── hadoop
│   ├── hadoop.cmd
│   ├── hdfs
│   ├── hdfs.cmd
│   ├── mapred
│   ├── mapred.cmd
│   ├── oom-listener
│   ├── test-container-executor
│   ├── yarn
│   └── yarn.cmd
├── etc
│   └── hadoop
│       ├── core-site.xml
│       ├── hadoop-env.sh
│       ├── hdfs-site.xml
│       ├── log4j.properties
│       ├── mapred-site.xml
│       ├── workers
│       ├── yarn-env.sh
│       └── yarn-site.xml
├── include
├── lib
│   └── native
│       ├── examples
│       ├── libhadoop.a
│       ├── libhadooppipes.a
│       ├── libhadoop.so -> libhadoop.so.1.0.0
│       ├── libhadoop.so.1.0.0
│       ├── libhadooputils.a
│       ├── libnativetask.a
│       ├── libnativetask.so -> libnativetask.so.1.0.0
│       └── libnativetask.so.1.0.0
├── logs
│ 
├── sbin
│   ├── hadoop-daemon.sh
│   ├── httpfs.sh
│   ├── mr-jobhistory-daemon.sh
│   ├── refresh-namenodes.sh
│   ├── start-all.sh
│   ├── start-balancer.sh
│   ├── start-dfs.sh
│   ├── start-secure-dns.sh
│   ├── start-yarn.sh
│   ├── stop-all.cmd
│   ├── stop-all.sh
│   ├── stop-balancer.sh
│   ├── stop-dfs.sh
│   ├── stop-secure-dns.sh
│   ├── stop-yarn.sh
│   ├── workers.sh
│   ├── yarn-daemon.sh
│ 
└── share
    ├── doc
    │   └── hadoop
    └── hadoop
        ├── client
        ├── common
        ├── hdfs
        ├── mapreduce
        ├── tools
        └── yarn


 Step 5:  Add the Hadoop and Java paths in the bash file (.bashrc).

update ~/.bashrc
export HADOOP_HOME=$HOME/sachinPB/hadoop-3.2.0
export HADOOP_CONF_DIR=$HOME/sachinPB/hadoop-3.2.0/etc/hadoop
export HADOOP_MAPRED_HOME=$HOME/sachinPB/hadoop-3.2.0
export HADOOP_COMMON_HOME=$HOME/sachinPB/hadoop-3.2.0
export HADOOP_HDFS_HOME=$HOME/sachinPB/hadoop-3.2.0
export YARN_HOME=$HOME/sachinPB/hadoop-3.2.0
export PATH=$PATH:$HOME/sachinPB/hadoop-3.2.0/bin

#set Java Home

export JAVA_HOME=/opt/ibm/java-ppc64le-80
export PATH=$PATH:/opt/ibm/java-ppc64le-80/bin

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH

source .bashrc

 Step 6: Edit the Hadoop Configuration files as per your application requirements.
              HOW TO  CONFIGURE AND RUN BIG DATA APPLICATIONS ?

              Configuration files at : $HOME/sachinPB/hadoop-3.2.0/etc/Hadoop

 

Step 7: Open core-site.xml and edit the property mentioned below inside configuration tag.

SET NAMENODE LOCATION

core-site.xml

-----------------------------------------------------------------------------

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>

<name>fs.default.name</name>

<value>hdfs://hadoopNode1:9000</value>

</property>

</configuration>

-----------------------------------------------------------------------------

Step 8: Edit hdfs-site.xml and edit the property mentioned below inside configuration tag.

SET PATH FOR HDFS

hdfs-site.xml

-----------------------------------------------------------------------------

<?xml version="1.0" encoding="UTF-8"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

 <property>

   <name>dfs.replication</name>

   <value>1</value>

 </property>

 <property>

   <name>dfs.namenode.name.dir</name>

   <value>file:$DATA_DIR/hadoop/hdfs/nn</value>

 </property>

 <property>

   <name>dfs.datanode.data.dir</name>

   <value>file:$DATA_DIR/hadoop/hdfs/dn</value>

 </property>

 <property>

   <name>dfs.permissions</name>

   <value>false</value>

 </property>

</configuration>

----------------------------------------------------------------------------

Step 9: Edit the mapred-site.xml file and edit the property mentioned below inside the configuration tag.

SET YARN AS JOB SCHEDULER

mapred-site.xml

-------------------------------------------------------------------------

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

    <property>

        <name>mapreduce.framework.name</name>

        <value>yarn</value>

    </property>

    <property>

        <name>yarn.app.mapreduce.am.env</name>

        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

    </property>

    <property>

        <name>mapreduce.map.env</name>

        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

    </property>

    <property>

        <name>mapreduce.reduce.env</name>

        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>

    </property>

</configuration>

-----------------------------------------------------------------------------

Step 10: Edit yarn-site.xml and edit the property mentioned below inside configuration tag.

CONFIGURE YARN

yarn-site.xml

----------------------------------------------------------------

<?xml version="1.0"?>

<configuration>

<property>

    <name>yarn.nodemanager.aux-services</name>

    <value>mapreduce_shuffle</value>

</property>

</configuration>

-----------------------------------------------------------------

Step 11: Edit hadoop-env.sh/yarn-env.sh and add the Java Path as mentioned below.

 export JAVA_HOME=$JAVA_PPC64LE_PATH

 export HADOOP_HOME=$HOME/sachinPB/hadoop-3.2.0

Step 12 : The file workers is used by startup scripts to start required daemons on all nodes. This a very new change in Hadoop3 ( Please check)

CONFIGURE SLAVES

workers

-------------------------------

hadoopNode1

hadoopNode2

---------------------------------

check Java and hadoop version

[sachinpb@hadoopNode1 hadoop]$ java -version

openjdk version "1.8.0_181"

OpenJDK Runtime Environment (build 1.8.0_181-b13)

OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

Check Hadoop version:

[sachinpb@hadoopNode1 hadoop]$ hadoop version

Hadoop 3.2.0

This command was run using $HOME/sachinPB/hadoop-3.2.0/share/hadoop/common/hadoop-common-3.2.0.jar

[sachinpb@hadoopNode1 hadoop]$

-----------------------------------------------

Step 13: Next,  format the NameNode.

HDFS needs to be formatted like any classical file system. On node-master, run the following command:  "hdfs namenode -format"

[sachinpb@hadoopNode1]$ hdfs namenode -format -clusterId CID***-XYZ

2019-05-07 03:44:04,380 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = hadoopNode1/$HOST1_IPADDRESS

STARTUP_MSG:   args = [-format, -clusterId, CID***-XYZ]

STARTUP_MSG:   version = 3.2.0

STARTUP_MSG:   classpath = $HOME/sachinpb/sachinPB/hadoop-3.2.0/etc/hadoop:$HOME/sachinpb/sachinPB/hadoop-.

.

.

.

.

 [$DATA_DIR/hadoop/hdfs/nn/current/VERSION, $DATA_DIR/hadoop/hdfs/nn/current/seen_txid, $DATA_DIR/hadoop/hdfs/nn/current/fsimage_0000000000000000000.md5, $DATA_DIR/hadoop/hdfs/nn/current/fsimage_0000000000000000000, $DATA_DIR/hadoop/hdfs/nn/current/edits_0000000000000000001-0000000000000000002, $DATA_DIR/hadoop/hdfs/nn/current/edits_0000000000000000003-0000000000000000004, $DATA_DIR/hadoop/hdfs/nn/current/edits_0000000000000000005-0000000000000000006, $DATA_DIR/hadoop/hdfs/nn/current/edits_0000000000000000007-0000000000000000008, $DATA_DIR/hadoop/hdfs/nn/current/edits_inprogress_0000000000000000009]

2019-05-07 03:44:08,926 INFO common.Storage: Storage directory $DATA_DIR/hadoop/hdfs/nn has been successfully formatted.

2019-05-07 03:44:08,937 INFO namenode.FSImageFormatProtobuf: Saving image file $DATA_DIR/hadoop/hdfs/nn/current/fsimage.ckpt_0000000000000000000 using no compression

2019-05-07 03:44:09,063 INFO namenode.FSImageFormatProtobuf: Image file $DATA_DIR/hadoop/hdfs/nn/current/fsimage.ckpt_0000000000000000000 of size 401 bytes saved in 0 seconds .

2019-05-07 03:44:09,089 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

2019-05-07 03:44:09,104 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at hadoopNode1/$HOST1_IPADDRESS

************************************************************/

[sachinpb@hadoopNode1 logs]$

------------------------------------------

Step 14: Once the NameNode is formatted, go to Hadoop-3.0/sbin directory and start all the daemons.

Your Hadoop installation is now configured and ready to run big data application

Step 15: Start hdfs and yarn daemons:

Form directory : $HOME/sachinPB/hadoop-3.2.0/sbin

NOTE: Copy the Hadoop home directory  across all the nodes in your cluster (if its not available on shared directory)

[sachinpb@hadoopNode1 sbin]$ ./start-all.sh

WARNING: Attempting to start all Apache Hadoop daemons as sachinpb in 10 seconds..

Starting namenodes on [hadoopNode1]

hadoopNode1: Welcome to hadoopNode1!

hadoopNode1:

hadoopNode1: namenode is running as process 146418. 

Starting datanodes

hadoopNode1: Welcome to hadoopNode1!

hadoopNode1:

hadoopNode2: Welcome to hadoopNode2!

hadoopNode2:

hadoopNode1: datanode is running as process 146666.

hadoopNode2: datanode is running as process 112502. 

Starting secondary namenodes [hadoopNode1]

hadoopNode1: Welcome to hadoopNode1!

hadoopNode1:

hadoopNode1: secondarynamenode is running as process 147091.

[sachinpb@hadoopNode1 sbin]$

Step 16:  All the Hadoop services are up and running. [on other platforms, you could use jps command  to see hadoop daemons. IBM JAVA does not provide jps or jstat. So  you need to check the Hadoop process by ps command]

[sachinpb@hadoopNode1 sbin]$ ps -ef | grep NameNode

sachinpb   105015      1  0 02:59 ?        00:00:27 $JAVA_PPC64LE_PATH/bin/java -Dproc_namenode -Djava.library.path=$HOME/sachinpb/sachinPB/hadoop-3.2.0/lib -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dyarn.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dyarn.log.file=hadoop-sachinpb-namenode-hadoopNode1.log -Dyarn.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dyarn.root.logger=INFO,console -Dhadoop.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dhadoop.log.file=hadoop-sachinpb-namenode-hadoopNode1.log -Dhadoop.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dhadoop.id.str=sachinpb -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.namenode.NameNode

-----

sachinpb   105713      1  0 02:59 ?        00:00:12 $JAVA_PPC64LE_PATH/bin/java -Dproc_secondarynamenode -Djava.library.path=$HOME/sachinpb/sachinPB/hadoop-3.2.0/lib -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dyarn.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dyarn.log.file=hadoop-sachinpb-secondarynamenode-hadoopNode1.log -Dyarn.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dyarn.root.logger=INFO,console -Dhadoop.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dhadoop.log.file=hadoop-sachinpb-secondarynamenode-hadoopNode1.log -Dhadoop.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dhadoop.id.str=sachinpb -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode

 [sachinpb@hadoopNode1 sbin]$ ps -ef | grep DataNode

sachinpb   105268      1  0 02:59 ?        00:00:19 $JAVA_PPC64LE_PATH/bin/java -Dproc_datanode -Djava.library.path=$HOME/sachinpb/sachinPB/hadoop-3.2.0/lib -Dhadoop.security.logger=ERROR,RFAS -Dyarn.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dyarn.log.file=hadoop-sachinpb-datanode-hadoopNode1.log -Dyarn.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dyarn.root.logger=INFO,console -Dhadoop.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dhadoop.log.file=hadoop-sachinpb-datanode-hadoopNode1.log -Dhadoop.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dhadoop.id.str=sachinpb -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.datanode.DataNode

------

 [sachinpb@hadoopNode1 sbin]$ ps -ef | grep ResourceManager

sachinpb   106257      1  1 02:59 pts/3    00:00:50 $JAVA_PPC64LE_PATH/bin/java -Dproc_resourcemanager -Djava.library.path=$HOME/sachinpb/sachinPB/hadoop-3.2.0/lib -Dservice.libdir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/share/hadoop/yarn,$HOME/sachinpb/sachinPB/hadoop-3.2.0/share/hadoop/yarn/lib,$HOME/sachinpb/sachinPB/hadoop-3.2.0/share/hadoop/hdfs,$HOME/sachinpb/sachinPB/hadoop-3.2.0/share/hadoop/hdfs/lib,$HOME/sachinpb/sachinPB/hadoop-3.2.0/share/hadoop/common,$HOME/sachinpb/sachinPB/hadoop-3.2.0/share/hadoop/common/lib -Dyarn.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dyarn.log.file=hadoop-sachinpb-resourcemanager-hadoopNode1.log -Dyarn.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dyarn.root.logger=INFO,console -Dhadoop.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dhadoop.log.file=hadoop-sachinpb-resourcemanager-hadoopNode1.log -Dhadoop.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dhadoop.id.str=sachinpb -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.yarn.server.resourcemanager.ResourceManager

[sachinpb@hadoopNode1 sbin]$ ps -ef | grep NodeManager

sachinpb   106621      1  1 02:59 ?        00:01:08 $JAVA_PPC64LE_PATH/bin/java -Dproc_nodemanager -Djava.library.path=$HOME/sachinpb/sachinPB/hadoop-3.2.0/lib -Dyarn.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dyarn.log.file=hadoop-sachinpb-nodemanager-hadoopNode1.log -Dyarn.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dyarn.root.logger=INFO,console -Dhadoop.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dhadoop.log.file=hadoop-sachinpb-nodemanager-hadoopNode1.log -Dhadoop.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dhadoop.id.str=sachinpb -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.yarn.server.nodemanager.NodeManager

Ckeck similarly the status of hadoop daemons on other worker node [hadoopNode2]:

[sachinpb@hadoopNode2 ~]$ ps -ef | grep hadoop

sachinpb    77718      1  7 21:52 ?        00:00:07 $JAVA_PPC64LE_PATH/bin/java -Dproc_datanode -Djava.library.path=$HOME/sachinpb/sachinPB/hadoop-3.2.0/lib -Dhadoop.security.logger=ERROR,RFAS -Dyarn.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dyarn.log.file=hadoop-sachinpb-datanode-hadoopNode2.log -Dyarn.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dyarn.root.logger=INFO,console -Dhadoop.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dhadoop.log.file=hadoop-sachinpb-datanode-hadoopNode2.log -Dhadoop.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dhadoop.id.str=sachinpb -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.datanode.DataNode

sachinpb    78006      1 12 21:52 ?        00:00:11 $JAVA_PPC64LE_PATH/bin/java -Dproc_nodemanager -Djava.library.path=$HOME/sachinpb/sachinPB/hadoop-3.2.0/lib -Dyarn.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dyarn.log.file=hadoop-sachinpb-nodemanager-hadoopNode2.log -Dyarn.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dyarn.root.logger=INFO,console -Dhadoop.log.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0/logs -Dhadoop.log.file=hadoop-sachinpb-nodemanager-hadoopNode2.log -Dhadoop.home.dir=$HOME/sachinpb/sachinPB/hadoop-3.2.0 -Dhadoop.id.str=sachinpb -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.yarn.server.nodemanager.NodeManager

[sachinpb@hadoopNode2 ~]$

 Step 17:  Now open the Browser and go to localhost:9870/dfshealth.html to check the NameNode interface.

NOTE:  In Hadoop 2.x, web UI port is 50070 but in Hadoop3.x, it is moved to 9870. You can access HDFS web UI from localhost:9870

Step 18: run Hadoop application – Example: wordcount mapreduce program

[sachinpb@hadoopNode1 hadoop-3.2.0]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /user/sachinpb/helloworld /user/sachinpb/helloworld_out

2019-05-07 04:04:35,044 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032

2019-05-07 04:04:36,137 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: $MY_DIR/hadoop-yarn/staging/sachinpb/.staging/job_1557225898252_0003

2019-05-07 04:04:36,374 INFO input.FileInputFormat: Total input files to process : 1

2019-05-07 04:04:36,486 INFO mapreduce.JobSubmitter: number of splits:1

2019-05-07 04:04:36,536 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled

2019-05-07 04:04:36,728 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1557225898252_0003

2019-05-07 04:04:36,729 INFO mapreduce.JobSubmitter: Executing with tokens: []

2019-05-07 04:04:36,939 INFO conf.Configuration: resource-types.xml not found

2019-05-07 04:04:36,939 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.

2019-05-07 04:04:36,997 INFO impl.YarnClientImpl: Submitted application application_1557225898252_0003

2019-05-07 04:04:37,029 INFO mapreduce.Job: The url to track the job: http://hadoopNode1:8088/proxy/application_1557225898252_0003/

2019-05-07 04:04:37,030 INFO mapreduce.Job: Running job: job_1557225898252_0003

2019-05-07 04:04:45,137 INFO mapreduce.Job: Job job_1557225898252_0003 running in uber mode : false

2019-05-07 04:04:45,138 INFO mapreduce.Job:  map 0% reduce 0%

2019-05-07 04:04:51,189 INFO mapreduce.Job:  map 100% reduce 0%

2019-05-07 04:04:59,223 INFO mapreduce.Job:  map 100% reduce 100%

2019-05-07 04:04:59,232 INFO mapreduce.Job: Job job_1557225898252_0003 completed successfully

2019-05-07 04:04:59,348 INFO mapreduce.Job: Counters: 54

        File System Counters

                FILE: Number of bytes read=41

                FILE: Number of bytes written=443547

                FILE: Number of read operations=0

                FILE: Number of large read operations=0

                FILE: Number of write operations=0

                HDFS: Number of bytes read=126

                HDFS: Number of bytes written=23

                HDFS: Number of read operations=8

                HDFS: Number of large read operations=0

                HDFS: Number of write operations=2

                HDFS: Number of bytes read erasure-coded=0

        Job Counters

                Launched map tasks=1

                Launched reduce tasks=1

                Data-local map tasks=1

                Total time spent by all maps in occupied slots (ms)=3968

                Total time spent by all reduces in occupied slots (ms)=4683

                Total time spent by all map tasks (ms)=3968

                Total time spent by all reduce tasks (ms)=4683

                Total vcore-milliseconds taken by all map tasks=3968

                Total vcore-milliseconds taken by all reduce tasks=4683

                Total megabyte-milliseconds taken by all map tasks=4063232

                Total megabyte-milliseconds taken by all reduce tasks=4795392

        Map-Reduce Framework

                Map input records=1

                Map output records=3

                Map output bytes=29

                Map output materialized bytes=41

                Input split bytes=109

                Combine input records=3

                Combine output records=3

                Reduce input groups=3

                Reduce shuffle bytes=41

                Reduce input records=3

                Reduce output records=3

                Spilled Records=6

                Shuffled Maps =1

                Failed Shuffles=0

                Merged Map outputs=1

                GC time elapsed (ms)=288

                CPU time spent (ms)=4030

                Physical memory (bytes) snapshot=350552064

                Virtual memory (bytes) snapshot=3825860608

                Total committed heap usage (bytes)=177668096

                Peak Map Physical memory (bytes)=226557952

                Peak Map Virtual memory (bytes)=1911750656

                Peak Reduce Physical memory (bytes)=123994112

                Peak Reduce Virtual memory (bytes)=1914109952

        Shuffle Errors

                BAD_ID=0

                CONNECTION=0

                IO_ERROR=0

                WRONG_LENGTH=0

                WRONG_MAP=0

                WRONG_REDUCE=0

        File Input Format Counters

                Bytes Read=17

        File Output Format Counters

                Bytes Written=23

[sachinpb@hadoopNode1 hadoop-3.2.0]$

------------------------

Step 19: Verify the output file in HDFS:

[sachinpb@hadoopNode1 hadoop-3.2.0]$ hdfs dfs -cat /user/sachinpb/helloworld_out/part-r-00000

---------------------

2019    4

hello    6

world   7

---------------------

Step 20:  MONITOR YOUR HDFS CLUSTER

[sachinpb@hadoopNode1]$ hdfs dfsadmin -report

Configured Capacity: 1990698467328 (1.81 TB)

Present Capacity: 1794297528320 (1.63 TB)

DFS Remaining: 1794297511936 (1.63 TB)

DFS Used: 16384 (16 KB)

DFS Used%: 0.00%

Replicated Blocks:

        Under replicated blocks: 0

        Blocks with corrupt replicas: 0

        Missing blocks: 0

        Missing blocks (with replication factor 1): 0

        Low redundancy blocks with highest priority to recover: 0

        Pending deletion blocks: 0

Erasure Coded Block Groups:

        Low redundancy block groups: 0

        Block groups with corrupt internal blocks: 0

        Missing block groups: 0

        Low redundancy blocks with highest priority to recover: 0

        Pending deletion blocks: 0

-------------------------------------------------

Live datanodes (2):

Name: $HOST1_IPADDRESS:9866 (hadoopNode1)

Hostname: hadoopNode1

Decommission Status : Normal

Configured Capacity: 995349233664 (926.99 GB)

DFS Used: 12288 (12 KB)

Non DFS Used: 118666317824 (110.52 GB)

DFS Remaining: 876682903552 (816.47 GB)

DFS Used%: 0.00%

DFS Remaining%: 88.08%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu May 09 23:23:50 PDT 2019

Last Block Report: Thu May 09 23:11:35 PDT 2019

Num of Blocks: 0

Name: $HOST2_IPADDESS:9866 (hadoopNode2)

Hostname: hadoopNode2

Decommission Status : Normal

Configured Capacity: 995349233664 (926.99 GB)

DFS Used: 4096 (4 KB)

Non DFS Used: 77734621184 (72.40 GB)

DFS Remaining: 917614608384 (854.60 GB)

DFS Used%: 0.00%

DFS Remaining%: 92.19%

Configured Cache Capacity: 0 (0 B)

Cache Used: 0 (0 B)

Cache Remaining: 0 (0 B)

Cache Used%: 100.00%

Cache Remaining%: 0.00%

Xceivers: 1

Last contact: Thu May 09 23:23:49 PDT 2019

Last Block Report: Thu May 09 23:20:58 PDT 2019

Num of Blocks: 0

NOTE : you could see two live data nodes (hadoopNode1 & hadoopNode2) in this cluster  with all details about allocated HDFS space and block size...etc .This way you can check the health of the above Hadoop cluster. Also, we tested the wordcount application on this cluster as shown above.

Step 21: How to stop the Hadoop daemons in cluster environmnet:

 cd to $HOME/sachinpb/sachinPB/hadoop-3.2.0/sbin

[sachinpb@hadoopNode1 sbin]$ ./stop-all.sh

WARNING: Stopping all Apache Hadoop daemons as sachinpb in 10 seconds.

WARNING: Use CTRL-C to abort.

Stopping namenodes on [hadoopNode1]

hadoopNode1: Welcome to hadoopNode1!

hadoopNode1:

Stopping datanodes

hadoopNode1: Welcome to hadoopNode1!

hadoopNode1:

hadoopNode2: Welcome to hadoopNode2!

hadoopNode2:

Stopping secondary namenodes [hadoopNode1]

hadoopNode1: Welcome to hadoopNode1!

hadoopNode1:

Stopping nodemanagers

Stopping resourcemanagers on []

[sachinpb@hadoopNode1 sbin]$

I hope this blog helped in understanding how to install Hadoop 3.x in a multinode setup i.e  cluster and how to perform operation on HDFS files. Overall, Hadoop is a powerful tool for big data processing and analysis, with a wide range of use cases in industries such as finance, healthcare, retail, and telecommunications.

----------------------------------------END-------------------------------------------
Reference:
1) https://hadoop.apache.org/docs/r3.0.3/hadoop-project-dist/hadoop-common/SingleCluster.html
2) https://hadoop.apache.org/docs/r3.0.3/hadoop-project-dist/hadoop-common/ClusterSetup.html
3) https://hadoop.apache.org/docs/r3.0.3/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html
4) http://www.sachinpbuzz.com/2014/01/big-data-hadoop-20yarn-multi-node.html