All about the Apache Hadoop Resource Manager

Within the Apache Hadoop open source distributed processing framework, YARN is one of the main components. The purpose of this resource management and task scheduling technology is to allocate system resources to different applications running in a Hadoop cluster. It is also used to schedule the execution of tasks on different clusters of clusters. Read More Points On Hadoop Online Training 

YARN: what is it?

The initials YARN refer to the term "Yet Another Resource Negotiator", a name given with humor by the developers. This technology became a sub-project of Apache Hadoop in 2012 and was added as a key feature of Hadoop with the 2.0 update deployed in 2013.

Before adding YARN, Hadoop could only run Map Reduce applications. YARN has therefore greatly increased the potential use cases of the framework. By decoupling resource management and planning from the Map Reduce data processing component, YARN has also enabled Hadoop to support more applications and different types of processing.

For example, Hadoop clusters are now able to launch real-time analytics, streaming data, and interactive queries on Apache Spark while letting Map Reduce run. Get More Points on Hadoop Training In Bangalore

YARN: what are the features?

YARN combines a central resource manager with containers, application coordinators, and agents to monitor the processing operations of different cluster nodes. YARN is able to dynamically allocate resources to applications based on their needs.

This component of Hadoop also offers several scheduling methods: FIFO Scheduler, Fair Scheduler, or Capacity Scheduler. In addition, the Reservation System feature allows users to reserve cluster resources in advance to ensure that important processing tasks are run smoothly.

Another notable feature, added with Hadoop 3.0, is YARN Federation. This increases the number of nodes that a single implementation of YARN can support by connecting different "sub-clusters" with their own resource managers. Get More Info On Hadoop Course

YARN consists of several main components. The purpose of the Resource Manager is to accept user-submitted tasks, schedule tasks, and allocate resources to them.

On each node, there is a Node Manager whose role is to monitor and report to the Resource Manager. There is also an Application Master, created for each application, to negotiate resources and work with the Node Manager to run and monitor tasks.

Finally, resource containers are controlled by Node Managers and assign resources allocated to individual applications. Generally, YARN containers are organized into nodes and programmed to perform tasks only if resources are available to do so. Under Hadoop 3.0, however, it is possible to create "opportunistic containers" that can be placed on hold until resources are released. This concept makes it possible to optimize the use of resources. Read More Info On Hadoop Certification