Download Cloudera.CCD-410.TestKing.2018-11-02.32q.vcex

Reducer has 3 primary phases:1. Shuffle The Reducer copies the sorted output from each Mapper using HTTP across the network. 2. Sort The framework merge sorts Reducer inputs by keys (since different Mappers may have output the same key). The shuffle and sort phases occur simultaneously i.e. while outputs are being fetched they are merged. SecondarySort To achieve a secondary sort on the values returned by the value iterator, the application should extend the key with the secondary key and define a grouping comparator. The keys will be sorted using the entire key, but will be grouped using the grouping comparator to decide which keys and values are sent in the same call to reduce. 3. Reduce In this phase the reduce(Object, Iterable, Context) method is called for each <key, (collection of values)> in the sorted inputs. The output of the reduce task is typically written to a RecordWriter via TaskInputOutputContext.write(Object, Object). The output of the Reducer is not re-sorted. Reference: org.apache.hadoop.mapreduce, Class Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT>

Reducer has 3 primary phases:1. Shuffle The Reducer copies the sorted output from each Mapper using HTTP across the network. 2. Sort The framework merge sorts Reducer inputs by keys (since different Mappers may have output the same key). The shuffle and sort phases occur simultaneously i.e. while outputs are being fetched they are merged. SecondarySort To achieve a secondary sort on the values returned by the value iterator, the application should extend the key with the secondary key and define a grouping comparator. The keys will be sorted using the entire key, but will be grouped using the grouping comparator to decide which keys and values are sent in the same call to reduce. 3. Reduce In this phase the reduce(Object, Iterable, Context) method is called for each <key, (collection of values)> in the sorted inputs. The output of the reduce task is typically written to a RecordWriter via TaskInputOutputContext.write(Object, Object). The output of the Reducer is not re-sorted. Reference: org.apache.hadoop.mapreduce, Class Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT>

See 3) below. Here is an illustrative example on how to use the DistributedCache:     // Setting up the cache for the application            1. Copy the requisite files to the FileSystem:           $ bin/hadoop fs -copyFromLocal lookup.dat /myapp/lookup.dat        $ bin/hadoop fs -copyFromLocal map.zip /myapp/map.zip        $ bin/hadoop fs -copyFromLocal mylib.jar /myapp/mylib.jar      $ bin/hadoop fs -copyFromLocal mytar.tar /myapp/mytar.tar      $ bin/hadoop fs -copyFromLocal mytgz.tgz /myapp/mytgz.tgz      $ bin/hadoop fs -copyFromLocal mytargz.tar.gz /myapp/mytargz.tar.gz            2. Setup the application's JobConf:           JobConf job = new JobConf();      DistributedCache.addCacheFile(new URI("/myapp/lookup.dat#lookup.dat"),                                     job);      DistributedCache.addCacheArchive(new URI("/myapp/map.zip", job);      DistributedCache.addFileToClassPath(new Path("/myapp/mylib.jar"), job);      DistributedCache.addCacheArchive(new URI("/myapp/mytar.tar", job);      DistributedCache.addCacheArchive(new URI("/myapp/mytgz.tgz", job);      DistributedCache.addCacheArchive(new URI("/myapp/mytargz.tar.gz", job);            3. Use the cached files in the Mapper      or Reducer:           public static class MapClass extends MapReduceBase        implements Mapper<K, V, K, V> {              private Path[] localArchives;        private Path[] localFiles;                public void configure(JobConf job) {          // Get the cached archives/files          localArchives = DistributedCache.getLocalCacheArchives(job);          localFiles = DistributedCache.getLocalCacheFiles(job);        }                public void map(K key, V value,                         OutputCollector<K, V> output, Reporter reporter)         throws IOException {          // Use data from the cached archives/files here          // ...          // ...          output.collect(k, v);        }      }       Reference: org.apache.hadoop.filecache , Class DistributedCache

Hadoop MapReduce for Parsing Weblogs Here are the steps for parsing a log file using Hadoop MapReduce:Load log files into the HDFS location using this Hadoop command:hadoop fs -put <local file path of weblogs>  <hadoop HDFS location> The Opencsv2.3.jar framework is used for parsing log records. Below is the Mapper program for parsing the log file from the HDFS location. public static class ParseMapper       extends Mapper<Object, Text, NullWritable,Text >{ private Text word = new Text(); public void map(Object key, Text value, Context context                ) throws IOException, InterruptedException {      CSVParser parse = new CSVParser(' ','\"');      String sp[]=parse.parseLine(value.toString());      int spSize=sp.length;      StringBuffer rec= new StringBuffer();      for(int i=0;i<spSize;i++){           rec.append(sp[i]);           if(i!=(spSize-1))           rec.append(",");      }      word.set(rec.toString());      context.write(NullWritable.get(), word); } } The command below is the Hadoop-based log parse execution. TheMapReduce program is attached in this article. You can add extra parsing methods in the class. Be sure to create a new JAR with any change and move it to the Hadoop distributed job tracker system. hadoop jar <path of logparse jar> <hadoop HDFS logfile path>  <output path of parsed log file> The output file is stored in the HDFS location, and the output file name starts with "part-".

YARN (Yet Another Resource Negotiator), as an aspect of Hadoop, has two major kinds of benefits:* (D) The ability to use programming frameworks other than MapReduce. / MPI (Message Passing Interface) was mentioned as a paradigmatic example of a MapReduce alternative * Scalability, no matter what programming framework you use. Note: * The fundamental idea of MRv2 is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs. * (B) The central goal of YARN is to clearly separate two things that are unfortunately smushed together in current Hadoop, specifically in (mainly) JobTracker:/ Monitoring the status of the cluster with respect to which nodes have which resources available. Under YARN, this will be global. / Managing the parallelization execution of any specific job. Under YARN, this will be done separately for each job. The current Hadoop MapReduce system is fairly scalable — Yahoo runs 5000 Hadoop jobs, truly concurrently, on a single cluster, for a total 1.5 – 2 millions jobs/cluster/month. Still, YARN will remove scalability bottlenecks Reference: Apache Hadoop YARN – Concepts & Applications

Configure the property using the -D key=value notation:-D mapred.job.name='My Job' You can list a whole bunch of options by calling the streaming jar with just the -info argument Reference: Python hadoop streaming : Setting a job name

The input types fed to the mapper are controlled by the InputFormat used. The default input format, "TextInputFormat," will load data in as (LongWritable, Text) pairs. The long value is the byte offset of the line in the file. The Text object holds the string contents of the line of the file. Note: The data types emitted by the reducer are identified by setOutputKeyClass() andsetOutputValueClass().  The data types emitted by the reducer are identified by setOutputKeyClass() and setOutputValueClass(). By default, it is assumed that these are the output types of the mapper as well. If this is not the case, the methods setMapOutputKeyClass() and setMapOutputValueClass() methods of the JobConf class will override these. Reference: Yahoo! Hadoop Tutorial, THE DRIVER METHOD

The fundamental idea of MRv2 (YARN) is to split up the two major functionalities of the JobTracker, resource management and job scheduling/monitoring, into separate daemons. The idea is to have a global ResourceManager (RM) and per-application ApplicationMaster (AM). An application is either a single job in the classical sense of Map-Reduce jobs or a DAG of jobs. Note: Let’s walk through an application execution sequence :A client program submits the application, including the necessary specifications to launch the application-specific ApplicationMaster itself. The ResourceManager assumes the responsibility to negotiate a specified container in which to start the ApplicationMaster and then launches the ApplicationMaster. The ApplicationMaster, on boot-up, registers with the ResourceManager – the registration allows the client program to query the ResourceManager for details, which allow it to  directly communicate with its own ApplicationMaster. During normal operation the ApplicationMaster negotiates appropriate resource containers via the resource-request protocol. On successful container allocations, the ApplicationMaster launches the container by providing the container launch specification to the NodeManager. The launch specification, typically, includes the necessary information to allow the container to communicate with the ApplicationMaster itself. The application code executing within the container then provides necessary information (progress, status etc.) to its ApplicationMaster via an application-specific protocol. During the application execution, the client that submitted the program communicates directly with the ApplicationMaster to get status, progress updates etc. via an application-specific protocol. Once the application is complete, and all necessary work has been finished, the ApplicationMaster deregisters with the ResourceManager and shuts down, allowing its own container to be repurposed. Reference: Apache Hadoop YARN – Concepts & Applications

As the Map operation is parallelized the input file set is first split to several pieces called FileSplits. If an individual file is so large that it will affect seek time it will be split to several Splits. The splitting does not know anything about the input file's internal logical structure, for example line-oriented text files are split on arbitrary byte boundaries. Then a new map task is created per FileSplit. When an individual map task starts it will open a new output writer per configured reduce task. It will then proceed to read its FileSplit using the RecordReader it gets from the specified InputFormat. InputFormat parses the input and generates key-value pairs. InputFormat must also handle records that may be split on the FileSplit boundary. For example TextInputFormat will read the last line of the FileSplit past the split boundary and, when reading other than the first FileSplit, TextInputFormat ignores the content up to the first newline. Reference: How Map and Reduce operations are actually carried out

Mapper maps input key/value pairs to a set of intermediate key/value pairs. Maps are the individual tasks that transform input records into intermediate records. The transformed intermediate records do not need to be of the same type as the input records. A given input pair may map to zero or many output pairs. Reference: Hadoop Map-Reduce Tutorial