Best unofficial Apache Server developers community

Answers

MapReduce shuffle/sort method

⇧

⇩

64 views

Somewhat of an odd question, but does anyone know what kind of sort MapReduce uses in the sort portion of shuffle/sort? I would think merge or insertion (in keeping with the whole MapReduce paradigm), but I'm not sure.

hadoop mapreduce hdfs

asked April 25, 2011 10:05 am CDT

SubSevn

posted via StackOverflow

1 Answers

⇧

⇩

It's Quicksort, afterwards the sorted intermediate outputs get merged together. Quicksort checks the recursion depth and gives up when it is too deep. If this is the case, Heapsort is used.

Have a look at the Quicksort class:

org.apache.hadoop.util.QuickSort

You can change the algorithm used via the map.sort.class value in the hadoop-default.xml.

answered April 28, 2011 7:43 am CDT

Thomas Jungblut

Your answer

Similar discussion threads

Get Shuffle and sort time
April 4, 2011 01:16:27 PM
Hi, Where can I get the shuffle and sort time of a reduce task?

How does Hadoop do sort(shuffle) after map exactly?
November 22, 2010 09:33:29 PM
Hi, I'm new to Hadoop and I want to know its implementation better. I was always wondering after mapping, how each reduce task get its input. It is said in google's paper and hadoop's documentation that a sort is done to aggregate the same key of…

Created: (MAPREDUCE-2308) Sort buffer size (io.sort.mb) is limited to < 2 GB
February 8, 2011 08:59:40 AM
Sort buffer size (io.sort.mb) is limited to < 2 GB

Created: (MAPREDUCE-2354) Shuffle should be optimized
March 4, 2011 05:02:02 AM
Shuffle should be optimized

Created: (MAPREDUCE-2168) We should implement limits on shuffle connections to TaskTracker per job
November 1, 2010 07:57:54 AM
We should implement limits on shuffle connections to TaskTracker per job

is configure( ) method of mapred api same as setup( ) method of mapreduce api
February 2, 2011 03:28:45 AM
Hi, setup( ), method present in mapred api, is called once at the start of each map/reduce task. Is it the same with configure( ) method present in mapreduce api ? Thanks, Giridhar.

Mapreduce sort by
February 4, 2011 03:27:58 AM
Hello, I did a map reduce and I get this result: Array ( [_id] => 2 [value] => 353 ) Array ( [_id] => 1 [value] => 230 ) Array ( [_id] => 3 [value] => 78 ) ..... I would like to sort the result by value, so I add this in my…

Re: Re: Mapreduce sort by
February 4, 2011 04:30:57 AM
It seems you don't understand how MapReduce works. It maps your data (sorting it makes no sense) to multiple separate chunks, which can be done in parallel, then reduces it (may be in parallel) and then stores result to some collection. You need to…

Could not evaluate: undefined method `sort' for :absent:Symbol
March 10, 2011 05:49:50 AM
Hi, Yesterday I implemented a module for managing Nagios configurations. It has turned out to be quite a complex beast, but in my test environment, it seemed to do everything it should. Now, in production, managing some 30 hosts, I get the…

Fail on mapreduce with sort/limit, please help
February 26, 2011 07:10:11 PM
I tried to use map and reduce to find the most visited files within any given range of time. However mongo replied with "best guess plan requested, but scan and order required:..." error if I added the "sort"/"limit" options. My environment is…

MapReduce sort/limit combination
August 21, 2010 07:17:40 PM
It is my understanding that the 'sort' parameter for map reduce sorts the results from the query when it's fed into the map reduce. However, I am getting strange results from a print statement in my map reduce that when I run a query, sort the…

work_mem = 900MB but Sort Method: external merge Disk: 304008kB
February 22, 2011 10:42:16 AM
What am I missing that causes this to resort to sorting on disk? obc=# select version(); version

Created: (MAPREDUCE-2229) Initialize reader in Sort example
December 22, 2010 09:21:29 PM
Initialize reader in Sort example

Buildr idea / idea7x failed on ruby 1.9.x -- String no long has method sort
January 11, 2011 01:43:56 PM
I ran into an issue today where idea / idea7x project generations uses method sort on String. This won't work for ruby 1.9.x: $ buildr idea7x (in /home/camyhsu/code/bug-submit/buildr/ruby1.9-idea, development) Writing …

Created: (MAPREDUCE-2397) Allow user to sort jobs in different sections (Completed, Failed, etc.) by
March 18, 2011 11:26:15 PM
Allow user to sort jobs in different sections (Completed, Failed, etc.) by the various columns available

Created: (MAPREDUCE-2295) Typo in method name getProtocolSigature
February 1, 2011 03:21:20 PM
Typo in method name getProtocolSigature

Created: (MAPREDUCE-2188) The new API MultithreadedMapper doesn't call the initialize method of the
November 15, 2010 05:58:00 PM
The new API MultithreadedMapper doesn't call the initialize method of the RecordReader

Created: (MAPREDUCE-2203) Wong javadoc for TaskRunner's appendJobJarClasspaths method
November 27, 2010 09:07:28 PM
Wong javadoc for TaskRunner's appendJobJarClasspaths method

Created: (MAPREDUCE-1996) API: Reducer.reduce() method detail misstatement
August 5, 2010 12:07:42 PM
API: Reducer.reduce() method detail misstatement

Shuffle tasks getting killed
September 22, 2010 11:53:49 PM
Hi, I continuously run a series of batch job using Hadoop Map Reduce. I also have a managing daemon that moves data around on the hdfs making way for more jobs to be run. I use capacity scheduler to schedule many jobs in parallel. I see an issue…

Questions and Answers are licensed under cc-wiki license.

MapReduce shuffle/sort method

1 Answers

Your answer

Join with account you already have