Best unofficial Apache Server developers community |
Hive, hadoop, and the mechanics behind hive.exec.reducers.max
|
In context of this other question here Using hive.exec.reducers.max directive has truely baffled me. From my perspective I thought hive worked on some sort of logic like, I have N # of blocks in a desired query so I need N maps. From N I will need some sensible range of reducers R which can be anywhere from R = N / 2 to R = 1. For the hive report I was working on, there was 1200+ maps and without any influence hive made a plan for about 400 reducers which was fine except I was working on a cluster that only had 70 reducers total. Even with the fair job scheduler, this caused a backlog that would hang up other jobs. So I tried a lot of different experiments until I found hive.exec.reducers.max and set it to something like 60. The results was that a hive job that took 248 minutes, finished in 155 minutes with no changes in the result. What's bothered me is, why not have hive default to N never being greater then the clusters reducer capacity and seeing as I can roll over several terabytes of data with a reduced set of reducers then what hive thinks is correct, is it better to always try and tweak this count?
posted via StackOverflow
|
2 Answers
|
|
You may want to look at(which talks about optimizing the number of slots): http://wiki.apache.org/hadoop/LimitingTaskSlotUsage Here is my opinion on the same: 1) Hive would ideally try optimize the number of reducers based on the expected amount of data that gets generated after the map task. It would expect the underlying cluster to be configured to support the same. 2) Regarding whether it may not be a good idea to tweak this count or not:
Case1: Hive is using 400 reducers Problem: Only 70 reducers can run at a given point of time.
Case2: Hive is using 70 reducers - Both the problems get addressed by setting this number. I guess its better to set the number of maximum available reducers. But, I am no expert at this. Would let the experts comment on this. |
|
|
In my experience with hive setting mapred.job.reuse.jvm.num.tasks to a healthy number (in my case, 8) helps with a lot of these ad-hoc queries. It takes around 20-30 seconds to spawn a JVM, so reuse can help out quite a bit with mappers and reducers that are short lived (< 30 seconds). |
