mapreduce - What is the number of reducer slots on GCE Hadoop worker nodes? -

i testing scaling of mapreduce jobs on google compute engine's hadoop cluster, , finding unexpected results. in short, i've been told behavior may explained having multiple number of reducer slots per each worker node in hadoop cluster.

can confirm number of reducer slots per worker node (worker vm) mapreduce jobs on gce's hadoop cluster? using hadoop2_env.sh deployment.

https://groups.google.com/a/cloudera.org/forum/#!topic/oryx-user/afiu2pe2g8o provides link background discussion behavior experiencing, additional details if desired.

thanks!

in bdutil, number of reduce slots function of total number of cores on machine , environment variable cores_per_reduce_task, applied inside configure_hadoop.sh:

export num_cores="$(grep -c processor /proc/cpuinfo)" export map_slots=$(python -c "print int(${num_cores} // \     ${cores_per_map_task})") export reduce_slots=$(python -c "print int(${num_cores} // \     ${cores_per_reduce_task})")  <...>  # mapreduce v2 (and yarn) configuration if [[ -x configure_mrv2_mem.py ]];   temp_env_file=$(mktemp /tmp/mrv2_xxx_tmp_env.sh)   ./configure_mrv2_mem.py \       --output_file ${temp_env_file} \       --total_memory ${total_mem} \       --available_memory_ratio ${nodemanager_memory_fraction} \       --total_cores ${num_cores} \       --cores_per_map ${cores_per_map_task} \       --cores_per_reduce ${cores_per_reduce_task} \       --cores_per_app_master ${cores_per_app_master}   source ${temp_env_file}   # leave tmp_env_file around debugging purposes. fi 

you can see in hadoop2_env.sh default 2 cores per reduce slot:

cores_per_reduce_task=2.0 

optimal settings may vary based on workload, cases these default settings should fine. mentioned in thread linked, general approach can follow in actual workload, set computation-layer.parallelism approximately equal number of reduce slots have. if you're using default settings, take number of machines have, times cores-per-machine, divide 2 know number of slots. if want 1 reduce slot per machine, set cores_per_reduce_task equal cores-per-machine.

i approximately because there additional advanced settings around setting number of reduce tasks in job, including "speculative execution" settings; typical recommendation set reduce parallelism bit less, perhaps 0.95 times number of reduce slots; allows bit of headroom failed or stuck reduce tasks.

additionally, may have seen cases of faster performance when increased parallelism beyond number of reduce slots, despite expected slowdown due needing perform multiple "waves", due large variance in speed of different reduce tasks. in workloads there high variance, second "wave" can run concurrently slowest tasks of first "wave"; hadoop wiki gave rule of thumb of setting reduce parallelism either 0.95 or 1.75 times number of available reduce slots. here's further discussion on topic; poster there correctly points out these applicable single-tenant clusters.

if want share large cluster concurrently lots of users, these rules of thumb don't apply, since should assign parallelism purely based on size , characteristics of workload, since wouldn't want hog 100% of cluster resources. however, recommended approach in cloud environment indeed have multiple, smaller single-tenant clusters, since can tune each cluster workloads want , don't need worry resource-packing among lots of different use cases.