Title :

Self-Balancing Job Parallelism and Throughput in Hadoop

Author(s) :

Zhang, Bo [Auteur]
Self-adaptation for distributed services and large software systems [SPIRALS]
Université de Lille, Sciences et Technologies
Křikava, Filip [Auteur]
Faculty of Information Technology [Prague] [FIT CTU]
Rouvoy, Romain [Auteur]
Self-adaptation for distributed services and large software systems [SPIRALS]
Université de Lille, Sciences et Technologies
Seinturier, Lionel [Auteur]
Self-adaptation for distributed services and large software systems [SPIRALS]
Université de Lille, Sciences et Technologies
Institut universitaire de France [IUF]

Scientific editor(s) :

Márk Jelasity
Evangelia Kalyvianaki

Conference title :

16th IFIP WG 6.1 International Conference on Distributed Applications and Interoperable Systems (DAIS)

City :

Heraklion, Crete

Country :

Grèce

Start date of the conference :

2016-06-06

Book title :

Lecture Notes in Computer Science

Journal title :

Distributed Applications and Interoperable Systems

Publisher :

Springer

English keyword(s) :

Hadoop
Map Reduce
Self-optimisation

HAL domain(s) :

Informatique [cs]/Génie logiciel [cs.SE]
Informatique [cs]/Informatique ubiquitaire
Informatique [cs]/Informatique mobile
Informatique [cs]/Web
Informatique [cs]/Système d'exploitation [cs.OS]

English abstract : [en]

In Hadoop cluster, the performance and the resource consumption of MapReduce jobs do not only depend on the characteristics of these applications and workloads, but also on the appropriate setting of Hadoop configuration ...
Show more >In Hadoop cluster, the performance and the resource consumption of MapReduce jobs do not only depend on the characteristics of these applications and workloads, but also on the appropriate setting of Hadoop configuration parameters. However, when the job workloads are not known a priori or they evolve over time, a static configuration may quickly lead to a waste of computing resources and consequently to a performance degradation. In this paper, we therefore propose an on-line approach that dynamically reconfigures Hadoop at runtime. Concretely, we focus on balancing the job parallelism and throughput by adjusting Hadoop capacity scheduler memory configuration. Our evaluation shows that the approach outperforms vanilla Hadoop deployments by up to 40% and the best statically profiled configurations by up to 13%.Show less >

Language :

Anglais

Peer reviewed article :

Oui

Audience :

Internationale

Popular science :

Non

Collections :

• Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189

Source :

Harvested from HAL