Self-Balancing Job Parallelism and Throughput ...
Type de document :
Communication dans un congrès avec actes
Titre :
Self-Balancing Job Parallelism and Throughput in Hadoop
Auteur(s) :
Zhang, Bo [Auteur]
Université de Lille, Sciences et Technologies
Self-adaptation for distributed services and large software systems [SPIRALS]
Křikava, Filip [Auteur]
Faculty of Information Technology [Prague] [FIT CTU]
Rouvoy, Romain [Auteur]
Université de Lille, Sciences et Technologies
Self-adaptation for distributed services and large software systems [SPIRALS]
Seinturier, Lionel [Auteur]
Institut universitaire de France [IUF]
Université de Lille, Sciences et Technologies
Self-adaptation for distributed services and large software systems [SPIRALS]
Université de Lille, Sciences et Technologies
Self-adaptation for distributed services and large software systems [SPIRALS]
Křikava, Filip [Auteur]
Faculty of Information Technology [Prague] [FIT CTU]
Rouvoy, Romain [Auteur]
Université de Lille, Sciences et Technologies
Self-adaptation for distributed services and large software systems [SPIRALS]
Seinturier, Lionel [Auteur]
Institut universitaire de France [IUF]
Université de Lille, Sciences et Technologies
Self-adaptation for distributed services and large software systems [SPIRALS]
Éditeur(s) ou directeur(s) scientifique(s) :
Márk Jelasity
Evangelia Kalyvianaki
Evangelia Kalyvianaki
Titre de la manifestation scientifique :
16th IFIP WG 6.1 International Conference on Distributed Applications and Interoperable Systems (DAIS)
Ville :
Heraklion, Crete
Pays :
Grèce
Date de début de la manifestation scientifique :
2016-06-06
Titre de l’ouvrage :
Lecture Notes in Computer Science
Titre de la revue :
Distributed Applications and Interoperable Systems
Éditeur :
Springer
Mot(s)-clé(s) en anglais :
Hadoop
Map Reduce
Self-optimisation
Map Reduce
Self-optimisation
Discipline(s) HAL :
Informatique [cs]/Génie logiciel [cs.SE]
Informatique [cs]/Informatique ubiquitaire
Informatique [cs]/Informatique mobile
Informatique [cs]/Web
Informatique [cs]/Système d'exploitation [cs.OS]
Informatique [cs]/Informatique ubiquitaire
Informatique [cs]/Informatique mobile
Informatique [cs]/Web
Informatique [cs]/Système d'exploitation [cs.OS]
Résumé en anglais : [en]
In Hadoop cluster, the performance and the resource consumption of MapReduce jobs do not only depend on the characteristics of these applications and workloads, but also on the appropriate setting of Hadoop configuration ...
Lire la suite >In Hadoop cluster, the performance and the resource consumption of MapReduce jobs do not only depend on the characteristics of these applications and workloads, but also on the appropriate setting of Hadoop configuration parameters. However, when the job workloads are not known a priori or they evolve over time, a static configuration may quickly lead to a waste of computing resources and consequently to a performance degradation. In this paper, we therefore propose an on-line approach that dynamically reconfigures Hadoop at runtime. Concretely, we focus on balancing the job parallelism and throughput by adjusting Hadoop capacity scheduler memory configuration. Our evaluation shows that the approach outperforms vanilla Hadoop deployments by up to 40% and the best statically profiled configurations by up to 13%.Lire moins >
Lire la suite >In Hadoop cluster, the performance and the resource consumption of MapReduce jobs do not only depend on the characteristics of these applications and workloads, but also on the appropriate setting of Hadoop configuration parameters. However, when the job workloads are not known a priori or they evolve over time, a static configuration may quickly lead to a waste of computing resources and consequently to a performance degradation. In this paper, we therefore propose an on-line approach that dynamically reconfigures Hadoop at runtime. Concretely, we focus on balancing the job parallelism and throughput by adjusting Hadoop capacity scheduler memory configuration. Our evaluation shows that the approach outperforms vanilla Hadoop deployments by up to 40% and the best statically profiled configurations by up to 13%.Lire moins >
Langue :
Anglais
Comité de lecture :
Oui
Audience :
Internationale
Vulgarisation :
Non
Collections :
Source :
Fichiers
- https://hal.inria.fr/hal-01294834/document
- Accès libre
- Accéder au document
- https://hal.inria.fr/hal-01294834/document
- Accès libre
- Accéder au document
- https://hal.inria.fr/hal-01294834/document
- Accès libre
- Accéder au document
- document
- Accès libre
- Accéder au document
- zhang-dais16.pdf
- Accès libre
- Accéder au document