Practical Online Debugging of Spark-like ...
Document type :
Communication dans un congrès avec actes
Title :
Practical Online Debugging of Spark-like applications
Author(s) :
Marra, Matteo [Auteur]
Software Languages Lab [SLL]
Polito, Guillermo [Auteur]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Analyses and Languages Constructs for Object-Oriented Application Evolution [RMOD]
Gonzalez Boix, Elisa [Auteur]
Software Languages Lab [SLL]
Software Languages Lab [SLL]
Polito, Guillermo [Auteur]
Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 [CRIStAL]
Analyses and Languages Constructs for Object-Oriented Application Evolution [RMOD]
Gonzalez Boix, Elisa [Auteur]
Software Languages Lab [SLL]
Conference title :
IEEE QRS 2021 : International Conference on Software Security and Reliability
City :
Hainan Island
Country :
Chine
Start date of the conference :
2021-12-06
Journal title :
IEEE QRS 2021 : International Conference on Software Security and Reliability
HAL domain(s) :
Informatique [cs]/Langage de programmation [cs.PL]
English abstract : [en]
Apache Spark is a framework widely used for writing Big Data analytics applications that offers a scalable and fault-tolerant model based on rescheduling failing tasks on other nodes. While this is well-suited for hardware ...
Show more >Apache Spark is a framework widely used for writing Big Data analytics applications that offers a scalable and fault-tolerant model based on rescheduling failing tasks on other nodes. While this is well-suited for hardware and infrastructure errors, it is not for application errors as they will reappear in the rescheduled tasks. As a result, applications are killed, losing all the progress and forcing developers to restart them from scratch. Despite the popularity of such a failure-recovery model, understanding and debugging Sparklike applications remain challenging. When an error occurs, developers need to analyze huge log files or undergo timeconsuming replays to find the bug. To address these concerns, we present an online debugging approach tailored to Big Data analytics applications. Our approach includes local debugging of remote parallel exceptions through dynamic local checkpoints, extended with domain-specific debugging operations and live code updating functionality. To deal with data-cleaning errors, we extend our model to easily allow developers to automatically ignore exceptions that happen at runtime. We validate our solution through performance benchmarks that show how our debugging approach is comparable or better than state-of-theart debugging solutions for Big Data. Furthermore, we conduct a user study to compare our approach with another state-of-theart debugging approach, and results show a lower time to find the solution to a bug using our approach, as well as a generally good perception of the features of the debugger.Show less >
Show more >Apache Spark is a framework widely used for writing Big Data analytics applications that offers a scalable and fault-tolerant model based on rescheduling failing tasks on other nodes. While this is well-suited for hardware and infrastructure errors, it is not for application errors as they will reappear in the rescheduled tasks. As a result, applications are killed, losing all the progress and forcing developers to restart them from scratch. Despite the popularity of such a failure-recovery model, understanding and debugging Sparklike applications remain challenging. When an error occurs, developers need to analyze huge log files or undergo timeconsuming replays to find the bug. To address these concerns, we present an online debugging approach tailored to Big Data analytics applications. Our approach includes local debugging of remote parallel exceptions through dynamic local checkpoints, extended with domain-specific debugging operations and live code updating functionality. To deal with data-cleaning errors, we extend our model to easily allow developers to automatically ignore exceptions that happen at runtime. We validate our solution through performance benchmarks that show how our debugging approach is comparable or better than state-of-theart debugging solutions for Big Data. Furthermore, we conduct a user study to compare our approach with another state-of-theart debugging approach, and results show a lower time to find the solution to a bug using our approach, as well as a generally good perception of the features of the debugger.Show less >
Language :
Anglais
Peer reviewed article :
Oui
Audience :
Internationale
Popular science :
Non
Collections :
Source :
Files
- https://hal.archives-ouvertes.fr/hal-03398772/document
- Open access
- Access the document
- https://hal.archives-ouvertes.fr/hal-03398772/document
- Open access
- Access the document
- https://hal.archives-ouvertes.fr/hal-03398772/document
- Open access
- Access the document
- document
- Open access
- Access the document
- paper.pdf
- Open access
- Access the document