Implementing a Flexible, Fault Tolerant Job Management System for Science Gateways

Implementing a Flexible, Fault Tolerant Job Management System for Science Gateways. Wannipurage, D., Marru, S., Piece, M., Abeysinghe, E., Pamidighantam, S., Christie, M., Shenoy, G., Dhamnaskar, A., & Jayathilaka, L. In Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (learning) - PEARC '19, pages 1-8, 2019. ACM Press.

Paper

Implementing a Flexible, Fault Tolerant Job Management System for Science Gateways [link]

Website doi abstract bibtex

This paper summarizes our experiences evaluating and deploying a new task execution management system within the open source Apache Airavata framework for science gateways. We base our choices on our operational requirements and experiences running Airavata software as a multi-tenanted production service for multiple gateway clients. Our considerations include integrating semi-independent components, making major upgrades to those components while retaining the system's overall functionality, and choosing between integrating third party and in-house developed components. While we focus on Apache Airavata as the platform for evaluation, our results should be of general interest. After considering the options of extensions to our previous, in-house job management system using Apache Kafka or replacing it with Kubernetes, we ultimately chose Apache Helix, primarily for its ability to execute multiple tasks coupled into directed acyclic graphs. We have integrated this approach into Apache Airavata and have tested extensively over several months with many thousands of jobs, both from our internal throughput testing and operational tests with early adopter science gateway clients. The new system has proven to be at least as reliable as the previous system with the advantages that we now have simplified maintenance, do not need to support an in-house system that required extensive developer training to modify, and can support more sophisticated job execution scenarios.

@inproceedings{
 title = {Implementing a Flexible, Fault Tolerant Job Management System for Science Gateways},
 type = {inproceedings},
 year = {2019},
 pages = {1-8},
 websites = {http://dl.acm.org/citation.cfm?doid=3332186.3332233},
 publisher = {ACM Press},
 city = {New York, New York, USA},
 id = {81ea3925-59cb-30fb-8332-2af82619daec},
 created = {2019-10-01T17:21:29.424Z},
 accessed = {2019-09-12},
 file_attached = {true},
 profile_id = {42d295c0-0737-38d6-8b43-508cab6ea85d},
 last_modified = {2020-05-11T14:43:46.062Z},
 read = {false},
 starred = {false},
 authored = {true},
 confirmed = {true},
 hidden = {false},
 citation_key = {Wannipurage2019},
 private_publication = {false},
 abstract = {This paper summarizes our experiences evaluating and deploying a new task execution management system within the open source Apache Airavata framework for science gateways. We base our choices on our operational requirements and experiences running Airavata software as a multi-tenanted production service for multiple gateway clients. Our considerations include integrating semi-independent components, making major upgrades to those components while retaining the system's overall functionality, and choosing between integrating third party and in-house developed components. While we focus on Apache Airavata as the platform for evaluation, our results should be of general interest. After considering the options of extensions to our previous, in-house job management system using Apache Kafka or replacing it with Kubernetes, we ultimately chose Apache Helix, primarily for its ability to execute multiple tasks coupled into directed acyclic graphs. We have integrated this approach into Apache Airavata and have tested extensively over several months with many thousands of jobs, both from our internal throughput testing and operational tests with early adopter science gateway clients. The new system has proven to be at least as reliable as the previous system with the advantages that we now have simplified maintenance, do not need to support an in-house system that required extensive developer training to modify, and can support more sophisticated job execution scenarios.},
 bibtype = {inproceedings},
 author = {Wannipurage, Dimuthu and Marru, Suresh and Piece, Marlon and Abeysinghe, Eroma and Pamidighantam, Sudhakar and Christie, Marcus and Shenoy, Gourav and Dhamnaskar, Ajinkya and Jayathilaka, Lahiru},
 doi = {10.1145/3332186.3332233},
 booktitle = {Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (learning) - PEARC '19}
}

Downloads: 0