Harp-DAAL for High Performance Big Data Computing. Qiu, J. Parallel Universe, 3, 2018. ![Harp-DAAL for High Performance Big Data Computing [pdf]](https://bibbase.org/img/filetypes/pdf.svg "pdf")
Website abstract bibtex Large-scale data analytics is revolutionizing many business and scientific domains. Easy-to-use scalable parallel techniques are necessary to process big data and gain meaningful insights. We introduce a novel HPC-Cloud convergence framework named Harp-DAAL and demonstrate that the combination of Big Data (Hadoop) and HPC techniques can simultaneously achieve productivity and performance. Harp is a distributed Hadoop-based framework that orchestrates efficient node synchronization [1]. Harp uses Intel ® Data Analytics Accelerator Library (DAAL) [2], for its highly optimized kernels on Intel ® Xeon and Xeon Phi architectures. This way the high-level API of Big Data tools can be combined with intra-node fine-grained parallelism that is optimized for HPC platforms. We illustrate this framework in detail with K-means clustering, a computation-bounded algorithm used in image clustering. We also show the broad applicability of Harp-DAAL by discussing the performance of three other big data algorithms: Subgraph Counting by color coding, Matrix Factorization and Latent Dirichlet Allocation. They share issues such as load imbalance, irregular structure, and communication issues that create difficult challenges. Figure 1 Cloud-HPC interoperable software for High Performance Big Data Analytics at Scale The categories in Figure 1 illustrate a classification of data intensive computation into five computation models that map into five distinct system architectures. It starts with Sequential, followed by centralized batch architectures corresponding exactly to the three forms of MapReduce: Map-Only, MapReduce and Iterative MapReduce. Category five is the classic MPI model. Harp brings Hadoop users the benefits of supporting all 5 classes of data-intensive computation, from pleasingly parallel to machine learning and simulations. We have expanded the applicability of Hadoop (with Harp plugin) for more classes of Big Data applications, especially complex data analytics such as machine learning and graph. We redesign a modular software stack with native kernels (with DAAL) to effectively utilize scale-up servers for machine learning and data analytics applications. Harp-DAAL shows how simulations and Big Data can use common programming environments with a runtime based on a rich set of collectives and libraries.
@article{
title = {Harp-DAAL for High Performance Big Data Computing},
type = {article},
year = {2018},
websites = {https://software.intel.com/sites/default/files/parallel-universe-issue-32.pdf},
month = {3},
id = {362b4d16-bf65-3c60-a6d4-aacdd96ef897},
created = {2019-10-01T17:20:53.992Z},
file_attached = {true},
profile_id = {42d295c0-0737-38d6-8b43-508cab6ea85d},
last_modified = {2020-05-11T14:43:32.906Z},
read = {false},
starred = {false},
authored = {true},
confirmed = {false},
hidden = {false},
citation_key = {Qiu2018},
private_publication = {false},
abstract = {Large-scale data analytics is revolutionizing many business and scientific domains. Easy-to-use scalable parallel techniques are necessary to process big data and gain meaningful insights. We introduce a novel HPC-Cloud convergence framework named Harp-DAAL and demonstrate that the combination of Big Data (Hadoop) and HPC techniques can simultaneously achieve productivity and performance. Harp is a distributed Hadoop-based framework that orchestrates efficient node synchronization [1]. Harp uses Intel ® Data Analytics Accelerator Library (DAAL) [2], for its highly optimized kernels on Intel ® Xeon and Xeon Phi architectures. This way the high-level API of Big Data tools can be combined with intra-node fine-grained parallelism that is optimized for HPC platforms. We illustrate this framework in detail with K-means clustering, a computation-bounded algorithm used in image clustering. We also show the broad applicability of Harp-DAAL by discussing the performance of three other big data algorithms: Subgraph Counting by color coding, Matrix Factorization and Latent Dirichlet Allocation. They share issues such as load imbalance, irregular structure, and communication issues that create difficult challenges. Figure 1 Cloud-HPC interoperable software for High Performance Big Data Analytics at Scale The categories in Figure 1 illustrate a classification of data intensive computation into five computation models that map into five distinct system architectures. It starts with Sequential, followed by centralized batch architectures corresponding exactly to the three forms of MapReduce: Map-Only, MapReduce and Iterative MapReduce. Category five is the classic MPI model. Harp brings Hadoop users the benefits of supporting all 5 classes of data-intensive computation, from pleasingly parallel to machine learning and simulations. We have expanded the applicability of Hadoop (with Harp plugin) for more classes of Big Data applications, especially complex data analytics such as machine learning and graph. We redesign a modular software stack with native kernels (with DAAL) to effectively utilize scale-up servers for machine learning and data analytics applications. Harp-DAAL shows how simulations and Big Data can use common programming environments with a runtime based on a rich set of collectives and libraries.},
bibtype = {article},
author = {Qiu, Judy},
journal = {Parallel Universe}
}
Downloads: 0
{"_id":"W6Nprv4koZNxqJXp9","bibbaseid":"qiu-harpdaalforhighperformancebigdatacomputing-2018","authorIDs":[],"author_short":["Qiu, J."],"bibdata":{"title":"Harp-DAAL for High Performance Big Data Computing","type":"article","year":"2018","websites":"https://software.intel.com/sites/default/files/parallel-universe-issue-32.pdf","month":"3","id":"362b4d16-bf65-3c60-a6d4-aacdd96ef897","created":"2019-10-01T17:20:53.992Z","file_attached":"true","profile_id":"42d295c0-0737-38d6-8b43-508cab6ea85d","last_modified":"2020-05-11T14:43:32.906Z","read":false,"starred":false,"authored":"true","confirmed":false,"hidden":false,"citation_key":"Qiu2018","private_publication":false,"abstract":"Large-scale data analytics is revolutionizing many business and scientific domains. Easy-to-use scalable parallel techniques are necessary to process big data and gain meaningful insights. We introduce a novel HPC-Cloud convergence framework named Harp-DAAL and demonstrate that the combination of Big Data (Hadoop) and HPC techniques can simultaneously achieve productivity and performance. Harp is a distributed Hadoop-based framework that orchestrates efficient node synchronization [1]. Harp uses Intel ® Data Analytics Accelerator Library (DAAL) [2], for its highly optimized kernels on Intel ® Xeon and Xeon Phi architectures. This way the high-level API of Big Data tools can be combined with intra-node fine-grained parallelism that is optimized for HPC platforms. We illustrate this framework in detail with K-means clustering, a computation-bounded algorithm used in image clustering. We also show the broad applicability of Harp-DAAL by discussing the performance of three other big data algorithms: Subgraph Counting by color coding, Matrix Factorization and Latent Dirichlet Allocation. They share issues such as load imbalance, irregular structure, and communication issues that create difficult challenges. Figure 1 Cloud-HPC interoperable software for High Performance Big Data Analytics at Scale The categories in Figure 1 illustrate a classification of data intensive computation into five computation models that map into five distinct system architectures. It starts with Sequential, followed by centralized batch architectures corresponding exactly to the three forms of MapReduce: Map-Only, MapReduce and Iterative MapReduce. Category five is the classic MPI model. Harp brings Hadoop users the benefits of supporting all 5 classes of data-intensive computation, from pleasingly parallel to machine learning and simulations. We have expanded the applicability of Hadoop (with Harp plugin) for more classes of Big Data applications, especially complex data analytics such as machine learning and graph. We redesign a modular software stack with native kernels (with DAAL) to effectively utilize scale-up servers for machine learning and data analytics applications. Harp-DAAL shows how simulations and Big Data can use common programming environments with a runtime based on a rich set of collectives and libraries.","bibtype":"article","author":"Qiu, Judy","journal":"Parallel Universe","bibtex":"@article{\n title = {Harp-DAAL for High Performance Big Data Computing},\n type = {article},\n year = {2018},\n websites = {https://software.intel.com/sites/default/files/parallel-universe-issue-32.pdf},\n month = {3},\n id = {362b4d16-bf65-3c60-a6d4-aacdd96ef897},\n created = {2019-10-01T17:20:53.992Z},\n file_attached = {true},\n profile_id = {42d295c0-0737-38d6-8b43-508cab6ea85d},\n last_modified = {2020-05-11T14:43:32.906Z},\n read = {false},\n starred = {false},\n authored = {true},\n confirmed = {false},\n hidden = {false},\n citation_key = {Qiu2018},\n private_publication = {false},\n abstract = {Large-scale data analytics is revolutionizing many business and scientific domains. Easy-to-use scalable parallel techniques are necessary to process big data and gain meaningful insights. We introduce a novel HPC-Cloud convergence framework named Harp-DAAL and demonstrate that the combination of Big Data (Hadoop) and HPC techniques can simultaneously achieve productivity and performance. Harp is a distributed Hadoop-based framework that orchestrates efficient node synchronization [1]. Harp uses Intel ® Data Analytics Accelerator Library (DAAL) [2], for its highly optimized kernels on Intel ® Xeon and Xeon Phi architectures. This way the high-level API of Big Data tools can be combined with intra-node fine-grained parallelism that is optimized for HPC platforms. We illustrate this framework in detail with K-means clustering, a computation-bounded algorithm used in image clustering. We also show the broad applicability of Harp-DAAL by discussing the performance of three other big data algorithms: Subgraph Counting by color coding, Matrix Factorization and Latent Dirichlet Allocation. They share issues such as load imbalance, irregular structure, and communication issues that create difficult challenges. Figure 1 Cloud-HPC interoperable software for High Performance Big Data Analytics at Scale The categories in Figure 1 illustrate a classification of data intensive computation into five computation models that map into five distinct system architectures. It starts with Sequential, followed by centralized batch architectures corresponding exactly to the three forms of MapReduce: Map-Only, MapReduce and Iterative MapReduce. Category five is the classic MPI model. Harp brings Hadoop users the benefits of supporting all 5 classes of data-intensive computation, from pleasingly parallel to machine learning and simulations. We have expanded the applicability of Hadoop (with Harp plugin) for more classes of Big Data applications, especially complex data analytics such as machine learning and graph. We redesign a modular software stack with native kernels (with DAAL) to effectively utilize scale-up servers for machine learning and data analytics applications. Harp-DAAL shows how simulations and Big Data can use common programming environments with a runtime based on a rich set of collectives and libraries.},\n bibtype = {article},\n author = {Qiu, Judy},\n journal = {Parallel Universe}\n}","author_short":["Qiu, J."],"urls":{"Website":"https://software.intel.com/sites/default/files/parallel-universe-issue-32.pdf"},"biburl":"https://bibbase.org/service/mendeley/42d295c0-0737-38d6-8b43-508cab6ea85d","bibbaseid":"qiu-harpdaalforhighperformancebigdatacomputing-2018","role":"author","metadata":{"authorlinks":{}},"downloads":0},"bibtype":"article","creationDate":"2019-09-12T13:19:08.411Z","downloads":0,"keywords":[],"search_terms":["harp","daal","high","performance","big","data","computing","qiu"],"title":"Harp-DAAL for High Performance Big Data Computing","year":2018,"biburl":"https://bibbase.org/service/mendeley/42d295c0-0737-38d6-8b43-508cab6ea85d","dataSources":["zgahneP4uAjKbudrQ","ya2CyA73rpZseyrZ8"]}