MONA: A generic big data management methodology for the high-level and automatic building of FAIR observatories, exploratory studies, and information profiling

Abstract

Big data science is a popular technology that empowers domain experts to uncover patterns within large datasets. It enables data-driven decision-making (DDDM) users to conduct exploratory analyses and extract actionable insights from vast volumes of information. However, adapting these systems to deliver customized and diverse information products for DDDM users remains a challenge, often requiring substantial manual effort and advanced technical expertise. This paper presents MONA, a big data management methodology to establish a transparent bridge between domain experts and DDDM users, facilitating seamless data exploration and enabling the extraction of meaningful insights. MONA begins with a data profiling model for domain experts to define big data pipelines at a high level. By integrating a profiling-based composition model with mappings of analytical processes, along with a pipeline orchestration framework, it transparently constructs customized big data pipelines that integrate and coordinate the processing, indexing, and storage within a unified data management. This approach enables DDDM experts to design FAIR observatories at a high level by focusing on queries and data visualization needs. MONA was implemented as a data platform and evaluated through the automatic construction of information product observatories, using datasets such as 18 years of Mexico's Pollutant Release Registries fused with Mexican Economic Sectors records, and 23 years of suicide registers in Mexico. The quantitative and qualitative evaluations revealed the suitability of MONA methodology to build FAIR observatories for transforming large and heterogeneous datasets, curated and prepared by domain experts, into vast amounts of interpretable information products, suitable for decision-making on DDDM processes.