NASA's Earth Science Data Systems (ESDS) Program is committed to making the data it oversees findable, accessible, interoperable, and reusable (FAIR). With the implementation of NASA’s Earth System Observatory (ESO), a coordinated series of complementary missions designed to obtain measurements of multiple Earth processes to help address and mitigate climate change, the more than 100 petabytes (PB) of data in NASA's Earth Observing System Data and Information System (EOSDIS) archive are on the verge of exponential growth. Current estimates project that the volume of NASA Earth science data could grow to nearly 600 PB by 2030. In keeping with NASA’s open data policies, data from ESO and other projects will be available as early in the mission process as feasible.
Developing a Mission Data Processing System (MDPS) that will ensure the most transparent processing and delivery of future NASA mission and project data is vital. The MDPS is the set of algorithms, software, compute infrastructure, operational procedures, documentation, and teams that process raw instrument data into science quality data products. The MDPS also includes the software tools that support the development of processing algorithms and the validation and analysis of processed data.
The Challenge
In the current NASA Earth science data processing architecture, each mission receives its raw instrument data through the multi-mission Near Space Network and processes data using a mission-specific MDPS. Created data products are then delivered to NASA’s EOSDIS to be archived and distributed.
This siloed data processing architecture creates several difficulties for upcoming high-data volume missions. It is not conducive to facilitating coordinated missions, it creates barriers to enabling broad and early access to science and related software, and it complicates intra-mission and instrument science.
To address these issues, NASA Chief Science Data Officer Kevin Murphy issued a challenge to the broad mission processing community to identify the best model for an open MDPS to support upcoming missions. Specifically, Murphy challenged the community to identify a data processing architecture that not only meets mission science processing objectives and supports Earth system science, but also promotes open science principles and enables data system efficiencies.
Finding the Right Mission Data Processing System
Addressing Murphy's challenge is being accomplished through the Multi-Mission Data Processing System Study. The study comprises several phases:
- Phase 1: Concept Study (October 2021 to March 2023)
- Phase 2: Design Review and Architecture Study (March 2023 through September 2024)
- Phase 3: System Development
- Phase 4: System Implementation
Open Results from an Open Study
Open science is a foundational objective of NASA’s Science Mission Directorate (SMD), and is the principle and practice of making research products and processes available to all, while respecting diverse cultures, maintaining security and privacy, and fostering collaborations, reproducibility, and equity.
In keeping with these guidelines, the Multi-Mission Data Processing System Study is fully open and encourages participation by the broader science and data processing communities. It is guided by open science practices and constraints:
- The study features public workshops and open requests for information, and includes broad social media and outreach efforts
- Public workshops enable community participation, engage key stakeholders, and promote diverse and inclusive discussions
- The study solicits input from a broad and diverse set of flight project teams, industry partners, open science experts, and stakeholders across a wide spectrum of the science mission data systems community
- Representatives from upcoming missions participate as members of the study team
- The recommended MDPS is aligned with agency open data, software, algorithm, and publication policies (such as NASA’s SPD-41a)