How Automation has Improved the Performance of the MMS SDC
Journal Article
Overview
abstract
The Magnetospheric Multiscale (MMS) Science Data Center (SDC) at the; Laboratory for Atmospheric and Space Physics (LASP), at the University; of Colorado, has managed MMS science and ancillary data processing and; distribution since MMS launched in March 2015. The MMS SDC employs; automation in nearly every part of its operations. Automation is used to; start up processing “runners” that listen on queues for new processing; jobs, which are triggered by configurable timing rules, including cron; and operational events, or certain data/data files being available. A; separate set of SDC code then automatically creates processing jobs and; tracks its progress. The MMS SDC runs processing jobs for each; instrument (47 different job types in total), ranging from levels l1a to; l3, for “survey” and “burst” modes, plotting, and cdf creation. The; SDC runs anywhere from a few hundred to over 2,000 jobs per day (on; average, 1,000 jobs per day). Processing jobs are scheduled in a few; different ways, from running based on a fixed schedule in cron, to being; triggered by certain mission events, to being triggered by the; appearance of new files. Several fail-safes have been added into the; code over time to ensure failures are caught and handled, however,; situations do arise where failures occur and are dealt with when; something in the SDC does not work as expected. Added to these; complexities is the fact that the MMS mission is incredibly; time-sensitive, and requires the SDC to be available and ready to handle; issues 24/7/365, which can be challenging due to the limited staffing on; MMS a few years into the mission. The importance of automation in MMS; SDC processing is clear. Not only does automated processing relieve some; of the load from the software engineers working on the SDC, but it; ensures continued smooth operation of the MMS SDC. This then allows; scientists to continue their research efforts unhindered. As time has; progressed, various areas for improvement, and extra automation, in this; process have been implemented. This poster will focus on automation; improvements to keep the system running smoothly with almost no human; involvement.
