CHORDS, Cloud-Hosted Real-time Data Services for the Geosciences, is an Earthcube Building Block project first funded in 2014 to provide real-time data services infrastructure for small research teams to acquire, navigate, and distribute real-time data streams via cloud services and the Internet. After an initial round of workshops to gather requirements from the geoscience community, the team handily proved their concept across a variety of instrumentation via a 2014 pilot project, and in 2016 the CHORDS team was awarded an Expanded Implementation to focus on heavy testing in broader applications, and connecting the resulting data streams into downstream services, such as archives, sensor webs and modeling.
Initial Successes (2014-2016)
Though technology to gather data has been steadily and exponentially improving, widespread use of real-time data across the geosciences is still often hampered by the expense of deploying and operating existing data platforms. Many academic geoscience research groups are small with limited funding and simply cannot foot the bill for the software staff required to develop, set up, and maintain extensive data services to support their observations. CHORDS is unique among real-time data platforms available to the multi-disciplinary geosciences community in that is “shrink-wrapped”, non-proprietary, and available to a wide variety of research applications. In the mind of PI Mike Daniels, the mandate for the project was clear from the start. “We wanted to create a system that is so easy, researchers can manage it themselves.”
The initial CHORDS project tested real-time data streams in various use cases: hydrology, solid earth, volcanoes, meteorology, and was even was deployed for use on the NSF GV Research Aircraft during the ORCAS field project. Even in the early days the team was able to demonstrate how simple it was to get an arbitrary datastream on the internet. The instances were easy to set up and required minimal overhead or IT support, freeing researchers to focus on actual research, rather than managing complex data systems and adhering to standards.
Charlie Martin, CHORDS Senior Architect, thinks the most compelling aspect of CHORDS is how easy it is to integrate real-time data into any workflow. “Most science professionals are fairly comfortable with and work primarily in spreadsheets, or writing small programs or web pages. CHORDS makes it really easy for them to get their data into whatever tools that they are using. Without software staff. Without budgets. Without IT departments.”
Access to real-time data is especially critical when sampling from remote or geographically inhospitable environments, such as at the base of a volcano. D. Sarah Stamps, assistant professor, and Josh Jones, a graduate student, both in the Virginia Tech Department of Geosciences’ Geodesy and Tectonophysics Laboratory, test equipment before setting up an observatory at the base of Tanzania’s Ol Doinyo Lengai volcano. Read more about their research here.
Not only was CHORDS reliable and robust in the field, the data could easily be used to help researchers make quick decisions in mission critical situations. Daniels, who has worked to support field researchers for decades, was well aware of the desire for this capability. “I’ve heard for years that scientists would like to bring real-time data directly into forecasting models to help steer mobile observations systems (like planes, mobile radars, vehicles, etc.) to optimal sampling locations based on a model that is being continuously updated with real-time data, forming a constantly improving feedback loop,” he said. “In addition, recent research has indicated that infrasound sensors, typically used by the solid earth community, may also be beneficial to atmospheric scientists to detect and track tornadoes. Through CHORDS, these and other data streams will be made visible and accessible to researchers in the field so that their experiment can benefit from them.” he said.
Democratization of Data
The low cost, ease of use, and rapid spin up of CHORDS’ online data streams means it is far easier for researchers to sample data sparse regions, then quickly make those data available to forecasters or farmers in local communities. In 2015, just a year after the CHORDS project began, NCAR scientist Paul Kucera was looking for a useful platform to support the transfer, archiving, and display of observations to monitor the 3D Printed Automatic Weather Stations (3d-PAWS) he and his team were in the process of installing in remote places in Zambia, Kenya, and the Caribbean. The CHORDS team thought they would be uniquely qualified to help Kucera meet his goals, and Kucera agreed. “Once they explained the system, it seemed very straightforward to implement the CHORDS API in our weather station software,” he said.
NCAR scientist Paul Kucera describes the various components of the 3D-PAWS at the Sirua Aulo Maasai High School, including CHORDS, which is used to manage the weather stations’ data streams. (©UCAR. Photo by Kristin Wegner. This image is freely available for media & nonprofit use.)
CHORDS has shown to be a great tool to detect problems with stations quickly; Kucera’s team didn’t have to wait weeks or months to find out that a station had problems with its data stream. This ensured maximum uptime of observational instrumentation, and much reduced risk of wasted time. In addition, CHORDS has made the process of research easier by giving Kucera’s team quick access to the data. “We are now able to download the observations and perform various analyses quickly. This really has helped for the evaluation of the systems and for making quick plots after a significant, high impact event has occurred,” Kucera said.
CHORDS was also used to quickly download and perform preliminary analysis shown in an article Kucera wrote for Meteorological International, published in the September 2016 issue, and for the World Meteorological Center in April 2017 (Innovative Technology for Low-Cost Surface Atmospheric Observations (UCAR).)
Advances in observational capabilities have, for the first time ever, made it possible for people in developing nations to reap the benefits of having weather and atmospheric data to make decisions affecting crops, livestock, or even detect changes like flash floods or other catastrophic weather events with enough time to save lives.
CHORDS can already address a variety of research needs:
Systems Reliability: CHORDS is ideal for “risk reduction”: Monitor in real-time to insure instruments are working, working correctly, and that data are being gathered.
Operational Decision Making: Make rapid decisions based on observed conditions. The collaborating decision makers have automatic shared access to the same data.
Field Research Guidance: Modify their experimental design based on the up-to-date streaming data sources in a region of interest.
Instant Data Exploration: Easily connect models to CHORDS and quickly be able to explore, forecast, or analyze. No longer have to wait for the typically delayed observational dataset.
CHORDS Expanded Implementation (2016-2019)
Now that CHORDS has proven its field usability and is nearing production-ready mode, the team is working with downstream data services so the real-time data being collected can be used by others. In the CHORDS Expanded Implementation, Daniels and his team are testing CHORDS with a variety of new data streams, then forwarding data to higher level services, which will provide capabilities such as integration and analysis with other disciplines, data federation, data discovery, and permanent archiving. The team recently completed the first demonstration of this capability in a lab setting, in which 22 new CHORDS instances were created and the resulting streams were connected to CUAHSI data services.
Essential to the continued success of the Expanded Implementation will be the team’s ability to forge collaborations with communities across the geosciences disciplines that have use for the data that has been collected, and the team is keeping an eye out for teams such those implementing arrays of inexpensive sensors or modeling groups. Their goal is to engage with a broad range of users, to ensure they can build something that is general enough to be configured to meet the unique needs of each community.
Call for Collaboration
The CHORDS team is busy locating and reaching out to producers and users of real-time data, configuring instances of CHORDS to engage with them with the data sources they currently have. “We have all these sensors out in the field in other countries [such as Kucera’s 3D printed weather stations] and now we’re exploring how CHORDS can be used in new ways beyond just by the instrument teams using them to ensure data quality or their particular research,” Daniels said. “We want to broaden the access to these streams.”
Daniels would welcome input from scientists about how such access to real-time data that CHORDS provides could assist their own research. “It’s hard for us to really imagine all of the potential uses of these data. Most importantly, we want scientists to know that these live streams exist so they might consider how best to use them to improve their science or experiments.”
There are a few challenges the CHORDS team is addressing to make their data more attractive to the modeling community: 1) the CHORDS data may be too fine of a resolution for the models and 2) there is a need for data quality verification before these data are ingested, and 3) CHORDS would need to adapt to standards or APIs that these communities have built over the years.
Martin believes there are virtually endless number of communities that CHORDS could help: urban hydrology, forest fire watershed science, tectonics, local weather, third world government weather services, to name a few. “I’d like to see CHORDS connect with university Earth Science departments that are doing field research. And I’d like to see CHORDS feeding the geophysical “Big Data” players; i.e. the organizations which are consolidating, mining, archiving, integrating, and distributing Earth Science data, such as ESIP members.”
Communities Outside NSF?
CHORDS’ usability could expand beyond the NSF geoscience communities, though they are the primary users of CHORDS. Efforts so far have been to ensure that CHORDS covers the diversity of real-time data that are coming from the NSF/GEO communities as best it can.
However, the difficulties inherent in fully utilizing real-time data streams is not limited to the academic geosciences community. Other research communities also suffer from a lack of general purpose infrastructure for supporting arbitrary field measurements. Martin stated, “individual investigators have to spin up their own instrumentation, and data communications and real-time distribution are usually not even on their radar. Just getting instrumentation together is a big challenge.”
Though NSF’s geosciences research needs take priority, CHORDS can easily serve a much broader community, including other agencies, or even commercial and non-profit research organizations. “NOAA, DOE, NASA, Army, Air Force all have extensive earth science research programs with field observations, yet are often just as hamstrung by lack of real-time technology, and there are plenty of independent small research labs that could benefit as well,” said Martin.
Contact the CHORDS team or visit the CHORDS Website if you are interested in collaboration or would like to see a demo.
CHORDS PIs: Mike Daniels, National Center for Atmospheric Research (NCAR); Branko Kerkez, University of Michigan; V. Chandrasekar, Colorado State University; D. Sarah Stamps, Virginia Tech; Sara Graves, University of Alabama/Huntsville.
CHORDS Team: Charlie Martin, Senior Architect, NCAR; Mike Dye, NCAR; Josh Jones, Virginia Tech; Ryan Gooch, Colorado State University; Matt Bartos, University of Michigan; Casey Calamaio, University of Alabama/Huntsville; Ken Keiser, University of Alabama/Huntsville.