Maksym


Session

11-02
17:30
60min
Inter-operable Data and Compute for Astronomical Research
verbenajl, Christof Buchbender, Maksym

Background

Astronomical research increasingly requires access to datasets from multiple observatories and substantial computational resources. Among the challenges we face, two related problems particularly limit efficient use of available infrastructure. First, data often remains scattered across individual archives with limited cross-center access. While IVOA standards and FAIR principles provide a great foundation, researchers still spend significant effort downloading, organizing, and combining datasets manually.

Modern science platforms like that of Rubin Observatory show promising approaches. Initiatives like PUNCH and NFDI in Germany are developing access layers and federation approaches. New astronomical data centers are being established, such as the one for the CCAT observatory in Cologne or at the German Center for Astrophysics (DZA). However, we would benefit from community agreement and blueprints on practical implementation at the data center level. For example, building CCAT's data center in Cologne, we find ourselves collecting lessons learned and advice from many people rather
than following established blueprints.

Second, HPC resources are fragmented across institutions, often with bureaucratic access barriers. Taking our experience of developing the CCAT Data Center in Cologne as an example again: we face a change in policies of the funding agencies, that now require community use for large computing infrastructure, impeding us from requesting funding for local HPC resources for our data center. This shows that our specific use case is not well represented in these policies, not least because using external HPC resources involves complex user access procedures, software deployment difficulties, and priority conflicts that make seamless resource sharing impractical or at least very burdensome.

Objectives

This BOF will examine these inter-operability challenges from both perspectives and identify practical next steps. We distinguish this from IVOA efforts by focusing on operational implementation and resource sharing policies rather than technical protocols. The goal is to understand what must be implemented at the core level when designing data centers and science platforms to enable easier, less bureaucratic cooperation between centers and institutions.

Format

We propose splitting into two working groups - one examining data inter-operability between archives and platforms, the other exploring compute resource sharing across HPC centers. Each group will identify existing solutions, ongoing initiatives, and priority areas for development. We can also explore differences in approaches between countries and communities. Groups will reconvene to share findings and develop a joint summary document outlining concrete recommendations for the community.

Expected Outcomes

  • Overview of existing solutions and ongoing initiatives for data and compute inter-operability
  • Differences in approaches between countries and communities
  • Actions needed to improve inter-operability
  • Successful models and technical requirements
  • Policy considerations for enabling better resource sharing across astronomical centers
Building and operating science platforms and workflows in the Petabyte Era
Seminar Room 4