Astronomy is entering an era in which several major facilities are simultaneously operating, or standing up, science platforms designed to deliver petabyte-scale data and computing to their communities. As missions and observatories move from data collection to data delivery at scale, they are independently converging on similar architectural choices — and independently running into similar operational challenges — around federated access, compute-to-data, authentication, and long-term sustainability. This BoF is proposed by confirmed panelists from Euclid/ESA Datalabs, Vera Rubin Observatory, the SKA Regional Centre Network (SRCNet), Roman Telescope, CADC/CANFAR, NASA/Fornax and LOFAR. Other platforms are invited to join the discussion to compare architectures, operational lessons, and user-facing services for large-scale science platforms and workflows.

With Euclid's Data Release 1 (DR1), a full year of Vera Rubin Observatory operations, the launch of the Roman Space Telescope, and SKA Low's science verification getting underway next year, the timing is ideal to take stock of what's working, what isn't, and where the community could converge — on topics such as federated data access, compute-to-data models, interoperability standards, and the balance between bespoke platform development and shared infrastructure. Panellists will open the session with a short position statement on how their platform is architected and operated today, before the floor opens to a moderated discussion aimed at surfacing common challenges and concrete opportunities for collaboration across observatories.

For a few years now, some astronomical data centers have observed an increasing number of incoming requests. This usually indicates that a service is becoming more popular and useful. However, the number and rate of these requests is so high that the services are becoming increasingly overwhelmed, ending up by a saturation and complete unavailability. Worryingly, this trend is speeding up. Consequently, services are becoming unavailable more often and more quickly than before.

The recently observed growth in activity actually hides a different user profile. Until now, humans and indexing crawlers were the only users of our services. Now, there is a new category of users: AI bots. Unlike the other users, they are much more aggressive. There are two main reasons for this. First, they rely on cloud infrastructures. This allows them to massively parallelize the web scrapping operations and use an enormous number of different IP addresses, making them impossible to block with the usual methods. Second, they do not adhere to any best practices, such as the well-known robots.txt, which indexing crawlers (search engines) read and follow.

Through this BoF, CDS (France), ESAC (Spain), and Data Central (Australia) aim to discuss beyond the negative impact of this recent practice on their infrastructures and explore strategies to maintain the high availability and quality of their services while allowing AI to reasonably interact with them. Everyone is welcome to join the discussion and share experiences, solutions (e.g. mirrors, proxy, authentication, ban with well-identified and shared IP ranges), and, more generally, ideas.