For a few years now, some astronomical data centers have observed an increasing number of incoming requests. This usually indicates that a service is becoming more popular and useful. However, the number and rate of these requests is so high that the services are becoming increasingly overwhelmed, ending up by a saturation and complete unavailability. Worryingly, this trend is speeding up. Consequently, services are becoming unavailable more often and more quickly than before.

The recently observed growth in activity actually hides a different user profile. Until now, humans and indexing crawlers were the only users of our services. Now, there is a new category of users: AI bots. Unlike the other users, they are much more aggressive. There are two main reasons for this. First, they rely on cloud infrastructures. This allows them to massively parallelize the web scrapping operations and use an enormous number of different IP addresses, making them impossible to block with the usual methods. Second, they do not adhere to any best practices, such as the well-known robots.txt, which indexing crawlers (search engines) read and follow.

Through this BoF, CDS (France), ESAC (Spain), and Data Central (Australia) aim to discuss beyond the negative impact of this recent practice on their infrastructures and explore strategies to maintain the high availability and quality of their services while allowing AI to reasonably interact with them. Everyone is welcome to join the discussion and share experiences, solutions (e.g. mirrors, proxy, authentication, ban with well-identified and shared IP ranges), and, more generally, ideas.