Modern AI tools have made code cheap to produce. Trust in the results
has not become cheap, and in science and critical systems it is essential. Without
trust, the speed that AI offers is not an advantage.

I lead a small team building the Data Center for the CCAT Observatory,
including the complete data life cycle after observations, from the
operations database and archive through the analysis workflows,
HPC infrastructure and data access frontend. We were early adopters of
AI tools and moved from consulting chat systems about implementation
details, to discussing architecture, to handing coding work completely
over to agentic AI. With a team size of two FTEs, this let us build more
of what I had envisioned for the data center, earlier and to a higher
standard. One example was a rewrite of our calibration software into a
working, parallelized Rust implementation in one week, with parity
against its predecessor and a substantial speedup. But our team was
skeptical this new code really calibrates the data reliably. Before
agentic AI tools the slow development and frequent discussions built
trust in the code, for both the developer and the team. Now, a fast
rewrite happens on one person's computer, and without precautions leaves
colleagues behind, and oneself with a tool of uncertain behavior on an
unfamiliar code base.

I will present the workflow I built for us to reach that trust without
giving up development speed. Its purpose is to keep humans and agents
working from the same understanding. We let the AI agents use
programming languages and frameworks that we know well. Where exact
results matter we create deterministic testing harnesses that we can
audit. Because AI sessions are ephemeral, I designed the workflow so
that machine-readable definitions force each session to leave clear
artifacts in well defined places: implementation plans, decision
records, and documentation. These artifacts inform and onboard
colleagues, give the next agent its context, and let us later understand
what we built and why. However, we noted that artifacts that satisfy the
agent are often unreadable for humans. We therefore invest in shaping
their language and aligning on a shared, defined vocabulary, so that the
documents can be understood without the context of the sessions or
native-level English. I will show what has worked for us, where it has
not, and what we are still doing about it.

Looking ahead, I am unsure where agentic AI will leave us as developers
and scientists. The calibration rewrite was fast, but only because I understood the old
software well after working with it for 10 years and we had a test suite. Work has been
offloaded before, for example we trusted libraries to do complex jobs,
however this is the first time that judgement can seemingly be offloaded
to AI. Before, this judgement could only be built by study, failure and
discussion. This leads to one central question I have: how does the next
generation acquire that judgement and experience if they never have to
earn it first?  Or do we hand judgement over altogether, willingly or not?

I am convinced that using AI agents for software engineering can improve
the science, and the support of it, tremendously, but only if we keep
people and our own judgement in the loop.