What I Learned Shipping ML to Production (the Hard Way)

A model’s offline accuracy number is the least interesting fact about whether it’s going to work in production. I learned this the expensive way, watching a model with a great validation AUC slowly degrade over a few months while every dashboard that only tracked accuracy stayed quiet.

Notebook accuracy is a snapshot, production is a stream

Offline evaluation answers “how good was this model on this fixed dataset.” Production asks a different question every day: “how good is this model on data that didn’t exist when it was trained.” Those are not the same question, and the gap between them is exactly where drift lives.

What actually caught the problem

Not the accuracy dashboard. What caught it was a much simpler set of checks:

• Input distribution monitoring. Tracking the distribution of each feature against its training-time distribution and alerting on drift, independent of whether we had ground-truth labels yet.
• Prediction distribution monitoring. If the model’s output distribution shifts meaningfully with no corresponding shift in inputs, something in the serving path is broken, and it’s worth knowing that immediately rather than after the next label-delayed accuracy report.
• Label-delay-aware evaluation. For anything where ground truth arrives weeks later, building an explicit “shadow accuracy” pipeline that backfills evaluation as labels arrive, instead of assuming no news is good news.

The actual lesson

Production reliability isn’t a property of the model — it’s a property of the monitoring and operational discipline wrapped around the model. A mediocre model with excellent observability will outlast a brilliant model with none, because you’ll actually know when the brilliant one starts quietly failing.