Observability for AI Systems: Tracing, Drift, and SLAs
DOI:
https://doi.org/10.15662/IJRAI.2025.0802002Keywords:
AI Observability, Model Drift, Data Drift, Concept Drift, Distributed Tracing, Service-Level Agreements (SLAs), Statistical Drift Detection, Uncertainty Estimation, MLOps, Model MonitoringAbstract
Modern AI systems—particularly those driving critical applications—must operate with transparency, reliability, and performance consistency. Observability for AI systems extends beyond infrastructure metrics, encompassing traceability of inference workflows, detection of concept and data drift, and adherence to Service-Level Agreements (SLAs). This paper presents a framework that integrates logging, tracing, drift detection, and SLA enforcement to maintain the trustworthiness of deployed AI systems. We examine approaches for distributed tracing of model pipelines, allowing root-cause analysis across data ingestion, feature processing, and inference stages. Drift detection methodologies, such as statistical tests (e.g., Kolmogorov– Smirnov, PSI) and uncertainty estimation (e.g., bootstrapped intervals with explainability tools), enable proactive identification of model degradation. We also address SLA-oriented observability, focusing on meeting latency, throughput, and accuracy guarantees, employing operational dashboards and alerting mechanisms. Our research methodology combines system design principles, simulations with varying drift scenarios, and evaluations using real-world deployment examples. Performance metrics include detection latency, false positive rates, SLA compliance, and trace-query efficiency. Results demonstrate that federated tracing techniques coupled with statistical drift tests can detect drift within seconds, allow rapid retraining triggers, and proactively prevent SLA violations. However, challenges remain in data volume, alert fatigue, and early detection accuracy. In conclusion, AI-aware observability is essential for maintaining reliability, transparency, and business alignment in AI systems. We outline future directions involving integrated model governance (ModelOps), causal tracing, workloadefficient drift detection, and unified observability pipelines that bridge MLOps and traditional system observability.
References
1. Nigenda, D., Karnin, Z., Zafar, M. B., et al. (2021). Amazon SageMaker Model Monitor: A System for Real-Time Insights into Deployed Machine Learning Models. arXiv preprint arXiv
2. Eck, B., Kabakci-Zorlu, D., Chen, Y., et al. (2022). A monitoring framework for deployed machine learning models with supply chain examples. arXiv preprint arXiv
3. Mougan, C., & Nielsen, D. S. (2022). Monitoring Model Deterioration with Explainable Uncertainty Estimation via Non-parametric Bootstrap. arXiv preprint arXiv
4. Bhaskhar, N., Rubin, D. L., & Lee-Messer, C. (2022). TRUST-LAPSE: An Explainable and Actionable Mistrust Scoring Framework for Model Monitoring. arXiv preprint arXiv
5. Fiddler AI documentation. ML Observability definitions and practices. Fiddler AI
6. Coralogix: AI Observability: Key Components & Challenges. Coralogix
7. General best practices in drift detection and MLOps ecosystem. ResearchGate
8. Microsoft AzureML Observability – scalable drift detection. TECHCOMMUNITY.MICROSOFT.COM
9. Foundational concept drift overview. Wikipedia
10. Definition and scope of ModelOps.
