ARCHITECTING EVENT-DRIVEN DATA PIPELINES FOR REAL-TIME SUPPLY CHAIN DECISIONING

Hariprakash Pasumarthi

doi:10.15662/2bx02q53

Authors

Hariprakash Pasumarthi Senior Application Dev Analyst, BJS Wholesale Club, United States of America Author

DOI:

https://doi.org/10.15662/2bx02q53

Keywords:

Event-Driven Architecture, Real-Time Data Pipelines, Supply Chain Analytics, Stream Processing, , Distributed Messaging Systems, Real-Time Decisioning, Data Streaming Platforms, Operational Intelligence, Data Lakehouse, Microservices Architecture

Abstract

Modern supply chains operate in highly dynamic environments where disruptions, fluctuating demand, and global logistics complexities require organizations to make decisions in near real time. Traditional batch-oriented data processing architectures often introduce latency that limits the ability of enterprises to respond quickly to operational events such as shipment delays, inventory shortages, or supplier disruptions. Event-driven data pipeline architectures provide a scalable and responsive solution for enabling continuous data flow and real-time analytics across distributed systems

This paper presents a generalized architectural framework for designing event-driven data pipelines that support real-time supply chain decisioning. The study examines the integration of streaming platforms, distributed messaging systems, microservices, and scalable data processing frameworks to build resilient and low-latency pipelines. It explores how event streams generated from logistics platforms, warehouse management systems, IoT sensors, and enterprise resource planning (ERP) systems can be processed and analyzed in real time to support operational intelligence

The paper further discusses architectural considerations such as event schema management, fault tolerance, data consistency, scalability, and security in event-driven ecosystems. A conceptual reference architecture is proposed to demonstrate how event ingestion, stream processing, and analytics layers interact to enable proactive decision-making. Additionally, the research highlights the role of modern data lakehouses, real-time dashboards, and machine learning models in transforming event streams into actionable supply chain insights

The findings suggest that event-driven data pipelines significantly improve supply chain responsiveness, reduce operational delays, and enhance predictive decision-making capabilities. By adopting event-driven architectures, organizations can achieve higher levels of supply chain visibility, resilience, and operational agility in increasingly complex global logistics networks

References

[1] H. Isah, T. Abughofa, S. Mahfuz, D. Ajerla, F. Zulkernine, and S. Khan, "A survey of distributed data stream processing frameworks," IEEE Access, vol. 7, pp. 154300-154316, 2019.

[2] W. Z. Khan, E. Ahmed, S. Hakak, I. Yaqoob, and A. Ahmed, "Edge computing: A survey," Future Generation Computer Systems, vol. 97, pp. 219-235, 2019.

[3] S. Sankaranarayanan, M. Salehe, and R. Buyya, "Data flow and distributed deep neural network-based low-latency IoT-edge computation model for big data environments," Engineering Applications of Artificial Intelligence, vol. 94, 2020.

[4] E. Mehmood et al., "Challenges and solutions for processing real-time big data streams: A systematic literature review," IEEE Internet of Things Journal, vol. 7, no. 9, pp. 8132-8147, 2020.

[5] Y. Yang, J. Xu, and C. Zhang, "Big data-driven edge-cloud collaboration architecture for cloud manufacturing," IEEE Access, vol. 8, pp. 45938-45950, 2020.

[6] H. Hernandez, B. Xiao, and V. Tudor, "Enabling intelligent data pipelines for IoT-based application systems," in Proceedings of the IEEE International Conference on Pervasive Computing and Communications, 2020.

[7] S. P. Veluru, "Streaming data pipelines for AI at the edge: Architecting for real-time intelligence," International Journal of Artificial Intelligence, Data Science and Machine Learning, vol. 3, no. 2, pp. 56-71, 2021.

[8] S. Kumar Devarasetty, "Integrating AI and data engineering in IoT ecosystems: Streaming data management for smart devices," ComputerTech Journal, vol. 5, no. 2, pp. 61-72, 2021.

[9] S. Singu, "Designing scalable data engineering pipelines using cloud-based distributed processing frameworks," ESP Journal of Engineering and Technology Advancements, vol. 1, no. 2, pp. 176-187, 2021.