Governing Cloud Data Pipelines with Agentic AI
Cloud data pipelines increasingly operate under dynamic workloads, evolving schemas, cost constraints, and strict governance requirements. Despite advances in cloud-native orchestration frameworks, most production pipelines rely on static configurations and reactive operational practices, resulting in prolonged recovery times, inefficient resource utilization, and high manual overhead. This paper presents Agentic Cloud Data Engineering, a policy-aware control architecture that integrates bounded AI agents into the governance and control plane of cloud data pipelines. In Agentic Cloud Data Engineering platform, specialized agents analyze pipeline telemetry and metadata, reason over declarative cost and compliance policies, and propose constrained operational actions such as adaptive resource reconfiguration, schema reconciliation, and automated failure recovery. All agent actions are validated against governance policies to ensure predictable and auditable behavior. We evaluate Agentic Cloud Data Engineering platform using representative batch and streaming analytics workloads constructed from public enterprise-style datasets. Experimental results show that Agentic Cloud Data Engineering platform reduces mean pipeline recovery time by up to 45%, lowers operational cost by approximately 25%, and decreases manual intervention events by over 70% compared to static orchestration, while maintaining data freshness and policy compliance. These results demonstrate that policy-bounded agentic control provides an effective and practical approach for governing cloud data pipelines in enterprise environments.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
SAGA: A Security Architecture for Governing AI Agentic Systems
Large Language Model (LLM)-based agents increasingly interact, collaborate, and delegate tasks to one another autonomously with minimal human interaction. Industry guidelines for agentic system governance emphasize the n…
Large Language ModelPDEFlow: Autonomous Agentic PDE Pipelines for Neural Operator Learning and Solver-Free Inference
We present PDEFlow, an autonomous agentic framework that turns user-level ODE and PDE descriptions into solver-backed neural-operator pipelines. The workflow links problem specification, data generation, operator trainin…
Thoughts without Thinking: Reconsidering the Explanatory Value of Chain-of-Thought Reasoning in LLMs through Agentic Pipelines
Agentic pipelines present novel challenges and opportunities for human-centered explainability. The HCXAI community is still grappling with how best to make the inner workings of LLMs transparent in actionable ways. Agen…
QoS-Aware Token Scheduling and Private Data Valuation for Multi-Modal Agentic Networks
In agentic systems, human-generated data records anchor the value of AI services. Yet cloud compute pipelines centralize processing on remote servers. Data centralization reduces personal data sovereignty and may potenti…
Image ReconstructionSee, Hypothesize, Validate: Multimodal Agentic Framework for Discovering Governing PDEs
Discovering governing partial differential equations (PDEs) from observational data remains a core challenge across the sciences. Existing sparse-regression, symbolic-regression, and LLM-based approaches can be constrain…