Enterprise AI agents today face a fundamental timing problem: They can't easily act on critical business events because they aren't always aware of them in real-time.
The challenge is infrastructure. Most enterprise data lives in databases fed by extract-transform-load (ETL) jobs that run hourly or daily — ultimately too slow for agents that must respond in real time.
One potential way to tackle that challenge is to have agents directly interface with streaming data systems. Among the primary approaches in use today are the open source Apache Kafka and Apache Flink technologies. There are multiple commercial implementations based on those technologies, too, Confluent, which is led by the original creators behind Kafka, being one of them.
Today, Confluent is introducing a real-time context engine designed to solve this latency problem. The technology builds on Apache Kafka, the distributed event streaming platform that captures data as events occur, and open-source Apache Flink, the stream processing engine that transforms those events in real time.
The company is also releasing an open-source framework, Flink Agents, developed in collaboration with Alibaba Cloud, LinkedIn and Ververica. The framework brings event-driven AI agent capabilities directly to Apache Flink, allowing organizations to build agents that monitor data streams and trigger automatically based on conditions without committing to Confluent's managed platform.
"Today, most enterprise AI systems can't respond automatically to important events in a business without someone prompting them first," Sean Falconer, Confluent's head of AI, told VentureBeat. "This leads to lost revenue, unhappy customers or added risk when a payment fails or a network malfunctions."
The significance extends beyond Confluent's specific products. The industry is recognizing that AI agents require different data infrastructure than traditional applications. Agents don't just retrieve information when asked. They need to observe continuous streams of business events and act automatically when conditions warrant. This requires streaming architecture, not batch pipelines.
Batch versus streaming: Why RAG alone isn't enough
To understand the problem, it's important to distinguish between the different approaches to moving data through enterprise systems and how they can connect to agentic AI.
In batch processing, data accumulates in source systems until a scheduled job runs. That job extracts the data, transforms it and loads it into a target database or data warehouse. This might occur hourly, daily or even weekly. The approach works well for analytical workloads, but it creates latency between when something happens in the business and when systems can act on it.
Data streaming inverts this model. Instead of waiting for scheduled jobs, streaming platforms like Apache Kafka capture events as they occur. Each database update, user action, transaction or sensor reading becomes an event published to a stream. Apache Flink then processes these streams to join, filter and aggregate data in real time. The result is processed data that reflects the current state of the business, updating continuously as new events arrive.
This distinction becomes critical when you consider what kinds of context AI agents actually need. Much of the current enterprise AI discussion focuses on retrieval-augmented generation (RAG), which handles semantic search over knowledge bases to find relevant documentation, policies or historical information. RAG works well for questions like "What's our refund policy?" where the answer exists in static documents.
But many enterprise use cases require what Falconer calls "structural context" — precise, up-to-date information from multiple operational systems stitched together in real time. Consider a job recommendation agent that requires user profile data from the HR database, browsing behavior from the last hour, search queries from minutes ago and current open positions across multiple systems.
"The part that we're unlocking for businesses is the ability to essentially serve that structural context needed to deliver the freshest version," Falconer said.
The MCP connection problem: Stale data and fragmented context
The challenge isn't simply connecting AI to enterprise data. Model Context Protocol (MCP), introduced by Anthropic earlier this year, already standardized how agents access data sources. The problem is what happens after the connection is made.
In most enterprise architectures today, AI agents connect via MCP to data lakes or warehouses fed by batch ETL pipelines. This creates two critical failures: The data is stale, reflecting yesterday's reality rather than current events, and it's fragmented across multiple systems, requiring significant preprocessing before an agent can reason about it effectively.
The alternative — putting MCP servers directly in front of operational databases and APIs — creates different problems. Those endpoints weren't designed for agent consumption, which can lead to high token costs as agents process excessive raw data and multiple inference loops as they try to make sense of unstructured responses.
"Enterprises have the data, but it's often stale, fragmented or locked in formats that AI can't use effectively," Falconer explained. "The real-time context engine solves this by unifying data processing, reprocessing and serving, turning continuous data streams into live context for smarter, faster and more reliable AI decisions."
The technical architecture: Three layers for real-time agent context
Confluent's platform encompasses three elements that work together or adopted separately.
The real-time context engine is the managed data infrastructure layer on Confluent Cloud. Connectors pull data into Kafka topics as events occur. Flink jobs process these streams into "derived datasets" — materialized views joining historical and real-time signals. For customer support, this might combine account history, current session behavior and inventory status into one unified context object. The Engine exposes this through a managed MCP server.
Streaming agents is Confluent's proprietary framework for building AI agents that run natively on Flink. These agents monitor data streams and trigger automatically based on conditions — they don't wait for prompts. The framework includes simplified agent definitions, built-in observability and native Claude integration from Anthropic. It's available in open preview on Confluent's platform.
Flink Agents is the open-source framework developed with Alibaba Cloud, LinkedIn and Ververica. It brings event-driven agent capabilities directly to Apache Flink, allowing organizations to build streaming agents without committing to Confluent's managed platform. They handle operational complexity themselves but avoid vendor lock-in.
Competition heats up for agent-ready data infrastructure
Confluent isn't alone in recognizing that AI agents need different data infrastructure.
The day before Confluent's announcement, rival Redpanda introduced its own Agentic Data Plane — combining streaming, SQL and governance specifically for AI agents. Redpanda acquired Oxla's distributed SQL engine to give agents standard SQL endpoints for querying data in motion or at rest. The platform emphasizes MCP-aware connectivity, full observability of agent interactions and what it calls "agentic access control" with fine-grained, short-lived tokens.
The architectural approaches differ. Confluent emphasizes stream processing with Flink to create derived datasets optimized for agents. Redpanda emphasizes federated SQL querying across disparate sources. Both recognize agents need real-time context with governance and observability.
Beyond direct streaming competitors, Databricks and Snowflake are fundamentally analytical platforms adding streaming capabilities. Their strength is complex queries over large datasets, with streaming as an enhancement. Confluent and Redpanda invert this: Streaming is the foundation, with analytical and AI workloads built on top of data in motion.
How streaming context works in practice
Among the users of Confluent's system is transportation vendor Busie. The company is building a modern operating system for charter bus companies that helps them manage quotes, trips, payments and drivers in real time.
"Data streaming is what makes that possible," Louis Bookoff, Busie co-founder and CEO told VentureBeat. "Using Confluent, we move data instantly between different parts of our system instead of waiting for overnight updates or batch reports. That keeps everything in sync and helps us ship new features faster.
Bookoff noted that the same foundation is what will make gen AI valuable for his customers.
"In our case, every action like a quote sent or a driver assigned becomes an event that streams through the system immediately," Bookoff said. "That live feed of information is what will let our AI tools respond in real time with low latency rather than just summarize what already happened."
The challenge, however, is how to understand context. When thousands of live events flow through the system every minute, AI models need relevant, accurate data without getting overwhelmed.
"If the data isn't grounded in what is happening in the real world, AI can easily make wrong assumptions and in turn take wrong actions," Bookoff said. "Stream processing solves that by continuously validating and reconciling live data against activity in Busie."
What this means for enterprise AI strategy
Streaming context architecture signals a fundamental shift in how AI agents consume enterprise data.
AI agents require continuous context that blends historical understanding with real-time awareness — they need to know what happened, what's happening and what might happen next, all at once.
For enterprises evaluating this approach, start by identifying use cases where data staleness breaks the agent. Fraud detection, anomaly investigation and real-time customer intervention fail with batch pipelines that refresh hourly or daily. If your agents need to act on events within seconds or minutes of them occurring, streaming context becomes necessary rather than optional.
"When you're building applications on top of foundation models, because they're inherently probabilistic, you use data and context to steer the model in a direction where you want to get some kind of outcome," Falconer said. "The better you can do that, the more reliable and better the outcome."
