Consider maintaining and developing an e-commerce platform that processes millions of transactions every minute, generating large amounts of telemetry data, including metrics, logs and traces across multiple microservices. When critical incidents occur, on-call engineers face the daunting task of sifting through an ocean of data to unravel relevant signals and insights. This is equivalent to searching for a needle in a haystack. This makes observability a source of frustration rather than insight. To alleviate this major pain point, I started exploring a solution to utilize the Model Context Protocol (MCP) to add context and draw inferences from the logs and distributed traces. In this article, I’ll outline my experience building an AI-powered observability platform, explain the system architecture and share actionable insights learned along the way.\n\nIn modern software systems, observability is not a luxury; it’s a basic necessity. The ability to measure and understand system behavior is foundational to reliability, performance and user trust. As the saying goes, "What you cannot measure, you cannot improve."\n\nYet, achieving observability in today’s cloud-native, microservice-based architectures is more difficult than ever. A single user request may traverse dozens of microservices, each emitting logs, metrics and traces. The result is an abundance of telemetry data: tens of terabytes of logs per day, tens of millions of metric data points and pre-aggregates, millions of distributed traces, and thousands of correlation IDs generated every minute. The challenge is not only the data volume, but the data fragmentation. According to New Relic’s 2023 Observability Forecast Report, 50% of organizations report siloed telemetry data, with only 33% achieving a unified view across metrics, logs and traces.\n\nLogs tell one part of the story, metrics another, traces yet another. Without a consistent thread of context, engineers are forced into manual correlation, relying on intuition, tribal knowledge and tedious detective work during incidents.\n\nBecause of this complexity, I started to wonder: How can AI help us get past fragmented data and offer comprehensive, useful insights? Specifically, can we make telemetry data intrinsically more meaningful and accessible for both humans and machines using a structured protocol such as MCP? This project’s foundation was shaped by that central question.\n\nAnthropic defines MCP as an open standard that allows developers to create a secure two-way connection between data sources and AI tools. This structured data pipeline includes: Contextual ETL for AI: Standardizing context extraction from multiple data sources. Structured query interface: Allows AI queries to access data layers that are transparent and easily understandable. Semantic data enrichment: Embeds meaningful context directly into telemetry signals. This has the potential to shift platform observability away from reactive problem solving and toward proactive insights.\n\nBefore diving into the implementation details, let’s walk through the system architecture.\n\nIn the first layer, we develop the contextual telemetry data by embedding standardized metadata in the telemetry signals, such as distributed traces, logs and metrics. Then, in the second layer, enriched data is fed into the MCP server to index, add structure and provide client access to context-enriched data using APIs. Finally, the AI-driven analysis engine utilizes the structured and enriched telemetry data for anomaly detection, correlation and root-cause analysis to troubleshoot application issues. This layered design ensures that AI and engineering teams receive context-driven, actionable insights from telemetry data.\n\nLet’s explore the actual implementation of our MCP-powered observability platform, focusing on the data flows and transformations at each step.\n\nFirst, we need to ensure our telemetry data contains enough context for meaningful analysis. The core insight is that data correlation needs to happen at creation time, not analysis time.\n\n# Generate correlation id\norder_id = f"order-{uuid.uuid4().hex[:8]}"\nrequest_id = f"req-{uuid.uuid4().hex[:8]}"\n\n# Initialize context dictionary that will be applied\ncontext = { \n  "user_id": user_id, \n  "order_id": order_id, \n  "request_id": request_id, \n  "cart_item_count": len(cart_items), \n  "payment_method": payment_method, \n  "service_name": "checkout", \n  "service_version": "v1.0.0" \n}\n\n# Start OTel trace with the same context\nwith tracer.start_as_current_span( "process_checkout", attributes={k: str(v) for k, v in context.items()} ) as checkout_span:\n\n# Logging using same context\nlogger.info(f"Starting checkout process", extra={"context": json.dumps(context)})\n\n# Context Propagation\nwith tracer.start_as_current_span("process_payment"): \n# Process payment logic…\nlogger.info("Payment processed", extra={"context": json.dumps(context)})\n\nThis approach ensures that every telemetry signal (logs, metrics, traces) contains the same core contextual data, solving the correlation problem at the source.\n\nNext, I built an MCP server that transforms raw telemetry into a queryable API. The core data operations here involve the following:\n\nIndexing: Creating efficient lookups across contextual fields\nFiltering: Selecting relevant subsets of telemetry data\nAggregation: Computing statistical measures across time windows\n\n@app.post("/mcp/logs", response_model=List[Log])\ndef query_logs(query: LogQuery): """Query logs with specific filters"""\nresults = LOG_DB.copy()\n\n# Apply contextual filters\nif query.request_id:\n  results = [log for log in results if log["context"].get("request_id") == query.request_id]\nif query.user_id:\n  results = [log for log in results if log["context"].get("user_id") == query.user_id]\n\n# Apply time-based filters\nif query.time_range:\n  start_time = datetime.fromisoformat(query.time_range["start"])\n  end_time = datetime.fromisoformat(query.time_range["end"])\n  results = [log for log in results if start_time <= datetime.fromisoformat(log["timestamp"]) <= end_time]\n\n# Sort by timestamp\nresults = sorted(results, key=lambda x: x["timestamp"], reverse=True)\nreturn results[:query.limit] if query.limit else results\n\nThis layer transforms our telemetry from an unstructured data lake into a structured, query-optimized interface that an AI system can efficiently navigate.\n\nThe final layer is an AI component that consumes data through the MCP interface, performing:\n\nMulti-dimensional analysis: Correlating signals across logs, metrics and traces.\nAnomaly detection: Identifying statistical deviations from normal patterns.\nRoot cause determination: Using contextual clues to isolate likely sources of issues.\n\ndef analyze_incident(self, request_id=None, user_id=None, timeframe_minutes=30): """Analyze telemetry data to determine root cause and recommendations."""\n\n# Define analysis time window\nend_time = datetime.now()\nstart_time = end_time – timedelta(minutes=timeframe_minutes)\ntime_range = {"start": start_time.isoformat(), "end": end_time.isoformat()}\n\n# Fetch relevant telemetry based on context\nlogs = self.fetch_logs(request_id=request_id, user_id=user_id, time_range=time_range)\n\n# Extract services mentioned in logs for targeted metric analysis\nservices = set(log.get("service", "unknown") for log in logs)\n\n# Get metrics for those services\nmetrics_by_service = {}\nfor service in services:\n  for metric_name in ["latency", "error_rate", "throughput"]:\n    metric_data = self.fetch_metrics(service, metric_name, time_range)\n\n# Calculate statistical properties\nvalues = [point["value"] for point in metric_data["data_points"]]\nmetrics_by_service[f"{service}.{metric_name}"] = { \n  "mean": statistics.mean(values) if values else 0, \n  "median": statistics.median(values) if values else 0, \n  "stdev": statistics.stdev(values) if len(values) > 1 else 0, \n  "min": min(values) if values else 0, \n  "max": max(values) if values else 0 \n}\n\n# Identify anomalies using z-score\nanomalies = []\nfor metric_name, stats in metrics_by_service.items():\n  if stats["stdev"] > 0: # Avoid division by zero\n    z_score = (stats["max"] – stats["mean"]) / stats["stdev"]\n    if z_score > 2: # More than 2 standard deviations\n      anomalies.append({ \n        "metric": metric_name, \n        "z_score": z_score, \n        "severity": "high" if z_score > 3 else "medium" \n      })\nreturn { \n  "summary": ai_summary, \n  "anomalies": anomalies, \n  "impacted_services": list(services), \n  "recommendation": ai_recommendation \n}\n\nThe potential benefits include:\n\nFaster anomaly detection, resulting in reduced minimum time to detect (MTTD) and minimum time to resolve (MTTR).\nEasier identification of root causes for issues.\nLess noise and fewer unactionable alerts, thus reducing alert fatigue and improving developer productivity.\nFewer interruptions and context switches during incident resolution, resulting in improved operational efficiency for an engineering team.\n\nHere are some key insights from this project that will help teams with their observability strategy:\n\nContextual metadata should be embedded early in the telemetry generation process to facilitate downstream correlation.\nStructured data interfaces create API-driven, structured query layers to make telemetry more accessible.\nContext-aware AI focuses analysis on context-rich data to improve accuracy and relevance.\nContext enrichment and AI methods should be refined on a regular basis using practical operational feedback.\n\nThe amalgamation of structured data pipelines and AI holds enormous promise for observability. We can transform vast telemetry data into actionable insights by leveraging structured protocols such as MCP and AI-driven analyses, resulting in proactive rather than reactive systems. Lumigo identifies three pillars of observability — logs, metrics, and traces — which are essential. Without integration, engineers are forced to manually correlate disparate data sources, slowing incident response.\n\nHow we generate telemetry requires structural changes as well as analytical techniques to extract meaning.

From terabytes to insights: Real-world AI obervability architecture

The article details the development of an AI-powered observability platform designed to address the challenges of fragmented telemetry data in modern software systems. By leveraging the Model Context Protocol (MCP), the platform embeds contextual metadata into logs, metrics, and traces at the source, enabling structured, queryable data that supports AI-driven anomaly detection, root-cause analysis, and proactive insights. The system architecture includes three layers: contextual telemetry generation, MCP-based indexing and querying, and an AI analysis engine that performs multi-dimensional analysis, detects anomalies using z-scores, and provides actionable recommendations. Key benefits include reduced time to detect and resolve incidents, fewer false alerts, and improved engineering productivity through contextualized insights.