How to Maximize Observability in Systems

🚀 Building Observable Infrastructure: Why Automation + Instrumentation = Production Excellence and Customer Success After building our platform's infrastructure and application automation pipeline, I wanted to share why combining Infrastructure as Code with deep observability isn't optional—it's foundational as shown in screenshots implemented on Google Cloud. The Challenge: Manual infrastructure provisioning and application onboarding creates consistency gaps, slow deployments, and zero visibility into what's actually happening in production. When something breaks at 3 AM, you're debugging blind. The Solution: Modular Terraform + OpenTelemetry from Day One with our approach centered on three principles: 1️⃣ Modular, Well architected Terraform modules as reusable building blocks. Each service (Argo CD, Rollouts, Sonar, Tempo) gets its own module. This means: 1. Consistent deployment patterns across environments 2. Version-controlled infrastructure state 3. Self-service onboarding for dev teams 2️⃣ OpenTelemetry Instrumentation of every application during onboarding as a minimum specification. This allows capturing: 1. Distributed traces across our apps / services / nodes (Graph) 2. Golden signals (latency, traffic, errors, saturation) 3. Custom business metrics that matter. 3️⃣ Single Pane of Glass Observability Our Grafana dashboards aggregate everything: service health, trace data, build pipelines, resource utilization. When an alert fires, we have context immediately—not 50 tabs of different tools. Real Impact: → Application onboarding dropped from days to hours → Mean time to resolution decreased by 60%+ (actual trace data > guessing) → nfrastructure drift: eliminated through automated state management → Dev teams can self-service without waiting on platform engineering Key Learnings: → Modular Terraform requires discipline up front but pays dividends at scale. → OpenTelemetry context propagation consistent across your stack. → Dashboards should tell a story by organising by user journey. → Automation without observability is just faster failure. You need both. The Technical Stack: → Terraform for infrastructure provisioning → ArgoCD for GitOps-based deployments → OpenTelemetry for distributed tracing and metrics → Tempo for trace storage → Grafana for unified visualisation The screenshot shows our command center : → Active services → Full trace visibility → Automated deployments with comprehensive health monitoring. Bottom line: Modern platform engineering isn't about choosing between automation OR observability. It's about building systems where both are inherent to the architecture. When infrastructure is code and telemetry is built-in, you get reliability, velocity, and visibility in one package. Curious how others are approaching this? What's your observability strategy look like in automated environments? #DevOps #PlatformEngineering #Observability #InfrastructureAsCode #OpenTelemetry #SRE #CloudNative

LLM systems don’t fail silently. They fail invisibly. No trace, no metrics, no alerts - just wrong answers and confused users. That’s why we architected a complete observability pipeline in the Second Brain AI Assistant course. Powered by Opik from Comet, it covers two key layers: 𝟭. 𝗣𝗿𝗼𝗺𝗽𝘁 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 → Tracks full prompt traces (inputs, outputs, system prompts, latencies) → Visualizes chain execution flows and step-level timing → Captures metadata like model IDs, retrieval config, prompt templates, token count, and costs Latency metrics like: Time to First Token (TTFT) Tokens per Second (TPS) Total response time ...are logged and analyzed across stages (pre-gen, gen, post-gen). So when your agent misbehaves, you can see exactly where and why. 𝟮. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 → Runs automated tests on the agent’s responses → Uses LLM judges + custom heuristics (hallucination, relevance, structure) → Works offline (during dev) and post-deployment (on real prod samples) → Fully CI/CD-ready with performance alerts and eval dashboards It’s like integration testing, but for your RAG + agent stack. The best part? → You can compare multiple versions side-by-side → Run scheduled eval jobs on live data → Catch quality regressions before your users do This is Lesson 6 of the course (and it might be the most important one). Because if your system can’t measure itself, it can’t improve. 🔗 Full breakdown here: https://lnkd.in/dA465E_J

Buying an observability platform doesn't give you observability. Just like buying a gym membership doesn't make you fit. Tools matter, but observability is a system made up of people, processes, and instrumentation. It requires consistency, conventions, and collaboration across teams. Observability becomes a system when you have: 🔹 Instrumentation discipline: Services emit structured, meaningful telemetry and not whatever each developer prefers. 🔹 Semantic conventions: Attributes, span names, and error formats are consistent across services. 🔹 A reliable pipeline: OpenTelemetry Collectors route data predictably and safely. 🔹 Operational workflows: Engineers know how to investigate outages, not just where to click. 🔹 Ownership: Teams maintain what they instrument and review observability as part of delivery. Without these pieces, even the best tool becomes little more than a data sink. 🧩 Example: When Observability Fails as a Tool Imagine a company buys a premium observability platform. They hook up a few logs and metrics. Dashboards are created. Alerts are set. Then an incident happens. Engineers jump into dashboards and see CPU spikes but no correlated traces. They search logs, but every service logs differently. They pull up metrics, but have no context for which user flows are impacted. Everyone spends hours guessing. Why? Because they bought a tool, but never built a system. 🧩 Example: When Observability Works as a System Another team invests in: • Consistent OTel instrumentation across services • Shared semantic conventions • A unified collector pipeline • Playbooks for incident response • Regular observability reviews in sprint cycles When something breaks, engineers instantly see: • The failing service • The impacted user flows • The exact span where latency spikes began • Related logs with matching attributes • Recent deployments that touched that code path They don't just detect the issue, they understand it. That's observability as a system. 🎯 Bottom Line Observability isn't what you buy. It's what you build over time. Tools give you capabilities. Systems give you outcomes. 💬 How have you built observability beyond just tools in your organization? #Observability #OpenTelemetry #PlatformEngineering #SRE #O11yEngineering

I recently had the opportunity to work with a large financial services organization implementing OpenTelemetry across their distributed systems. The journey revealed some fascinating insights I wanted to share. When they first approached us, their observability strategy was fragmented – multiple monitoring tools, inconsistent instrumentation, and slow MTTR. Sound familiar? Their engineering teams were spending hours troubleshooting issues rather than building new features. They had plenty of data but struggled to extract meaningful insights. Here's what made their OpenTelemetry implementation particularly effective: 1️⃣ They started small but thought big. Rather than attempting a company-wide rollout, they began with one critical payment processing service, demonstrating value quickly before scaling. 2️⃣ They prioritized distributed tracing from day one. By focusing on end-to-end transaction flows, they gained visibility into previously hidden performance bottlenecks. One trace revealed a third-party API call causing sporadic 3-second delays. 3️⃣ They standardized on semantic conventions across teams. This seemingly small detail paid significant dividends. Consistent naming conventions for spans and attributes made correlating data substantially easier. 4️⃣ They integrated OpenTelemetry with Elasticsearch for powerful analytics. The ability to run complex queries across billions of spans helped identify patterns that would have otherwise gone unnoticed. The results? Mean time to detection dropped by 71%. Developer productivity increased as teams spent less time debugging and more time building. They could now confidently answer "what's happening in production right now?" Interestingly, their infrastructure costs decreased despite collecting more telemetry data. The unified approach eliminated redundant collection and storage systems. What impressed me most wasn't the technology itself, but how this organization approached the human elements of the implementation. They recognized that observability is as much about culture as it is about tools. Have you implemented OpenTelemetry in your organization? What unexpected challenges or benefits did you encounter? If you're still considering it, what's your biggest concern about making the transition? #OpenTelemetry #DistributedTracing #Observability #SiteReliabilityEngineering #DevOps

Are metrics an essential pillar of your Observability strategy? Or did you implement logging and call it a day? The Value of Metrics 💵 Many underestimate metrics (OTEL, StatsD, or Prometheus), but metrics add tremendous value by providing insights into your platform's health and operation. With metrics, you can view the system as a whole or drill down to a single instance; that kind of visibility is empowering. Being thoughtful about your collected metrics is the key to unlocking their value. What Metrics to Collect 🕵️♂️ 📊 Application Metrics These metrics provide insights into how your application is performing. Examples might be thread usage, garbage collection time, or heap space utilization. With application metrics, you can see low-level performance details. 💻 System Metrics Infrastructure performance visibility is just as important as application metrics. It is imperative to be able to answer questions about your I/O wait time, CPU utilization, number of network connections, etc., at any time, historically, and live. Applications only run well if the underlying infrastructure runs well; system metrics provide insights into your infrastructure. ⚙️ Application Events The events within your application are probably the second most valuable metrics to collect. These include the number of HTTP requests, database calls, scheduled task executions, etc. Seeing application events across an entire platform can provide some fantastic operational insights. But it’s essential to collect these metrics in the right way. Track the number of these events and their execution time, and categorize them using labels. With the right metrics, you should be able to see how long each database call took and what its purpose was. You should be able to see how many HTTP requests a specific endpoint received, how long it took to respond, and what response code was provided. All application events are essential and should be tracked. 💼 Business Events While you might be able to derive business events from application events, it is better to create specific metrics to track business events. When you create these metrics, ask yourself: What is the purpose of this application, and why do clients use it? What background events does my application perform that could impact business operations? What are the crucial aspects of my business events? Is it speed, number of requests, or success rate? Like application events, it’s essential to categorize business events appropriately. Use labels with your metrics to build more granularity in events. Know what clients are doing, what activities are being performed, why, and how. Combining them all 🧩 While many of these metrics could be derived from logging or tracing, metrics give you real-time and historical perspectives with less overhead. Implementing metrics can provide unique insights into your platforms and products.

One of the hardest parts of data engineering is getting a pipeline into production. The next hardest part is proving every day that it is still telling the truth. Most on-call pages are not about jobs failing. They are about dashboards lying. - A spike that should not exist. - A drop nobody can explain. - Business teams asking, "Is this real or a bug?" If you cannot answer that quickly, you do not have observability. You just have logs that are not useful to anyone. Here is how I think about pipeline observability in six layers. Each layer comes from the graphic you see, but translated into how we operate in real systems. Layer 1: Data checks to confirm the drop is real Before you debug anything, prove the alert is not a false positive. - Compare to the same weekday, same time window, not just “yesterday vs today”. - Check for late data, upstream throttling, backfills, or missing partitions. - Look at job runtimes and lag. Did a job run late, or not at all? Goal: you want to stop waking people up at 2 AM because your API was 4 hours late, not because the business collapsed. Layer 2: Lineage and drivers for the metric Once the drop is real, you ask, “Which layer broke?” - Decompose the KPI into 2 to 4 driver checks. Example: Health = Freshness + Completeness + Correctness. - Use lineage to trace which inputs, joins, and filters feed each driver. - Add cheap driver metrics: row counts, null ratios, uniqueness, expected ranges. Goal: isolate which part of the pipeline is sick instead of staring at the final dashboard. Layer 3: Trend monitoring for each driver Do not just look at the current value. Look at the story over time. - Plot each driver for the same time range. - Find the first timestamp where one driver diverged. - Correlate that moment with deploys, config changes, or upstream experiments. Goal: find the “start of failure” moment, not just the loudest symptom. Layer 4: Slice-level anomaly detection Now you zoom in. Averages hide the crime. - Break the failing driver into slices: source system, region, event type, partition, SDK version, and so on. - Look for the segment where the drop is sharpest. - Keep slicing until you find the smallest segment that behaves differently. Goal: localize the blast radius. You want “US web events since version 3.2.1” instead of “events are broken”. Layer 5: Controlled comparisons between healthy and failing segments Debugging becomes much easier when you compare two concrete worlds. - Pick a healthy slice that stayed normal and a failing slice that dropped. - List everything that is the same: ingest path, schedule, infra. - Then list what is different: schema version, SDK version, region, feature flags. Goal: turn a vague incident into a controlled experiment. The differences between the two slices are your clue list. Continued in comments:

This is what we built SigNoz for. 🚀 → 357TB of spans ingested per month → >10K EC2 instances monitored → ~200Mbps continuous ingestion → 100M+ unique metrics at peak Shivee Gupta from Dream11 just published one of the most detailed production observability case studies I've seen — and it's honestly humbling to see what the team has built. The scale they're running: → 100M+ unique metrics at peak → 1M rows/sec into ClickHouse → 357TB of spans ingested per month → >10K EC2 instances monitored → ~200Mbps continuous ingestion But what makes this post special isn't just the numbers. Most tutorials stop at "here's how to install it." Shivee goes into the real stuff — the 2 AM debugging sessions, the ClickHouse merge tuning, the Kafka buffer optimizations, the memory pressure battles. The things that only show up when you're actually running observability at production scale. Some gems from the post: → How they built a Kafka-backed metric collection layer for self-observability → Custom OTEL instrumentation for legacy systems (Vert.x 3.9!) → Tail-based sampling strategies for traces → Why they chose push-based OTEL over Prometheus pull model → Practical configs that go beyond the docs Huge shoutout to Shivee and the entire Dream11 team for not just building this, but taking the time to document and share it with the community. Link to the full post in comments

You can't fix what you don't measure. That old adage about reliability still applies to AI systems. However, Rootly's research indicates that more than 50% of teams haven't adapted their metrics to AI evaluations. I sat down with Sylvain Kalache to discuss why AI observability requires fundamentally rethinking system reliability. Traditional monitoring metrics (latency, error rates, uptime) only tell you so much about AI system quality. How do you know if today's summaries are better than yesterday's? How do you detect when your RAG system is citing irrelevant context? When does an agent action actually advance the user's goal versus going on a sidebar? At Galileo, we encourage AI engineers to leverage Evaluation-Driven Development (EvDD or EDD) to answer these questions. Instead of static unit tests, EvDD uses continuous evaluation across metrics like: - Context adherence and completeness - Chunk attribution and utilization - Goal progression for agentic systems (action advancement) - LLM-as-a-judge panels (OpenAI, Claude, Llama, Qwen weighted differently) And human feedback isn't optional here. Despite all the automation we're building, including our Luna-2 small language models for real-time evaluation, human oversight remains essential. Whether it's SME reviews, data labeling, or few-shot examples that help models extrapolate improvements, keeping humans in the loop creates continuous learning that scales. For regulated industries? Multiple guardrails are non-negotiable. Input guardrails for prompt injection detection, and output guardrails for PII leakage: we need both. Infrastructure teams require a new breed of monitoring and observability that integrates seamlessly with their current systems, as well as their emerging AI systems. Ignoring it means flying blind on your most critical systems. 🎧 Full conversation in the comments #AI #Observability #SRE #EvaluationDrivenDevelopment

You’re not ready for K8s observability until you separate logs from metrics. Most teams jump into Prometheus + Grafana and wonder why things feel noisy, slow, or expensive. TL;DR that actually works in prod: Two problems, two pipelines. • Logs = what happened (events, errors) • Metrics = how it’s performing (rates, latency, saturation) Logs pipeline (scale-friendly): Pod → collector (Fluent Bit/Otel) → CloudWatch (landing) → Lambda (normalize/enrich) → Kinesis Firehose (batch) → OpenSearch (hot, 7d) → S3 (cold, years) Why: fast search + sensible cost + compliance. Metrics pipeline (reliable by design): App exposes /metrics → Prometheus scrapes via ServiceMonitors (not push) → Grafana visualizes (mix with CloudWatch/OpenSearch for one pane). Keep Prom retention short; archive with Thanos if you need long history. What to get right: Instrument your app (counters, histograms) before you chase dashboards. Use ServiceMonitors for auto-discovery; 30s scrape is a sane default. Treat CloudWatch as the ingest/bridge, not your search engine at scale. Define SLOs (latency, availability) and tie alerts to error budget burn, not single noisy metrics. Start with community Grafana dashboards, then customize for your domain. Starter checklist: Std log format (ts, level, svc, trace_id). Sidecar or node collector—pick one and stick to it. Normalize logs before index (Lambda/Otel proc). Hot (OpenSearch 7d) / Warm (CloudWatch 30d) / Cold (S3) retention. App metrics: req rate, errors, duration (p50/p95/p99), queue depth, saturation. Alerts on SLO burn, not raw CPU. Question: If you had to cut one thing today to reduce observability cost without losing insight, what would it be—log retention, label cardinality, or scrape frequency? #Kubernetes #Observability #Prometheus #Grafana #OpenSearch #FluentBit #OpenTelemetry #SRE #DevOps #EKS #CloudWatch #Kinesis #Thanos #SLO #ErrorBudgets #PlatformEngineering

Today, we're introducing eBPF-based OpenTelemetry tracing alongside a remotely controlled Better Stack Collector. eBPF is ready for prime time. Here's the playbook for adopting it. What's eBPF? "extended Berkeley Packet Filter" is a Linux kernel technology that lets you run sandboxed programs inside the kernel safely and efficiently. Thanks to eBPF, you can now instrument your clusters with OpenTelemetry without changing any application code 🤯 The eBPF ecosystem has matured significantly over the past few months and many Better Stack customers are already using it in production. Until now, deploying eBPF to production has been tricky. We're simplifying it today by bundling the best of the open source eBPF sensors into a single remotely controlled Better Stack collector you can deploy with a single command. Better Stack collector gives you granular control over what exactly gets instrumented. Get the service map of your cluster, RED metrics for individual services, see network flows, and aggregate your application and system logs out of the box. Without changing any code. Observability tools are only useful if you actually ingest all relevant data. Today, we're making that simpler and more convenient than ever. The eBPF OpenTelemetry playbook™ = "Do the easy thing before doing the hard thing" 1. In your staging environment. 2. Deploy the eBPF collector into your distributed cluster. 3. In 98% of cases: Declare victory, your app is now instrumented. 4. In 2% of cases: You notice a particular service has slowed down. For example, the CPU utilization on a high-throughput Redis instance handling millions of operations per second got noticeably higher. Better be safe, so you disable eBPF for this single instance while keeping it enabled for the other 98% of services. 5. If needed, use the OpenTelemetry SDK auto-instrumentation to instrument the last 2% of applications. Most teams today still start with step 5. If you're revisiting your observability stack, I encourage you to give eBPF a chance: it has matured significantly and is better than you might expect. Better Stack encourages combining OpenTelemetry traces from the OTel SDK, eBPF, and your frontend. That's the only way to get the clearest picture of what's actually happening in your application. Want to chat eBPF? Catch me at KubeCon in Amsterdam next week!