This might feel like an insurmountable hurdle at first glance.

One of those “it is what it is” situations. After all:

The more data we collect → the more noise we face → the less trust we have in our alerts → the slower we respond → the more incidents worsen → the more data we collect to compensate.

In MIT’s world of System Dynamics, this noise amplification problem is what we’d call a reinforcing loop.

It’s a spiralling up of information overload as the loop continues to reinforce, or metaphorically snowball, onto itself. But here’s the thing…

Observability (o11y) isn’t just telemetry.

It’s also who interprets, triages, and learns from the telemetry: a balancing loop.

In a healthy system, every new signal that enters should trigger an equal and opposite stabilizing action, essentially a check-and-balance. That’s the balancing loop at work.

For example, when noise increases, teams should automatically slow alert creation or tighten signal thresholds to wait until trust recovers.

When signal quality improves, they can loosen up again.

Without that feedback control, the system loses balance, and the painful reinforcing loop that I mentioned earlier takes over.

If your team doesn’t trust the data, or worse, doesn’t have time to translate it, your observability system isn’t truly “seeing everything”.

That’s why engineers with Staff+ potential treat incident retros and observability reviews like a process tuning. They ask:

• Who sees which o11y signals, and when?

• What incentives drive our attention to o11y signals?

• Where does learning from outputs feed back into o11y design?

Small interventions like taking the time to prune unhelpful alerts can have an outsized impact in the long run because they restore the balancing loop between data and actionability.

This should be your takeaway: reliability improves when observability helps people modify their impact from using the system, not just seeing the outputs of their services.

There’s another issue — data quality. Imagine going into an incident and being unsure about the reliability of the data you’re using.

That’s what can happen when you have observability data quality issues.

The scary thing is — it’s really easy to end up with low-quality data.

We’ll first explore how data quality issues manifest in observability.

We’ll then look at the specific problems in data quality.

And finally, we’ll work through a few potential solutions.

Sound good? Let’s get started.

Unpacking poor data quality

In plain English, you risk getting dodgy or unreliable data.

It’s like trying to fill a swimming pool from a ground well that you don’t know much about managing well.

In this scenario:

• the swimming pool is your observability data lake and

• the water is observability data **

You might be able to fill your observability data lake or warehouse with the unverified water source, but the pipeline is filling it with swirls of dirt and sludge.

The data is tainted with errors or inaccuracies.

End result and risk: your observability data is filled with dirty or useless data, making it hard to trust the insights you gain from it.

It can manifest because of issues like high data cardinality, noisy data, and weak sampling strategy.

Let’s now explore a few of these data quality problems…

High data cardinality

High cardinality is when you have a large number of distinct values in your dataset.

It can happen if you pick something with very unique values like user IDs. That’s never recommended. It can also happen for more strategic fields like instance ID.

How cardinality can make your data quality issues worse by making it harder to handle duplications and inconsistencies. Bottlenecks can result in weak data output, which will give you a lo-fi picture of a hi-fi software system.

The key here is to avoid excessive cardinality data.

I explore this idea in more detail in the article The Cardinality Conundrum in Observability.

Share Boost software reliability with SREpath

Noisy Data

A system running at scale can generate a gargantuan amount of data.

But it’s almost certain that you don’t need all of this data to understand your system’s behaviors.

Noisy data is the phenomenon when an excessive amount of data overshadows the meaningful signals you need to solve problems.

For example, a logging system might track all events including routine informational types and a myriad of overcautious warnings.

You’re essentially wading through a flood of blogs to try and find the important error messages.

You can try to resolve this by implementing filters to reduce the noise at collection or analysis.

You can also set up better thresholds and apply better sampling.

Weak sampling strategy

Sampling is a common practice in observability. You are essentially picking out small slivers of your full observability data to analyze and query.

This is useful when using all the data would too much data to handle.

You can try to complete the puzzle of your full data through strategic sampling.

But a bad sampling strategy can hit your observability data quality the wrong way.

Let’s explore a few of the characteristics of a bad sampling strategy:

Non-random sampling

It’s surprisingly easy to do this.

You might be used to picking out data samples from certain time periods such as business hours.

You might also have a set of criteria to do this.

Either way, you are not allowing random sampling to occur.

End result: skewed insights that don’t accurately represent your system's behavior.

Sampling too small

You need to consider the dataset that you have to work out what size you need to sample.

Taking 10 entries out of a 1000 entry database for example will give you very skewed results.

For small datasets of under 100 entries, you could viably sample 20-50% of the data.

For larger datasets, you can reduce the sampling size to 5-10% of the data.

For mega datasets with 100,000+ entries — like the ones you typically see in systems at scale — the sampling size can be as low as 1%.

Keep in mind, these numbers are guidelines. Follow engineering judgment for your context.

Ignoring dependencies

It’s easy to overlook the fact that your system is many different parts connected together.

Ignore these connections at your peril. Or at least your sampling’s peril.

Having an idea of the relationships between parts lets you sample effectively.

Imagine you're responsible for diagnosing latency issues in a web application with components like web servers, databases, and caching systems.

You already know that these components are dependent on each other.

Their interactions can affect the overall performance of the application.

Here’s what can happen without considering these dependencies:

You randomly select data points to analyze without considering how the components interplay.

You notice spikes in latency but struggle to understand why they occur.

Hard to solve this problem.

Here’s what can happen if you consider dependencies:

Latency in your web application is highly affected by interactions between the web server and the database.

So you focus your sampling on periods when there are simultaneous spikes in web server usage and database queries.

A few telltale signs come up from sampling moments when all 3 areas are at higher loads.

You discover that certain database queries are causing delays in the web server response time.

This is something you can work on resolving.

Wrapping up

As you now know, data quality in observability is an ongoing issue.

It requires handling challenges like high cardinality, noisy data, and weak sampling strategies.

Some of the highlights of what I suggested include avoiding excessive cardinality, implementing effective noise reduction, and considering dependencies in sampling.

Maybe then you can get more accurate insights for solving problems in your complex system.

The scary thing is — it’s really easy to fall into this situation.

We’ll first explore how data flow issues manifest in observability.

We’ll then look at the specific problems in data flow.

And finally, we’ll work through a few potential solutions.

Sound good? Let’s get started.

Unpacking poor data flow

In plain English, you risk not getting enough data.

It’s like trying to fill a swimming pool with water using leaky or clogged piping.

In this scenario:

• the swimming pool is your observability data lake

• the water is observability data and

• the clogged piping is your observability data pipeline.

With leaky piping, the data you need might not be coming through properly to fill the pool.

End result and risk: your big observability data pool isn’t as full as it should be, leaving you with an incomplete picture.

Before we dive into specific issues, let’s do a refresher on how observability data flows…

Refresher on data flow in observability

I mentioned in the guide Key Observability Concepts Explained that observability data flows through 4 stages. It flows like so:

I also mentioned in that guide that the arrows have meaning to them.

They denote how easy (or not) the flow is from one stage to the next.

Let’s explore these flow relationships:

Instrumentation ⇒ Ingestion

This stage is when the data goes from instrumented code and components via collectors to ingestion at the central observability system.

It should be straightforward but can get a little tricky if you have a large variety of data sources, poorly formatted data, or real-time requirements

Ingestion ⇒ Storage

This stage is less tricky as the data is already in a format that databases like and simply needs to flow into storage.

But you can get issues here when there is a large volume of data to push or data coming in at high velocity.

Storage ⇒ Usage

You can expect a myriad of issues at this stage because data retrieval can be a pain.

Query complexity can make time-to-usability slower than you’d like. On top of that, visualizations need intensive processing power.

<aside> ℹ️ Data flow is not the only observability data issue. Check out this guide on the other major issue afflicting **observability data: data quality.

</aside>

This is a high-level view of how the data flows and some kinks that can slow things down.

But there are more serious issues that can plague data flow.

Let’s explore them now…

What are the 3 main data flow issues?

We can put data flow issues into 3 buckets:

1. Siloed Data

2. Latency & Delays

3. Incomplete Data

We’ll explore each of these buckets in detail in a moment.

Before we do that, let me explain the star rating system I will use later on to guide you through the tasks inside each bucket.

I will rate each task with the following on a 5-star (★★★★★) scale:

• ease of doing — how easy would it be to implement and keep doing

• cost — factoring in engineer time and potential vendor costs

• impact — what level of difference it can make to data flow in observability systems

(Spoiler alert: nothing gets 5 stars for impact because that is reserved for tasks with exceptionally high or transformative impact)

My ratings are based on the assumptions that:

• the work will get assigned to SREs with at least 1-3 years of experience

• the team aims to get open-source tooling rather than pay vendors from Day 1 and

• that observability is considered pivotal to providing system insights

For each of the 3 issue types, I’ll give a quick overview then break down the associated tasks.

Let’s begin!

Overview of incomplete data

This implies that your observability data has gaps. The magnitude depends on the number of issues that are plaguing it. By issues, I mean:

1. technical errors

2. misconfigurations and

3. infrastructure limitations

How to reduce incomplete observability data

Summary concept map for solving incomplete observability data issues

Let’s look into 6 ways you can resolve the above 3 issues:

Reduce technical errors

1. Check your error logs

   Step 1 is having a robust error logging process in the first place. You should then have a simple mechanism for checking errors, exceptions, and failures.

   Ease of Doing ★★★★☆

   • Setting up a robust error-logging process is manageable, and checking errors is relatively easy but consistently doing so requires manual effort

   Cost: ★★☆☆☆

   • The cost of setting up error logging is reasonable. Manual checks incur some operational overhead but don’t take up excessive time

   Impact: ★★★★☆

   • Regular checking of error logs has a high impact on identifying and resolving issues promptly, providing valuable insights into system health

2. Monitor for errors

   Implement monitoring tools to track and alert when technical errors occur in the collection stage. This helps in the real-time identification of issues.

   Ease of Doing ★★★★☆

   • Monitoring tools involve some setup and configuration, which can be straightforward but only if you’re already familiar with monitoring tools. making it moderately straightforward. This process requires familiarity with monitoring tools.

   Cost: ★★★☆☆

   • Monitoring tools may come with vendor costs, but the investment is justifiable because it helps with real-time insights needed to assure reliability

   Impact: ★★★★☆

   • Real-time identification of errors has a high impact on system reliability, enabling proactive issue resolution and minimizing downtime.

Reduce misconfigurations

1. Audit configurations.

   Set a regular cadence to check if your system is configured correctly.

   Ease of Doing: ★★★★☆

   • After the initial troubleshooting and training period, conducting regular audits of configurations becomes more straightforward

   Cost: ★★☆☆☆

   • The cost of manual audits is low to moderate in most situations — so long as your Director of SRE doesn’t get involved

   Impact: ★★★☆☆

   • Regular audits are like the annual health checkup that help maintain system reliability

2. Automated audits.

   Ease of Doing: ★★★☆☆

   • Automated audits may require some scripting and/or integration work — you gotta code it, so it takes a moderate level of effort for an SRE with coding skills

   Cost: ★★★☆☆

   • Automation tools come with costs from engineering time, but the efficiency gained in automated checks justifies the investment

   Impact: ★★★★☆

   • Automated audits enhance efficiency by proactively identifying and resolving misconfigurations, contributing to system stability

Overcome infrastructure limitations

1. Monitor for bottlenecks.

   Use performance monitoring on the observability system itself to find where the bottlenecks are

   Ease of Doing: ★★★★☆

   • SREs are adept at running performance monitoring tools and addressing bottlenecks

   Cost: ★★★☆☆

   • The cost of performance monitoring tools varies and the time taken to implement and support them too but it’s worth the impact

   Impact: ★★★★☆

   • Getting rid of bottlenecks can have a high impact on data flow

2. Scalability planning.

   Scale the observability infrastructure to handle the increase in data volume if it is showing consistent patterns

   Ease of Doing: ★★★☆☆

   • SREs with at least 1-2 years of experience should be familiar with capacity planning and system dynamics that govern it

   Cost: ★★★★☆

   • The cost of scalability planning involves strategic investment in infrastructure to handle increased data volumes

   Impact: ★★★★☆

   • Proper scalability planning means the system will be able to handle increased observability data volumes

A few notes:

• With all of the above tasks, you are mitigating a problem that has already occurred

• The order in which you tackle the tasks depends on the problems you find in your observability system’s data flow

• It’s not easy to prevent any of these except perhaps persistent infrastructure scaling issues through autoscaling (super tricky work)

That’s why you should remain vigilant of errors and misconfigurations in the observability system

Overview of latency and delays

You’ll find that one of the most useful aspects of modern observability systems is that they can give us real-time insights.

But this implies your data pipeline is capable of pushing real-time data through all stages in a timely fashion. Latency and delays in data flow will hinder this.

But the good news is that latency and delay issues can be prevented with good practices, which we will cover now:

How to reduce latency & delay issues

Summary concept map for solving observability latency and delay issues

We are trying to improve the timeliness of data and minimize bottlenecks. To do that, we can:

1. Implement real-time monitoring.

   Not for your software, but the observability system itself to see how its data flow is performing at various points

   Ease of Doing: ★★★★☆

   • Real-time monitoring for the observability system’s data flow itself is generally achievable with modern monitoring tools

   Cost: ★★★☆☆

   • Explainer: The cost is moderate because it involves vendor costs and the possibility of needing additional infrastructure

   Impact: ★★★★☆

   • Real-time monitoring significantly boosts the observability system's ability to detect issues promptly for swift resolution

2. Improve load balancing.

   Implement load-balancing strategies including not limited to:

   • Weighted round-robin balancing — distribute more requests to nodes weighted to have more processing capacity

   • Least connections balancing — route requests to nodes with the least active connections at a given time

   • Geographic load balancing — route requests to the nearest node to minimize latency

   Ease of Doing: ★★★☆☆

   • SREs usually stay on top of a few load-balancing strategies for actual application workloads, but it can take time to review the observability system architecture

   Cost: ★★★☆☆

   • Costs are mainly associated with engineering time to set up load-balancing configurations but might also include commercial load-balancing tools

   Impact: ★★★★☆

   • Improving load balancing has a significant impact on system performance through more efficient distribution of requests, lower latency, and less overloads

3. Optimize data pipelines.

   Set a regular cadence to review and enhance data processing pipelines to reduce bottlenecks and streamline processes

   Ease of Doing: ★★☆☆☆

   • Optimizing data pipelines requires a systematic approach to reviewing data processing stages and making adjustments to configurations and workflows

   Cost: ★★★☆☆

   • The cost is moderate because it mainly involves engineer time for the initial review work and then periodic reviews and enhancements

   Impact: ★★★★☆

   • Optimizing data pipelines has a significant impact on reducing pipeline bottlenecks

Overview of siloed data

Observability systems pull data from multiple sources including applications, infrastructure, networks, and more.

Because all of these components are separate, there’s a risk that all your observability data can end up sitting in siloes next to each component.

This defeats the whole purpose of observability systems which are supposed to bring together data from multiple disparate sources to give you a big picture.

Lord help you if you have to solve this problem because doing so almost turns you into a data engineer! Almost…

How to prevent siloed data

Summary concept map for busting siloes in observability data

Siloed data requires bringing in data tools and techniques to unite data. You may need to:

1. Regularly audit existing integrations.

   Define policies that guide the data your system should generate and then regularly check against them.

   Ease of Doing: ★★★★☆

   • Regularly auditing existing integrations is straightforward once you have properly defined your policies

   Cost: ★★★☆☆

   • The cost is mainly engineer time in doing the initial integration inspections, creating the policy, and then the ongoing checks

   Impact: ★★★☆☆

   • Regular audits are a simpler and moderate impact way to solve data flow issues stemming from integration issues

2. Centralize metadata management.

   Metadata can be useful for identifying various sources of data to pull into the central observability system, so keep a system that tracks it.

   Ease of Doing: ★★★★☆

   • Track metadata is a well-documented practice and can be done with ready-to-use tooling like AWS Glue DataBrew and Apache Atlas

   Cost: ★★☆☆☆

   • The cost can be reasonable and mainly involves the engineer's time to set up and maintain the metadata management system

   Impact: ★★★☆☆

   • Centralizing metadata management is impactful for identifying and tracking data sources to organize and push it in the right direction

3. Implement data integration middleware.

   This is not as uncommon as you may think, with the possibility of observability systems streaming data through tooling like Kafka

   Ease of Doing: ★★☆☆☆

   • Setting up and configuring middleware tools comes with a degree of complexity. It's a cakewalk for SREs who know Kafka, but those without this skill might need more time

   Cost: ★★★☆☆

   • The cost is moderate as it involves engineer time to learn Kafka for setup and maintenance of middleware tooling, a lot of which doesn’t have vendor fees

   Impact: ★★★★☆

   • This integration can have a high level of impact if you identify data flow issues due to disparate sources

4. Explore data federation models.

   You could implement techniques that help you do real-time aggregation of data from various sources as needed

   Ease of Doing: ★★☆☆☆

   • The concept of data federation models is not super complex, but it requires you to have some background with it to hit the ground running

   • Cost: ★★★★☆

     • You may need to invest in tools or technologies — like customized open source — that support data federation but it is a strategic investment if you end up needing to do it

   • Impact: ★★★★☆

     • Data federation models have a significant impact on aggregating real-time data from various sources

Wrapping up

This is kind of meta, but observability systems need their own observability — meta-observability! I mentioned monitoring several times to prevent or resolve data flow issues.

The solutions I’ve mentioned are just springboards for you to explore each area in more detail on your own. If you have any specific questions, feel free to ask.

It’s important that we address the challenges I’ve outlined. That’s what it will take to make observable data flow as smooth and pure as freshly melted butter.

It’s a framework that promises to solve several messy problems in observability.

But at its core, it solves one thing unlike anything else out there.

Fragmented instrumentation and collection.

It can instrument all kinds of services for different kinds of data like metrics, logs, and traces and then process them for you.

In other words, OpenTelemetry can make it painless to get the data you need on how your software system is performing.

It helps you generate and capture telemetry data that contributes to better observability.

You’ll also hear some people say it democratizes the process of collecting observability data.

In plain English, it can cut the risk of vendor lock-in. We’ll talk about this later.

What was observability like before OpenTelemetry?

Victor Farcic of the DevOps toolkit has said that “OpenTelemetry can help fix the observability mess.”

His words make sense when you think of having to configure and maintain a multitude of tools just to get your telemetry data.

Before OpenTelemetry, it was not unusual to find 5 to 10 different tools and methods for collecting telemetry data in one software system.

Before OpenTelemetry, there was a mix and match of tooling that collected disparate telemetry data

You had a variety of SDKs, collectors, protocols, APIs, and agents trying to work together.

Each new tool came with an additional layer of complexity because Tool A is configured differently from Tool B is configured differently from Tool C etc.

So engineers had to stay on top of it all.

You’ll hear people talk about this situation with descriptions like lacking standardization and interoperability. Big words highlighting big problems.

The lack of standardization meant engineers struggled to get consistent outcomes across different languages, frameworks, and environments.

The worst aspect of it all was — sure, you could put in the hard work to keep everything running — but it was all fragmented data.

You then faced serious challenges in integrating the data and all kinds of other data flow issues.

By unifying instrumenting, collecting, and exporting data, OpenTelemetry promises to change this for good.

Its interoperability also allows you to change vendors for your querying and visualization work.

Key benefits of OpenTelemetry

1. Reduces fragmentation from disparate observability tooling

2. Eliminates the risk of vendor lock-in

3. Offers comprehensive observability covering logs, metrics, and traces

4. Ensures a consistent approach to observability across the organization

5. Works well within complex cloud-native environments

How does OpenTelemetry prevent vendor lock-in?

You might hear from people that OTel makes vendor lock-in a thing of the past.

Or at least makes it easier for you to switch out components because nothing is proprietary.

The absence of proprietary elements means that users can switch out instrumentation libraries, collectors, or exporters with ease.

The key idea here is that you aren’t relegated to any vendor’s specific tech stack.

You don’t have to recode or instrument your services when you decide to change tooling or vendors.

Which might be a nice thing when you’re looking to try something else.

Now you might be thinking: why would an observability vendor want this to happen?

It could be that they think OpenTelemetry will enable organizations to instrument more services, leading to a larger volume of observability data.

Most commercial plans are priced based on data storage and usage. And there you are.

It’s still a win for engineers because you don’t have to use commercial solutions. You could go with unsupported open source, but that rarely works for actual organizations.

Where does OpenTelemetry collect data from?

The simple answer is almost everywhere you can think of.

OpenTelemetry pulls data from:

• your frontend and backend — supporting a whole bunch of languages like Go, Java, Python, Ruby, Javascript, and even Erlang

• containerized environments like Docker and Kubernetes

• all the major cloud providers including AWS, Azure, and GCP

• existing observability tools like Prometheus, Jaeger, and more

What kind of data does OpenTelemetry collect?

It can handle the 3 core signals of observability at varying levels of compatibility.

By core signals, I mean logs, metrics, and traces.

Traces are the best covered pillar across different programming languages.

That might have something to do with the much older OpenTracing folding into OTel.

OpenTelemetry’s working groups are also investigating other signal types like continuous profiling.

That’s when you continuously collect data on the application’s runtime. That might cover aspects like CPU usage, memory usage, function calls, and more.

What is OpenTelemetry’s architecture like?

In the least possible words, I’d say OpenTelemetry is a loosely coupled framework.

You can use all of it or some of it. It’s super flexible!

You could just use the SDK to instrument your services and then use different collectors like Jaeger or Zipkin.

Or you could go the other way around and use only the OTel collector. When would you do this? Let’s say you have Kafka as a middleware to stream data from various services.

You could stream data into the collector to push out to your exporter and then the visualization tool of choice.

For simplicity, I’m going to break OTel into 3 separate areas: instrumentation, deployment, and integration.

Let’s talk about instrumentation first because… that’s the first part of the process.

(Phase 1 of 3) Instrumentation

This initial phase marks where you instrument (or place) OpenTelemetry code within your services.

It gives your applications and components the ability to emit telemetry data.

Like I mentioned earlier, OTel has most of the popular programming languages covered.

Your job at this point is to incorporate the right SDK or instrumentation libraries into your application components.

Depending on what kind of data you need from the specific application, you will need to:

• create spans for trace data

• instrument to collect performance metrics and/or

• add logging statements to generate logs

You can do all of this using API calls within the SDK you’re using.

You can choose how you instrument your system’s components in 3 modes:

1. auto instrumentation

2. manual instrumentation

3. a hybrid of manual and auto instrumentation

The choice you make will make a huge difference to the amount of time and energy you spend on the initial OTel setup.

Let’s explore each of these in more detail:

Auto Instrumentation

This is when the instrumentation mechanism automatically attaches to your applications' runtime and injects code to capture telemetry data.

When would you do this?

• When you want rapid deployment to instrument your application without making extensive changes to the code

• When your application or other component follows common patterns that work well with auto instrumentation

• When you need to capture telemetry data from a service that you don’t directly control e.g. a cloud provider’s database

When might you consider against it?

• Auto instrumentation can have limitations in capturing domain-specific e.g. conversion rate in eCommerce or custom metrics that fit only your use case

Manual Instrumentation

This is when you add the instrumenting code to the application yourself — without any automation.

When would you do this?

• When you need fine-grained control over what and how you instrument, like the custom metrics I mentioned, or when your application doesn’t follow standard patterns

• It can also come in useful with legacy systems or unique architectures

When might you consider against it?

• If you’re pressed for time or don’t have the resources to do it because manual instrumentation implies manual work

• When you don’t have direct access to the codebase of the service you need to instrument

Hybrid of Manual and Auto Instrumentation

A mix of the automatic and manual modes I highlighted earlier

When would you do this?

• When you have a clear idea of which components are eligible for auto instrumentation and which are more suitable for manual tweaks

• A hybrid approach is useful for transitioning from manual to auto instrumentation or vice versa

When would you consider against it?

• When you lack the planning capability to assign the right mode of instrumentation for components

• This process requires consistent maintenance work as components evolve or completely change

Instrumenting your application doesn’t push the data out for collection at this point.

That comes in the next phase — deployment.

Advice for instrumenting complex software systems

Before you start to instrument your code, you need to know how OpenTelemetry will best complement your system.

So you need to audit your stack.

Have a clear mental or (ideally) written model outlining:

1. Languages in your software system

2. Kinds of signals needed for each component or service (group them to make it easier)

3. Which protocols you will use (OpenTelemetry’s default OTLP or another)

4. Which analytics tools you will use

What does OTel instrumented code look like?

OpenTelemetry’s CNCF ambassadors like Adriana Villela are better people to follow for specific examples of OpenTelemetry instrumentation.

Listen to our podcast episode interview where she discusses OpenTelemetry.

I’ll show you a simple example to get you thinking about instrumentation mechanics.

Below you can see 3 different code samples.

We are instrumenting a Node.js application with OpenTelemetry for tracing that then exports span data to Jaeger, a dedicated tracing tool.

The first code sample installs OpenTelemetry packages for our Node.js application:

npm install \\
  @opentelemetry/api \\
  @opentelemetry/context-base \\
  @opentelemetry/core \\
  @opentelemetry/exporter-jaeger \\
  @opentelemetry/instrumentation \\
  @opentelemetry/instrumentation-express \\
  @opentelemetry/instrumentation-http \\
  @opentelemetry/sdk-trace-node \\
  @opentelemetry/tracing \\
  @opentelemetry/resources

This second code sample installs the exporter needed to send data to Jaeger

npm install @opentelemetry/exporter-jaeger

This third code sample is within our Node.js application showing the instrumentation steps

// Step 1: Import OpenTelemetry
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { SimpleSpanProcessor, ConsoleSpanExporter } = require('@opentelemetry/tracing');
const { registerInstrumentations } = require('@opentelemetry/instrumentation');

// Step 2: Import Jaeger exporter
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');

// Step 3: Create and configure a TracerProvider with Jaeger exporter
const tracerProvider = new NodeTracerProvider();
tracerProvider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter())); // For console logging
tracerProvider.addSpanProcessor(new SimpleSpanProcessor(new JaegerExporter({ serviceName: 'example-service' }))); // Export to Jaeger
tracerProvider.register();

// Step 4: Instrumentation - Import instrumentations for popular libraries
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');

// Step 5: Register instrumentations with the tracer provider
registerInstrumentations({
  instrumentations: [
    HttpInstrumentation,
    ExpressInstrumentation,
    // Add more instrumentations as needed
  ],
});

// Step 6: Use OpenTelemetry APIs in your code
const { context, trace } = require('@opentelemetry/api');

// Function to simulate a simple HTTP request
async function makeHttpRequest() {
  const span = trace.getTracer('example-tracer').startSpan('makeHttpRequest');

  // Simulate some work
  await new Promise(resolve => setTimeout(resolve, 100));

  span.end();
}

// Sample application logic
async function main() {
  const span = trace.getTracer('example-tracer').startSpan('main');

  // Simulate some work
  await makeHttpRequest();

  span.end();
}

// Step 7: Execute the application
main();

(Phase 2 of 3) Deployment

After your application has been successfully instrumented, OpenTelemetry needs to run alongside the application’s runtime.

It can do this in various ways including sidecars, daemonsets, gateways, etc.

You might hear about these as “Collectors”.

This is a critical step that cannot be messed up.

It’s the point when telemetry data is gathered during the execution of your application.

Now that we’ve covered deployment, let’s look at the integration phase:

(Phase 3 of 3) Integration

This phase involves receiving, transforming, and then pushing the data out for human use.

Some vendors will have an “Observability Agent” or “Agent” for short that will do all of this.

You can mix and match depending on your programming language, OS, etc.

We can break OpenTelemetry’s integration phase into components like:

1. Receivers

2. Processors

3. Exporters

4. OTLP

Let’s talk about each of these components now:

1. Receivers

Receivers collect raw telemetry data from your instrumented applications or system components.

They are the entry points for your incoming observability data.

2. Processors

Processors manipulate (or better put) transform the raw telemetry data taken in by receivers.

You can transform the data by filtering, sampling, and enriching it.

3. Exporters

Exporters take your processed telemetry data and send it to external systems. This can be your observability dashboard for example.

You can also direct the data to storage for future querying or visualization needs.

4. OTLP

OTLP is not so much a stage for doing anything to the data as much as the protocol that moves it around. It stands for OpenTelemetry Protocol.

In a way, this is the secret sauce that keeps the data consistent. It’s the common language for communicating within the OTel framework.

There are 2 ways the data can be transported: as protobuf or JSON.

Protobuf is a compact and efficient way of encoding structured data, so it comes as the default.

JSON is available if you want the data to be human readable, but you will increase the data size.

Keep in mind that you don’t have to use OTLP at all for transporting the data.

You can use other protocols like Jaeger Thrift for tracing, Prometheus exposition format for metrics, and more.

Challenges with taking up OpenTelemetry

Not all of it will work well with your architecture

The most important thing you have to keep in mind is that OTel is made up of various components, and not every component is stable or even available.

For example, as of December 29th 2023, logs were not available for Go-based services.

But you can be confident in OTel’s tracing capabilities as that part is mature across most languages.

You can easily check if OTel will work with your various system components.

Go to opentelemetry.io/status and follow the instructions there.

You will see that various OTel components are marked as stable, experimental, in development, or not yet started.

Once a component is stable, you’re good to use it in most situations because stable implies:

1. long term support — the component is well-tested and ready for production use

2. dependency isolation — designed to minimize dependencies and provide clear APIs

3. backwards compatibility — future updates will strive to avoid breaking existing functionality

Some systems just won’t work with OpenTelemetry

Now, I said that a stable component can be used in most situations.

There are some situations where you might not be able to or want to use it.

The first instance is when you’re looking to instrument a legacy system.

You’ll have to make considerations about whether that component will work with your legacy system effectively.

Another instance is when you have a low-latency system like a trading platform.

The performance overhead of OpenTelemetry might call for further looking into how it affects your system’s performance.

OpenTelemetry has a learning curve

While it’s a lot easier than learning 5 or even 10 different tools, OpenTelemetry is still intricate and you will need to learn how to use it well.

A few things I’d recommend for you to do are:

1. deep dive into the documentation on the OpenTelemetry website

2. talk with developer advocates at the observability provider you use

3. reach out to CNCF ambassadors or maintainers who are focused on OpenTelemetry

While I’m on that train of thought, a quick shout out to the hardworking maintainers who maintain OpenTelemetry’s codebase and are working to make every component stable.

I’d say they are the minimum you need for effective observability.

There are a lot more signals you can pick up from your systems, but these 3 are the core ones you should master before you move on to others.

Logs

What are logs?

Logs are text-based records of events and messages that your software system emits.

Why would I use logs?

Logs are an important activity in developing software.

Debugging software is much harder without knowing what kind of error messages your software is producing.

You can also create automated alerts to trigger when a specific log level e.g. WARN or ERROR occurs.

More modern logging systems can even support your efforts to collect metrics and traces, which we’ll get to later.

What kinds of logs exist?

Here are some examples:

What is found in a log?

You will typically find the following elements in a log:

• Timestamp — the time the event or message occurred

• Source identifier — this could be your microservice’s name or instance ID

• Levels — helps you know the importance of a record e.g. INFO, DEBUG, WARN, ERROR, etc.

• Description — gives further details of the event or message

What do the various log levels mean?

INFO means you are getting details about the normal functioning of your app or system

DEBUG can help developers trace an execution flow for troubleshooting purposes

WARN will indicate that something is starting to go wrong or deviate from normal

ERROR will indicate that there is some kind of issue or failure related to the source

How is log data structured?

Log data for most of its existence has been unstructured data stored in plain text files.

Here’s an example of unstructured log data:

2022-01-08 15:30:00 | Error | Application crashed
2022-01-08 15:35:20 | Info  | User logged in
2022-01-08 15:40:45 | Warning | Disk space low

Now, the modern way is JSON to represent log data.

Here’s an example of JSON-based log data:

[
  {
    "timestamp": "2023-01-08T15:30:00",
    "level": "Error",
    "message": "Database overload!"
  },
  {
    "timestamp": "2023-01-08T15:35:20",
    "level": "Info",
    "message": "User logged out"
  },
  {
    "timestamp": "2023-01-08T15:40:45",
    "level": "Warning",
    "message": "Disk space approaching 90%"
  }
]

How do you gather logs on your systems?

A lot of popular frameworks like Ruby, Python, and Javascript come with their own logging libraries, so be sure to check them out.

You can also add logging functionality through external frameworks like Log4j (for Java).

OpenTelemetry also has SDKs to instrument for logging in several popular languages.

3 golden rules for maintaining your logs

Let’s unpack each of these a little further:

Rule 1: Push all logs into a central repo

Life was simpler when monoliths were around. You could get all of its log data in one place.

But it’s no longer so simple, now that most software systems have distributed components like VMs, databases, and more.

A fragmented view of the system’s performance is one of the biggest risks from log data being held near each of these components.

It also increases the complexity of solving log issues like storage and security.

Log aggregation can help push all the fragmented log data into a centralized repository.

This can drastically simplify storage and management of logs.

Rule 2: Stay on top of log storage

The simple aspect of this rule is the more log data you store, the harder it gets to manage them.

Logs can come with an operational overhead in terms of capacity planning work, storage costs, and complexity.

I know of companies that are spending high 6-figures every month to store their log data when they could cut those costs down to low 5-figures by applying good techniques.

Techniques like:

• Log compression — reducing the log data size using compression algorithms

• Log retention policies — having a set policy for when log entries should get purged ensures ongoing control of log data size

• Automated log file reduction — automatically trim older log entries at certain time intervals or file size limits

Rule 3: Keep your logs secure

Logs can contain sensitive information like error and warning messages that can tell bad actors about your system’s architecture and its inner (non)workings.

They can work out the breadth of your system and where they can find the weakest spots to infiltrate with measures like brute force attacks, evasion-based testing, and more.

You can apply some of these methods to reduce your logs’ security risk:

1. Sanitize logs regularly — clean up any sensitive information e.g. user input or tokens from logs once it's a second past its usefulness for debugging and incident handling

2. Restrict your logging levels — sensitive levels like DEBUG (which can link to a vulnerability) should have restricted access

3. Regular audits of logs — check your logs to make sure that sensitive information is not staying on long-term and that access is restricted

Log data lifecycle

3 ways to make analyzing log data easier

Logs are a firehose of data since they collect it from all kinds of components in your system.

You need to use a few analysis techniques to make sure you get good value out of them. Techniques like:

1. Search and filtering

   • Helps you sift through the big data of logs fast by finding or filtering down to log entries with keywords, time ranges, or levels

   • Tools that can help include Elasticsearch, Splunk, or ELK

   • As an example, you could use this ability to find entries with the type ERROR throughout the system so that you can track issues faster

2. Parsing

   • This is very useful if you haven’t got JSON-based logs and your data is unstructured

   • You can break log data into meaningful attributes to extract valuable insights

   • Of course, you can always turn your logs into JSON to make it easier to process

3. Trend analysis

   • You will benefit most from your log data when you can identify patterns and trends within

   • One of the easier ways to do this is to visualize the data on dashboards and graphs

   • You can then see how key numbers like ERROR levels are moving and more

Metrics

What are metrics?

Metrics are measurements that give you an indication of how a certain aspect of your software system is doing or performing.

They are always number-based i.e. quantitative measurements.

Why do I need metrics?

Metrics are important data for your software system.

They can help guide your decision-making as to how to improve your system.

You can use metrics to guide capacity planning efforts which is a big part of Site Reliability Engineering (SRE) work.

They also come in very handy during incident response to see where issues are occurring, which once again is a huge part of SRE work.

They are the main data contributor to alerts.

What kind of metrics exist?

Here are some examples of metrics:

You can develop and consume metrics for:

• applications

• infrastructure

• networks and other system components

You will encounter 4 types of measures when it comes to metrics:

• Counters — measure the number of times a particular action or event occurs e.g. http_requests_total gives a count of the number of HTTP requests

• Gauges — provide a snapshot of the current level of a metric at a particular time e.g. memory usage_mb gives a measurement of memory usage at a particular time

• Timers — measure the duration or latency of an event or process e.g. request_duration measures the length of time it takes to initiate and complete a request

• Histograms — provide a distribution of values broken down into buckets or set intervals e.g. response_time_histogram shows the breakdown of times falling into 0ms, 10ms, 20ms, etc

What is found in a metric?

You will find that metrics are composed of various key:value pairs like above.

The example above shows a metric called “Page Load Time” that is measured in seconds with the average being 2.5 seconds and an acceptable limit of 3 seconds. Because the average is well below 3 seconds, the status is considered optimal.

You will rarely see a simple metric like this in your system, but I wanted to keep it simple to explain the breakdown of a metric to you.

You will more likely see metrics like this:

{
  "metric_name": "http_requests",
  "region": "us-east",
  "http_method": "GET",
  "status_code": 200,
  "response_time": 25.3,
  "bucket": "low",
  "timestamp": 1642267200
}

How do you gather metrics from your system?

You can use various open-source and commercial tools for gathering metrics.

Open-source options include Prometheus, Grafana, and more.

You can also use commercial tools like Datadog and New Relic.

OpenTelemetry once again has stable SDKs to instrument for metrics in several popular languages. You can read an intro guide to OpenTelemetry here.

3 golden rules for building your metrics

Let’s unpack each of these a little further:

Rule 1: Define your metrics clearly

Your metrics should fit a very specific purpose that supports you in solving a problem, known incident type, or business need.

A metric can consist of multiple key:value pairs that relate to a specific aspect of the metric.

You can precisely define your metrics by stating your key:value pairs as:

1. what is being measured e.g. latency

2. how it’s calculated e.g. time elapsed from start to end of request and

3. the unit of measurement e.g. milliseconds

A metric can get complicated fast with a rise in key:value pairs.

Pay special attention to the cardinality of each key:value pair.

High cardinality key:value pairs like user IDs can cause a surge in data leading to storage issues.

Rule 2: Establish baselines and thresholds

It’s all well and good to define what data you want to collect, but a lot of the time you want the system to tell you if a metric is not getting optimal results.

You can do this by establishing baselines and thresholds.

Your baseline number outlines your comfort level for the specific measurement.

Thresholds set the level where you start getting red flags about the metric you’re collecting.

Let’s look at the metric example below:

{
  "metric_name": "http_requests",
  "region": "us-east",
  "http_method": "GET",
  "status_code": 200,
  "response_time": 25.3,
  "timestamp": 1642267200,
  "baseline": {
    "average_response_time": 20,
    "acceptable_status_codes": [200, 201, 204]
  },
  "threshold": {
    "response_time": 30,
    "error_rate": 5
  }
}

Your baseline has an average response time of 20ms.

We’ve also added status codes within the 2xx range that signify successful requests.

Now, let's talk thresholds.

If we see the response time going beyond 30ms, that's a signal that something might be off.

But that's not all – if this happens and we notice error codes repeating more than 5 times, it's a double whammy, and we consider it a problem.

Now you might be thinking: why can’t I just set thresholds in my alerting tool or monitoring system? You can, and that’s what a lot of people do.

But setting the threshold within the metric itself protects it from multiple individuals setting different alerts over time for the same metric. That can reduce alert fatigue risk.

Rule 3: Know the context of your metrics

Context is your secret decoder ring.

It turns raw data into meaningful insights.

Not having the right context means you might think your site is crashing. But in reality, it's just handling the Black Friday rush like a champ.

When you have context, you can distinguish normal behavior from the unusual.

You can look into the context in terms of:

• Day or time

• Special events

• Typical user behavior

Metrics data lifecycle

How metrics are consumed

1. Alerting

   • By far the most critical use of metrics

   • These help identify when system and application issues are occurring in real-time

   • They continuously assess values and trigger when thresholds are hit

2. Dashboards

   • One of the most common ways metrics are consumed

   • They can provide real-time and historical views of metrics through charts, counts, and graphs

   • You can use this to help resolve chronic or wide-spanning issues

3. Machine learning and data analysis

   • One of the more promising uses of metrics data

   • Can analyze and uncover hidden patterns and predict trends — without human intervention

   • Helps you get deeper insights into system behavior than your manual analysis work

There are a lot more areas that use metrics data, but the above 3 can take most engineers very far in their work.

Tracing

What are traces?

Traces are like detailed records that show every step a request or process takes in your system, helping you understand and fix any issues.

In essence you are understanding the lifecycle of a request or process.

Why would I use traces?

Traces can help you find bottlenecks within a particular process or request chain.

By nature, request chains can have multiple requests within. That can make it very hard to pinpoint where the error is occurring.

Tracing breaks the chain down to work out how each individual request is performing as it initiates and hands off to the next request.

This makes it a lot easier to work out where in the request chain you are experiencing excessive latency, for example.

It could mean finding the issue with a database that forms a critical step in a request.

In another situation, it could be a particular piece of code thats causing trouble.

What kind of traces exist?

You can find traces in two forms, but one can fit inside the other.

You can trace internally within a service e.g. within a backend service.

You can trace across multiple services e.g. from frontend to backend to database and back.

The latter is known as distributed tracing and is the more common way to run tracing.

What is found in a trace

Original image via jaegertracing.io

The core element of tracing is spans.

Spans are called the “unit of work” within a trace. They measure the duration of an activity i.e. how long it takes a particular action to start and complete.

In most situations, a span is measured in milliseconds. Each span comes with a span ID to make it easy to deepdive into later on.

Spans can be children of other spans. In the above example, the various spans under Inventory are children of the Inventory span above them.

How do you gather tracing data from your system?

Open-source options include Jaeger and Zipkin.

Tracing is an area that OpenTelemetry excels in since it incorporates the very mature OpenTracing framework.

You should be able to find stable SDKs to instrument for tracing in several popular languages.

3 golden rules for managing your tracing

Let’s unpack each of these a little further:

Rule 1: Define your tracing context

This means you need to set unique identifiers and propagate them consistently across your request. You need these to be clear with your spans.

You can’t link spans together without this work, which makes it hard to get full visibility into a particular request.

So to do this right and define the context:

1. Set a unique trace ID for each incoming request

2. Give each component within the request a unique span ID

3. Propagate these identifiers by including them in headers or context objects

Rule 2: Instrument thoughtfully

Tools like OpenTelemetry make it easy for you to instrument everything for tracing, but that doesn’t mean you should.

You want to gain meaningful insights from important requests.

You don’t want to overwhelm your system with excessive data.

So how do you instrument thoughtfully? Here’s how:

1. Work out what services and components are critical to knowing your system’s behavior

2. Instrument only the ones that will help you understand:

   1. system performance

   2. diagnosing bottlenecks or

   3. improving latency

3. Avoid the temptation to instrument every component to prevent noisy tracing data

Rule 3: Correlate traces with logs & metrics

One of the most important aspects of observability is that it integrates multiple data types to give you a bigger, stronger picture of the system.

Doing so can help you identify root causes more effectively and often faster.

Not doing so means tracing data is own its own when really, it could benefit from log data to support its context.

So how do you make this integration happen? Here’s how:

1. Ensure that logs include trace IDs

2. Be consistent with how you tag components and services so you can link up their traces, logs, and metrics

3. Leverage tracing and logging libraries that integrate with observability platforms

How logs, metrics and traces work in unison

Picture logs, metrics, and traces as the dynamic trio in the backstage of your software.

There are a bunch of other signals, but these 3 are the core foundation of your observability work.

Logs keep a record of every event, significant or not.

Metrics help you keep a scorecard on system health.

Traces map out the journey of each request from start to finish.

Bringing these three together is like weaving a compelling story — traces connect the dots, metrics add the numbers, and logs provide the backstage commentary.

It's your backstage pass to understanding and fine-tuning your system.

This trio has your back when you need to work out, “What’s going wrong here?”

They are your ticket to more visibility during troubleshooting and improvement work.

If I had to explain observability to you for the first time, I’d say it like this:

Observability is a practice that helps us collect, refine and use data on how our software system is behaving. We can query or visualize patterns in the data to solve problems.

You can also call it o11y to save writing observability over and over again.

Why the 11s? Those are the number of characters between the o and y in observability.

I sometimes pronounce it as “Olly” to people who I think would get the context.

My favorite “textbook” 🤓 definition of observability

If I were to make things a little harder and give you a textbook definition of observability, I’d pick this one:

Observability is the measure of a system's ability to allow operators or engineers to understand its internal state and behavior. It involves the collection, analysis, and visualization of relevant telemetry data, such as logs, metrics, traces, and events. This data helps facilitate effective monitoring, debugging, and performance optimization of the software system.

Both definitions give us an idea of what observability can offer, but only one will cut the mustard on Reddit. I’ll let you guess which one.

One other factor is important

Observability is designed to support the real-time nature of needing access to data-based insights. It is also more suited to the nature of cloud-based software.

What kinds of questions can observability answer?

Important ones is going to be my upfront response!

All the effort behind observability is to get solid data to answer critical questions like:

• Are there issues affecting users’ ability to complete requests?

• How do we NOT fly blind when an outage or security incident occurs?

• How can we scale our system to meet changing demand?

• Are there any bottlenecks in our system performance and where?

• Is our service meeting SLOs?

• How do we provide evidence of service uptime in an SLA dispute?

• How can we investigate system performance or uptime issues faster?

The big picture of observability

It's a complex area with much to do, a lot of data generated, and a fair risk of messing up.

But it's worth it. Observability not only helps us respond to incidents with the confidence of data.

It supports works that enhance the system. Works like improving other focus areas like system design, release engineering, performance tuning, and more.

SREpath's Island of Observability highlights the big picture of this focus area.

Observability vs monitoring vs APM

I’ve heard a myriad of engineers in different roles mix up the terms just above.

They think monitoring is observability or that their APM tool can fully handle observability.

Nope!

APM and monitoring are older paradigms from simpler times.

They don’t encompass the breadth of observability, but all 3 of these terms are interrelated. We’ll get to that in a moment.

I think it’ll help to define what each paradigm stands for first. Let’s start with APM.

What does APM mean?

APM stands for Application Performance Monitoring. It focuses on the application layer and debugging any problems within applications.

In other words, APM tooling gives you data on how your application performs at a given time. It does not cover other parts of the system like infrastructure or networks.

This model worked well for monolithic architectures where the APM tool was part of the application’s code.

Can you see how this would be problematic to do in cloud-native architecture?

You might have heard of companies like Splunk and New Relic. They started off with APM offerings but are now also focused on observability for the cloud-native reason.

APM is a subset of monitoring, so let’s define monitoring.

What does monitoring mean?

Monitoring within a software system is all about picking up its health data.

Prometheus is an example of a tool solely focused on monitoring.

You can gather data like error rate, traffic levels, latency, and saturation.

I chose this example because latency is my favorite monitoring data — maybe I should have worked as a performance engineer 🤔

By the way, these are known as the ****4 golden signals in the Site Reliability Engineering (2016) book.

Think of monitoring as something that’s always reactive. It’s like a radar that lets you see what’s in your field of view at that point in time.

The key purpose of monitoring for a long time has been to support predefined alerting.

I’ll share an analogy with you in a moment that will highlight this alerting use case, but I need to explain observability again for that. Let me throw a bombshell first…

Monitoring is a part of observability systems, so let’s define observability again now that we are comparing it with others.

What is observability when compared to APM and monitoring?

Unlike APM, observability can collect data from everywhere in the software system. This includes applications, databases, storage, networks, and more.

Observability factors in the complex nature of cloud computing that APM does not.

Unlike monitoring, observability captures data from a proactive standpoint and helps engineers anticipate where issues will arise. It also gives context to alerts when issues occur.

Keep this in mind: APM is a part of monitoring. Monitoring is a part of observability.

Non-software story to highlight observability vs monitoring

Pavan Elthepu gave a brilliant example in the context of a hospital situation.

Imagine a bedbound patient is connected to a heart rate monitor.

The monitor’s heart rate reading suddenly spikes, causing an alarm to sound in the hospital ward.

A nearby doctor rushes to the scene to investigate the issue.

She looks at the monitor and notices the spike in heart rate, but that’s all it can tell her.

What should her next step be to prevent the situation from worsening?

The doctor is well-seasoned and starts looking into medication charts and historical health data.

She finds that the patient had been administered a new pain management drug recently.

Her conclusion is that the drug is causing an increase in heart rate.

The nursing staff then work to administer countering treatments to resolve the issue.

In this situation, the heart rate monitor is analogous to monitoring in that it gathers data to give reactive alerts.

The medication charts and historical health data are analogous to observability data in that they give context to the alert. They helped pinpoint the underlying issue.

You can see from this analogy that observability can be extremely useful for adding context during incidents like outages and security breaches.

What does observability data look like?

You might have heard of logs, metrics, and traces, and events.

These are the main types of data you will find in an observability system.

Each has its use case for helping you investigate and resolve system problems.

Log data definition and example

Logs are recorded messages or events the system generates, providing a chronological record of its activities.

Here’s a simplified example of what logs can look like:

2023-12-01 15:30:45 [INFO] User 123 logged in successfully.
2023-12-01 15:32:10 [ERROR] Database connection timeout.
2023-12-01 15:33:25 [WARNING] Disk space is running low (90% used).

Metrics data definition and example

Metrics are quantitative measures of various aspects of the system and its underlying performance

Here’s a simplified example of what metrics can look like:

cpu_usage_percent: 85
memory_used_mb: 1200
http_requests_total: 1500
error_rate: 0.05

Trace data definition and example

Traces are data that cover the underlying responses within a request as it moves through different components of a distributed system.

It consists of data points known as span, which we explore in another guide that deep dives into logs, traces, and metrics.

Here’s a simplified example of what trace span data can look like:

Trace ID: 12345
  Span 1: Service A (Start Time: 15:30:00, Duration: 500ms)
    Span 2: Service B (Start Time: 15:30:05, Duration: 300ms)
      Span 3: Database Query (Start Time: 15:30:07, Duration: 100ms)
    Span 4: Service C (Start Time: 15:30:10, Duration: 200ms)
  Span 5: Service D (Start Time: 15:30:12, Duration: 150ms)

Event data definition and example

Event data in observability provides fine-grained details about notable events.

Each event includes a timestamp, event type, contextual information, and a payload.

Events are crucial support data for incident response and root cause analysis.

Here’s a simplified example of what event data can look like:

{
  "timestamp": "2023-05-01T12:34:56Z",
  "eventType": "UserLogin",
  "context": {
    "userId": "12345",
    "location": "Homepage"
  },
  "payload": {
    "status": "Success",
    "ipAddress": "192.168.1.100"
  },
  "source": "AuthService"
}

Observability data can come from multiple sources

Catchpoint’s 2023 SRE Report found that over 53.5% of its 550+ respondents in the reliability space worked in an environment with 3 or more sources feeding their observability system.

What’s the significance of this finding?

It’s the recognition that having multiple data sources is crucial for obtaining a comprehensive and accurate understanding of the system's dynamics.

The risk of having a single source of truth (that IT vendors love selling) is that you risk missing the complete picture of what’s happening.

What kinds of sources can we get data flowing in from?

Here are 5 examples that are most relevant to software in production:

1. applications

2. infrastructure

3. network

4. front-end UX monitoring

5. client-side device monitoring

Imagine having only infrastructure observability data and no information about how applications behave. Such a scenario would mean doing guesswork to pinpoint issues.

This is why it’s best to cover infrastructure and applications at a minimum.

Where is all this observability data going?

Observability data follows a journey to usefulness

⚠️ The shape of the arrows means something. I’ll explain after we go through the flow process.

Let’s do a simplified rundown of what happens at each stage:

1. Instrumentation

• You insert monitoring code (listeners) into the services you want to collect metrics from

• This code gathers data about the service's performance, errors, etc

• Examples of tooling at this stage include Prometheus, OpenTelemetry

ℹ️ A quick aside on OpenTelemetry. CNCF is the open-source cloud computing tooling network, and OpenTelemetry is the second most contributed project there. Its purpose is to make instrumenting your code for data collection painless.

2. Ingestion

• The listener code then sends collected metrics data to a data collection service

• These are often called collectors. These collectors organize and process the incoming data

• Examples of tooling at this stage include Prometheus, Kafka, FluentD

3. Storage

• The metrics data then gets pushed to designated databases for storage

• These databases are often designed to store time-series data, common in monitoring and observability systems

• Examples of tooling at this stage include Prometheus, OpenTSDB, InfluxDB

4. Usage

• This is the final stage and where you can use the data for several purposes

• You can use it for alerting, creating dashboards, and supporting machine querying

• This helps you stay on top of system behavior, performance, and issues

As I mentioned under the last image, data flow issues can happen in observability.

I will cover them more in-depth in an upcoming guide called How to Solve Poor Data Flow in Observability.

How is observability data stored?

Most modern observability tools work with time series databases (TSDBs).

These databases help us understand what a metric is up to at specific moments in time.

You might catch Comp Sci PhDs calling these time series a "temporal aspect."

Components of a time-series database (TSDB)

In a TSDB, time series data is organized into individual series.

Definition: we are ordering a sequence of data points in terms of time. Hence time series.

Each data point usually has a timestamp and corresponding value/s.

Here’s an example of what time series data looks like:

Metric Name Dimensions (labels) Timestamp Value http_requests_total {status=”200”, method=”GET”} 2023-11-24 09:00:00 1010 http_requests_total {status=”404”, method=”GET”} 2023-11-24 09:00:00 239 http_requests_total {status=”200”, method=”GET”} 2023-11-24 09:00:01 1028 http_requests_total {status=”500”, method=”GET”} 2023-11-24 09:00:02 10383

The above example captures what time series data could have looked like for some eCommerce sites at the beginning of the Black Friday sale on November 24th, 2023.

Let’s break it down:

• At exactly 9 am, they had 1010 successful “200 OK” requests and 239 unsuccessful “Page not found” 404 requests.

• A second later, they had 1028 successful “200 OK requests”

• But only a second after that, they had 10383 requests that got a “500 internal server error” response — possibly due to more requests than server capabilities

Series? Huh?

Think of a series like this. It is the unique combination of a metric name (what you are trying to gather) and the key-value pairs that make it up.

The cherry on top is a timestamp that marks: “This is what happened at this point.”

That’s why it’s called a time-series database 😉

You can also see adding each new key-value pair as adding a dimension.

You will hear these terms being used in observability conversations a lot:

Don’t add a new dimension to your series unless you know its cardinality is worth it.

You can explore the important concept of cardinality here.

Let’s break down a series

Remember it has the metric name and the key:values that compose it.

Here’s a common example of a metric name:

http_request_duration_seconds

Dimensions can include:

• method (e.g., GET, POST)

• status_code (e.g., 200, 404, 500)

• endpoint (e.g., /api/v1/resource)

• server (e.g., server-1, server-2)

Every series will give a response on a new line like:

1. http_request_duration_seconds{method="GET", status_code="200", endpoint="/api/v1/resource", server="server-1"}

   • Duration: 0.235 seconds

2. http_request_duration_seconds{method="POST", status_code="404", endpoint="/api/v1/user", server="server-2"}

   • Duration: 0.540 seconds

3. http_request_duration_seconds{method="GET", status_code="500", endpoint="/api/v1/resource", server="server-3"}

   • Duration: 1.120 seconds

How is observability data being used?

It’s going places, don’t you worry about that. I mean real practical uses.

We are ingesting, storing, and then querying it for 3 purposes:

1. Alerting — helps us respond to outages and other incidents

2. Dashboards — gives SREs and other rockstars a big-picture view of system health

3. Intelligence — machine learning pulls data to analyze for predicting trends and proactive work

How does observability add value to software operations?

As I mentioned earlier, observability is often used as a conduit to doing other software operations activities better. It can help with:

1. Incident response — Observability provides real-time insights into system behavior, which helps minimize the impact of incidents through rapid identification and resolution of issues

2. Improving system design — System architects and engineers can gain a deep understanding of system internals to enhance its design by analyzing observability data

3. Capacity planning — Capacity planners like SREs can use observability data to see trends in resource usage to make informed decisions for scaling systems up or down

4. Assuring smooth releases. Observability can help identify and cut the risk of potential issues that can cause post-release incidents, resulting in smoother software deployments.

5. Performance tuning — Observability data can help pinpoint system bottlenecks and inefficiencies that performance engineers and SREs can use to optimize components

What are some risks in observability practice?

I mentioned several data flow issues earlier, but many of these can be kept in check by staying on top of the constant upgrades in observability tooling.

Some other issues can be on top of mind in observability.

The most critical one I’ve encountered is the high cardinality issue.

I wrote a full guide on the cardinality conundrum to give you a view of what it means.

The high cost of observability systems (some reaching 7-figure costs) is also a bugbear for many software-dependent organizations.

Data quality issues can also come and can be directly related to the flow issues I mentioned earlier, but sometimes they’re a beast of their own.

And finally, the one I’ve been asked about more than once is alert fatigue. Poorly configured alerting can cause burnout in engineers from excessive cognitive load.

Whether it’s purely an observability-related issue is up for debate. I’ll leave it here.

Wrapping up

Observability is not as straightforward as it may be marketed to be.

There are a lot of nuances in making it work well and using the data to its full potential.

Keep the data flowing well, and it will serve your reliability work just as well.

Cardinality is a term you’ll hear over and over again if you’re looking into how to do observability.

And especially if you are talking with vendors! They love this topic!

A lot of people have been thinking about high cardinality for a while and for a good reason.

Because high cardinality can cost you a LOT of money and time if you go about it wrong… but really, what we want to do is cut down excessive cardinality.

The kind that doesn’t add value to your querying and intelligence.

That’s why dealing with cardinality is not a straightforward solution. Cutting labels willy-nilly is not the answer. We’ll get onto ways to deal with this later on.

But we should first talk about what cardinality actually means.

What is cardinality?

Because it’s not something you think about every day… unless you’re an observability engineer or vendor.

The last time I heard “cardinality” being used this often was in my SQL classes, which was a while back.

So I did a refresher and went down a rabbit hole of math, logic, and all that fun stuff.

Cardinality refers to how many unique values there are in a data set.

You’re essentially looking for how diverse — and ultimately complex — the data is.

How do you differentiate between low and high cardinality data?

• Low cardinality means low complexity with few dimensions to data. This is fine for analyzing aggregate data but lacks granularity that engineers often seek to solve system problems

• High cardinality contains more data dimensions. This lets you slice and dice data for more detailed analysis, but you then have to deal with complexity issues.

By dimensions, I mean attributes like method, status_code, instance_id, etc.

Let’s run through a simple example to cement the difference:

Let’s say you have the table fruits with only the fruit apple in your database. We want to add a color key to identify each apple in your table.

Now, the data on apples is low cardinality if you only find red or green apples like so:

But it becomes higher cardinality if you have apples with colors like orange, blue, red, pink, green, violet, black etc.

The more color options you have for apple, the higher the cardinality of the color data.

Examples of high cardinality

Here are some examples of high cardinality data:

• email addresses (never append these to metrics!)

• user IDs (”Observability engineer: why is this in my beautiful TSDB?!”)

• IP addresses (sometimes appended for AppSec purposes)

• instance name (fair use case for identifying instance issues)

• pod s (like the kind you find within Kubernetes)

3 key benefits of high cardinality in observability data

High cardinality enables these traits within observability data:

1. Granularity

With granular data, you can slice and dice to deeper and deeper levels to precisely pinpoint where issues like performance degradation and outages are happening.

For example, with each 500 error, you may want to dig a little further.

With a high cardinality metric, you can dig into the 50x for GET, POST, and DELETE requests.

2. Segmentation

Helps you collect disparate data and organize it into chunks for easier human processing.

Examples include:

• payments by age group e.g. 18-25, 25-34, 35-49, etc.

• payments by market region e.g. North America, South East Asia, LatAm, EU, etc.

3. Performance evaluation

High cardinality means you can go beyond “success” and “fail” for responses.

You can classify it as performance levels to improve reliability for different instances and groups.

Imagine what you can work out by putting responses into buckets of 0-10 scoring.

Before we continue, I will assume that you know the answers to the following questions:

What does observability data look like?

How does observability data flow?

Where observability data gets stored and how

How do observability data types do in terms of cardinality?

Metrics are the main data type when we think about cardinality. But let’s still cover each of the 3 main observability data types to see its cardinality issue:

High cardinality in logs

Logs of yesteryear are typically less affected by high cardinality than metrics and traces.

Some engineers still use unstructured logs for small-scale systems.

At the commercial scale, modern logs are structured or at least shifting toward that. JSON shows the most promise through that.

So why shift to a structured format? A few things come into play like:

• Readability

• Better ability to group or segment logs by keys for better more refined analytical outcomes

• Speed to read as less string manipulation or string processing has to take place during the reading of the aggregated log data

You can derive metrics from logs now!

Logs in a commercial setting can experience high cardinality, just like metrics and traces.

High cardinality in metrics

Among the trio, metrics are the most significantly affected by high cardinality.

Metrics get complicated by the fact that they can have many dimensions.

You can have a multiplier effect when you add a new dimension.

This multiplication is what defines the high cardinality we are talking about.

You benefit in one way from this through more granularity that helps deeper analysis.

But then you also impact storage, querying, and visualization performance.

High cardinality in traces

Traces can also be affected by high cardinality.

Picture a trace to be the bus route from New York to Los Angeles and back.

This route can be divided into sections — when you plan a trip along this route you may want to stop for a breather, food, or a bathroom break.

Each section of this route or round trip is equivalent to what in tracing we would call a span.

A common dimension for spans is the time or duration of that specific section of work.

We can add more detail or context for each span by appending metadata to the span.

You get high cardinality within tracing when you use spans like user IDs and specific names of what happened.

All this once again impacts storage and querying performance.

Calculating the impact of cardinality within a metric

Adding a dimension to a metric does not cause high cardinality.

It’s what that key-value pair stands for that determines this.

So adding a 6th dimension doesn’t increase cardinality if it’s a boolean or a “success”/”fail” type.

But if it were something like instance ID and you have 100 or 1000 instances, that would.

We do not need to calculate cardinality unless we are sitting in a math class.

We are more interested in how many time series

Calculating time series count for a metric

Let’s run through a simple example.

Say you have a metric called network_latency_distribution covering 100 instances with 10 buckets, 10 possible response codes, and 10 network paths.

**Calculating the series would look like this:**
= 100 instances * 10 buckets * 10 response codes * 10 paths
= 100,000 series

This is a reasonable size for a series, but things can get out of hand as you add more dimensions.

Adding a dimension like region can significantly increase cardinality.

**Say we have 6 regions to choose from. The calculation would become:**
= 100 instances x 10 buckets x 10 response codes x 10 paths x 6 regions
= 600,000 series (! 😕)

Okay. It doesn’t look excessive, but can significantly increase querying time!

I’ll share some query time data with you in a minute.

The dimension of pods (within containers) can increase cardinality by a huge factor, too.

**Say we have a low 1,000 pods in action. The calculation would become:**
= 100 instances x 10 buckets x 10 response codes x 10 paths x 1,000 pods
= 100,000,000 series (!! 😦)

Excessive cardinality comes into the picture when you add a dimension like user_id.

**Even with a modest user count of 10,000, the time series blows out with:**
= 100 instances * 10 buckets * 10 response codes * 10 paths * 10000 users
= 1,000,000,000 series (!!! 😱)

We’ll talk about Excessive cardinality in more detail later on.

🚨 PSA: The more types of data you collect, the more fragmented view you have of the whole piece. And if you have a fragmented view, that reduces your ability to TAKE ACTION on the data. Remember that Quality of data > Quantity of data.

High cardinality data is rising because of system trends

Shift from monolith to microservices architecture

Where you had one humongous service emitting metrics, you now have 10, 20, 50, 100+ microservices with each emitting its metrics. ‘Nuf said.

Shift from VM-based to container-based infrastructure

Who would’ve thought that life was easier when we depended on VM infrastructure rather than containers? </sarcasm>

Containers have their benefits but generate a whole bunch of time-series data.

For every 1 VM in an older system, there are 10s of containers to match workloads.

The key culprit behind this is their high ephemerality.

The ephemeral nature of containers refers to when they start, shut down and then a new container takes their place.

All this stop-start emits a lot of data for metrics collectors to ingest and push to storage.

Serverless functions as a large part of the system

Every time you invoke a Lambda function, that induces a metric with a time series. Depending on the particular service you are running on serverless, this time series number can be HUGE.

A whole bunch of factors come into play like:

1. how often you're invoking the lambda function

2. the number of serverless functions in your system

3. concurrency of handling requests and scale

Your time series data incurs higher and higher cardinality as these 3 factors rise in frequency or occurrence (depending on the factor).

When to deal with high cardinality

Remember how observability data flows. There are 4 distinct stages:

Dealing with high cardinality at the Instrumentation stage is not ideal. You will not confidently know what dimensions to filter out at this stage.

It’s also not ideal at the Usage stage. It’s too late to deal with cardinality issues because you’ve incurred high ingestion and storage costs, so…

Deal with high cardinality at the Ingestion & Storage stages. This is when you can put in practices like in-flight aggregation, cardinality isolation, and cardinality limiters.

When to ditch high cardinality data

High cardinality data is a necessary evil in some situations. As I mentioned earlier, it helps with granularity, segmentation, and deeper performance evaluation.

If it does not add value to your querying, intelligence, or alerting. Ditch it.

It’s just an ornament to give you pretty data.

What I mean by this is it looks like it could be important.

But in reality:

• it’s giving you limited analytical value

• rarely contributes to usable insight and

• with either of the above two is resource intensive.

Because if resource intensiveness did not cost money or time, we wouldn’t care about it.

We’d just let high cardinality data sit there and do its thing.

But it does cost money in terms of storage and processing power.

And it costs time in terms of how long you have to wait before you can start working with the data.

And it’s definitively something you need to look into if it’s slowing things down to the point where you’re pushing past your MTTR target.

(MTTR = your mean time to recovery, repair, resolution, whatever you want to call it).

Now, you might be saying thank you, Captain Obvious.

But think about how often this is still a real problem in software systems.

You need to pose it as a challenge for people to think about and solve it.

The problem with excessive cardinality

The working group behind the open-source Prometheus monitoring tool has warned about this for a while.

Excerpt from Prometheus.io documentation on Metric and label naming

A quick TLDR of their alert message on key-value pairs:

“Every unique combination of key-value pairs represents a new time-series. This significantly increases the data stored.”

An ever-increasing number of possible key-value pairs or dimensions does something sinister within time series databases (TSDBs).

The number of series for a single metric will explode 💥 to the point that your querying eventually slows down to a crash.

Query performance data by Chris Marchbanks (ex-Splunk, Grafana Labs) says it all.

This is how long it took to query quantities of time series:

• 100,000 series took 1.5 seconds (acceptable)

• 200,000 series took 5 seconds (slow-ish)

• 10,000,000 series took 15 minutes (!)

Keep this in mind: high or (better put) excessive cardinality is a data problem at its core.

Not only does it slow down query time to a grind, but it can also cause trouble like:

• data being wasted on dimensions that are not needed for system improvements

• need for engineers to constantly maintain performance of the observability system

• higher storage needs and processing resources means higher cost to run

We will think through a few solution starters toward the end to prevent or at least reduce this risk.

What’s contributing to excessive cardinality in a software system?

Bad dimension selection

Remember my calculation example above?

Putting in user or request IDs can skyrocket your observability metrics’ cardinality.

Each unique identifier will contribute to a new entry, increasing the overall cardinality of the logs.

Improper sample practices

Observability systems generate a ton of data all the time.

It’s not an easy task to query all of that data all the time.

This is where you want to bring sampling practices to select a portion of the data to analyze.

But if you don’t use the right sampling techniques, you will deal with high cardinality data and more series than your dashboard or query tool can handle.

Unbounded event types

Metrics are not the sole culprits for pushing out excessive cardinality data.

Putting weak boundaries around your event data can do the same to logs and traces.

A good system will have few event types while a system suffering from excessive cardinality will have numerous possible event types.

The more event types there are, the more logs and spans you’ll have to push to ingestion.

This is not an exhaustive list of ways excessive cardinality can happen. I want to illustrate the idea that there are several ways you can end up with it.

How to solve excessive cardinality

Be selective about dimensions

I feel like I’ve mentioned it several times already, but never, ever use dimensions like email address, user ID, transaction ID, or anything with overly unique data in your metrics.

Split the metric into smaller metrics

Ask yourself these two questions:

Do I need to have this single metric with all these dimensions?

Can I split it into two separate metrics that can still help me answer the questions I will pose at querying, and still give me the necessary alerting?

An example might better highlight why you’d want to do this:

Let’s return to our metric called network_latency_distribution, which covers 100 instances with 10 buckets, 10 possible response codes, and 10 network paths.

**Calculating the series would look like this:**
= 100 instances * 10 buckets * 10 response codes * 10 paths
= 100,000 series

Now what would happen if we were to split this metric into 2 individual metrics, one without paths and one without buckets?

**Here's the first metric without paths:**
= 100 instances * 10 buckets * 10 response codes 
= 10,000 series

**Here's the second metric without buckets:**
= 100 instances * 10 response codes * 10 paths
= 10,000 series

This gives us a grand total of 20,000 series and a whopping 80,000 series reduction!

This split works perfectly if we don’t need to correlate paths with buckets to solve system issues.

How much faster would it be to query 20,000 vs 100,000 series?

Answer: enough to feel instantaneous vs. time spent hearing, “Please wait while we process your query.”

Allow high(er) cardinality for high-value metrics only

It can still make sense to generate a whole bunch of series for metrics that add business value.

You may have some metrics that need 100,000 series rather than being split into multiple mini-metrics and losing their strength.

How do we define a high-value metric? It is this if it supports:

• critical decision-making processes

• developing actionable insights or

• enhancement of overall system performance

What hits one of these criteria depends on your software architecture, industry context, and the problems you’re looking to solve.

Here are a few considerations to make:

1. These kinds of metrics might not be suitable for time-of-essence needs like alerting

2. You can try and save the designation of high-value metrics to ad-hoc queries, low-usage dashboards for specialist or special interest groups, and periodic reports

3. You still need to weigh the impact vs querying & visualization time

4. Even a high-value metric can start looking too expensive with the way observability is priced these days, so work out the costings vs the value you attain and discuss with management

We must still remember to control cardinality levels to meet our cost and time-to-productivity constraints.

Wrapping up

If you can only remember one thing from this guide, I want it to be this 👇🏼

Cardinality is valuable, but excessive cardinality is expensive — in terms of time to query, cost to store, and resource consumption to process & analyze. Keep it in check.

I’ve decided to focus this newsletter on publishing written-only content in the future. The podcast writeups I used to do will not continue after Episode #35.

That episode’s titled Boosting Your Observability Data’s Usability. Check it out if you are interested in observability and maximizing its value.

There’s a write-up on the episode in the second half of this newsletter.

To stay on top of the podcast, follow us on your favorite player (or bookmark this link to the web player on the SREpath website).

Here are a few of the top podcast players that you’ll find the SREpath podcast on:

• Spotify

• Apple Podcasts

• Amazon Music

• Google Podcasts

Episode 35 [SREpath Podcast]

About this episode

The observability (o11y) revolution is underway with many organizations already instrumenting services. But are we getting the most from the data that is being collected?

Richard Benwell thinks we have room for improvement in this area, especially at the usage stage where we query and visualize the o11y data.

He is the founder and CEO of SquaredUp, a dashboard software company based out of Maidenhead, UK with over 10 years of experience in the monitoring space.

Richard highlighted the importance of converging human intuition with technical o11y implementations and moving from a narrow focus on collecting data to leveraging it for actionable insights.

You can connect with Richard via LinkedIn

Key concepts we explored

Our conversation touched on fascinating observability concepts like:

1. Overemphasis on Collecting Observability Data. The key theme of our conversation was that observability shouldn't solely focus on collecting vast amounts of data. The data should be used judiciously for insights and actionable intelligence rather than just being stored.

2. A Human Approach to Observability. Richard shared the need for human intuition and engagement in observability. Engineers should prioritize understanding how to use data effectively for engaging human stakeholders rather than solely focusing on the technical aspects.

3. Socio-Technical Perspective of Observability. Observability is not just a technical challenge but also a socio-technical one. It involves understanding human behavior, collaboration, and organizational dynamics alongside technical infrastructure.

4. Visualization Increases Engagement with O11y Data. It can make observability data more meaningful and engaging for stakeholders. The core premise of this is to simplify complex data into simple visuals that provide clear and rapid insights as well as give context.

5. Observability in Big Tech vs. Other Industries. We need to recognize that most of the ideas being shared are by BigTech for BigTech. While big tech focuses heavily on real-time data and rapid innovation, other industries need to prioritize stability, reliability, and business continuity.

6. Dashboards Should Be Interactive. While dashboards are valuable for sharing information, they are not very robust as static views. Aim for interactive dashboards that allow for exploration and drilling down into observability data.

7. Keep your Stakeholders Engaged. Lack of engagement in observability leads to a vicious cycle of declining data quality and relevance. Engaging users and stakeholders ensures continuous feedback and improvement in observability outcomes.

8. Make Observability Data Relevant to Business Context. Tailor your querying and visualizations to the specific needs and context of the business. You could use approaches like representing data in the form of a customer journey, to make data more relatable and understandable for stakeholders.

9. Don't Sleep on Continuous Improvement in o11y. Just like anything in technology, observability is an ongoing process. It requires continuous improvement and adaptation to changing business needs, technological advancements, and user feedback.

You will not want to miss Richard's insights that can improve the usability and ROI of your observability data.

In Episode #35 of the SREpath podcast, Richard Benwell gives ideas on making the most of your observability data [Spotify link]

Keep scrolling for a full write-up on this topic below 👇🏼

Show notes

How critical is observability (o11y) to SRE work?

To me, observability is the core foundation practice for all other SRE practice areas. Without it, you're flying blind. 

David Caudill is not afraid of making controversial viewpoints about this area. But he doesn't do it for glory. To him, it's about driving better practices. 

After all, it's not in his interest to promote (sell) shiny objects. He runs engineering teams at Capital One, one of America's largest banks, so he wants a functional offering more than overly done hype. 

He believes that delusions are getting in the way of our success in observability. We explore some of them in this episode of the SREpath podcast.

You can connect with David via LinkedIn

More about our conversation

David's stance is simple: observability itself is not bad; it's just that we are often not doing it with the right mindset! We are seeking elegant technical solutions to complicated real world problems. Notice the disconnect?

I had a chat with him a few weeks back about all of this.

Our conversation touched on ideas you likely won't have heard anywhere else including:

➡️ David's analogy for observability of software architectures "from monoliths to rocks, pebbles and gaseous clouds"

➡️ The need to handle cognitive load effectively when it comes to your observability system by simplifying measures (inspired by Google)

➡️ Moving toward event-based SLOs rather than leaning too heavy on time-based metrics for your observability 

Let's unpack each of these:

Observability of software architectures "from monoliths to rocks, pebbles, and gaseous clouds"

Did I mention that David likes to use fun and peculiar analogies to highlight his ideas? I love it, and you might too. 

But what does he mean by this particular analogy (above)? 🤔

David put it like this (quote truncated for brevity):

"They start with a monolith, and they decide, 'we could break this up into a few service oriented architecture' blobs [rocks]... then go to microservices [pebbles], and then into lambdas [gaseous clouds]. Before long, you're down to really, really intense atomicity in this architecture."

This is a common enough pattern in more than a few organizations I know inside details of. It makes sense as microservices is the shift that cloud pundits have been pushing for a decade now. 

Serverless these days is not the quiet one in the corner.

So all seems well and good. How much can atomicity hurt?

Well, it can be painful if your observability system can't handle it.

David added the fact that this kind of atomicity adds cardinality to observability. It adds overhead that itself needs to be observed. This overhead can start to feel as large as the application itself.

I've heard people echo David's sentiment that this overhead can become a completely invisible cloud of noise around your application that has nothing to do with your application.

A cautionary tale indeed for carefully reworking your software architecture. Be sure to keep your o11y capabilities in mind when doing so!

Handling cognitive load when it comes to observability systems 

Our discussion of architectural woes in o11y drove me to bring up an area that is so critical to successful engineering work but is often neglected: cognitive load. 

Observability systems are overloading engineers with too much data and things to see. 

David's suggestion to my assertion around cognitive load made me chuckle and grimace at the same time:

"I'm very much of the opinion that you want to preserve simplicity as long as you can. There's a lot of anticipatory architecture changes that happen really on the optimism like, 'Oh, it's going to blow up!' I have worked in environments where it didn't blow up and it was still really complex because we were prepared for a legion of people that never showed up."

He added that engineers need to "avoid complexity like the plague" and keep things as simple as they possibly can. 

Moving toward event-based SLOs rather than solely time-based metrics

Drawing from experience, David recalls his attempts at making time-based SLOs work. Over time and several failures, he learned that event-based SLOs are more appropriate in many situations. 

He stresses the rationale for not using time-based SLOs through this quote:

"Because not every minute of time is the same as every other minute. And if your service goes down when no one is using it, who cares? That's not a problem. And, you know, you're not, in most cases, contractually obligated to provide this many minutes a month."

Doing so can become "a really confusing side quest" that's difficult to connect to reality. David emphasizes that SLOs are a sociotechnical construct and because of this, they need people to buy into what you're trying to achieve. 

His experience has shown that it's a lot easier to get people behind simpler SLOs like error rate rather than time-based SLOs.For example, you could set SLOs for the number of 50x that are occurring in your services.

Your service might normally return a 0.1% error rate, but if that number doubles, it can start a conversation. The key behind doing any of this is knowing that you can't do bottom-up with SLOs. You need senior leadership support to make it happen.

If you can't get that buy in and that can be really tough. This could explain the sheer volume of failed SLOs in the industry today.  

You need to invest your time in SLOs, but keep it simple. David recommends to not jump into the whole hype surrounding SLOs from the market.

Here are 10 more takeaways from the show:

1. Understand your observability billing model: Gain a clear understanding of your service provider's billing model to avoid unexpected expenses. This helps in planning and optimizing costs associated with high cardinality data and log retention.

2. Employ cost optimization tactics: tactics like sampling and selective logging can help manage costs without sacrificing the visibility needed for effective monitoring. This is particularly valuable in high-traffic scenarios where full resolution logs are not financially justifiable.

3. Enhance your log retention policies: Use automated tools to manage log retention, ensuring that you're only keeping what's necessary and utilizing cost-effective storage solutions like Amazon S3 for long-term storage.

4. Make querying and data access cost-effective: Look into using tools like Amazon Athena for querying large datasets at a low cost, which enables you to access your logs as needed without incurring high storage and processing fees.

5. Take the time to prioritize your data: Focus on logging and monitoring the most critical aspects of your system to avoid information overload and reduce costs. This involves understanding what data is truly valuable for your specific context.

6. Learn to differentiate status vs. diagnostic information: Status indicators and diagnostic data often give mixed up as solving the same problem. They don't. Simplify your monitoring by providing clear status indicators (red, yellow, green) for quick assessments and reserve detailed diagnostic data for deeper analysis.

7. Effectively leverage the power of metrics: Develop work metrics that reflect the actual performance and health of your platform. This approach helps in quickly identifying issues without sifting through irrelevant data.

8. Simplify your incident management: Aim for a "single pane of glass" dashboard that offers a straightforward view of system health to reduce time spent in identifying and debating the presence and scope of issues.

9. Educate your stakeholders and manage their expectations: Help your various stakeholders understand the cost-benefit analysis behind observability practices including data logging and retention strategies that foster support for cost-effective practices.

10. Go beyond the metrics mindset: Recognize the value of logs and stack traces as critical diagnostic tools that can support your monitoring and alerting. Ensure that your observability strategy integrates these elements effectively for pinpointing issues.

You will not want to miss his insights (and his charismatic humor).

In Episode #30 of the SREpath podcast, David Caudill gives me his take on clearing delusions in observability.

I’m sure your team will appreciate you clearing any accidental misgivings they may have developed about observability 😁

Ash Patel interviews Adriana Villela who is a CNCF ambassador, OpenTelemetry contributor, and senior developer advocate at Lightstep.

Adriana talks about her experiences discovering observability, life as a team leader, and the promise of OpenTelemetry.

She sheds light on the importance of observability practices and the role of OpenTelemetry in standardizing instrumentation collection, enabling organizations to achieve comprehensive observability in complex systems.

Episode Transcript

Ash Patel: Adriana, great to have you. I'm looking forward to learning what you have to teach us about all things SRE, or at least a few interesting things you've been working on recently.

And I know you've been talking about OpenTelemetry quite a bit. So, looking forward to talking about that. But let's do a quick introduction of who you are and what you do.

What do you do SRE-wise in terms of you and your company?

Adriana Villela: I guess I'm in the SRE adjacent space if you will. I guess it is part of SRE.

So I'm a Senior developer advocate at LightStep, which now is part of ServiceNow. So it's actually called ServiceNow Cloud Observability. And Lights or ServiceNow Cloud Observability, I'm, I'm still getting used to the new name is an observability tool. And observability I would say is one of the important aspects of SRE.

What I do as part of my job is to educate folks on observability practices with a focus on OpenTelemetry, which is a great open-source CNCF project that aims to standardize how we collect instrumentation from our systems so that we can achieve those beautiful observability goals.

Ash Patel: OpenTelemetry has got to be one of the hottest projects out there on CNCF right now.

Adriana Villela: Mm hmm. I think it's got the second-highest contributions behind Kubernetes, which is pretty wild when you think about it.

Ash Patel: I was gonna say. I know Kubernetes is number one for obvious reasons.

Actually, not obvious reasons, but you know what I mean. Yeah. Yeah, open telemetry is such an important area.

I say to a lot of people that observability is the foundation practice that any cloud operation needs to have to be effective.

Adriana Villela: Yes, yes, absolutely.

It is such a foundational practice. I think one of my pet peeves, and I had to edit myself when I was introducing myself and talk about SRE because I think a lot of the industry tends to treat observability as a separate thing when in fact it's part of the overall practice.

When you consider the fact that we are in this microservices world, right? I think a lot of organizations have moved away from monoliths and have embraced the microservices model, which, you know, has its ups and downs.

But one of its downs is you have so many different services. And depending on the size of your organization, you can have like ton of microservices interacting with each other in ways that you would never even predict. Right? And so observability helps to untangle those interactions.

All of a sudden, You're like, oh, I see how data flows from point A to point B of my application.

Ash Patel: You mentioned monoliths. What are your thoughts on a lot of companies talking about how they're switching back to monoliths, but I don't know why they would do that. Is it to save money or is it to cut down on their spending on doing things like observability? What are your thoughts on monoliths?

Adriana Villela: I'm kind of happy to hear that because I do feel like a lot of organizations went on this microservices craze because it was a fad.

And so I feel like some C suite executive attended some seminar or whatever where they were talking about microservices and then all of a sudden it's like, Oh, we must do this at, our organization, it's going to solve all of our problems and feel like a lot of companies rushed onto the microservice bandwagon without really thinking about does this actually do what I need it to do?

So I'm kind of happy that some companies are taking a step back to the monolith.

I think what people conflate with monoliths and microservices is that there are still efficiencies to be made when it comes to monoliths where I think if you structure your project properly, you don't have to have some of those huge long build times that you would normally see with your monoliths in the past. And I think that's one of the things that I think folks try to alleviate with the microservices model. Cause it's like, you just change one little thing. You don't have to build the entire frigging project.

But I think if you structure your monolithic app properly, it doesn't have to be painful either.

Ash Patel: Exactly. You don't actually have to do a monolith as a waterfall project either. Mm. Mm. Which I think people construe a few things together. They say, hey, microservices is agile, monoliths are waterfall.

And I'm saying, no. But anyway, we could talk about architectural decisions all night long.

How did you get to working with people in observability in the first place? How did you get into the space in terms of your work history, your personal history with it?

Adriana Villela: So I'll say I got into the observability space.

I caught wind of Charity Major's tweets when I was doing some research for something else. I had like a little side hustle with, with a friend of mine and I don't know, we were just doing some late-night research and then I happened upon one of her tweets, which I don't think even had anything to do with observability.

And then my friend, he started obsessing and reading all her tweets. And then he's like, Hey, you should check out this observability thing. It's really cool. And I'm like, okay, but I don't get what it is. And he tried to explain it to me a bunch and I'm like, I sorta get it, but I don't really get it.

And then like the company where I,

I was out of the time.

I was at the time they were exploring some observability products. They had some microservices and they were like looking into getting into Kubernetes. They, they were like hung up on this one product.

I'm like, you know, we should check out these other products around observability. Me saying this with like a vague notion of what observability was about. And then a couple of months later I found myself working at Tucows where I was managing two teams. It was a platform engineering team where, instead of using Kubernetes, they were using Nomad.

So I had to learn that. But I was also managing an observability team, which was kind of funny because I had a vague notion of observability from my, you know, like, my chats with, with my friend doing, during our side hustle. But also, I was trying to wrap my head around and when I got this job I started thinking I'm like, I'm like leading a team of people.

I better know what observability is because otherwise how am I supposed to like lead an entire organization on this journey, right? So I set out to do what I do best, which is learn the thing that I don't know so I can do my job. And as part of that, I started blogging about it.

And I already had a blog at the time on Medium which was focused on, like, more the Kubernetes, Argo CD side of things. And so I started exploring observability. Blogging about it.

As I learned more, then I was able to start setting a direction for my team. So this team that I inherited It was an observability team.

I'll say this in air quotes. And really what the team did was manage a set of, of open source tools in house. But at the same time, the organization was sending its observability data to a SaaS vendor. So I'm sitting here thinking, okay, we're already paying somebody to receive our data. Why the hell is this team managing a set of tools in house?

At this point, I had a better idea of what observability was all about, and I said, let's pivot this team completely. I want to focus on observability practices and not on observability tooling. Leave the tooling to the experts. So then I sought to educate my team on like, what are proper observability practices?

And that's around the time that I got wind of open telemetry. Started looking at it a little bit closer. So the company before I was there was like that, I think that was the first time I'd heard of open telemetry, but it's. It seemed like, I don't know, it, it, it seemed like a very small thing. And then when I, when I went to my job where I was managing this observability team It was something that I started looking at more closely as I started reading blog posts about it and whatnot and educating myself and I'm like, huh, I feel like this is the future of observability.

So I started pushing really hard for the organization to adopt OpenTelemetry, which was very difficult because I found myself in a position, this was 2021, where OpenTelemetry, like, the tracing specification was not yet in general availability. And this was like summer, fall of 2021. And I'm sitting there thinking, no, no, this is going to be the next big thing.

I assure you it's going to be the next big thing. But how can I convince these folks that OpenTelemetry is the next big thing when tracing isn't even in general availability?

So I thought, I, I need to like, calm these folks concerns. So what I did, which is what I do best, I think, in my which I've been able to leverage in my capacity of as a developer advocate now, which is like make connections with people.

And so I had made some friends in the open telemetry space.

I'd talk to a couple of folks from two different vendors that we were also considering at the same time, because we were, you know, we had the SaaS vendor, but I'm like, well, is, is this the right fit for us? We should explore other vendors.

I reached out to these folks who were, who worked for other vendors, but they put on their OpenTelemetry hats for this, right? They said, please come speak with us on a unified front under the OpenTelemetry umbrella to answer questions and concerns for folks in the organization who are hesitant about adopting OpenTelemetry.

And to my surprise, they went along with my idea, I, I honestly thought, this is a total long shot, this is not happening. But they went with it and I remember telling my team, we need some seed questions in case this thing is going to be crickets, otherwise I'm going to look like total ass.

So we came up with some seed questions, but guess what?

For the whole hour, the folks from the dev teams, they had one question after another, and we didn't even have to ask any of our seed questions, which I was very relieved to hear. To me, it showed that there was an interest in what was being done.

There was a genuine interest in OpenTelemetry, and I think at the end of the day, mission accomplished. We managed to quell those fears as a result. And so the company was a lot more comfortable in terms of adopting OpenTelemetry. So that was my original foray into it. And then I ended up getting hired by one of the observability vendors as a developer advocate because of my blog posts around observability and OpenTelemetry, my explorations.

Which for me, honestly, is, is something that I've dreamed of.

I love educating folks through my blogging and I love digging into technical topics. So I feel like it was the best use of my, of my skills. And it's been great because also like this job has been my first opportunity to contribute to open source, which I'd never done before.

And it's scary if you've never contributed to open source cause you're being vulnerable. You're putting yourself out there. It is terrifying. Like what if someone says your thing is crap, but I've been lucky cause in the OpenTelemetry community, no one goes and says your PR is crap.

They always have very thoughtful suggestions always useful suggestions. So, even though I'm scared every single time, my fears are always put at ease because the folks are just genuinely nice about it.

That's been my journey into OpenTelemetry and observability.

Ash Patel: That's the great thing about open source communities. It's psychologically safe in that everyone's trying to learn and trying to improve what they're working on and they'll share ideas, but in a constructive way.

Adriana Villela: Yes, absolutely.

Ash Patel: And I said that because I'm calling out to people to try and do the same as Adriana and contribute to open source if you can, and if you have the time to do it.

Adriana Villela: I do want to mention one thing actually on that same vein, like when, when I was at Tucows, I myself wasn't practicing what I was preaching, but I did encourage my team as a whole, like my observability team, like when they were running into issues with, Oh, I'm waiting on this feature for OpenTelemetry and blah, blah, blah.

I'm like, dude, just try to work on it if you can, if you feel like you, you can like put in a fix, do it. It took me a little bit longer and, you know, switching roles to like muster the courage to do it myself. But I, I totally agree with you.

I think it's so important for us to contribute to open source.

And for our organizations to encourage contributing to open source, especially if we are consumers of that open source software.

Ash Patel: I have a question about your thoughts on how organizations that are non tech native feel about open source. We'll get to that in a second, but I want to actually go back one step and talk about how OpenTelemetry fits into the whole observability area of software.

Adriana Villela: I think the important distinction to make is that open telemetry is not equal to observability. Open telemetry is one of those tools that enables observability. And honestly, like, you know, before open telemetry, there were things like open tracing and open census. OpenTelemetry is at the forefront now because it is the one that all of the big vendors have rallied behind and, and for a little bit of history, open source projects like OpenCensus and OpenTracing, they actually merged to form OpenTelemetry.

But I wouldn't be surprised if, like, somebody for funsies decided to, like, create their own tracing library or whatever. But at the end of the day, like, observability is about, you have that information. What do you do with the information about your system so that you can follow the breadcrumbs to answer the question, why is this happening?

OpenTelemetry enables that to happen, right? It's the thing that supplies that information.

I would encourage folks to use OpenTelemetry as the standard for that sort of thing because if we get to the point where, like, we're all using different tools, we end up losing out on the standardization and the work that has gone into OpenTelemetry, if that makes any sense.

But a tool like this is only as good as the user base. And if we don't have that standardization, if we don't rally as a community to standardize around how we get our observability how we emit the observability data then we end up with all these different factions.

I don't want to be super prescriptive about it, but I think it ends up losing its worth if, if we go all like everyone on their own, do whatever they want for these sorts of things. Right?

Ash Patel: Yeah, I would be totally against it myself.

If I were managing a team or multiple teams, which like I have in the past, I always used to tell my teams, don't reinvent the wheel. Look at what's available out there. And try and use that first. And then if we have to make a solution from scratch, then we'll look at it then. But start learning from what everyone else has done, and then we can work with that.

And this is how OpenTelemetry is doing. It's giving you all this capability from the get go.

Adriana Villela: Yeah, totally. And the other thing that I want to emphasize is that because, you know, all the major observability vendors are behind it. It means that these vendors are not competing on the data that is ingested. They're just competing on how they're rendering the data.

We all speak the same language. The data that we send to whatever vendor, vendor X and vendor Y, it's the same data. Vendor X will probably render it differently from vendor Y, and so as a consumer. It's up to you to decide which one is more useful for me for my troubleshooting purposes.

Ash Patel: You actually have a bit of experience with running teams. So, I'd be interested to know in terms of your observability team, how you managed them, how you hired them, how you managed the performance. .

Adriana Villela: When I joined Tucows, I was lucky enough that there were some folks on both of my teams already, and it was, like I said, it was managing two teams, and they were similar but different.

And so there were already folks on those teams, but I needed to also hire a headcount. At that point, I already knew what I wanted out of my teams. And in particular, I'll talk about the observability team.

I wanted to hire somebody who either appreciated what observability had to offer either as an expert in observability or somebody who maybe they didn't come from that observability background, maybe they came from a more traditional monitoring background, but saw the importance of observability and were willing to do a paradigm shift.

It was a weird role because it was a technical role that wasn't terribly hands on. So my team our mandate was to be able to educate folks at the organization on observability practices.

We taught them the overarching best practices, but also we assisted them as far as unpacking some of the gnarly bits of OpenTelemetry so that they could instrument their code.

Now that was very different from actually instrumenting the code themselves, because the slippery slope that you can get into by having an observability team, and it wasn't just my team you can end up getting asked to instrument your developer's code. Which is a very bizarre ask, because I know nothing about your code.

I don't know what's important to you for debugging purposes. I don't know what the code does in general. I might have a vague idea from a high level. But realistically speaking, it is not my code. I don't have that kind of vested interest. I will help you in terms of, like, Hey, these might be the things that you might want to instrument.

And I can provide some assistance on how to instrument in whatever language. But as far as instrumenting your own code, that was something that was on purpose outside of the realm of my team, because we wanted to encourage developers to instrument their own code.

So hiring for the team was... was tricky in that sense, because I had a lot of candidates applying for the position who came from like an infrastructure as code background and were super gung ho on managing tools and I'm like that's not exactly what we do.

I wanted somebody who had an appreciation from an infrastructure as code background, somebody who had an appreciation from like an SRE type of background, you know, someone who felt the pain so that they could then go on and advocate for those observability practices.

In a way, I guess I was hiring for an internal team of advocates in the organization. So it makes it a very tricky type of role to hire for because it's a more squishy role that's not completely hands on, but I need you to be an engineer. I need you to understand the pain around this sort of thing.

That was on the hiring side, but then for the folks who I had inherited was a little bit tricky too, because all of a sudden I come in, I'm saying we're pivoting the team's mandate. That went well for some folks and for other folks they're like, but I want to just do infrastructure as code.

That is my thing. I enjoy it. So I had to have some hard conversations with some of my team members saying, look, I understand that this is your preference and that I have gone ahead and shifted what our team's priorities are.

And I am okay. If we don't align, if you want to go look elsewhere in the organization, I will support you in that search, to ensure that you find something that's more in line with your career goals in the organization.

It's kind of a tough conversation to have because it means that person admitting, yeah, I don't really want to do this and me saying I'm okay letting you go because I think at the end of the day, when you're managing a team you kind of have to all toe the party line together.

You have to be aligned as to what the overarching team goals are. It's okay to question certain things. I think it's always important to question direction. Hey, maybe you should be doing X instead of Y. That's totally fine, but to have folks who are constantly questioning what you're doing as if you're butting heads because you don't see eye to eye on the philosophies and where the team is going, I think it ends up being very unhealthy for the team, and so you're better off parting ways because you're better off parting ways.

All it takes is one person to start sowing dissent on the team to ruin the team cohesion. And then that becomes a very toxic work environment for everybody, right? Because then you start creating factions and people aren't working together, they're working against each other, and that's not something that you can have.

So, I mean, at the end of the day, being aligned on what our mandate was was very important, whether it was somebody already existing on the team or somebody being hired into the team.

Ash Patel: That was very insightful about a lot of things that you can apply beyond what your team issue was in terms of there's so many teams that are facing this exact thing. There are people already there, there are people that you need, and you need to all align on whatever mandate you as a manager have to create to align with goals.

Coming back to now, what you're doing right now, you'd be having some interesting conversations with a lot of people every day about what their challenges are. So what kind of person typically reaches out to you or you end up having conversations with around observability and OpenTelemetry?

Adriana Villela: So we talked to, I'd say, a lot of people who are new to OpenTelemetry, who are just trying to figure this thing out. I had a conversation recently with somebody who was just starting to wrap their head around it. They knew that observability is important. They knew that OpenTelemetry is important, but they weren't sure how to get started.

So one of the things that I like to do is, I'll give them a little overview and I'm happy to, anyone who wants to talk about observability, OpenTelemetry, like reach out to me. Let's have that conversation. You know, I work for a vendor, but I'm not here to sell you a vendor product.

I'm here to sell you on observability first and foremost. I'm here to sell you on observability, OpenTelemetry, the power of that.

And I think that's the most important thing we, we need to do right by, by our systems and make sure that we have the tools necessary to be able to troubleshoot in production.

So having those conversations with folks to understand what their needs are, what their difficulties are. I can come because I have the experience of having been on the other side of looking at a couple of different vendors and really just focusing on observability practices.

I can have that conversation with folks around what do you need to do in order to be successful in terms of bringing observability and open telemetry to your organization? Just, just like wrapping your mind around observability. I think with a lot of things in tech it's very easy to get into various anti patterns.

We saw that with the DevOps movement. I think one of my biggest pet peeves is still the idea of, like, DevOps as a job rather than as a practice because I think we, we ended up moving away from what DevOps was trying to promise.

And I feel like, you know, it, it can be a similar slippery slope with observability where I have used OpenTelemetry, therefore my system is observable, and I've encountered situations where you can instrument till you're blue in the face, but if you're instrumenting the wrong thing that's useless.

Recently I'm one of the co leads of the Open Telemetry End User Working Group, and we had one person come in and present to us, Hazel Weekly, and she was talking about a story where one of the organizations where she was in.

They instrumented their code. They over instrumented their code to the point where like, great, we've got open telemetry. It is useless. We have no idea what is going on here. It's not giving us any meaningful information. So she actually had to spend some time unraveling that big mess and trying to get it to a state where the instrumentation was more meaningful.

We need to be aware of the initial lack of knowledge around both observability and open telemetry - both from a I don't know where to get started, to, Oh, I've come in guns blazing and I've over instrumented and it's still not helping me.

We really need to take a step back and really, really focus on the practices.

Ash Patel: Okay Adriana. Where do you see the category of observability heading in the next two, three, five years?

Adriana Villela: This is my, my hope and I, I, I hope that it, it follows my hope. So, in terms of observability, I hope to see more integrated with the SRE practice and not so much of this separate adjacent practice. We need to start talking about it from a more unified standpoint. The other thing that I wanted to mention is that I really want us to look at observability as a team sport.

And not as just the thing that is the burden of the SRE. Because when you think about it, yes, the SRE is probably going to be your first line of defense when there's an issue happening in production. But, let's think back to who instrumented that code in the first place, the developer.

So the developer, you would expect, would want to have a vested interest in making sure that the instrument, the code properly so that our poor friend, the SRE has an easier time of troubleshooting issues in production. But I think we can take a step further and say, the developers should take advantage of these observability platforms to even help troubleshoot their own code while they're debugging, while they're unit testing.

But let's take it a step further than that, which is before it goes into production, we send it over for testing, right, to our friends, the QAs and this idea of: "why don't we empower QAs to be able to use observability data to make sense of what is happening in the application when they're testing the code so that when there is an error, then the QA can go to the developer and say, "Hey, there's bug and I can tell you where it is".

Or if there's a bug and the QA can't tell the developer where it is, that's fine too because they can say to the developer, " Hey, your system's not emitting enough information. I don't know what the problem is. I think you're missing some instrumentation. Go back. And then the other piece of that puzzle is leveraging those traces that you already have in place and creating trace based tests.

Which is basically, as the name implies, you use traces to create automated integration tests, and there are some tools out there that already do that. The most talked about one in the space is TraceTest. It's an open source tool that facilitates trace based testing.

There's another one called Maliby: it's more of a JavaScript based trace based testing tool, and I don't think it's been maintained for the last little while. And then Helios I believe they were looking at trace based testing definitely last year. I don't know where things are at now, but these are three different types of tools that you might want to look at if you want to explore trace based testing.

I think it's a really, really cool concept and that's something like I want to see more. I want to see us talking more about trace based testing in the future because I think trace based testing can give us extra superpowers in terms of achieving more reliable systems especially because now you're not having to worry about wrangling different testing frameworks to be able to do end to end tests of your systems, especially if you know, you're dealing with a system where the microservice is written in different languages.

Now trace based testing is like your single unified language for writing your your integration tests, which is awesome because the unified language itself is the trace.

So again, we need to make sure that we have good quality traces so that we can leverage those traces to really make them work for us.

Ash Patel: It's interesting you're talking about tracing because I was crossing University Avenue in Toronto and I overheard two developers talking about tracing and testing. It was a little haphazard.

They were talking about trace testing, but they were very confused about what they had just heard in a meeting prior. So there's a lot of, it was surprising to me because then I stopped them. I said, Hey, what do you guys know about observability? And they said. Not much. And I asked them, do you need to learn more?

They said, yes, please. Well, they didn't say it like two boys in a Charles Dickens novel, but you get the idea that they were pretty keen on figuring out what is this thing that's now suddenly become part of our responsibility in this increasing you build it, you run it model.

It's interesting that these kinds of things are being brought out. Trace testing was just mentioned. I'm gonna say one thing, and I'm not sure whether you're gonna agree with me. It's that all this needs to be brought together cohesively, so that when it's shared with people, they can see this is where this fits into the observability practice.

Where do you see SRE heading in the same way in the future? Where do you see it heading in the next few years?

Adriana Villela: I think for SRE, I'd like to see more conversations happening around things like self service provisioning.

Actually Anna and I have a talk coming up at KubeCon where that is our subject matter. The idea that there's the rise of platform engineering, which this may or may not be a popular opinion, but in, in the definition that we're using, we see platform engineering as an extension of SRE.

Where you're not only concerned about the external customer, but the internal customer, i. e. the developer. And this idea that in platform engineering, as a developer, it's so annoying when you're trying to get your job done, and you need certain tools to get your job done, and you gotta wait on this platform team to provision stuff for you.

And so you're sitting there waiting, waiting, waiting, thinking, Frigman, why can't I just use, like my own Kubernetes cluster? I don't want to use this stupid Kubernetes cluster that's, you know, in the cloud and blah, blah, blah. And, like, this is taking forever. I need to get my job done. And by having self service tooling available, it means that developers can then provision the resources that they need to get their jobs done on demand.

These resources are provided by the platform team. However, the platform team automates the provisioning of those resources, right? They're not having to like, you know, execute on these requests manually but on top of that, this whole idea of being packaged, so that it's compliant within the organization.

So security concerns are addressed. It's using standard configurations. So you're not deviating from what the standards within the organization are. So the thing that's delivered to you that you're able to use as part of your tooling is wrapped in a nice little neat bow.

It's got all the configurations you need and you don't have to wait days and days and days to get the thing that you need to be able to do your job. So having the self service provisioning.

And then the other thing that is important that I think follows in with that is whenever we go from one organization to another, I can't tell you how many times you figure out this automation, then you go to the next organization.

Oh crap. I got to do this exact same automation. Like, come on. We keep reinventing the same automations over and over and over again. Wouldn't it be nice if we just had this marketplace of standardized automations where we can just pick and choose, right?

And then make some customizations here and there, but we're not having to like reinvent these friggin automations every single time for our SRE practice.

The last one is policy as a service where I feel like we're always at odds with InfoSec. InfoSec has a very tough job because they're protecting us from all these attacks in our systems making sure that everything's safe.

And I feel like there's... And they can't even talk about their triumphs, right, because if you talk about, like, oh, I, I blocked this attack from happening, I blocked that attack from happening, that already, like, that's not secure by nature. So, you, you can't even, like, you can't even gloat about your accomplishments but to be able to work together with Infosec so that we have certain policies in place that are codified, and again, so we don't have to go through these long drawn out processes of, Oh, I need access to XYZ and have like all these approvals.

Well, yes, we've already had the pre approvals because we already have all of these policies, like pre packaged, pre approved. Everything's compliant. And so we're not mad at our InfoSec friends. They're just trying to do their jobs and we're getting these compliant things. These things that are properly secured. Our Kubernetes clusters that are properly secured and are cloud resources that have the right permissions without having to like wait forever and ever and do this song and dance with InfoSec.

Those are the areas that I, I would like to see like SRE going towards.

Ash Patel: We can only wait and see.

Adriana Villela: I know, right? Fingers crossed.

Ash Patel: What advice would you give to SREs who are dealing with a lot of challenges right now to help them in their working lives?

Adriana Villela: I would say never stop being curious. Whether or not you're an SRE, I think that will take you very far in your career. Because by being curious, by always looking to do things a better way. It helps us grow in our practice. I think it allows us to innovate. I think we risk stagnation when we stop caring about innovating in our technologies, in our jobs.

So let's continue to be curious.

Ash Patel: There we have it. Never stop being curious. Thank you so much for joining me, Adriana.

Adriana Villela: Thanks for having me, Ash.

Before we get into tracing, let's define observability.

What is observability?

Observability is a comprehensive means of gaining data on how software services perform in production.

This data gives you a picture of the health and performance of individual services, as well as the cloud infrastructure that supports them.

It can be broken down into 3 actions: logging, tracing, and monitoring. Our focus in this article will be on tracing.

What is tracing?

Tracing is an action that tracks a request from initiation to completion within a microservices architecture.

It usually starts when a user or service starts a request which moves along a chain of interconnected services needed to fulfill the request.

With tracing enabled, software engineers and SREs can pinpoint any issues within the chain of requests among the various involved services.

Where Jaeger fits into the tracing paradigm

What is Jaeger tracing?

Jaeger is an open-source tracing tool that allows engineers to track request performance and issues among 10s, 100s, and even 1000s of services and their dependencies. It collects tracing data and then populates Grafana dashboards.

The key benefit of this is that it highlights downtime/load-time risks and errors. This makes it an essential component of a strong observability practice.

Jaeger's origin story

Jaeger was created in 2015 by an engineer at Uber, Yuri Shkuro, who wanted to help engineers work out where issues were popping up. This emerged as a critical need at Uber over time.

The Uber app may seem simple to its end users, but behind the facade runs a complex network of microservices. Many of these services depend on other services and their sub-services.

Weaknesses in the service chain can risk the whole user request falling apart i.e. no ride.

In business terms, Uber risks losing ride fares at a large scale if one or some component services fail or slow down.

“In deep distributed systems, finding what is broken and where is often more difficult than why“

— Yuri Skhuro, Founder & Maintainer, CNCF Jaeger

Jaeger tracing helps engineers find out what services are experiencing issues and where. That way, they can fix small issues before they snowball into serious problems or crises.

Do your observability needs justify using Jaeger?

You might be wondering whether you even need Jaeger. After all, your use case might not be as complex as Uber’s. Jaeger was designed to make sense of a complex web of services and up to millions of daily requests.

Tracing is not an absolute must-have for simpler software architectures. However, it is useful for finding bottlenecks if you have more than a handful of services. Having more than 10 services is a fair threshold of need.

Would the following situation ever pose a problem for your software? Your application has more than 10 services and suddenly gets a traffic spike. A large volume of requests has not been completed.

How will you find the culprit fast enough to fix the issue?

If this compels your need for tracing, let's explore how Jaeger tracing works from a high-level view:

How Jaeger tracing works

Step 1

Jaeger Agent gathers “span data” by sampling parts of UDP packets transmitted by microservices

Step 2

Data (service name, start time, duration) gets sent on to the Collector

Step 3

Collector sends data to 2 places: Analytics and Visual Dashboard

Et voilà!

Now let's explore how to install Jaeger on a Kubernetes cluster

How to setup Jaeger

2 ways to install Jaeger on Kubernetes

I will assume that you know how Kubernetes clusters are structured in terms of containers, nodes, pods, sidecars, etc.

Jaeger Agent can run on a Kubernetes cluster in two distinct ways: as a daemon or sidecar. Let’s compare both of them.

Setup Jaeger as a daemonset

Mechanism: Jaeger Agent runs as a pod and collects data from all other pods within the same node

Useful for: single tenant or non-production clusters

Benefits: lower memory overhead, more straightforward setup

Risk: security risk if deployed on a multi-tenant cluster

LEARN BY DOING: simple Jaeger setup tutorial via Digital Ocean

Setup Jaeger as a sidecar

Mechanism: Jaeger Agent runs as a container alongside the service container within every pod

Useful for: multi-tenant clusters, public cloud clusters

Benefits: granular control, higher security potential

Risk: more DevOps supervision required

LEARN BY DOING: deploy Jaeger as a sidecar via Jaeger's Github

Remember from earlier that Jaeger samples parts of UDP packets transmitted by services?

There are 2 sampling methods for sampling UDP packets: heads-based sampling and tails-based sampling. Each has its benefits and downsides. Let’s explore:

Heads-based sampling

Also known as upfront sampling

Mechanism: sampling decision is made before request completion

Useful for: high-throughput use cases, looking at aggregated data

Benefits: cheaper sampling method – lower network and storage overhead

Risk: potential to miss outlier requests due to less than 100% sampling

Work required: easy setup, supported by Jaeger SDKs

Configuration notes: sampling based on flip-of-coin or until a certain rate is achieved

Tails-based sampling

Also known as response sampling

Mechanism: sampling decision is made after the request has been completed

Useful for: catching anomalies in latency, failed requests

Benefits: more intelligent approach to looking at request data

Risk: temporary storage for all traces – more infra overhead, a single node only

Work required: extra work – connect to a tool that supports tail-based sampling like Lightstep

Config notes: sampling based on latency criteria and tags

Now that you've picked your sampling method, you will also need to consider that Jaeger's collector has a finite data capacity.

Prevent Jaeger's collector from getting clogged

Jaeger’s collector holds data temporarily before it writes onto a database. The visual dashboard then queries this database. But the collector can get clogged if the database can’t write fast enough during high-traffic situations.

Problem:

• Collector’s temporary storage model becomes problematic during traffic spikes

• Some data gets dropped so the collector can stay afloat from the flood of incoming request data

• Your tracing may look patchy in areas because of the gaps in sampling data

• Risk of missing failed or problematic requests if they were in the sampling that gets dropped

Solution:

• Consider asynchronous span ingestion technique to solve this problem

• This means adding a few components between your collector and database:

  1. Apache Kafka – real-time data streaming at scale

  2. Apache Flink – processes Kafka data asynchronously

  3. 2 jaeger components – jaeger-ingester and jaeger-indexer – push Flink output to storage

Once these components are in place, the collector will be less likely to get overloaded and dump data.

Implementation reading:

These links – access them in order – might help you get started with your implementation:

Youtube – Jaeger straight-to-DB vs asynch write method

Youtube – Apache Kafka videos by Confluent

Practical overview (with example) of Apache Flink

Wrapping up

This concludes our article on Jaeger and the promise it holds for distributed tracing of microservices, as well as the wider observability apparatus.