Sentry stepping into the middle of an app’s outgoing HTTP request to an external API and injecting the baggage header

Overview

Turning Sentry on doesn’t just bring error collection with it. Distributed tracing comes on as well, and from that point every outgoing request carries trace headers.

There is no such line in your app code. Nothing shows up in the logs either. So when something goes wrong, no amount of reading the code will reveal it.

This article covers what those headers are, how to see them, and how to narrow the range they attach to.

Plenty of places use Sentry to track errors and bugs. It is a tool that collects exceptions thrown in an application and shows you when, where and how often they happened. A few lines are all it takes to wire up, and since it works regardless of language or framework, it is common to install it by default.

The part that attaches headers is not error collection. When you turn Sentry on, distributed tracing comes along with it. That is the part putting a header on every outgoing request.

What the feature is for

This header exists for distributed tracing. When a single request passes through several services, it is there so that journey can be seen as one.

Say service A calls B and B calls C. Without the header, all three leave separate records. You may know that A’s response was slow, but you cannot tell that C was the cause. When the header carries a trace_id, the records from all three merge into the same trace.

The Sentry docs describe the purpose like this.

track software performance and measure throughput & latency, while seeing the impact of errors across multiple systems

The two headers play different roles.

  • sentry-trace points at a position. It takes the form <trace_id>-<span_id>-. It tells you which point of which trace this request is.
  • baggage carries context. It is a standard header defined by the W3C, and Sentry puts the values used for dynamic sampling into it. Things like sentry-environment and sentry-release.
1sentry-trace: 41c74c2ea9f2bfb184f86939de5b97aa-399b3e5cc8b83494-1
2baggage:      sentry-trace_id=41c74c2ea9f2bfb184f86939de5b97aa,
3              sentry-environment=production,
4              sentry-release=1.4.2,
5              sentry-sample_rate=0.2

Passing the sampling decision made by the first service down to the ones after it is also baggage’s job.

a sampling decision is made in the originating service and passed to subsequent services

If a trace discarded up front were recorded further down, you would be left with half a record. So the decision is carried along in the value.

Isn’t it unnecessary with a single server?

It is meaningless without a receiving side. The receiving side here means a service that sends data to the same Sentry organization.

That said, there are cases where it is useful even with one server. If you have put Sentry in the browser, your frontend and backend are already two. Work that passes through a queue or a worker is the same.

On the other hand, it is almost never useful on requests going out to someone else’s service. The public data portal has no reason to receive our trace_id. Yet the default for the server SDK is to attach it to every outgoing request. tracePropagationTargets, covered later, is the tool for narrowing that default.

What goes out

To see what goes out, build the receiving side yourself. Spin up a throwaway server, send a request to it from a process with Sentry enabled, and print the headers it receives.

 1// print-headers.js
 2const Sentry = require('@sentry/node');
 3const http = require('http');
 4
 5Sentry.init({
 6  dsn: process.env.SENTRY_DSN,
 7  environment: 'production',
 8  release: '1.4.2',
 9  tracesSampleRate: 0,
10});
11
12// A throwaway server that prints whatever reaches it.
13const server = http.createServer((req, res) => {
14  console.log(req.headers);
15  res.end('ok');
16  server.close();
17});
18
19server.listen(0, '127.0.0.1', () => {
20  const { port } = server.address();
21  fetch(`http://127.0.0.1:${port}/`, { headers: { Accept: 'application/json' } });
22});

Run it with node print-headers.js and you get this.

1accept: application/json
2
3sentry-trace: 6406c1516309404d89d3d094f704f600-8cb266148716b50a
4baggage: sentry-environment=production,
5         sentry-release=<release>,
6         sentry-public_key=<public key>,
7         sentry-trace_id=6406c1516309404d89d3d094f704f600,
8         sentry-org_id=<org id>,
9         sentry-sample_rand=0.730917246418646

The only header our code passed was Accept. The two lines below it were added by Sentry. Sentry wraps the HTTP layer during initialization and attaches trace headers to every request that goes out afterwards. That line appears nowhere in the app code.

How it gets attached

There is no line in the code that adds a header, yet the header goes out. The mechanism that makes this possible differs by platform. Comparing three of them shows why this kind of thing is particularly hard to find on Node.

Node.js and NestJS — it swaps the module out

The Sentry Node SDK hooks the module loader. For CommonJS that is require-in-the-middle, and for ESM it is import-in-the-middle. The official docs put it this way.

all packages are wrapped under the hood by import-in-the-middle to aid instrumenting them

So when your code imports http, what comes back is not the original but the http that Sentry has wrapped. The call site is unchanged, but http.request is already a different function. The header is attached inside it.

This is why you are told to put Sentry.init() before any other import. A module that is already loaded cannot be caught by the hook. Set registerEsmLoaderHooks to false and the tracing instrumentation goes off along with the hooks.

NestJS works the same way. You put Sentry.init() in instrument.ts and import it first at the top of main.ts. A comment in the docs nails down that order.

1// Import this first!
2import "./instrument";
3
4// Now import other modules
5import { NestFactory } from "@nestjs/core";

There is no registration point anywhere in the code. That is why reading the code never reveals the header.

Java — framework interceptors or bytecode injection

The default approach is to slot into the place the framework already provides. With Spring and Spring Boot, enabling the integration attaches the headers to outgoing requests automatically.

The Java SDK will attach the sentry-trace and baggage headers to all outgoing requests by default.

Not every client is automatic, though. For OkHttp you add the interceptor yourself.

1private val client = OkHttpClient.Builder()
2  .addInterceptor(SentryOkHttpInterceptor())
3  .eventListener(SentryOkHttpEventListener())
4  .build()

There is also a path that leaves your code untouched: attaching sentry-opentelemetry-agent with -javaagent.

1java -javaagent:/path/to/sentry-opentelemetry-agent.jar -jar app.jar

The docs describe what the agent does like this.

dynamically injects bytecode into your application / Inject tracing information into outgoing requests and produced messages

With the agent, no trace is left in the code, just as with Node. Instead, -javaagent is visible in the launch command.

.NET — it inserts a handler into the HttpClient pipeline

A DelegatingHandler called SentryHttpMessageHandler is placed into the HttpClient handler chain. The header is attached as the request passes through that handler.

DI takes care of the inserting. The SDK registers an IHttpMessageHandlerBuilderFilter implementation as a singleton.

1// Sentry.Extensions.Logging/Extensions/DependencyInjection/ServiceCollectionExtensions.cs
2services.AddSingleton<IHttpMessageHandlerBuilderFilter, SentryHttpMessageHandlerBuilderFilter>();

That filter puts the handler into every HttpClient created through IHttpClientFactory.

1// Injects Sentry's HTTP handler into HttpClientFactory
2handlerBuilder.AdditionalHandlers.Add(
3    new SentryHttpMessageHandler(hub)
4);

So a client obtained through IHttpClientFactory gets it automatically, while one you build yourself with new HttpClient() does not. You can turn it off with the DisableSentryHttpMessageHandler option.

In summary

Only on Node does the standard module itself get replaced. In the others, some trace is left somewhere that someone put it into the pipeline. A dependency, a DI registration, a launch command — there is always a thread to follow.

The mechanism differs but the result is the same. sentry-trace and baggage ride along on outgoing requests. And all three can narrow the targets with tracePropagationTargets.

References

It attaches even with tracing off

There is one more thing worth noting. This header is attached even when tracesSampleRate is 0. It marks the trace as excluded from sampling and propagates it anyway. The assumption that it doesn’t matter because tracing is off is wrong.

Narrowing the targets

Turning Sentry off is not the answer. Narrow the targets that get trace headers instead. The tracePropagationTargets option does that job.

The official docs state the server-side default like this.

On the server, all outgoing requests will be propagated by default.

Give it a list and the headers attach only to matching addresses. Put in your own domain and localhost and you’re done.

 1Sentry.init({
 2  dsn: process.env.SENTRY_DSN,
 3  environment: 'production',
 4  release: '1.4.2',
 5  tracesSampleRate: 0,
 6  // Without this the SDK attaches trace headers to every outgoing request.
 7  tracePropagationTargets: [
 8    /^https?:\/\/(localhost|127\.0\.0\.1)(:\d+)?\//,
 9    /^https?:\/\/([a-z0-9-]+\.)*plzhans\.com(:\d+)?\//,
10  ],
11});

Anchor your regular expressions at both ends. Leave them loose and an address that merely brushes past a fragment will match. Something like https://example.com/?ref=plzhans.com does exactly that.

Once applied, check again with the print-headers.js from earlier. Send to an address that isn’t in the target list and sentry-trace and baggage are gone.

Error collection is unaffected. This option deals only with trace headers, so captureException keeps working as before.

Wrapping up

Instrumentation tools change your requests quietly. The code shows one header line while three go out on the wire. That fact is not in the logs, so no amount of reading the code will show it. When you want to know what goes out, building the receiving side yourself and printing it is the fastest way.

If you turn Sentry on for a server that calls external APIs, it is better to configure tracePropagationTargets along with it. It prevents incidents like this one, and it also stops your release version and commit hash from going out to other people’s servers.

There is a case where an external API rejected requests outright because of this header. I’ve written it up in the article below.