I recently worked on a new ecommerce project which was using my favorite OpenTelemetry vendor Honeycomb.io, and the stack running in NodeJS/NextJS on a combination of Azure Containers Apps and Azure Functions. Unfortunately, when inspecting the state of the traces I found the following issues: A mix of Sentry tracing/samplers and default OpenTelemetry setup Frontend and backend traces are not linked Traces are missing parent spans/traces None of the Azure Functions are instrumented Luckily I’ve now done multiple projects where I fixed most of these issues, or know how to solve them (or so I thought).
It turns out NODE_OPTIONS does not support most options in Azure functions. I’m currently working on a migration from AWS to Azure. While adding observability to Azure Functions (we use NODE_OPTIONS to bootstrap OpenTelemetry so it monkeypatches http/https properly) our functions stopped working. Logs in Azure only showed it was preparing to start up and nothing else. Using func start --verbose locally it finally said ‘unsupported option for NODE_OPTIONS’. The Azure Function documentation shows you can use languageWorkers__node__arguments instead.
I’ve been stuck multiple times in OpenTelemetry hell where things didn’t work or suddenly break. At some point our HTTP client instrumentation traces (which set the incoming root span on our GraphQL gateway) randomly did or did not include specific manually instrumented span attributes on different environments. So it would work locally and on test, but sometimes fail on acceptance / production. We use these for dashboards, triggers and general querying so we had to find a fix.