Description
The transport/http
package by default wraps the HTTP transport with an OpenTelemetry ochttp.Transport
. That transport wrapper will record HTTP traces for a subset of all requests if OpenCensus tracing is enabled.
However, it is misconfigured. The Transport
type has this field, which is unset google-api-go-client
:
// NameFromRequest holds the function to use for generating the span name
// from the information found in the outgoing HTTP Request. By default the
// name equals the URL Path.
FormatSpanName func(*http.Request) string
The default is for the transport to name each span after the request URL path. In practice, this means a proliferation of differently named spans. For example, if you use the Google Cloud Storage library, you will end up with a span for every file you access. This is not how spans are supposed to be named - they should be named after a function or API endpoint, perhaps including some small, finite set of user-specified options.
This becomes pathological if you actually record the traces somewhere. For instance, if you have the OpenCensus zpages
debug endpoints enabled (common in production systems), a sample of those traces will be stored in a local store. While old traces for a given span are purged, the spans themselves are never purged, and so a running process will accumulate traces indefinitely for every HTTP request path made by google-api-go-client
. Loading the OpenCensus tracez
page in such cases is pretty funny: a production service of mine had accumulated several gigabytes of traces, and its tracez
endpoint served so much HTML that it crashed my browser.
I think the fix here is to specify a FormatSpanName
function when setting up the ochttp.Transport
. I'm not sure how exactly the spans should be named - probably in some service-specific way - but an immediate remedy would be to give all requests the same span name (google-api-go-client
?).