Skip to main content

3.3. Signals in the Sauce

A recipe is only as good as its ingredients—and so are your traces. In this lab, you'll extract ingredient names as span attributes for the catalog service. The result? Traces that are flavorful and easy to follow.

Extracting attributes​

We currently receive spans with paths in the name like GET /api/ingredients/pineapple from the catalog service. For this challenge, we want to extract the ingredient portion of the URL to a span attribute named ingredient.

Documentation Reference

The documentation for the OpenTelemetry Collector span processor may prove useful for this challenge, but this doesn't discount the use of other components.

  1. Edit the lab_receive_telemetry pipeline and add your new component.

  2. Create your span processor component

    Need a hint?

    Check out this example from the docs. By crafting a regular expression for the span names, we can extract the ingredient as an attribute and remove it from the name (reducing cardinality)

    Tip: regex101.com is a great way to test regular expressions interactively (use the Golang flavor).

    Regex making you mad?

    It happens to the best of us. Here is a pattern that will capture the ingredient from the URL:

    rules = [`^GET \/api\/ingredients\/(?P<ingredient>.+)$`]

    Note that using `backticks` allows us to define string literals in Alloy. This avoids the backslashes in our regular expression from being interpreted by Alloy.

  3. Save the pipeline and confirm to roll out changes.

  4. Try using Alloy's live debugging UI to confirm if your updates worked. You may want to create a few recipes if you're not seeing spans.

  5. We can also confirm this using Explore Traces and some TraceQL!

    1. Navigate to Explore in the left navigation of Grafana Cloud.

    2. Type traces into the Datasources box and select the one containing your username (i.e. not demoinfra).

    3. Run the query below to see a count of requests by ingredient:

      {.service.name = "catalog" && .ingredient != ""} | count_over_time() by (.ingredient)