Transformations
Before you begin: You must already know about connecting data in Scenes apps before continuing with this guide.
Transformations are a powerful way to manipulate data returned by the SceneQueryRunner
object before scenes render a visualization. Using transformations, you can:
- Rename fields
- Join time series data
- Perform mathematical operations across queries
- Use the output of one transformation as the input for another transformation
With transformations you can query data once, manipulate it, and display it in a scene.
Learn more about Grafana transformations in the official Grafana documentation.
Transform query results in a scene
Step 1. Create a scene
Create a scene with a single Table panel and a Prometheus query. The example query returns the average duration of HTTP requests for Prometheus API endpoints. The resulting table consists of three columns: Time
, handler
, and Value
.
const queryRunner = new SceneQueryRunner({
$timeRange: new SceneTimeRange(),
datasource: {
type: 'prometheus',
uid: '<PROVIDE_GRAFANA_DS_UID>',
},
queries: [
{
refId: 'A',
expr: 'sort_desc(avg by(handler) (rate(prometheus_http_request_duration_seconds_sum {}[5m]) * 1e3))',
format: 'table',
instant: true,
},
],
});
const scene = new EmbeddedScene({
$data: queryRunner,
body: new SceneFlexLayout({
direction: 'column',
children: [
new SceneFlexItem({
body: PanelBuilders.table().setTitle('Average duration of HTTP request').build(),
}),
],
}),
});
Step 2. Configure a data transformation
The resulting table from the previous step will look similar to the one that follows:
Time | handler | Value |
---|---|---|
2023-05-09 14:00:00.000 | /metrics | 1.10 |
2023-05-09 14:00:00.000 | /api/v1/label/:name/values | 0.361 |
2023-05-09 14:00:00.000 | /api/v1/metadata | 0.113 |
2023-05-09 14:00:00.000 | /api/v1/query_range | 0.0847 |
2023-05-09 14:00:00.000 | /api/v1/query | 0.14 |
2023-05-09 14:00:00.000 | /api/v1/labels | 0.0194 |
2023-05-09 14:00:00.000 | /api/v1/series | 0 |
2023-05-09 14:00:00.000 | /api/v1/status/buildinfo | 0 |
Add the Organize fields transformation to hide the Time
field:
Create a SceneDataTransformer
object:
const transformedData = new SceneDataTransformer({
$data: queryRunner,
transformations: [
{
id: 'organize',
options: {
excludeByName: {
Time: true,
},
indexByName: {},
renameByName: {},
},
},
],
});
Objects used in transformations
are the same transformation configuration objects you would see in a typical dashboard panel when you view the JSON tab in the panel inspect drawer. To access this tab, click Inspect > Panel JSON in the panel edit menu.
Use the newly created transformedData
object in place of the previously used SceneQueryRunner
:
const scene = new EmbeddedScene({
$data: transformedData,
body: new SceneFlexLayout({
direction: 'column',
children: [
new SceneFlexItem({
body: PanelBuilders.table().setTitle('Average duration of HTTP request').build(),
}),
],
}),
});
The resulting table will look similar to the one that follows:
handler | Value |
---|---|
/metrics | 1.10 |
/api/v1/label/:name/values | 0.361 |
/api/v1/metadata | 0.113 |
/api/v1/query_range | 0.0847 |
/api/v1/query | 0.14 |
/api/v1/labels | 0.0194 |
/api/v1/series | 0 |
/api/v1/status/buildinfo | 0 |
Step 3. Configure multiple transformations
SceneDataTransformer
allows you to configure multiple transformations. The transformations are executed in the same order as they are added to the transformations
array.
Modify the transformedData
object and add Rename by regex transformations to change the field names: handler
to Handler
and Value
to Average duration
:
const transformedData = new SceneDataTransformer({
$data: queryRunner,
transformations: [
{
id: 'organize',
options: {
excludeByName: {
Time: true,
},
indexByName: {},
renameByName: {},
},
},
{
id: 'renameByRegex',
options: {
regex: 'handler',
renamePattern: 'Handler',
},
},
{
id: 'renameByRegex',
options: {
regex: 'Value',
renamePattern: 'Average duration',
},
},
],
});
The resulting table will look similar to the one that follows:
Handler | Average duration |
---|---|
/metrics | 1.10 |
/api/v1/label/:name/values | 0.361 |
/api/v1/metadata | 0.113 |
/api/v1/query_range | 0.0847 |
/api/v1/query | 0.14 |
/api/v1/labels | 0.0194 |
/api/v1/series | 0 |
/api/v1/status/buildinfo | 0 |
Add custom transformations
In addition to all the transformations available in Grafana, scenes allow you to create custom transformations.
SceneDataTransformer
accepts CustomTransformOperator
as an item of the transformations
array:
transformations: Array<DataTransformerConfig | CustomTransformOperator>;
CustomTransformOperator
is a function that returns the RxJS Operator, which transforms data:
type CustomTransformOperator = (context: DataTransformContext) => MonoTypeOperatorFunction<DataFrame[]>;
Read more about RxJS operators in the RxJS official documentation.
Step 1. Create a custom transformation
Create a custom transformation that will apply to the handler
field and prefix all values with a URL:
Custom transformations depend heavily on manipulating internal Grafana data objects called data frames. Learn more about data frames in the official Grafana documentation.
import { DataFrame } from '@grafana/data';
import { Observable } from 'rxjs';
import { map } from 'rxjs/operators';
const prefixHandlerTransformation: CustomTransformOperator = () => (source: Observable<DataFrame[]>) => {
return source.pipe(
map((data: DataFrame[]) => {
return data.map((frame: DataFrame) => {
return {
...frame,
fields: frame.fields.map((field) => {
if (field.name === 'handler') {
return {
...field,
values: field.values.map((v) => 'http://www.my-api.com' + v),
};
}
return field;
}),
};
});
})
);
};
Step 2. Use a custom transformation
Add a custom transformation to the previously created transformedData
object:
const transformedData = new SceneDataTransformer({
$data: queryRunner,
transformations: [
prefixHandlerTransformation,
{
id: 'organize',
options: {
excludeByName: {
Time: true,
},
indexByName: {},
renameByName: {},
},
},
{
id: 'renameByRegex',
options: {
regex: 'handler',
renamePattern: 'Handler',
},
},
{
id: 'renameByRegex',
options: {
regex: 'Value',
renamePattern: 'Average duration',
},
},
],
});
The prefixHandlerTransformation
custom transformation is added as the first one because it applies to the handler
field that's renamed to Handler
in the following transformations. You can modify the custom transformation implementation so that it doesn't have to be used prior to other transformations.
The resulting table will look similar to the one that follows:
Handler | Average duration |
---|---|
http://www.my-api.com/metrics | 1.10 |
http://www.my-api.com/api/v1/label/:name/values | 0.361 |
http://www.my-api.com/api/v1/metadata | 0.113 |
http://www.my-api.com/api/v1/query_range | 0.0847 |
http://www.my-api.com/api/v1/query | 0.14 |
http://www.my-api.com/api/v1/labels | 0.0194 |
http://www.my-api.com/api/v1/series | 0 |
http://www.my-api.com/api/v1/status/buildinfo | 0 |
Combine and filter
One powerful thing you can do with transformations (custom and built-in) is share query results between panels in interesting ways. This allows you to place most of your queries in a single query runner that lives at the top of the scene. Then you can use a SceneDataTransformer
object on the VizPanel
level to join and filter the resulting data in different ways. Some panels may need the result of two of the queries, and another may need the results of all of them.
It's easy to filter the resulting DataFrame
array by which query they came from using the refId
property on DataFrame
.