Grafana data streaming meets Redis
Redis does not store its statistics long-term to keep its memory footprint to a minimum. How to monitor the Redis database without any third-party tools?
We introduced Redis Data Source for Grafana and included a pre-defined dashboard to monitor the Redis database. This Redis dashboard displayed various metrics and statistics using INFO, SLOWLOG, CLIENT, CLUSTER, and other commands. It gave users visibility to the Redis database and does not require any special configuration or tools. Unfortunately, it doesn’t show the dynamic for ops/sec, client connections, or network throughput changes.
What is a Streaming Data Source?
If you Google “Grafana streaming”, you most probably find Build a streaming data source plugin tutorial, an old blog, and a couple of youtube videos from Ryan McKinley and Sean Lafferty from GrafanaCon.
Looking at the Grafana GitHub repository, I found only one data source Loki, which has the option “streaming”: true enabled. Not a lot of information to work with and understand how cool this technology is!
By enabling streaming for your data source plugin, you can update your dashboard as soon as new data becomes available. For example, a streaming data source plugin can connect to a WebSocket, or subscribe to a message bus, and update the visualization whenever a new message is available.
To keep the existing functionality of the Redis Data Source and add Streaming capability, we added the Streaming option with two parameters to the query editor:
- Interval — Streaming interval in milliseconds. The default is 1000ms.
- Capacity — Values will be constantly added and will never exceed the given capacity. The default is 1000ms.
Based on the streaming interval browser will request data from the data source, and the returned value will be displayed on the panel and stored in memory. If you enable Refresh on the dashboard or refresh the browser window, data will be gone, and panels will start to collect data again.
How does streaming work?
Grafana uses RxJS to continuously send data from a data source to a panel visualization. All the work is done on the React frontend. The Golang backend may not know that it’s used for streaming unless data should be returned in a different format as we did for Command Statistics using command INFO COMMANDSTATS command.
The Grafana tutorial gives you some insight into what needs to be done, but it’s not enough to implement streaming for the fully functional backend data source:
/**
* Override query to support streaming
*/
query(request: DataQueryRequest<RedisQuery>): Observable<DataQueryResponse> {
const refA = head(request.targets);
/**
* Streaming is not enabled
*/
if (!refA?.streaming) {
return super.query(request);
}
/**
* Streaming enabled
*/
return new Observable<DataQueryResponse>((subscriber) => {
/**
* This dataframe can have values constantly added, and will never exceed the given capacity
*/
const frame = new CircularDataFrame({
append: 'tail',
capacity: refA?.streamingCapacity || 1000,
});
/**
* Set refId and Time field
*/
frame.refId = refA.refId;
frame.addField({ name: 'time', type: FieldType.time });
/**
* Interval
*/
const intervalId = setInterval(async () => {
let values: { [index: string]: number } = { time: Date.now() };
/**
* Run Query and filter time field out
*/
const fields = await super
.query(request)
.pipe(
switchMap$((response) => response.data),
map$((data: DataFrame) => data.fields.filter((field) => (field.name === 'time' ? false : true)))
)
.toPromise();
if (fields) {
/**
* Add fields to frame fields and return values
*/
fields.map((field) =>
field.values.toArray().map((value) => {
if (frame.fields.length < fields.length + 1) {
frame.addField({ name: field.name, type: field.type });
}
return (values[field.name] = value);
})
);
}
/**
* Add frame
*/
frame.add(values);
subscriber.next({
data: [frame],
key: refA.refId,
});
}, refA.streamingInterval || 1000);
return () => {
clearInterval(intervalId);
};
});
}To keep things simple — only A query is supported, and the command should return only one line of data. Support for multiple queries and lines may be added to the feature versions.
How to stream Application data?
Similarly, as we monitor the Redis database, we can stream application data:
- Sets using
SCARD, - Streams using
XLENcommands, - RedisTimeSeries series and many more.
What we did in the Pop-up store demo using RedisGears can be partially accomplished with data streaming in Grafana.
Instant feedback
Having streaming capabilities in Grafana and Redis Data Source supporting it allows monitoring of the Redis database and application data. If you need to store historical data, consider Prometheus, RedisTimeSeries, or other data storage. If you are interested in the most recent data, streaming is a great way to see changes as they happen.
Using the interactive Redis CLI panel, a part of the Redis Application plugin, you can manipulate data, configure the database, and see instant feedback on your dashboard.
To learn more about Real-time observability with Redis and Grafana, please look at our session presented at #ObservabilityCON.
Volkov Labs is an agency founded by long-time Grafana contributor Mikhail Volkov. We find elegant solutions for non-standard tasks.
Check out the latest plugins and projects at https://volkovlabs.io
