Forecasting Stocks and Crypto prices using Redis, Prophet, and Grafana

This article is not financial advice; it’s about the technology that helps to do it fast.

This article will demonstrate how to analyze Stocks and Crypto historical data stored as RedisTimeSeries using serverless engine RedisGears and Facebook’s Prophet Model to predict prices.

The Prophet Model needs to be trained to provide accurate results. Using Redis as a primary database to store time series and processing data will help prepare the model efficiently.

Using Grafana and Redis-Prophet Docker image to Forecast Stocks and Crypto prices.

Also, I will explain how to visualize forecasts and interact with Redis using Grafana and Redis plugins.

Redis plugins for Grafana

Build Financial Application on RedisTimeSeries

In the Redis Labs blog post Build Your Financial Application on RedisTimeSeries, Prasanna Rajagopal explained how to use RedisTimeSeries to store, aggregate, query stock prices, financial information, and technical indicators — and many other time-series data sets used by investors.

Build Your Financial Application on RedisTimeSeries | Redis Labs

RedisTimeSeries is schemaless, which means that you can load data without defining schema, add new fields on the fly, or change your data model should your business circumstances change. Its real-time performance and simple developer experience makes it fun to work with time-series data!

I would highly recommend reading it to learn more about RedisTimeSeries.

Facebook Prophet Model

The prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.

There are a lot of exciting articles describing how to use Prophet for forecasting and visualizing time series using standalone Python applications to get started.

Implementing Facebook Prophet efficiently

Forecasting Stock Prices using Prophet

Using RedisGears to forecast RedisTimeSeries data using the Prophet model

Redis Labs announced RedisGears 1.0 in May last year, and one of the examples was the ProphetGears project, visualizing RedisTimeSeries with RedisInsight, which was the only option at that time.

Announcing RedisGears 1.0: A Serverless Engine for Redis | Redis Labs

ProphetGears project visualizing Forecast using RedisInsight.

I already wrote about adopting RedisGears for other projects but have not had a chance to try it with Prophet until now.

Pop-up store demo using RedisTimeSeries, RedisGears, and Redis Data Source for Grafana

Prophet team recently (March 28, 2021) released Version 1.0 with many improvements, and I am excited to play with it.

Redis-Prophet Docker image

To start playing with Prophet, it should be installed using RedisGears and registered in the Redis database. It’s fast and easy to do on the Production server with high RAM, a powerful CPU, and Redis Enterprise. For development with limited resources, I prefer to use Docker containers that are easy to manage and recreate.

We developed a Docker image based on the latest version of RedisTimeSeries and RedisGears with Prophet pre-installed. This Docker image is automatically built every night using GitHub Action and can be used for any project.

To start the Redis container using the Redis-Prophet image run:

$ docker run -p 6379:6379 --name=redis-prophet ghcr.io/redisgrafana/redis-prophet:latest

RedisGears with Prophet may be warmed up to load dependencies and save the state in the database before running prediction in theredis-clitool:

RG.PYEXECUTE "GearsBuilder().run()" REQUIREMENTS Prophet

Or in the Grafana with Redis Application plugin installed:

RedisGears Script Editor for Grafana.

Execute RG.PYDUMPREQS command to validate that Prophet and other dependencies loaded correctly:

RG.PYDUMPREQS command executed in Redis CLI panel for Grafana.

Where to get Stocks and Crypto historical data?

Yahoo Finance provides historical data, which can be downloaded on-demand as a CSV file, parsed, and added to RedisTimeSeries.

Bitcoin USD (BTC-USD) Price, News, Quote & History - Yahoo Finance

This is what the downloaded file looks like:

Date,Open,High,Low,Close,Adj Close,Volume
2014-09-17,465.864014,468.174011,452.421997,457.334,457.334,21056800
2014-09-18,456.859985,456.859985,413.104004,424.440,424.440,34483200

To parse downloaded files, we created a simple Node.js script, which reads files line by line and creates RedisTimeSeries with labels:

• Symbol — BTC-USD, APPL, TSLA, etc.
• Type — OPEN, HIGH, LOW, CLOSE, ADJ, and VOLUME

await redis.send_command("TS.ADD", `{${symbol}:${ts}}`, date, value, "LABELS", "symbol", symbol, "type", ts).catch((err) => console.log(err));

After the script imported all files inside an import/ folder, you can scan all the keys using the SCAN command:

SCAN command executed using Redis CLI in Grafana.

Or the custom TMSCAN command, available in the Redis plugin for Grafana, displays keys with type and memory usage.

Custom TMSCAN command, available in Redis plugin for Grafana, displays keys with type and memory usage.

At this stage, we can already visualize RedisTimeSeries using TS.RANGE and TS.MRANGE commands following the “How to Use the New Redis Data Source for Grafana Plugin” blog post.

How to Use the New Redis Data Source for Grafana Plugin | Redis Labs

Let’s do a Forecast

Ok, the time series are ready and loaded to the Redis-Prophet instance. The project provides multiple scripts for a 30, 90, and 365 days forecast.

RedisGears Script Editor with the Predict function in Grafana.

When running this Python script in RedisGears, it will execute the Prediction function for all keys matching pattern {*CLOSE} — for closing prices. The weekends were eliminated in the future predictions for 90 days for accurate results.

As a result of the script, new time series with predictions will be created as separate keys with an additional label futurefor filtering.

• {TSLA:CLOSE}:365:yhat}
• {TSLA:CLOSE}:365:yhat_upper}
• {TSLA:CLOSE}:365:yhat_lower}

RedisTimeSeries information displayed in Redis CLI for Grafana.

This script can be executed by running RG.PYEXECUTE command with Unblocking option enabled to avoid timeouts waiting for the script to finish or registered and triggered with RG.TRIGGER.

Visualize Time-series in Grafana

RedisTimeSeries can be effortlessly visualized in Grafana across multiple keys using TS.MRANGE command with a filter for symbols chosen in the Grafana Dashboard’s variable.

Redis plugins for Grafana - Variables

symbol=(${symbol:raw}) type!=VOLUME

AAPL price prediction visualized in Grafana using Redis plugins.

Grafana does not have predefined relative time ranges for the future, so I am using now-3y to now+1y to query time series for four years period:

I am using now-3y to now+1y to query time series for four years period.

The final results are displayed on the dashboard with RedisGears Script editor and Redis CLI panels available to start experimenting.

Stock and Crypto symbols prices are visualized using repeat by variable.

What’s next?

There are many aspects I did not discuss in the article — how to train Prophet Model, use RedisGears and RedisTimeSeries to store time-series employing Redis Cluster with multiple shards for Performance, and Redundancy. I will continue training the Prophet Model and compare it with the results I can get from RedisAI.

RedisAI - A Server for Machine and Deep Learning Models

Join my RedisConf 21 session “Elevate your Redis experience with Redis plugins for Grafana” to learn more about Redis Data Source, Redis Application, and one more plugin, which I have never mentioned before. See you there!

Elevate your Redis experience with Redis plugins for Grafana

Volkov Labs is an agency founded by long-time Grafana contributor Mikhail Volkov. We find elegant solutions for non-standard tasks.

Check out the latest plugins and projects at https://volkovlabs.io