How to Deploy Your Jupyter Notebook As a Dashboard: A use case of visualizing stock data with AWS

~ Build an automated dashboarding system with AWS SageMaker, GitHub Actions, ECR, App Runner, and Mercury ~

Transform Jupyter Notebook into Dashboard

Introduction

Jupyter Notebook is one of the vital tools for data scientists. However, there are still difficulties in collaborating with your teammates and sharing your notebooks with your stakeholders.

I'll introduce an automated dashboarding system using AWS SageMaker and Mercury.

SageMaker enables us to edit notebooks in a hosted environment in AWS. It's easy to share your notebooks with your teammates, and no worries about building Jupyter server staff.

Mercury is a Python library that can transform Jupyter Notebooks into dashboards. You can easily publish your dashboard to your counterparts by using this library.

Architecture

Automated Dashboarding System Architecture

For quick understanding, here is the workflow of the whole architecture.

1. Developers edit Jupyter Notebooks in Amazon SageMaker.
2. Developers push their commits to GitHub.
3. GitHub detects the push and automatically runs GitHub Actions, a CI/CD.
4. During GitHub Actions, a Docker image is built and pushed to Amazon ECR, a container registry in AWS.
5. As soon as the Docker image is pushed to ECR, it will be deployed to Amazon App Runner immediately.
6. App Runner hosted Mercury server so that end users can access the dashboard.

Here are the service or software used in this article.

• Mercury
• Amazon App Runner
• Amazon SageMaker
• GitHub Actions
• Amazon ECR(Elastic Container Registry)
• Amazon S3
• Docker

A use case for visualizing stock data

Here is a use case of this dashboarding system which visualizes the S&P 500 ETF data.

Source code: https://github.com/koyaaarr/invest-analytics-aws

Store the stock data in S3

First, you need to store the data you want to visualize. There are several options, and I chose S3 in this use case.

stored stock data in S3

Edit a Jupyter Notebook in SageMaker

It's straightforward to use Jupyter Notebook in SageMaker, even if you're new to this like me.

You can launch the SageMaker Studio service and create a new notebook as you do in your local host.

Create a new notebook in SageMaker

Visualize the moving average and MACD of the S&P500 ETF

This is analyzing part, and I visualize some financial indicators. The analysis part is the most critical part of the actual use case, but I skip a detailed explanation since this is out of the scope of this article.

Visualize the MACD of the ETF

Add widgets for Mercury

To add widgets to the dashboard, you must add a particular cell on top of the notebook. You can check the official documentation if you want to know the detail.

https://github.com/mljar/mercury

Config of the widgets

I lay out some slider widgets in this dashboard, and the configuration will be generated like this.

Actual widgets

Commit your work

You can easily git commit your work in this tab.

Push your commit to GitHub

Once you finish your work, you can push your work to your GitHub. You must generate an access token in GitHub to git push the commits to the remote repository.

https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token

Create Dockerfile and requirements.txt

In lines 5 to 12, I install the "TA-LIB" library for stock analysis, so you can skip this part if you don't need it.

Create ECR private repository

Before creating GitHub Actions, you need to create an ECR repository. The name of your repository will be used in GitHub Actions.

Create yaml for GitHub actions

This configuration is set to execute when the master branch is changed. You can check the official documentation if you want to know the detail. Don't forget to set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ECR_REPO_NAMEas the environmental variables in your repository. You might need to create an IAM account with appropriate privileges to generate an access key.

https://github.com/aws-actions/amazon-ecr-login

Create App runner

Lastly, let's create an App Runner instance to host your application. The deployment will be started once you set up the service's configuration. It takes several minutes.

Access Dashboard

After all of the work is finished, you can access your dashboard by clicking the link described in your App Runner service.

You can set VPC and security groups to set access control.

Conclusion

I introduced an automated dashboarding system using SageMaker and Mercury. In addition, GitHub Actions can realize CI/CD, so your code change will automatically be reflected on the dashboard. I hope this article helps.