How to Deploy Your Jupyter Notebook As a Dashboard: A use case of visualizing stock data with AWS
~ Build an automated dashboarding system with AWS SageMaker, GitHub Actions, ECR, App Runner, and Mercury ~
Jupyter Notebook is one of the vital tools for data scientists. However, there are still difficulties in collaborating with your teammates and sharing your notebooks with your stakeholders.
I'll introduce an automated dashboarding system using AWS SageMaker and Mercury.
SageMaker enables us to edit notebooks in a hosted environment in AWS. It's easy to share your notebooks with your teammates, and no worries about building Jupyter server staff.
Mercury is a Python library that can transform Jupyter Notebooks into dashboards. You can easily publish your dashboard to your counterparts by using this library.
For quick understanding, here is the workflow of the whole architecture.
- Developers edit Jupyter Notebooks in Amazon SageMaker.
- Developers push their commits to GitHub.
- GitHub detects the push and automatically runs GitHub Actions, a CI/CD.
- During GitHub Actions, a Docker image is built and pushed to Amazon ECR, a container registry in AWS.
- As soon as the Docker image is pushed to ECR, it will be deployed to Amazon App Runner immediately.
- App Runner hosted Mercury server so that end users can access the dashboard.
Here are the service or software used in this article.
- Amazon App Runner
- Amazon SageMaker
- GitHub Actions
- Amazon ECR(Elastic Container Registry)
- Amazon S3
A use case for visualizing stock data
Here is a use case of this dashboarding system which visualizes the S&P 500 ETF data.
Source code: https://github.com/koyaaarr/invest-analytics-aws
Store the stock data in S3
First, you need to store the data you want to visualize. There are several options, and I chose S3 in this use case.
Edit a Jupyter Notebook in SageMaker
It's straightforward to use Jupyter Notebook in SageMaker, even if you're new to this like me.
You can launch the SageMaker Studio service and create a new notebook as you do in your local host.
Visualize the moving average and MACD of the S&P500 ETF
This is analyzing part, and I visualize some financial indicators. The analysis part is the most critical part of the actual use case, but I skip a detailed explanation since this is out of the scope of this article.
Add widgets for Mercury
To add widgets to the dashboard, you must add a particular cell on top of the notebook. You can check the official documentation if you want to know the detail.
I lay out some slider widgets in this dashboard, and the configuration will be generated like this.
Commit your work
You can easily git commit your work in this tab.
Push your commit to GitHub
Once you finish your work, you can push your work to your GitHub. You must generate an access token in GitHub to git push the commits to the remote repository.
Create Dockerfile and requirements.txt
In lines 5 to 12, I install the "TA-LIB" library for stock analysis, so you can skip this part if you don't need it.
Create ECR private repository
Before creating GitHub Actions, you need to create an ECR repository. The name of your repository will be used in GitHub Actions.
Create yaml for GitHub actions
This configuration is set to execute when the master branch is changed. You can check the official documentation if you want to know the detail. Don't forget to set
AWS_ECR_REPO_NAMEas the environmental variables in your repository. You might need to create an IAM account with appropriate privileges to generate an access key.
Create App runner
Lastly, let's create an App Runner instance to host your application. The deployment will be started once you set up the service's configuration. It takes several minutes.
After all of the work is finished, you can access your dashboard by clicking the link described in your App Runner service.
You can set VPC and security groups to set access control.
I introduced an automated dashboarding system using SageMaker and Mercury. In addition, GitHub Actions can realize CI/CD, so your code change will automatically be reflected on the dashboard. I hope this article helps.