A Practical Use-Case of Cloud-Native and Secured Dashboard with Google Cloud and Python Streamlit

Demo

Introduction

However, it’s also becoming more and more complicated to understand or utilize them.

This article will show the use-case of combining these technologies by building a secured dashboard managing my investment portfolio.

This article explains the application from three perspectives; business, data science, and engineering. These are often defined as essential skills in data science. Therefore, I intend to break down my explanation into these sections so you can read them in which you’re interested.

Data Science Skill’s Venn Diagram

Business Persipective: Requierments

Therefore, I will define some requirements before the implementation.

By the way, I’ve bought some ETFs monthly, but I’m not sure what’s going on in my portfolio. This is because the prices of ETFs are varied and go up and down day by day. In addition, I don’t check my portfolio frequently because I don’t want to spend much time watching the stock markets. These things remind me of creating an app satisfying the following requirements.

1. show specific ETFs I’m interested in to see whether each stock is a bargain or not
2. show the current value of my portfolio to check how good or bad
3. show the ratio of the types of ETFs (e.g., stock/bond/commodity) to help me to decide whether I need to rotate my portfolio according to the best ratio of the types of assets
4. update daily because I’ll check this daily at most
5. authentication is required to hide my tangible assets(This is IMPORTANT)!

What I want to see

In addition to the above requirements, UI should be handy but provide sufficient information. Just between iPhone’s stock app and TradingView is ideal for me.

Target Position

Data Science Perspective: Data Modeling and Build Data Pipeline

Data Model

There are two types of visualization needed, so I will create two data marts and a data warehouse that can provide enough data for data marts.

Data Pipeline

Now let’s get into the data schema of data marts. The first data mart is to plot a line chart of my portfolio and stocks, so historical values need to be prepared. The second one is to plot a pie chart of the ratio of my portfolio, so each stock’s ratio needs to be calculated.

Calculate Portfolio Value

Calculate Portfolio Ratio

The data modeling in detail is omitted due to space limitations. In the next article, I will introduce Google BigQuery and dbt in this data pipeline to explain modeling.

Ref: https://koyaaarr.medium.com/dbt-and-bigquery-in-practice-transform-stock-data-1771e2393319

Engineering Perspective: Architecture and Software

Architecture

Let me explain each component for each role.

Data Retrieve, Transform, Accumulate Script
- Cloud Function: for data retrieving, transforming, and accumulating
- Cloud Storage: data will be served from here via API
- pandas-datareader: to get stock data
- gcsfs: to get data from Cloud Storage

Data Visualize Application
- Cloud Run: run application containerized with Docker.
- Cloud IAP(Identity-Aware Proxy): add authentication to Cloud Run app without coding
- Streamlit: serve a dashboard quickly and nicely
- Plotly: plot graphs quickly and nicely

Operation, CI/CD
- Cloud Build: Connect with GitHub to automatically and immediately deploy to Cloud Run / Cloud Functions after git push
- Cloud Scheduler: trigger Cloud Function regularly
- Cloud Pub/Sub: the same purpose with scheduler

How to use it regularly

All I need to do is to modify the number of stocks I hold in this YAML file when I buy some stocks. After git push, Cloud build detects that and copies the YAML file to Cloud Storage automatically, then Cloud Function calculates according to the data so Cloud Run can fetch the latest data from there.

Operation

Lastly, what my dashboard looks like is this.

Authentication is required
Dashboard Overview

Conclusion

Reference

--

--

Data Scientist in Japan. Write about DS/ML/AI use-cases https://www.linkedin.com/in/ryo-koyajima/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store