A Practical Use-Case of Cloud-Native and Secured Dashboard with Google Cloud and Python Streamlit

Demo

Introduction

With rising cloud services and data scientist-friendly visualization tools, building a dashboard is getting easier and faster.

However, it’s also becoming more and more complicated to understand or utilize them.

This article will show the use-case of combining these technologies by building a secured dashboard managing my investment portfolio.

This article explains the application from three perspectives; business, data science, and engineering. These are often defined as essential skills in data science. Therefore, I intend to break down my explanation into these sections so you can read them in which you’re interested.

Data Science Skill’s Venn Diagram

Business Persipective: Requierments

Though this article focuses on technology, it wouldn’t be convincing if my app is not unpractical(even if this is only for personal use).

Therefore, I will define some requirements before the implementation.

By the way, I’ve bought some ETFs monthly, but I’m not sure what’s going on in my portfolio. This is because the prices of ETFs are varied and go up and down day by day. In addition, I don’t check my portfolio frequently because I don’t want to spend much time watching the stock markets. These things remind me of creating an app satisfying the following requirements.

1. show specific ETFs I’m interested in to see whether each stock is a bargain or not
2. show the current value of my portfolio to check how good or bad
3. show the ratio of the types of ETFs (e.g., stock/bond/commodity) to help me to decide whether I need to rotate my portfolio according to the best ratio of the types of assets
4. update daily because I’ll check this daily at most
5. authentication is required to hide my tangible assets(This is IMPORTANT)!

What I want to see

In addition to the above requirements, UI should be handy but provide sufficient information. Just between iPhone’s stock app and TradingView is ideal for me.

Target Position

Data Science Perspective: Data Modeling and Build Data Pipeline

I need to prepare a data mart for my dashboard to meet the above requirements. The data mart is one of the concepts in the data model, and this also includes the data warehouse and the data lake. These concepts have different purposes so let me explain them in the following table.

Data Model

There are two types of visualization needed, so I will create two data marts and a data warehouse that can provide enough data for data marts.

Data Pipeline

Now let’s get into the data schema of data marts. The first data mart is to plot a line chart of my portfolio and stocks, so historical values need to be prepared. The second one is to plot a pie chart of the ratio of my portfolio, so each stock’s ratio needs to be calculated.

Calculate Portfolio Value

Calculate Portfolio Ratio

The data modeling in detail is omitted due to space limitations. In the next article, I will introduce Google BigQuery and dbt in this data pipeline to explain modeling.

Ref: https://koyaaarr.medium.com/dbt-and-bigquery-in-practice-transform-stock-data-1771e2393319

Engineering Perspective: Architecture and Software

Finally, select appropriate software and services and combine them to realize my system. Here is the whole architecture.

Architecture

Let me explain each component for each role.

Data Retrieve, Transform, Accumulate Script
- Cloud Function: for data retrieving, transforming, and accumulating
- Cloud Storage: data will be served from here via API
- pandas-datareader: to get stock data
- gcsfs: to get data from Cloud Storage

Data Visualize Application
- Cloud Run: run application containerized with Docker.
- Cloud IAP(Identity-Aware Proxy): add authentication to Cloud Run app without coding
- Streamlit: serve a dashboard quickly and nicely
- Plotly: plot graphs quickly and nicely

Operation, CI/CD
- Cloud Build: Connect with GitHub to automatically and immediately deploy to Cloud Run / Cloud Functions after git push
- Cloud Scheduler: trigger Cloud Function regularly
- Cloud Pub/Sub: the same purpose with scheduler

How to use it regularly

I use the YAML file to simplify the operation of managing my portfolio. It is to configure my portfolio that contains the number of stocks I have and the details of each stock.

All I need to do is to modify the number of stocks I hold in this YAML file when I buy some stocks. After git push, Cloud build detects that and copies the YAML file to Cloud Storage automatically, then Cloud Function calculates according to the data so Cloud Run can fetch the latest data from there.

Operation

Lastly, what my dashboard looks like is this.

Authentication is required
Dashboard Overview

Conclusion

I intend to break down my application into three perspectives. There are a lot of valuable services like Cloud Run and Cloud IAP. These look complicated to use but are quite helpful in building an application quickly, so I strongly recommend diving into there. This article explains how to create the dashboard using Google Cloud and Python Streamlit. I hope you will find this helpful.

Reference

- Data Science Venn diagram (http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram)
- Trading View (https://www.tradingview.com/)
- iPhone Stocks app (https://apps.apple.com/us/app/stocks/id1069512882)
- Enabling IAP with Cloud Run (https://cloud.google.com/iap/docs/enabling-cloud-run)

--

--

--

Data Scientist in Japan. https://www.linkedin.com/in/ryo-koyajima/

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Going Local or Session in Storage

Dockerizing a React and Spring Boot application

CS373 Spring 2021: Final Entry

Conference And Learn Day 4, How To Tell A Story

Manage Risk, Not Buffers

R vs Stata: Which One is Best For Data Analysis?

Python’s naming rules and conventions

Patterns in Kotlin: Abstract Factory

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ryo Koyajima / 小矢島 諒

Ryo Koyajima / 小矢島 諒

Data Scientist in Japan. https://www.linkedin.com/in/ryo-koyajima/

More from Medium

Checking types and column names of a data frame with the schema information for BigQuery

How to read and write (sync) data from Google sheets to Bigquery

Load Twitter data into Google Sheet and automate it — Start a simple data pipeline

Data Science 3 — Unstructured Text Analysis