Skip to content

Commit d251033

Browse files
pareenavermapareenaverma
authored
Merge pull request #3011 from odidev/grpc_flyte
Build ML Workflow Pipelines with Flyte and gRPC on Google Cloud C4A Axion processors
2 parents 77f6f6c + 493482d commit d251033

12 files changed

Lines changed: 938 additions & 0 deletions

File tree

‎content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/_index.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
---
2+
title: Build ML Workflow Pipelines with Flyte and gRPC on Google Cloud C4A Axion processors
3+
4+
draft: true
5+
cascade:
6+
draft: true
7+
8+
minutes_to_complete: 30
9+
10+
who_is_this_for: This is an introductory topic for developers, data engineers, and ML engineers who want to build scalable machine learning workflow pipelines on Arm64-based Google Cloud C4A Axion processors using Flyte workflow orchestration and gRPC-based microservices.
11+
12+
learning_objectives:
13+
- Deploy Flyte workflow pipelines on Google Cloud C4A Axion processors
14+
- Build distributed machine learning pipelines using Flyte tasks
15+
- Implement gRPC-based services for feature engineering
16+
- Integrate Flyte workflows with distributed services
17+
- Run scalable ML pipelines on Arm-based cloud infrastructure
18+
19+
prerequisites:
20+
- A [Google Cloud Platform (GCP)](https://cloud.google.com/free) account with billing enabled
21+
- Basic familiarity with Python
22+
- Basic understanding of machine learning pipelines
23+
- Familiarity with Linux command-line operations
24+
25+
author: Pareena Verma
26+
27+
##### Tags
28+
skilllevels: Introductory
29+
subjects: ML
30+
cloud_service_providers:
31+
- Google Cloud
32+
33+
armips:
34+
- Neoverse
35+
36+
tools_software_languages:
37+
- Flyte
38+
- Python
39+
- gRPC
40+
41+
operatingsystems:
42+
- Linux
43+
44+
# ================================================================================
45+
# FIXED, DO NOT MODIFY
46+
# ================================================================================
47+
48+
further_reading:
49+
- resource:
50+
title: Google Cloud documentation
51+
link: https://cloud.google.com/docs
52+
type: documentation
53+
54+
- resource:
55+
title: Flyte documentation
56+
link: https://docs.flyte.org/
57+
type: documentation
58+
59+
- resource:
60+
title: gRPC documentation
61+
link: https://grpc.io/docs/
62+
type: documentation
63+
64+
- resource:
65+
title: Flyte GitHub repository
66+
link: https://github.com/flyteorg/flyte
67+
type: documentation
68+
69+
weight: 1
70+
layout: "learningpathall"
71+
learning_path_main_page: yes
72+
---

‎content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/_next-steps.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
# ================================================================================
3+
# FIXED, DO NOT MODIFY THIS FILE
4+
# ================================================================================
5+
weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation.
6+
title: "Next Steps" # Always the same, html page title.
7+
layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing.
8+
---

‎content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/architecture.md‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
---
2+
title: ML Pipeline Architecture
3+
weight: 8
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
# ML Pipeline Architecture
10+
11+
In this section, you explore the architecture behind the distributed machine learning pipeline built using Flyte and gRPC on Google Axion Arm-based infrastructure.
12+
13+
This architecture demonstrates how modern ML workflows are orchestrated using workflow engines while delegating specific tasks to distributed services.
14+
15+
Flyte manages the pipeline orchestration, while gRPC enables efficient communication between workflow tasks and external services.
16+
17+
18+
## System architecture
19+
20+
The ML pipeline consists of several tasks executed sequentially within the Flyte workflow.
21+
22+
```text
23+
Flyte Workflow Engine
24+
│
25+
▼
26+
Dataset Loader Task
27+
│
28+
▼
29+
Data Preprocessing Task
30+
│
31+
▼
32+
Feature Engineering Service (gRPC)
33+
│
34+
▼
35+
Model Training Task
36+
│
37+
▼
38+
Model Evaluation Task
39+
│
40+
▼
41+
Pipeline Result
42+
```
43+
44+
Each component in the workflow performs a specific function within the machine learning pipeline.
45+
46+
## Components
47+
48+
### Flyte workflow engine
49+
Flyte orchestrates the pipeline execution. It manages task dependencies, workflow execution, and data flow between tasks.
50+
51+
Key capabilities include:
52+
53+
- defining ML pipelines as Python workflows
54+
- managing task dependencies
55+
- enabling reproducible ML experiments
56+
- scaling pipeline execution
57+
58+
### Dataset loader
59+
The dataset loader task simulates loading a training dataset that will be used for model training.
60+
61+
In real ML systems, this step might include:
62+
63+
- loading datasets from object storage
64+
- retrieving data from data lakes
65+
- accessing distributed datasets
66+
67+
### Data preprocessing
68+
Data preprocessing transforms raw data into a format suitable for model training.
69+
70+
Typical preprocessing steps include:
71+
72+
- cleaning data
73+
- normalizing values
74+
- handling missing data
75+
- encoding categorical variables
76+
77+
### Feature engineering service (gRPC)
78+
Feature engineering is implemented as a gRPC microservice.
79+
80+
This design allows feature-generation logic to run independently of the workflow engine.
81+
82+
Benefits include:
83+
84+
- scalable feature generation
85+
- reusable feature services
86+
- independent scaling of compute resources
87+
- low-latency communication using gRPC
88+
89+
### Model training
90+
The training task uses generated features to train a machine learning model.
91+
92+
In production systems, this stage might include:
93+
94+
- training regression models
95+
- training classification models
96+
- training deep learning models
97+
98+
### Model evaluation
99+
The evaluation step measures model performance.
100+
101+
Typical evaluation metrics include:
102+
103+
- accuracy
104+
- precision
105+
- recall
106+
- F1 score
107+
108+
Based on the results, the workflow can determine whether to retrain the model.
109+
110+
## Pipeline execution flow
111+
112+
The ML pipeline follows this execution sequence.
113+
114+
```text
115+
Load Dataset
116+
│
117+
▼
118+
Preprocess Data
119+
│
120+
▼
121+
Feature Engineering (gRPC Service)
122+
│
123+
▼
124+
Model Training
125+
│
126+
▼
127+
Model Evaluation
128+
│
129+
▼
130+
Pipeline Result
131+
```
132+
133+
Each task executes sequentially while Flyte manages the workflow orchestration.
134+
135+
## Benefits of this architecture
136+
137+
This architecture provides several advantages:
138+
139+
- scalable ML pipeline orchestration
140+
- distributed feature engineering services
141+
- modular pipeline components
142+
- efficient task communication using gRPC
143+
- reproducible machine learning workflows
144+
145+
## Running on Axion
146+
This example demonstrates how machine learning workflows can run efficiently on Google Axion Arm-based processors.
147+
148+
Benefits include:
149+
150+
- high performance per watt
151+
- efficient execution of data pipelines
152+
- scalable infrastructure for ML workloads
153+
- optimized performance for modern cloud applications
154+
155+
## What you've learned
156+
157+
In this section, you explored the architecture behind the ML training pipeline.
158+
159+
You learned how:
160+
161+
- Flyte orchestrates ML workflows
162+
- gRPC services enable distributed feature engineering
163+
- pipeline tasks interact through workflow dependencies
164+
- ML pipelines can scale across distributed infrastructure
165+
166+
This architecture underpins modern distributed machine learning systems running on Arm-based cloud infrastructure.

‎content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/background.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
title: Get started with Flyte ML Workflow Pipelines with gRPC on Google Axion C4A
3+
weight: 2
4+
5+
layout: "learningpathall"
6+
---
7+
8+
## Explore Axion C4A Arm instances in Google Cloud
9+
10+
Google Axion C4A is a family of Arm-based virtual machines built on Google’s custom Axion CPU, which is based on Arm Neoverse-V2 cores. Designed for high-performance and energy-efficient computing, these virtual machines offer strong performance for data-intensive and analytics workloads such as big data processing, in-memory analytics, columnar data processing, and high-throughput data services.
11+
12+
The C4A series provides a cost-effective alternative to x86 virtual machines while leveraging the scalability, SIMD acceleration, and memory bandwidth advantages of the Arm architecture in Google Cloud.
13+
14+
These characteristics make Axion C4A instances well-suited for modern analytics stacks that rely on columnar data formats and memory-efficient execution engines.
15+
16+
To learn more, see the Google blog [Introducing Google Axion Processors, our new Arm-based CPUs](https://cloud.google.com/blog/products/compute/introducing-googles-new-arm-based-cpu).
17+
18+
## Explore Flyte ML Workflow Pipelines with gRPC on Google Axion C4A (Arm Neoverse V2)
19+
20+
Flyte is an open-source workflow orchestration platform used to build scalable and reproducible data and machine learning pipelines. It allows developers to define workflows as Python tasks, simplifying the management of complex ML processes such as data preparation, feature engineering, and model training.
21+
22+
gRPC enables fast communication between distributed services within these pipelines. Running Flyte with gRPC on Google Axion C4A Arm-based processors provides efficient, scalable infrastructure for executing modern ML workflows and distributed data processing tasks.
23+
24+
To learn more, visit the [Flyte documentation](https://docs.flyte.org/) and explore the [gRPC documentation](https://grpc.io/docs/) to understand how distributed service communication enables scalable machine learning workflows.
25+
26+
## What you've learned and what's next
27+
28+
In this section, you learned about:
29+
30+
* Google Axion C4A Arm-based VMs and their performance characteristics
31+
* Flyte as a workflow orchestration platform for machine learning pipelines
32+
* gRPC as a communication layer for distributed services
33+
* How Flyte and gRPC can be used together to build scalable ML training pipelines
34+
35+
Next, You will deploy Flyte tools, create a gRPC-based feature engineering service, and build a distributed ML workflow pipeline that orchestrates data processing and model training tasks on Axion infrastructure.

0 commit comments

Comments
 (0)