Skip to content

Commit 5a090a1

Browse files
authored
Merge pull request #3143 from madeline-underwood/flyte
Flyte reviewed
2 parents 244ffea + 5cc288a commit 5a090a1

7 files changed

Lines changed: 28 additions & 21 deletions

File tree

content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/_index.md

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,6 @@
22
title: Build ML Workflow Pipelines with Flyte and gRPC on Google Cloud C4A Axion processors
33
description: Learn how to build scalable machine learning workflow pipelines on Google Cloud C4A Axion processors using Flyte for workflow orchestration and gRPC for distributed service communication.
44

5-
draft: true
6-
cascade:
7-
draft: true
8-
95
minutes_to_complete: 30
106

117
who_is_this_for: This is an introductory topic for developers, data engineers, and ML engineers who want to build scalable machine learning workflow pipelines on Arm64-based Google Cloud C4A Axion processors using Flyte workflow orchestration and gRPC-based microservices.

content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/architecture.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,16 @@
11
---
2-
title: ML Pipeline Architecture
2+
title: Understand the distributed ML architecture
33
weight: 8
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
88

9+
## Analyze the ML pipeline architecture
10+
911
In this section, you explore the architecture behind the distributed machine learning pipeline built using Flyte and gRPC on Google Axion Arm-based infrastructure.
1012

11-
This architecture demonstrates how modern ML workflows are orchestrated using workflow engines while delegating specific tasks to distributed services.
13+
The architecture demonstrates how modern ML workflows are orchestrated using workflow engines while delegating specific tasks to distributed services.
1214

1315
Flyte manages the pipeline orchestration, while gRPC enables efficient communication between workflow tasks and external services.
1416

@@ -75,7 +77,7 @@ Typical preprocessing steps include:
7577
### Feature engineering service (gRPC)
7678
Feature engineering is implemented as a gRPC microservice.
7779

78-
This design allows feature-generation logic to run independently of the workflow engine.
80+
The design allows feature-generation logic to run independently of the workflow engine.
7981

8082
Benefits include:
8183

content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/background.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,14 @@
11
---
2-
title: Get started with Flyte ML Workflow Pipelines with gRPC on Google Axion C4A
2+
title: Understand Flyte and gRPC ML workflows on Google Axion
33
weight: 2
44

55
layout: "learningpathall"
66
---
77

8+
## Why Flyte and gRPC on Google Axion C4A
9+
10+
Flyte is an open-source workflow orchestration platform used to build scalable and reproducible data and machine learning pipelines. Combined with gRPC for efficient distributed service communication, Flyte enables developers to define workflows as Python tasks while delegating specific operations to independent microservices. Running this stack on Google Axion C4A Arm-based processors provides efficient, scalable infrastructure for executing modern ML workflows and distributed data processing tasks.
11+
812
## Explore Axion C4A Arm instances in Google Cloud
913

1014
Google Axion C4A is a family of Arm-based virtual machines built on Google’s custom Axion CPU, which is based on Arm Neoverse-V2 cores. Designed for high-performance and energy-efficient computing, these virtual machines offer strong performance for data-intensive and analytics workloads such as big data processing, in-memory analytics, columnar data processing, and high-throughput data services.
@@ -15,9 +19,9 @@ These characteristics make Axion C4A instances well-suited for modern analytics
1519

1620
To learn more, see the Google blog [Introducing Google Axion Processors, our new Arm-based CPUs](https://cloud.google.com/blog/products/compute/introducing-googles-new-arm-based-cpu).
1721

18-
## Explore Flyte ML Workflow Pipelines with gRPC on Google Axion C4A (Arm Neoverse V2)
22+
## Flyte and gRPC for distributed ML workflows
1923

20-
Flyte is an open-source workflow orchestration platform used to build scalable and reproducible data and machine learning pipelines. It allows developers to define workflows as Python tasks, simplifying the management of complex ML processes such as data preparation, feature engineering, and model training.
24+
Flyte allows developers to define workflows as Python tasks, simplifying the management of complex ML processes such as data preparation, feature engineering, and model training.
2125

2226
gRPC enables fast communication between distributed services within these pipelines. Running Flyte with gRPC on Google Axion C4A Arm-based processors provides efficient, scalable infrastructure for executing modern ML workflows and distributed data processing tasks.
2327

content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/create-grpc-service.md

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,18 @@
11
---
2-
title: Create Feature Engineering gRPC Service
2+
title: Build a gRPC feature engineering service
33
weight: 5
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
88

9+
## Create distributed feature engineering with gRPC
10+
911
In modern machine learning pipelines, feature engineering is often implemented as a separate service so it can scale independently from the training workflow.
1012

1113
In this section, you create a **gRPC-based feature engineering service** that generates features used by the machine learning pipeline.
1214

13-
This service will later be called by the Flyte workflow during pipeline execution.
15+
The Flyte workflow will call this service during pipeline execution.
1416

1517
## Architecture overview
1618

@@ -79,7 +81,7 @@ python -m grpc_tools.protoc \
7981
feature.proto
8082
```
8183

82-
This generates the following files:
84+
The command generates the following files:
8385

8486
```text
8587
feature_pb2.py
@@ -143,7 +145,7 @@ if __name__ == "__main__":
143145
serve()
144146
```
145147

146-
This service receives a value from the workflow and generates a derived feature used during model training.
148+
The service receives a value from the workflow and generates a derived feature used during model training.
147149

148150
## Run the feature service
149151

content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/install-flyte.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,22 @@
11
---
2-
title: Install Flyte and Dependencies
2+
title: Install Flyte and gRPC tools on Axion
33
weight: 4
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
88

9+
## Prepare the development environment
910

1011
In this section, you prepare a SUSE Linux Enterprise Server (SLES) arm64 virtual machine and install the core components required to build machine learning workflow pipelines using Flyte.
1112

1213
Flyte provides workflow orchestration for scalable ML pipelines, while gRPC enables efficient communication between distributed services used within those pipelines.
1314

14-
This environment ensures that the workflow orchestration tools and communication libraries run natively on Arm-based Axion processors.
15+
Running these tools natively on Arm-based Axion processors ensures efficient execution of ML workflows.
1516

1617
## Architecture overview
1718

18-
This architecture represents a single-node development environment used to build and run distributed machine learning workflows.
19+
The development environment consists of a single-node setup used to build and run distributed machine learning workflows.
1920

2021
```text
2122
SUSE Linux Enterprise Server (arm64)

content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/instance.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
---
2-
title: Create a Google Axion C4A arm64 virtual machine on GCP
2+
title: Create a Google Axion C4A Arm virtual machine
33
weight: 3
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
88

9-
## Provision a Google Axion C4A arm64 virtual machine
9+
## Provision Google Axion infrastructure
1010

1111
In this section you'll create a Google Axion C4A arm64 virtual machine on Google Cloud Platform. You'll use the `c4a-standard-4` machine type, which provides 4 vCPUs and 16 GB of memory. This virtual machine hosts your Flyte ML Workflow with gRPC applications.
1212

content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/run-ml-pipeline.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
---
2-
title: Run ML Training Pipeline
2+
title: Execute and validate the ML pipeline
33
weight: 7
44

55
### FIXED, DO NOT MODIFY
66
layout: learningpathall
77
---
88

9+
## Run the distributed ML workflow
10+
911
In this section, you execute the distributed machine learning pipeline built using **Flyte and gRPC**.
1012

1113
The ML workflow will:
@@ -107,7 +109,7 @@ Feature gRPC service running on port 50051
107109
Generating feature for: 20
108110
```
109111

110-
This confirms that the Flyte workflow successfully communicated with the gRPC service.
112+
The output confirms that the Flyte workflow successfully communicated with the gRPC service.
111113

112114
## What you've learned and what's next
113115

0 commit comments

Comments
 (0)