Skip to content

Commit 1c7c88e

Browse files
jasonrandrewsjasonrandrews
authored
Merge pull request #3056 from odidev/minio
Deploy and Use MinIO on Azure Cobalt 100 Arm64 Virtual Machines
2 parents 88b5e8b + 6108215 commit 1c7c88e

21 files changed

Lines changed: 827 additions & 0 deletions

‎content/learning-paths/servers-and-cloud-computing/minio-cobalt/_index.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
---
2+
title: Deploy and Use MinIO on Azure Cobalt 100 Arm64 Virtual Machines
3+
4+
minutes_to_complete: 30
5+
6+
who_is_this_for: This learning path is designed for developers, DevOps engineers, and platform engineers who want to deploy and use S3-compatible object storage using MinIO on Arm-based cloud environments.
7+
8+
learning_objectives:
9+
- Deploy MinIO on Azure Cobalt 100 Arm64 virtual machines
10+
- Configure and manage object storage using MinIO
11+
- Benchmark storage performance for high-throughput workloads
12+
- Validate S3 compatibility using Python SDK
13+
- Implement AI/ML dataset and model storage workflows using MinIO
14+
15+
prerequisites:
16+
- A [Microsoft Azure account](https://azure.microsoft.com/) with access to Cobalt 100 based instances (Dpsv6)
17+
- Basic knowledge of Linux command-line operations
18+
- Familiarity with SSH and remote server access
19+
- Basic understanding of cloud storage concepts
20+
21+
author: Pareena Verma
22+
23+
### Tags
24+
skilllevels: Introductory
25+
subjects: Databases
26+
cloud_service_providers:
27+
- Microsoft Azure
28+
29+
armips:
30+
- Neoverse
31+
32+
tools_software_languages:
33+
- MinIO
34+
- Python
35+
- MinIO Client (mc)
36+
37+
operatingsystems:
38+
- Linux
39+
40+
further_reading:
41+
- resource:
42+
title: MinIO Official Website
43+
link: https://min.io/
44+
type: website
45+
- resource:
46+
title: MinIO Documentation
47+
link: https://min.io/docs/minio/linux/index.html
48+
type: documentation
49+
- resource:
50+
title: MinIO GitHub Repository
51+
link: https://github.com/minio/minio
52+
type: documentation
53+
- resource:
54+
title: Azure Cobalt 100 processors
55+
link: https://techcommunity.microsoft.com/blog/azurecompute/announcing-the-preview-of-new-azure-vms-based-on-the-azure-cobalt-100-processor/4146353
56+
type: documentation
57+
58+
### FIXED, DO NOT MODIFY
59+
# ================================================================================
60+
weight: 1
61+
layout: "learningpathall"
62+
learning_path_main_page: "yes"
63+
---

‎content/learning-paths/servers-and-cloud-computing/minio-cobalt/_next-steps.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
# ================================================================================
3+
# FIXED, DO NOT MODIFY THIS FILE
4+
# ================================================================================
5+
weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation.
6+
title: "Next Steps" # Always the same, html page title.
7+
layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing.
8+
---

‎content/learning-paths/servers-and-cloud-computing/minio-cobalt/ai-ml-usecase.md‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
---
2+
title: Use MinIO for AI/ML Dataset and Model Storage
3+
weight: 7
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Use MinIO for AI/ML Dataset and Model Storage
10+
11+
In this section, you simulate a real-world AI/ML workflow using MinIO.
12+
13+
MinIO serves as an object storage backend for datasets and trained models used in machine learning pipelines.
14+
15+
This demonstrates how MinIO integrates into modern data-driven and AI/ML applications.
16+
17+
18+
## Architecture overview
19+
20+
This architecture represents a simple ML workflow using object storage.
21+
22+
```text
23+
Dataset / Model Files
24+
│
25+
▼
26+
MinIO Object Storage (S3-compatible)
27+
│
28+
▼
29+
Training / Inference Workloads
30+
```
31+
32+
## Scenario
33+
34+
You will simulate a typical ML pipeline:
35+
36+
- Upload a dataset
37+
- Upload a model artifact
38+
- Retrieve data for training or inference
39+
40+
## Create dataset
41+
42+
Create a sample dataset to represent structured training data.
43+
44+
```bash
45+
mkdir ai-dataset
46+
echo "id,name,score" > ai-dataset/data.csv
47+
echo "1,jon,90" >> ai-dataset/data.csv
48+
echo "2,nick,85" >> ai-dataset/data.csv
49+
echo "3,jack,95" >> ai-dataset/data.csv
50+
```
51+
52+
**Why this matters:**
53+
54+
- Represents structured data used in ML training
55+
- Simulates dataset ingestion into object storage
56+
- Mimics real-world data lake inputs
57+
58+
## Upload dataset
59+
60+
Upload the dataset to MinIO.
61+
62+
```bash
63+
mc cp ai-dataset/data.csv local/ml-datasets/
64+
```
65+
66+
## Verify upload
67+
68+
```bash
69+
mc ls local/ml-datasets
70+
```
71+
72+
You should see `data.csv` in the output.
73+
74+
The output is similar to:
75+
76+
```output
77+
[2026-03-24 04:28:25 UTC] 43B STANDARD data.csv
78+
[2026-03-24 04:29:59 UTC] 19B STANDARD model.bin
79+
[2026-03-24 04:16:22 UTC] 13B STANDARD test.txt
80+
[2026-03-24 05:21:04 UTC] 0B dataset/
81+
```
82+
83+
**Why this matters:**
84+
85+
- Confirms successful data ingestion
86+
- Validates object storage functionality
87+
88+
## Create model artifact
89+
90+
Simulate a trained machine learning model.
91+
92+
```bash
93+
mkdir model
94+
echo "fake-model-weights" > model/model.bin
95+
```
96+
97+
**Why this matters:**
98+
99+
- Represents the output of training pipelines
100+
- Mimics model registry storage
101+
102+
## Upload model artifact
103+
104+
```bash
105+
mc cp model/model.bin local/ml-datasets/
106+
```
107+
108+
## Download data for usage
109+
110+
Simulate retrieving data for training or inference.
111+
112+
```bash
113+
mkdir download-test
114+
mc cp --recursive local/ml-datasets download-test/
115+
```
116+
117+
## Verify downloaded data
118+
119+
```bash
120+
ls download-test/ml-datasets
121+
```
122+
123+
You should see:
124+
125+
```bash
126+
data.csv dataset model.bin test.txt
127+
```
128+
129+
Why this matters:
130+
131+
- Confirms data retrieval works correctly
132+
- Simulates how ML jobs access datasets and models
133+
134+
## What this demonstrates
135+
136+
| Real-world concept | Implementation |
137+
| ------------------ | -------------------------- |
138+
| Data lake | Dataset stored in MinIO |
139+
| Model registry | model.bin stored as object |
140+
| Training input | Dataset download |
141+
| Inference | Model retrieval |
142+
143+
144+
## Explanation
145+
146+
MinIO provides S3-compatible object storage for AI/ML workflows:
147+
148+
- Datasets are stored as objects in buckets
149+
- Training jobs read datasets from storage
150+
- Models are stored after training
151+
- Inference systems retrieve models when required
152+
153+
This workflow mirrors production-grade ML pipelines used in cloud environments.
154+
155+
## What you've learned
156+
In this section, you learned how to:
157+
158+
- Use MinIO as object storage for AI/ML datasets
159+
- Store and retrieve model artifacts
160+
- Simulate a real-world ML workflow
161+
- Understand how object storage fits into data pipelines

‎content/learning-paths/servers-and-cloud-computing/minio-cobalt/background.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
---
2+
title: "Overview of Azure Cobalt 100 and MinIO"
3+
4+
weight: 2
5+
6+
layout: "learningpathall"
7+
---
8+
9+
## Azure Cobalt 100 Arm-based processor
10+
11+
Azure’s Cobalt 100 is Microsoft’s first-generation, in-house Arm-based processor. Built on Arm Neoverse N2, Cobalt 100 is a 64-bit CPU that delivers strong performance and energy efficiency for cloud-native, scale-out Linux workloads such as web and application servers, data analytics, open-source databases, and caching systems. Running at 3.4 GHz, Cobalt 100 allocates a dedicated physical core for each vCPU, which helps ensure consistent and predictable performance.
12+
13+
To learn more, see the Microsoft blog [Announcing the preview of new Azure VMs based on the Azure Cobalt 100 processor](https://techcommunity.microsoft.com/blog/azurecompute/announcing-the-preview-of-new-azure-vms-based-on-the-azure-cobalt-100-processor/4146353).
14+
15+
## MinIO
16+
17+
MinIO is a high-performance, S3-compatible object storage platform designed for cloud-native applications, AI/ML workloads, and modern data infrastructure. It is lightweight, scalable, and optimized for high-throughput and low-latency object storage operations.
18+
19+
MinIO is commonly used as a storage backend for data lakes, machine learning pipelines, backup systems, and analytics platforms. It supports the Amazon S3 API, allowing seamless integration with existing tools and SDKs that are built for AWS S3.
20+
21+
MinIO is designed to run efficiently on Arm-based architectures such as Azure Cobalt 100, enabling cost-effective and energy-efficient storage solutions for large-scale workloads.
22+
23+
24+
25+
## Key features of MinIO
26+
27+
- **S3 Compatibility:** Fully compatible with the Amazon S3 API, enabling easy integration with existing applications and tools.
28+
- **High Performance:** Optimized for high-throughput workloads such as AI/ML datasets and analytics pipelines.
29+
- **Scalability:** Supports both standalone and distributed deployments for scaling storage capacity and performance.
30+
- **Lightweight and Cloud-Native:** Minimal resource footprint and designed for containerized and cloud environments.
31+
- **Data Protection:** Supports erasure coding, encryption, and access control for secure and reliable storage.
32+
33+
34+
## MinIO architecture components
35+
36+
MinIO deployments typically consist of the following components:
37+
38+
- **MinIO Server:** The core storage service that handles object storage operations and exposes S3-compatible APIs.
39+
- **Object Storage (Buckets):** Logical containers used to store data objects such as files, datasets, and model artifacts.
40+
- **MinIO Client (mc):** A command-line tool used to interact with MinIO for uploading, downloading, and managing objects.
41+
- **Web Console:** A browser-based interface for managing buckets, objects, users, and access policies.
42+
43+
## Use cases
44+
45+
MinIO is widely used in modern cloud and data environments:
46+
47+
- **AI/ML Workloads:** Store training datasets, model artifacts, and inference data
48+
- **Data Lakes:** Serve as a scalable backend for structured and unstructured data
49+
- **Backup and Archival:** Store backups and snapshots with high durability
50+
- **Cloud-Native Applications:** Provide object storage for microservices and distributed systems
51+
- **Analytics Pipelines:** Integrate with tools like Apache Spark, Presto, and Hadoop
52+
53+
To learn more about MinIO, see:
54+
55+
- [MinIO Official Website](https://min.io/)
56+
- [MinIO Documentation](https://min.io/docs/minio/linux/index.html)
57+
- [MinIO GitHub Repository](https://github.com/minio/minio)
58+
59+
## What you will learn
60+
61+
In this learning path, you will:
62+
63+
- Deploy MinIO on an Azure Cobalt ARM virtual machine
64+
- Configure and manage object storage using MinIO
65+
- Benchmark storage performance for high-throughput workloads
66+
- Validate S3 compatibility using standard SDKs
67+
- Implement a real-world AI/ML dataset and model storage workflow

0 commit comments

Comments
 (0)