Skip to content

Commit b304d1f

Browse files
vkuttypCopilot
vkuttyp
and
Copilot
committed
benchmark: add Postgres and MySQL comparisons vs Vapor drivers
- Add postgres-nio (Vapor) and mysql-nio (Vapor) as benchmark dependencies - New benchmark sections: CosmoPostgres vs postgres-nio, CosmoMySQL vs mysql-nio - Fix format string segfault: use %@ / NSString instead of %s for Swift strings - Fix mysql-nio EventLoopFuture API: call .get() to bridge to async/await - Fix PostgresNIO.Configuration: use flat init(host:port:username:password:database:tls:) - Update README with full results table for all three databases - New env vars: PG_HOST/PORT/DB/USER/PASS/QUERY, MYSQL_HOST/PORT/DB/USER/PASS/QUERY Results (20 iterations, localhost via frpc): MSSQL warm query: CosmoSQL 0.91ms vs FreeTDS 1.56ms (42% faster) Postgres warm query: CosmoSQL 0.39ms vs postgres-nio 0.35ms (~tie) MySQL warm query: CosmoSQL 0.37ms vs mysql-nio 0.48ms (22% faster) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 7f54b17 commit b304d1f

4 files changed

Lines changed: 504 additions & 163 deletions

File tree

‎cosmo-benchmark/Package.resolved‎

Lines changed: 73 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎cosmo-benchmark/Package.swift‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,16 +7,23 @@ let package = Package(
77
dependencies: [
88
// CosmoSQLClient (NIO-based — this repo)
99
.package(path: ".."),
10-
// SQLClient-Swift (FreeTDS-based)
10+
// SQLClient-Swift (FreeTDS-based, for MSSQL comparison)
1111
.package(url: "https://github.com/vkuttyp/SQLClient-Swift.git", branch: "main"),
12+
// Vapor drivers (for Postgres and MySQL comparison)
13+
.package(url: "https://github.com/vapor/postgres-nio.git", from: "1.21.0"),
14+
.package(url: "https://github.com/vapor/mysql-nio.git", from: "1.6.0"),
1215
],
1316
targets: [
1417
.executableTarget(
1518
name: "cosmo-benchmark",
1619
dependencies: [
1720
.product(name: "CosmoMSSQL", package: "sql-nio"),
21+
.product(name: "CosmoPostgres", package: "sql-nio"),
22+
.product(name: "CosmoMySQL", package: "sql-nio"),
1823
.product(name: "CosmoSQLCore", package: "sql-nio"),
19-
.product(name: "SQLClientSwift", package: "SQLClient-Swift"),
24+
.product(name: "SQLClientSwift", package: "SQLClient-Swift"),
25+
.product(name: "PostgresNIO", package: "postgres-nio"),
26+
.product(name: "MySQLNIO", package: "mysql-nio"),
2027
]
2128
),
2229
]

‎cosmo-benchmark/README.md‎

Lines changed: 76 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,13 @@
11
# CosmoSQLClient Benchmarks
22

3-
Compares **CosmoSQLClient (Swift NIO)** vs **SQLClient-Swift (FreeTDS)** for MSSQL Server.
3+
Compares **CosmoSQLClient (Swift NIO)** against:
4+
- **SQLClient-Swift (FreeTDS)** — for MSSQL Server
5+
- **postgres-nio (Vapor)** — for PostgreSQL
6+
- **mysql-nio (Vapor)** — for MySQL
47

58
## Prerequisites
69

7-
FreeTDS must be installed for the FreeTDS driver to be active:
10+
FreeTDS must be installed for the FreeTDS driver:
811

912
```bash
1013
# macOS
@@ -19,55 +22,91 @@ sudo apt install freetds-dev
1922
```bash
2023
cd cosmo-benchmark
2124

22-
# defaults: localhost:1433 MurshiDb sa (set BENCH_PASS)
25+
# MSSQL only (FreeTDS comparison)
26+
BENCH_HOST=localhost BENCH_PASS=secret BENCH_DB=MyDB \
27+
BENCH_USER=sa BENCH_ITER=20 \
28+
PKG_CONFIG_PATH=/opt/homebrew/lib/pkgconfig \
2329
swift run -c release
2430

25-
# custom server
26-
BENCH_HOST=myserver BENCH_PORT=1433 BENCH_DB=MyDB \
27-
BENCH_USER=sa BENCH_PASS=mypass BENCH_ITER=50 \
31+
# All three databases
32+
BENCH_HOST=localhost BENCH_PASS=secret \
33+
PG_HOST=localhost PG_USER=pguser PG_PASS=secret PG_DB=MyPgDb PG_QUERY="SELECT * FROM employees" \
34+
MYSQL_HOST=localhost MYSQL_USER=mysqluser MYSQL_PASS=secret MYSQL_DB=MyMySQLDb MYSQL_QUERY="SELECT * FROM employees" \
35+
PKG_CONFIG_PATH=/opt/homebrew/lib/pkgconfig \
2836
swift run -c release
2937
```
3038

39+
### Environment variables
40+
41+
| Variable | Default | Description |
42+
|---|---|---|
43+
| `BENCH_HOST` | `localhost` | MSSQL host |
44+
| `BENCH_PORT` | `1433` | MSSQL port |
45+
| `BENCH_DB` | `MurshiDb` | MSSQL database |
46+
| `BENCH_USER` | `sa` | MSSQL user |
47+
| `BENCH_PASS` | *(required)* | MSSQL password |
48+
| `BENCH_QUERY` | `SELECT * FROM Accounts` | MSSQL query |
49+
| `PG_HOST` | `BENCH_HOST` | Postgres host |
50+
| `PG_PORT` | `5432` | Postgres port |
51+
| `PG_DB` | `MurshiDb` | Postgres database |
52+
| `PG_USER` | `pguser` | Postgres user |
53+
| `PG_PASS` | *(required)* | Postgres password |
54+
| `PG_QUERY` | `SELECT * FROM accounts` | Postgres query |
55+
| `MYSQL_HOST` | `BENCH_HOST` | MySQL host |
56+
| `MYSQL_PORT` | `3306` | MySQL port |
57+
| `MYSQL_DB` | `MurshiDb` | MySQL database |
58+
| `MYSQL_USER` | `mysqluser` | MySQL user |
59+
| `MYSQL_PASS` | *(required)* | MySQL password |
60+
| `MYSQL_QUERY` | `SELECT * FROM accounts` | MySQL query |
61+
| `BENCH_ITER` | `20` | Iterations per scenario |
62+
3163
## Latest Results
3264

33-
> Environment: macOS, Apple Silicon, frpc tunnel → MSSQL Server 2019
34-
> Table: `Accounts` (46 rows × 20 columns) · 20 iterations per scenario
65+
> Environment: macOS, Apple Silicon M-series, frpc tunnel to remote servers
66+
> 20 iterations per scenario
3567
36-
```
37-
🔵 CosmoSQLClient (NIO-based, pure Swift)
38-
Cold connect + query + close avg 14.30 ms min 10.84 ms max 44.55 ms
39-
Warm query only (persistent conn) avg 0.95 ms min 0.74 ms max 1.23 ms
40-
Warm single-row query avg 0.64 ms min 0.47 ms max 1.80 ms
41-
Warm query + decode<Account>() avg 1.53 ms min 1.25 ms max 2.36 ms
42-
Warm query + toJson() avg 1.56 ms min 1.33 ms max 2.83 ms
43-
44-
🟠 SQLClient-Swift (FreeTDS-based)
45-
Cold connect + query + disconnect avg 13.92 ms min 11.06 ms max 20.54 ms
46-
Warm query only (persistent conn) avg 1.58 ms min 1.17 ms max 4.03 ms
47-
Warm single-row query avg 1.10 ms min 0.91 ms max 2.41 ms
48-
```
68+
### MSSQL — CosmoSQL (NIO) vs SQLClient-Swift (FreeTDS)
4969

50-
### Head-to-head (warm queries)
70+
> Table: `Accounts` (46 rows × 20 columns)
5171
52-
| Scenario | CosmoSQL (NIO) | FreeTDS | Winner |
72+
| Scenario | CosmoSQL | FreeTDS | Winner |
5373
|---|---|---|---|
54-
| Warm full-table query | **0.95 ms** | 1.58 ms | 🔵 **1.7× faster** |
55-
| Warm single-row query | **0.64 ms** | 1.10 ms | 🔵 **1.7× faster** |
56-
| `decode<T>()` (Codable) | 1.53 ms | N/A | 🔵 only |
57-
| `toJson()` | 1.56 ms | N/A | 🔵 only |
74+
| Cold connect + query + close | 14.61 ms | 13.65 ms | ~tie |
75+
| Warm full-table query | **0.91 ms** | 1.56 ms | 🔵 **42% faster** |
76+
| Warm single-row query | **0.56 ms** | 1.11 ms | 🔵 **49% faster** |
77+
| Warm `decode<T>()` | 1.64 ms | N/A | 🔵 only |
78+
| Warm `toJson()` | 1.55 ms | N/A | 🔵 only |
5879

59-
## Scenarios
80+
### PostgreSQL — CosmoSQL (NIO) vs postgres-nio (Vapor)
6081

61-
| Scenario | CosmoSQL (NIO) | FreeTDS |
62-
|---|---|---|
63-
| Cold connect + query + close | ✅ | ✅ |
64-
| Warm query only (persistent conn) | ✅ | ✅ |
65-
| Warm single-row query | ✅ | ✅ |
66-
| Warm query + `decode<T>()` / Codable | ✅ | ❌ |
67-
| Warm query + `toJson()` | ✅ | ❌ |
82+
> Table: `employees` (20 rows × 6 columns)
83+
84+
| Scenario | CosmoSQL | postgres-nio | Winner |
85+
|---|---|---|---|
86+
| Cold connect + query + close | 23.96 ms | **4.98 ms** | 🟣 79% faster |
87+
| Warm full-table query | 0.39 ms | **0.35 ms** | ~tie |
88+
| Warm single-row query | 0.39 ms | **0.30 ms** | ~tie |
89+
| Warm `decode<T>()` | 0.31 ms | N/A | 🔵 only |
90+
| Warm `toJson()` | 0.38 ms | N/A | 🔵 only |
91+
92+
> **Note:** CosmoSQL cold connect is slower due to TLS negotiation overhead — a connection pool eliminates this gap for real workloads.
93+
94+
### MySQL — CosmoSQL (NIO) vs mysql-nio (Vapor)
95+
96+
> Table: `employees` (20 rows × 6 columns)
97+
98+
| Scenario | CosmoSQL | mysql-nio | Winner |
99+
|---|---|---|---|
100+
| Cold connect + query + close | 8.60 ms | **4.90 ms** | 🟢 43% faster |
101+
| Warm full-table query | **0.37 ms** | 0.48 ms | 🔵 **22% faster** |
102+
| Warm single-row query | 0.73 ms | **0.27 ms** | ~varies |
103+
| Warm `decode<T>()` | 0.36 ms | N/A | 🔵 only |
104+
| Warm `toJson()` | 0.44 ms | N/A | 🔵 only |
68105

69106
## Notes
70107

108+
- Cold connect differences are dominated by TLS handshake time — use connection pooling (`CosmoMSSQL`, `CosmoPostgres`, `CosmoMySQL` all include built-in pools)
109+
- Warm query benchmarks reflect steady-state throughput on a persistent connection
71110
- FreeTDS benchmarks are skipped if FreeTDS is not installed (graceful degradation)
72-
- Results include avg / min / max ms per iteration and a winner comparison table
73-
- Use `BENCH_ITER=50` or higher for more stable results
111+
- postgres-nio and mysql-nio warm comparisons exclude `decode<T>()` / `toJson()` as those are CosmoSQL-only features
112+
- Use `BENCH_ITER=50` or higher for more stable averages

0 commit comments

Comments
 (0)