Skip to content

Commit ed9d6bc

Browse files
lohanidamodarlohanidamodar
committed
Refactor Usage class for improved metric handling
- Introduced increment and set methods for metrics, replacing log and logCounter. - Added in-memory buffers for increment and counter metrics with automatic flushing. - Implemented collect and collectSet methods for deferred metric accumulation. - Enhanced flush logic to handle both increment and set metrics, clearing buffers post-flush. - Added configuration options for flush thresholds and intervals. - Updated tests to reflect changes in metric logging and buffer management.
1 parent 929ab40 commit ed9d6bc

7 files changed

Lines changed: 753 additions & 398 deletions

File tree

‎README.md‎

Lines changed: 96 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,11 @@ Although this library is part of the [Utopia Framework](https://github.com/utopi
1414
- **Database Adapter**: Store metrics in any SQL database via utopia-php/database
1515
- **ClickHouse Adapter**: High-performance analytics storage for massive scale
1616
- **Flexible Periods**: Hourly (1h), Daily (1d), and Infinite (inf) periods
17-
- **Batch Operations**: Log multiple metrics efficiently
17+
- **Dual Upsert Semantics**: Additive (`increment`) and replace (`set`) upserts
18+
- **In-Memory Buffering**: Collect metrics and flush in batch for high-throughput scenarios
19+
- **Auto Period Fan-Out**: `increment()`, `set()`, `collect()`, `collectSet()` automatically write to all periods
20+
- **Batch Operations**: `incrementBatch()` and `setBatch()` for efficient bulk writes
21+
- **Async Inserts**: ClickHouse adapter supports server-side async inserts
1822
- **Rich Queries**: Filter, limit, offset, and aggregate metrics
1923
- **Tag Support**: Add custom tags for multi-dimensional analytics
2024

@@ -104,67 +108,95 @@ $adapter = new Database($database);
104108
$usage = Usage::withAdapter($adapter);
105109
$usage->setup();
106110
```
107-
**Log Usage**
108111

109-
A simple example for logging a usage metric.
112+
## Metric Types
113+
114+
The library supports two types of metrics with different upsert semantics:
115+
116+
### Increment (Additive Upsert)
117+
118+
Values are **summed** when the same metric/period/time bucket already exists. Use for event-driven counters like request counts, bandwidth, etc.
110119

111120
```php
112-
$metric = 'requests';
113-
$value = 100;
114-
$period = '1h'; // Supported periods: '1h', '1d', 'inf'
115-
$tags = ['region' => 'us-east', 'method' => 'GET'];
121+
// Single metric, auto fan-out to all periods (1h, 1d, inf)
122+
$usage->increment('requests', 1);
123+
$usage->increment('bandwidth', 5000, ['region' => 'us-east']);
124+
125+
// Batch with explicit periods
126+
$usage->incrementBatch([
127+
['metric' => 'requests', 'value' => 100, 'period' => '1h', 'tags' => ['method' => 'GET']],
128+
['metric' => 'bandwidth', 'value' => 50000, 'period' => '1h', 'tags' => ['region' => 'us-east']],
129+
]);
130+
```
116131

117-
$usage->log($metric, $value, $period, $tags);
132+
### Set (Replace Upsert)
133+
134+
Values **replace** the existing value when the same metric/period/time bucket already exists. Use for periodic recounts or resource gauges (e.g., current storage size, active user count).
135+
136+
```php
137+
// Single metric, auto fan-out to all periods (1h, 1d, inf)
138+
$usage->set('storage.size', 1048576);
139+
$usage->set('users.active', 42, ['plan' => 'pro']);
140+
141+
// Batch with explicit periods
142+
$usage->setBatch([
143+
['metric' => 'storage.size', 'value' => 1048576, 'period' => '1h', 'tags' => []],
144+
['metric' => 'users.active', 'value' => 42, 'period' => '1d', 'tags' => []],
145+
]);
118146
```
119147

120-
**Log Batch Usage**
148+
## In-Memory Buffering
121149

122-
Log multiple metrics in batch for better performance.
150+
For high-throughput scenarios (e.g., inside a request loop or worker), use `collect()` / `collectSet()` to accumulate metrics in memory and `flush()` to write them in batch.
123151

124152
```php
125-
$metrics = [
126-
[
127-
'metric' => 'requests',
128-
'value' => 100,
129-
'period' => '1h',
130-
'tags' => ['region' => 'us-east'],
131-
],
132-
[
133-
'metric' => 'bandwidth',
134-
'value' => 50000,
135-
'period' => '1h',
136-
'tags' => ['region' => 'us-east'],
137-
],
138-
];
139-
140-
$usage->logBatch($metrics);
153+
// Accumulate increment metrics (values are summed in-memory)
154+
$usage->collect('requests', 1);
155+
$usage->collect('requests', 1);
156+
$usage->collect('bandwidth', 5000);
157+
158+
// Accumulate set metrics (last-write-wins in-memory)
159+
$usage->collectSet('storage.size', 1048576);
160+
161+
// Check if flush is recommended (threshold or interval reached)
162+
if ($usage->shouldFlush()) {
163+
$usage->flush();
164+
}
165+
166+
// Or flush explicitly
167+
$usage->flush();
141168
```
142169

143-
**Get Usage By Period**
170+
### Flush Configuration
144171

145-
Fetch all usage metrics by period.
172+
```php
173+
// Flush when 5000 collect() calls have been made (default: 10,000)
174+
$usage->setFlushThreshold(5000);
175+
176+
// Flush when 10 seconds have elapsed since last flush (default: 20)
177+
$usage->setFlushInterval(10);
178+
```
179+
180+
## Querying Metrics
181+
182+
**Get Usage By Period**
146183

147184
```php
148185
$metrics = $usage->getByPeriod('requests', '1h');
149-
// Returns an array of all usage metrics for specific period
186+
// Returns an array of Metric objects
150187
```
151188

152189
**Get Usage Between Dates**
153190

154-
Fetch all usage metrics between two dates.
155-
156191
```php
157192
$start = '2024-01-01 00:00:00';
158193
$end = '2024-01-31 23:59:59';
159194

160195
$metrics = $usage->getBetweenDates('requests', $start, $end);
161-
// Returns an array of usage metrics within the date range
162196
```
163197

164198
**Count and Sum Usage**
165199

166-
Get counts and sums of usage metrics.
167-
168200
```php
169201
// Count total records
170202
$count = $usage->countByPeriod('requests', '1h');
@@ -173,9 +205,24 @@ $count = $usage->countByPeriod('requests', '1h');
173205
$sum = $usage->sumByPeriod('requests', '1h');
174206
```
175207

176-
**Purge Old Usage**
208+
**Find with Query Objects**
177209

178-
Delete old usage metrics.
210+
```php
211+
use Utopia\Usage\Query;
212+
213+
$metrics = $usage->find([
214+
Query::equal('metric', ['requests', 'bandwidth']),
215+
Query::greaterThan('value', 100),
216+
Query::orderDesc('time'),
217+
Query::limit(10),
218+
]);
219+
220+
$count = $usage->count([
221+
Query::equal('period', ['1h']),
222+
]);
223+
```
224+
225+
**Purge Old Usage**
179226

180227
```php
181228
use Utopia\Database\DateTime;
@@ -202,23 +249,21 @@ The Database adapter uses [utopia-php/database](https://github.com/utopia-php/da
202249
- Works with MySQL, MariaDB, PostgreSQL, SQLite
203250
- Full query support (filters, sorting, pagination)
204251
- ACID compliance for data consistency
205-
- Easy migration from existing databases
206-
207-
**Example**:
208-
```php
209-
$usage = Usage::withDatabase($database);
210-
```
252+
- Additive upsert via `upsertDocumentsWithIncrease`
253+
- Replace upsert via `upsertDocuments`
211254

212255
### ClickHouse Adapter
213256

214257
The ClickHouse adapter uses the HTTP interface to store metrics in ClickHouse for high-performance analytics.
215258

216259
**Features**:
217-
- Optimized for analytical queries
218-
- Handles millions of metrics per second
260+
- SummingMergeTree for additive upserts (`usage` table)
261+
- ReplacingMergeTree for replace upserts (`usage_snapshot` table)
219262
- Automatic partitioning by month
220263
- Efficient compression and storage
221264
- Bloom filter indexes for fast lookups
265+
- Async insert support for server-side batching
266+
- Deterministic IDs for correct merge behavior
222267

223268
**Example**:
224269
```php
@@ -230,10 +275,9 @@ $usage = Usage::withClickHouse(
230275
secure: true // Use HTTPS
231276
);
232277

233-
// Configure database and table (optional)
278+
// Enable async inserts (server-side batching)
234279
$adapter = $usage->getAdapter();
235-
$adapter->setDatabase('analytics');
236-
$adapter->setTable('metrics');
280+
$adapter->setAsyncInserts(true, waitForConfirmation: true);
237281

238282
$usage->setup();
239283
```
@@ -244,13 +288,16 @@ Extend the `Utopia\Usage\Adapter` abstract class and implement these methods:
244288

245289
- `getName(): string` - Return adapter name
246290
- `setup(): void` - Initialize storage structure
247-
- `log(string $metric, int $value, string $period, array $tags): bool` - Log single metric
248-
- `logBatch(array $metrics): bool` - Log multiple metrics
291+
- `healthCheck(): array` - Check adapter health
292+
- `incrementBatch(array $metrics, int $batchSize): bool` - Additive upsert batch
293+
- `setBatch(array $metrics, int $batchSize): bool` - Replace upsert batch
249294
- `getByPeriod(string $metric, string $period, array $queries): array` - Get metrics by period
250295
- `getBetweenDates(string $metric, string $startDate, string $endDate, array $queries): array` - Get metrics in date range
251296
- `countByPeriod(string $metric, string $period, array $queries): int` - Count metrics
252297
- `sumByPeriod(string $metric, string $period, array $queries): int` - Sum metric values
253298
- `purge(string $datetime): bool` - Delete old metrics
299+
- `find(array $queries): array` - Find metrics with query objects
300+
- `count(array $queries): int` - Count metrics with query objects
254301

255302
## System Requirements
256303

‎src/Usage/Adapter.php‎

Lines changed: 64 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,62 @@ abstract class Adapter
99
*/
1010
abstract public function getName(): string;
1111

12+
/**
13+
* Increment a metric across all periods (1h, 1d, inf).
14+
*
15+
* Uses additive upsert semantics: if a row with the same deterministic ID exists,
16+
* the value is added to the existing value (SummingMergeTree in ClickHouse,
17+
* upsertDocumentsWithIncrease in Database).
18+
*
19+
* @param string $metric Metric name
20+
* @param int $value Value to add (must be positive)
21+
* @param array<string,mixed> $tags Optional tags
22+
* @return bool
23+
* @throws \Exception
24+
*/
25+
public function increment(string $metric, int $value, array $tags = []): bool
26+
{
27+
$metrics = [];
28+
foreach (array_keys(Usage::PERIODS) as $period) {
29+
$metrics[] = [
30+
'metric' => $metric,
31+
'value' => $value,
32+
'period' => $period,
33+
'tags' => $tags,
34+
];
35+
}
36+
37+
return $this->incrementBatch($metrics);
38+
}
39+
40+
/**
41+
* Set a metric to an absolute value across all periods (1h, 1d, inf).
42+
*
43+
* Uses replace upsert semantics: if a row with the same deterministic ID exists,
44+
* the value replaces the existing value (ReplacingMergeTree in ClickHouse,
45+
* upsertDocuments in Database).
46+
*
47+
* @param string $metric Metric name
48+
* @param int $value Absolute value to set
49+
* @param array<string,mixed> $tags Optional tags
50+
* @return bool
51+
* @throws \Exception
52+
*/
53+
public function set(string $metric, int $value, array $tags = []): bool
54+
{
55+
$metrics = [];
56+
foreach (array_keys(Usage::PERIODS) as $period) {
57+
$metrics[] = [
58+
'metric' => $metric,
59+
'value' => $value,
60+
'period' => $period,
61+
'tags' => $tags,
62+
];
63+
}
64+
65+
return $this->setBatch($metrics);
66+
}
67+
1268
/**
1369
* Check adapter health and connection status
1470
*
@@ -22,34 +78,26 @@ abstract public function healthCheck(): array;
2278
abstract public function setup(): void;
2379

2480
/**
25-
* Log usage metric
81+
* Increment metrics in batch (additive upsert).
2682
*
27-
* @param array<string,mixed> $tags
28-
*/
29-
abstract public function log(string $metric, int $value, string $period = Usage::PERIOD_1H, array $tags = []): bool;
30-
31-
/**
32-
* Log multiple metrics in batch
83+
* Values with the same deterministic ID are summed together
84+
* (SummingMergeTree in ClickHouse, upsertDocumentsWithIncrease in Database).
3385
*
3486
* @param array<array{metric: string, value: int, period?: string, tags?: array<string,mixed>}> $metrics
3587
* @param int $batchSize Maximum number of metrics per INSERT statement
3688
*/
37-
abstract public function logBatch(array $metrics, int $batchSize = 1000): bool;
89+
abstract public function incrementBatch(array $metrics, int $batchSize = 1000): bool;
3890

3991
/**
40-
* Log usage counter metric (individual entry without aggregation)
92+
* Set metrics in batch (replace upsert).
4193
*
42-
* @param array<string,mixed> $tags
43-
*/
44-
abstract public function logCounter(string $metric, int $value, string $period = Usage::PERIOD_1H, array $tags = []): bool;
45-
46-
/**
47-
* Log multiple counter metrics in batch (individual entries without aggregation)
94+
* Values with the same deterministic ID are replaced (last write wins)
95+
* (ReplacingMergeTree in ClickHouse, upsertDocuments in Database).
4896
*
4997
* @param array<array{metric: string, value: int, period?: string, tags?: array<string,mixed>}> $metrics
5098
* @param int $batchSize Maximum number of metrics per INSERT statement
5199
*/
52-
abstract public function logBatchCounter(array $metrics, int $batchSize = 1000): bool;
100+
abstract public function setBatch(array $metrics, int $batchSize = 1000): bool;
53101

54102
/**
55103
* Get usage metrics by period

0 commit comments

Comments
 (0)