Skip to content

Commit fcc7183

Browse files
fix(billing): plan-change forced reset + atomic usage+extras outbox (#3582)
* fix(billing): forceRotateForPlanChange + atomic usage+extras outbox 🟠 B: BillingResetService.forceRotateForPlanChange() updates current week snapshot mid-period when plan changes (was no-op when week doc existed, keeping stale quota until next reset). Webhook plan.changed handler now calls forceRotate with preserveUsage:true default. Consumers needing clean-break downgrade pass {preserveUsage:false}. 🟠 C: Outbox pattern guarantees extras debit completes even if first attempt fails. incrementMeterWithOutbox() inserts BillingMeterOutbox row alongside usage increment; out-of-band cron retry-pending-extras-debit reconciles within 5min. After 5 failed attempts, row marked 'failed' + billing.extras_debit.exhausted event emitted for downstream alerting. unique index on idempotencyKey prevents double-create races. Documents both behaviors in modules/billing/README.md. * fix(billing): guard markFailedAttempt exhaustion against concurrent cron race Add status:'pending' filter to both findOneAndUpdate calls in markFailedAttempt so concurrent K8s CronJob instances cannot double-emit billing.extras_debit.exhausted. The second cron that loses the status-flip race receives null back and skips emit. Add regression test for the concurrent-nil path. * fix(billing): harden outbox + webhook concurrent-safety - crypto.randomInt for cron jitter (resolves Codacy critical) - markCommitted: add status:pending guard so committed/failed rows are immutable - emitExhausted: wrap billingEvents.emit in try/catch so listener throws cannot cause markFailedAttempt to double-count (Copilot thread 5) - webhook: forceRotateForPlanChange no longer suppresses resetWeek when period also changed (combined plan+period change now calls both, fixes Copilot thread 8) - tests: align to new markCommitted filter + combined plan+period webhook contract * fix(billing): cron exit code + alertedAt flags on clean-break rotation - cron: exhausted rows are a handled business outcome — exit 0, warn instead of exit 1 so K8s CronJob does not treat normal reconciliation as an infrastructure failure - repository: rotateWeekSnapshotForPlanChange with preserveUsage=false now also clears alertedAt80/alertedAt100 so the new quota window can trigger threshold alerts again
1 parent 9b46842 commit fcc7183

21 files changed

Lines changed: 1386 additions & 89 deletions

modules/billing/README.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,20 @@ step. Passing the cumulative total will double-charge costs already attributed i
8989
**Backward compat**: callers that only ever attribute once (no multi-step) continue to work
9090
unchanged — the default `stepKey='initial'` makes the idempotency key `${history._id}:initial`.
9191

92+
## Plan-change semantics
93+
94+
When Stripe `plan.changed` webhook fires, devkit calls `forceRotateForPlanChange(orgId, { preserveUsage: true })` by default:
95+
- Updates `meterQuota` and `planVersion` snapshot to the new plan
96+
- Preserves `meterUsed` (no refund, no double-charge)
97+
98+
Consumers wanting clean-break behavior on downgrade should pass `{ preserveUsage: false }`.
99+
100+
## Extras debit reliability
101+
102+
`attribute()` returns optimistically after usage increment + outbox row insert. Extras debit happens out of band; if it fails, cron `retry-pending-extras-debit` reconciles within 5min. After 5 failed attempts, the outbox row is marked `failed` and event `billing.extras_debit.exhausted` is emitted for alerting.
103+
104+
Consumers should NOT retry on `applied: true` — the outbox handles eventual consistency.
105+
92106
## Stripe — `automatic_tax` flag
93107

94108
```js

modules/billing/crons/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,13 +15,15 @@ No `node-cron` dependency — orchestration is handled by Kubernetes CronJob man
1515
| `billing.weeklyReset.js` | Reset meter counters for orgs whose billing period rolled over | Daily `0 1 * * *` |
1616
| `billing.extrasExpiration.js` | Expire topup ledger entries past their `expiresAt` date | Daily `0 2 * * *` |
1717
| `billing.dunningSweep.js` | Downgrade stale `past_due` subs (>14d) to `unpaid` + `free` | Daily `0 3 * * *` |
18+
| `retry-pending-extras-debit.cron.js` | Retry pending extras debits from the meter outbox | Every 5 minutes `*/5 * * * *` |
1819

1920
## Usage
2021

2122
```sh
2223
NODE_ENV=production node modules/billing/crons/billing.weeklyReset.js
2324
NODE_ENV=production node modules/billing/crons/billing.extrasExpiration.js
2425
NODE_ENV=production node modules/billing/crons/billing.dunningSweep.js
26+
NODE_ENV=production node modules/billing/crons/retry-pending-extras-debit.cron.js
2527
```
2628

2729
Exit code 0 = success (or meterMode disabled). Exit code 1 = at least one error or fatal failure.
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
/**
2+
* Cron script — retry pending extras debits from the meter outbox.
3+
*
4+
* No-op when config.billing.meterMode === false (default).
5+
* Intended to run as a Kubernetes CronJob every 5 minutes.
6+
*
7+
* Usage:
8+
* NODE_ENV=production node modules/billing/crons/retry-pending-extras-debit.cron.js
9+
*/
10+
11+
process.env.NODE_ENV = process.env.NODE_ENV || 'development';
12+
13+
const [{ default: config }, { default: mongooseService }] = await Promise.all([
14+
import('../../../config/index.js'),
15+
import('../../../lib/services/mongoose.js'),
16+
]);
17+
18+
if (!config?.billing?.meterMode) {
19+
console.log('[billing.retryPendingExtrasDebit] meterMode disabled — skipping.');
20+
process.exit(0);
21+
}
22+
23+
const { randomInt } = await import('node:crypto');
24+
const jitterMs = randomInt(0, 60_000);
25+
await new Promise((resolve) => setTimeout(resolve, jitterMs));
26+
27+
try {
28+
await mongooseService.loadModels();
29+
await mongooseService.connect();
30+
31+
const { default: BillingMeterOutboxService } = await import('../services/billing.meter.outbox.service.js');
32+
const result = await BillingMeterOutboxService.retryPendingExtrasDebits(5 * 60 * 1000, 100);
33+
34+
console.log(
35+
`[billing.retryPendingExtrasDebit] done — scanned: ${result.scanned}, committed: ${result.committed}, failedAttempts: ${result.failedAttempts}, exhausted: ${result.exhausted}`,
36+
);
37+
if (result.exhausted > 0) {
38+
// Exhausted rows are a handled business outcome (alert event already emitted),
39+
// not an operational cron failure — log for visibility without failing the job.
40+
console.warn(`[billing.retryPendingExtrasDebit] exhausted rows (alert emitted): ${result.exhausted}`);
41+
}
42+
process.exitCode = 0;
43+
} catch (err) {
44+
console.error('[billing.retryPendingExtrasDebit] fatal:', err);
45+
process.exitCode = 1;
46+
} finally {
47+
await mongooseService.disconnect?.();
48+
}
49+
process.exit(process.exitCode ?? 0);

modules/billing/lib/events.js

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,10 @@ import { EventEmitter } from 'events';
99
* Events:
1010
* - `plan.changed` — emitted when a subscription's plan changes
1111
* Payload: { organizationId, previousPlan, newPlan, subscription, isDowngrade }
12+
* - `billing.plan_change.rotated` — emitted after current-week meter snapshot refresh
13+
* Payload: { organizationId, oldQuota, newQuota, oldVersion, newVersion, preserveUsage }
14+
* - `billing.extras_debit.exhausted` — emitted when outbox extras debit retries fail 5 times
15+
* Payload: { organizationId, idempotencyKey, extrasUnits, attempts, lastError }
1216
* - `payment.failed` — emitted when an invoice payment fails (pastDueSince set on first failure)
1317
* Payload: { organizationId }
1418
*/
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
/**
2+
* Module dependencies
3+
*/
4+
import mongoose from 'mongoose';
5+
6+
const Schema = mongoose.Schema;
7+
8+
/**
9+
* Meter outbox model.
10+
*
11+
* Stores deferred extras debits created after meter usage crosses plan quota.
12+
* Pending rows are retried by the billing retry-pending-extras-debit cron.
13+
*/
14+
const BillingMeterOutboxMongoose = new Schema({
15+
organizationId: { type: Schema.ObjectId, required: true, index: true },
16+
idempotencyKey: { type: String, required: true, unique: true },
17+
extrasUnits: { type: Number, required: true },
18+
status: { type: String, enum: ['pending', 'committed', 'failed'], default: 'pending', index: true },
19+
attempts: { type: Number, default: 0 },
20+
lastError: { type: String, default: null },
21+
lastAttemptedAt: { type: Date, default: null },
22+
createdAt: { type: Date, default: () => new Date() },
23+
});
24+
25+
BillingMeterOutboxMongoose.index({ status: 1, lastAttemptedAt: 1 });
26+
27+
/**
28+
* Returns the hex string representation of the document ObjectId.
29+
* @returns {string} Hex string of the ObjectId.
30+
*/
31+
function addID() {
32+
return this._id.toHexString();
33+
}
34+
35+
BillingMeterOutboxMongoose.virtual('id').get(addID);
36+
BillingMeterOutboxMongoose.set('toJSON', {
37+
virtuals: true,
38+
});
39+
40+
mongoose.model('BillingMeterOutbox', BillingMeterOutboxMongoose);
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
/**
2+
* Module dependencies
3+
*/
4+
import { z } from 'zod';
5+
6+
/**
7+
* BillingMeterOutbox Zod schema — mirrors billing.meter.outbox.model.mongoose.js
8+
*/
9+
10+
const objectIdRegex = /^[a-f\d]{24}$/i;
11+
12+
const BillingMeterOutboxStatus = z.enum(['pending', 'committed', 'failed']);
13+
14+
const BillingMeterOutbox = z.object({
15+
organizationId: z.string().trim().regex(objectIdRegex, 'organizationId must be a valid ObjectId'),
16+
idempotencyKey: z.string().trim().min(1, 'idempotencyKey is required'),
17+
extrasUnits: z.number().int().min(1, 'extrasUnits must be >= 1'),
18+
status: BillingMeterOutboxStatus.default('pending'),
19+
attempts: z.number().int().min(0).default(0),
20+
lastError: z.string().nullable().default(null),
21+
lastAttemptedAt: z.coerce.date().nullable().default(null),
22+
createdAt: z.coerce.date().optional(),
23+
});
24+
25+
const BillingMeterOutboxCreate = z.object({
26+
organizationId: z.string().trim().regex(objectIdRegex, 'organizationId must be a valid ObjectId'),
27+
idempotencyKey: z.string().trim().min(1, 'idempotencyKey is required'),
28+
extrasUnits: z.number().int().min(1, 'extrasUnits must be >= 1'),
29+
});
30+
31+
export default {
32+
BillingMeterOutboxStatus,
33+
BillingMeterOutbox,
34+
BillingMeterOutboxCreate,
35+
};
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
/**
2+
* Module dependencies
3+
*/
4+
import mongoose from 'mongoose';
5+
6+
/**
7+
* @function BillingMeterOutbox
8+
* @description Lazily resolves the BillingMeterOutbox Mongoose model.
9+
* Deferred to keep unit tests importable before model registration.
10+
* @returns {import('mongoose').Model} The registered BillingMeterOutbox model.
11+
*/
12+
// biome-ignore lint/correctness/useQwikValidLexicalScope: false positive — Node.js repository, not Qwik
13+
const BillingMeterOutbox = () => mongoose.model('BillingMeterOutbox');
14+
15+
/**
16+
* @function create
17+
* @description Insert a pending outbox row for a deferred extras debit.
18+
* @param {Object} payload - Outbox row fields.
19+
* @param {string} payload.organizationId - Organization ObjectId.
20+
* @param {string} payload.idempotencyKey - Usage attribution idempotency key.
21+
* @param {number} payload.extrasUnits - Extras units to debit.
22+
* @param {Object} [options={}] - Optional write options.
23+
* @param {import('mongoose').ClientSession} [options.session] - Optional Mongo session.
24+
* @returns {Promise<Object>} Inserted outbox document.
25+
*/
26+
// biome-ignore lint/correctness/useQwikValidLexicalScope: false positive — Node.js repository, not Qwik
27+
const create = async ({ organizationId, idempotencyKey, extrasUnits }, options = {}) => {
28+
const docs = await BillingMeterOutbox().create(
29+
[{
30+
organizationId,
31+
idempotencyKey,
32+
extrasUnits,
33+
status: 'pending',
34+
}],
35+
options.session ? { session: options.session } : undefined,
36+
);
37+
return docs[0];
38+
};
39+
40+
/**
41+
* @function findPendingDue
42+
* @description Return pending outbox rows whose last attempt is due for retry.
43+
* Rows with lastAttemptedAt=null are due immediately.
44+
* @param {number} [thresholdMs=300000] - Retry backoff threshold in milliseconds.
45+
* @param {number} [limit=100] - Maximum rows to return.
46+
* @returns {Promise<Object[]>} Pending due outbox rows.
47+
*/
48+
// biome-ignore lint/correctness/useQwikValidLexicalScope: false positive — Node.js repository, not Qwik
49+
const findPendingDue = (thresholdMs = 5 * 60 * 1000, limit = 100) => {
50+
const dueBefore = new Date(Date.now() - thresholdMs);
51+
return BillingMeterOutbox()
52+
.find({
53+
status: 'pending',
54+
$or: [
55+
{ lastAttemptedAt: null },
56+
{ lastAttemptedAt: { $lt: dueBefore } },
57+
],
58+
})
59+
.sort({ lastAttemptedAt: 1, createdAt: 1 })
60+
.limit(limit)
61+
.lean();
62+
};
63+
64+
/**
65+
* @function markCommitted
66+
* @description Mark an outbox row as committed after a successful extras debit.
67+
* The `status:'pending'` filter makes this idempotent: committed or
68+
* failed rows are immutable and concurrent calls are no-ops.
69+
* @param {string} id - Outbox row id.
70+
* @returns {Promise<Object>} Mongo update result.
71+
*/
72+
// biome-ignore lint/correctness/useQwikValidLexicalScope: false positive — Node.js repository, not Qwik
73+
const markCommitted = (id) =>
74+
BillingMeterOutbox().updateOne(
75+
{ _id: id, status: 'pending' },
76+
{ $set: { status: 'committed', lastError: null, lastAttemptedAt: new Date() } },
77+
);
78+
79+
/**
80+
* @function markFailedAttempt
81+
* @description Record a failed debit attempt. The fifth failed attempt exhausts
82+
* the row and moves it to failed status atomically. The status
83+
* transition uses `{ status: 'pending' }` as a filter on the
84+
* exhaustion update so that concurrent cron runs cannot emit
85+
* duplicate exhausted events.
86+
* @param {string} id - Outbox row id.
87+
* @param {Error|string} error - Failure to record.
88+
* @returns {Promise<Object|null>} Updated outbox row after failure accounting.
89+
*/
90+
// biome-ignore lint/correctness/useQwikValidLexicalScope: false positive — Node.js repository, not Qwik
91+
const markFailedAttempt = async (id, error) => {
92+
const message = error?.message ?? String(error);
93+
const doc = await BillingMeterOutbox().findOneAndUpdate(
94+
{ _id: id, status: 'pending' },
95+
{
96+
$inc: { attempts: 1 },
97+
$set: {
98+
lastError: message,
99+
lastAttemptedAt: new Date(),
100+
},
101+
},
102+
{ returnDocument: 'after' },
103+
).lean();
104+
105+
if (!doc) return null;
106+
if (doc.attempts >= 5) {
107+
// Atomic exhaustion transition: filter on status:'pending' ensures only
108+
// the first concurrent caller wins the status flip and owns the event emit.
109+
return BillingMeterOutbox().findOneAndUpdate(
110+
{ _id: id, status: 'pending' },
111+
{ $set: { status: 'failed' } },
112+
{ returnDocument: 'after' },
113+
).lean();
114+
}
115+
return doc;
116+
};
117+
118+
export default {
119+
create,
120+
findPendingDue,
121+
markCommitted,
122+
markFailedAttempt,
123+
};

0 commit comments

Comments
 (0)