Merge pull request #11 from topcoder-platform/develop

Add handling of engagements to locked / consumed tabls

Author: jmgasper

.env.example

@@ -4,6 +4,16 @@ DATABASE_URL="postgresql://postgres:postgres@localhost:5432/topcoder-services?sc
 # Example: postgresql://postgres:postgres@localhost:5432/topcoder-services?schema=members
 MEMBER_DB_URL=""
 
+# Optional lookup stores used to resolve billing-account line-item names.
+# If unset, line items are still returned with externalName omitted.
+CHALLENGE_DB_URL="postgresql://postgres:postgres@localhost:5432/topcoder-services?schema=challenges"
+ENGAGEMENTS_DB_URL="postgresql://postgres:postgres@localhost:5432/engagements"
+
+# Historical engagement-payment backfill sources.
+# FINANCE_DB_URL points to tc-finance-api, PROJECTS_DB_URL points to projects-api-v6.
+FINANCE_DB_URL="postgresql://postgres:postgres@localhost:5432/topcoder-services?schema=finance"
+PROJECTS_DB_URL="postgresql://postgres:postgres@localhost:5432/topcoder-services?schema=projects"
+
 # Auth (tc-core-library-js)
 AUTH_SECRET=""                  
 AUTH0_URL=""                    # e.g. https://topcoder-dev.auth0.com/ (optional)

README.md

@@ -7,11 +7,11 @@
 - Endpoints:
   - `GET /billing-accounts` 
   - `POST /billing-accounts` 
-  - `GET /billing-accounts/:billingAccountId` (includes locked/consumed arrays + budget totals)
+  - `GET /billing-accounts/:billingAccountId` (includes locked/consumed arrays + budget totals; line items expose `amount`, `date`, `externalId`, `externalType`, `externalName`, and `challengeId` only for challenge compatibility)
   - `GET /billing-accounts/users/:userId` (list billing accounts accessible by the given Topcoder user ID — resolved via Salesforce resource object)
   - `PATCH /billing-accounts/:billingAccountId`
-  - `PATCH /billing-accounts/:billingAccountId/lock-amount` (0 amount = unlock)
-  - `PATCH /billing-accounts/:billingAccountId/consume-amount` (deletes locks for challenge, then upserts consumed)
+  - `PATCH /billing-accounts/:billingAccountId/lock-amount` (challenge-only typed reference; non-negative amount; 0 amount = unlock; rejects insufficient remaining funds)
+  - `PATCH /billing-accounts/:billingAccountId/consume-amount` (positive amount; challenge typed references delete the matching lock and overwrite consumed; engagement typed references append a consumed row; rejects insufficient remaining funds)
   - `GET /billing-accounts/:billingAccountId/users` (list users with access)
   - `POST /billing-accounts/:billingAccountId/users` (grant access; accepts `{ param: { userId } }` or `{ userId }`)
   - `DELETE /billing-accounts/:billingAccountId/users/:userId` (revoke access)
@@ -74,6 +74,38 @@ pnpm run build && pnpm start
     - Lookup the matching `user_account` by `user_account_id`, then query the Members DB by `user_name` (handle)
     - Upsert `BillingAccountAccess` for the resolved `userId`
 
+- Engagement payment consumed backfill:
+  - Required env:
+    - `DATABASE_URL` for billing-accounts-api-v6.
+    - `FINANCE_DB_URL` for tc-finance-api (`winnings` and `payment`).
+    - `ENGAGEMENTS_DB_URL` for engagements-api-v6 (`EngagementAssignment` and `Engagement`).
+    - `PROJECTS_DB_URL` for projects-api-v6 (`projects.billingAccountId`, matching the trusted project lookup used by engagements-api-v6).
+    - Optional `TGBillingAccounts` for finance-compatible TopGear exemptions. If unset, the script uses finance's default exempt accounts `80000062,80002800`.
+  - Dry run:
+    - `pnpm run backfill:engagementPayments -- --dry-run`
+    - Dry run is the default. It reads finance engagement payments, resolves assignment/project/billing account context, plans inserts/updates, and writes a JSON audit report under `scripts/output/`.
+  - Apply:
+    - `pnpm run backfill:engagementPayments -- --apply`
+    - Optional filters: `--assignmentId=<id>[,<id>]`, `--winningId=<id>[,<id>]`, `--since=<iso-date>`, `--until=<iso-date>`, `--limit=<n>`, `--report=<path>`.
+  - Behavior:
+    - Uses `payment.challenge_markup` first as a markup rate and computes `total_amount + (total_amount * challenge_markup)`.
+    - If `payment.challenge_markup` is absent and `payment.challenge_fee` is present, treats `challenge_fee` as an absolute fee amount, computes `total_amount + challenge_fee`, and records the row in the `absoluteFee` audit bucket.
+    - If neither finance value is present, falls back to the current `BillingAccount.markup` and records that fallback in the audit report.
+    - Skips automated consumed-row planning for TopGear-exempt billing accounts and records those payments in the `exemptBillingAccounts` audit bucket.
+    - Apply mode runs billing mutations, post-apply total calculation, and report writing inside one billing database transaction. If a write, verification read, or report write fails before commit, the billing mutations are rolled back.
+    - Reconciles one assignment-level aggregate row for historical idempotency. Existing correct rows are left alone; a single incorrect row is updated; multiple existing rows are only moved when their total already matches the legacy expected amount.
+    - Exceptions such as missing assignments, missing project billing accounts, missing billing accounts, and ambiguous consumed duplicates are reported without blocking resolvable assignments.
+  - Validation:
+    - The dry-run/apply report includes expected legacy totals and current/projected billing totals.
+    - `verify:engagementPayments` invokes `psql` directly, so export `DATABASE_URL` in the shell before running it.
+    - If the source schemas are visible in one Postgres session, run:
+      - `pnpm run verify:engagementPayments -- -v finance_schema=finance -v engagements_schema=public -v projects_schema=projects -v billing_schema=billing-accounts -v exempt_billing_account_ids=80000062,80002800`
+    - The verification SQL lists mismatches between expected legacy engagement-payment consumes and `ConsumedAmount` rows with `externalType = ENGAGEMENT`. TopGear-exempt accounts are reported separately as `topgear_exempt_*` statuses.
+  - Rollback:
+    - Use the JSON report from the apply run. A successful apply only commits after that report is written. Delete rows listed with `action: "insert"` and `createdId`; restore rows listed with `action: "update_single"` from `previous`; restore rows listed with `action: "move_existing_rows"` from their `existingRows` billing account ids.
+    - If apply mode exits with an error before writing the report, the transaction rolled back and there should be no partial billing-ledger changes from that run.
+    - If apply mode exits with an error after a report file exists, run the verifier or a dry run before rollback; the billing transaction should have committed all planned actions or none.
+
 ## Downstream Usage
 
 - This service is consumed (directly or indirectly) by multiple Topcoder apps. The pointers below help with debugging.

package.json

@@ -16,6 +16,8 @@
     "import:access": "ts-node scripts/import-ba-access.ts",
     "import:markup": "ts-node scripts/import-markup.ts",
     "backfill:clientIds": "ts-node scripts/backfill-client-ids.ts",
+    "backfill:engagementPayments": "ts-node scripts/backfill-engagement-payments.ts",
+    "verify:engagementPayments": "psql \"$DATABASE_URL\" -f scripts/verify-engagement-payment-backfill.sql",
     "generate:subcontractor-sql": "ts-node scripts/generate-subcontractor-sql.ts"
   },
   "dependencies": {

packages/ba-prisma-client/edge.js

@@ -158,7 +158,8 @@ exports.Prisma.BillingAccountScalarFieldEnum = {
 exports.Prisma.LockedAmountScalarFieldEnum = {
   id: 'id',
   billingAccountId: 'billingAccountId',
-  challengeId: 'challengeId',
+  externalId: 'externalId',
+  externalType: 'externalType',
   amount: 'amount',
   createdAt: 'createdAt',
   updatedAt: 'updatedAt'
@@ -167,7 +168,8 @@ exports.Prisma.LockedAmountScalarFieldEnum = {
 exports.Prisma.ConsumedAmountScalarFieldEnum = {
   id: 'id',
   billingAccountId: 'billingAccountId',
-  challengeId: 'challengeId',
+  externalId: 'externalId',
+  externalType: 'externalType',
   amount: 'amount',
   createdAt: 'createdAt',
   updatedAt: 'updatedAt'
@@ -204,6 +206,11 @@ exports.BAStatus = exports.$Enums.BAStatus = {
   INACTIVE: 'INACTIVE'
 };
 
+exports.BudgetEntryExternalType = exports.$Enums.BudgetEntryExternalType = {
+  CHALLENGE: 'CHALLENGE',
+  ENGAGEMENT: 'ENGAGEMENT'
+};
+
 exports.Prisma.ModelName = {
   Client: 'Client',
   BillingAccount: 'BillingAccount',