Skip to main content

Architecture Alignment — Work Log (audit trail)

Running, append-only log of every step taken on the fix/architecture-alignment branch, so the work can be audited later. Plan + workstream definitions live in ARCHITECTURE_ALIGNMENT_PLAN.md.

Branch: fix/architecture-alignment (based on origin/dev-v1.0.0). Git policy: commits ONLY when the user explicitly asks. Nothing pushed.

Legend: ✅ done+verified · 🚧 in progress · ⬜ todo

Local commits so far (NOT pushed)​

RefTitleContents
743ca62feat(equb,resilience): partner-bank escrow flow + custody gate + gRPC breakersvision-alignment pass, W1.1, W1.2, W6, W8
e124995feat(equb): support both payout methods — RANDOM lottery + FIXED rotationpayoutMethod domain+schema+repo+usecase+tests
735e30cfeat(equb): expose both payout methods on the HTTP APIcontroller/DTO wiring

(3 commits — all local. After these the user instructed: commit/push/add to GitHub ONLY when asked.)

Step-by-step log​

S0 — Vision alignment pass ✅​

  • Equb custody gate: EQUB_MONEY_MOVEMENT_ENABLED (default off, forbidden in prod via config superRefine); DisabledEqubLedgerAdapter seals the single EQUB_LEDGER_PORT chokepoint.
  • Wallet terminology clarified as a ledger read-model (NO rename — auditability/ADR-009): comments in private-equb-contribution.service.ts, me-equb.controller.ts.
  • Docs → one story: README Equb status, CLAUDE.md Equb-gate fact, archive-doc positioning banner.
  • Files: packages/shared/config/src/lib/config.ts, apps/api/src/products/equb/{disabled-equb-ledger.adapter.ts,equb-api.module.ts}, README, CLAUDE.md, docs/archive/SYSTEM_DOCUMENTATION.md.
  • Verify: api app typecheck 0 errors; shared-config tests pass.

W1.1 — Escrow foundation + verified binding ✅​

  • bank-sandbox: seeded FAKE-EQUB-ESCROW-001 (services/bank-sandbox/internal/store/store.go).
  • Seed: demo EqubCycle (DRAFT) + PartnerBankEscrowBinding → DASHEN/FAKE-EQUB-ESCROW-001 (apps/api/prisma/seed.ts).
  • equb-escrow.controller.ts createBinding now verifies the account at the partner via IdentityLookupPort (same pattern as employer payroll-bank linking) before ACTIVE; rejects unknown/invalid, 409 on partner-unavailable.
  • Side-fix: prisma generate (stale client from the merged attendance migration caused 13 false errors).
  • Verify: api app typecheck 0 errors.

W1.2 — CORPORATE contribution leg → escrow ✅​

  • payroll-equb-fanout.service.ts: requires an ACTIVE escrow binding (else skip — "escrow required" gate); posts equb:pool mirror PENDING; moves money employer-source → escrow via EWA DisbursementPort; confirms/fails the mirror on the bank's answer.
  • EqubLedgerPort gained confirmContribution/failContribution; EqubContributionInput.status (PENDING for escrow-backed). Threaded through RecordEqubContribution. Disabled adapter refuses these too (gate holds).
  • Updated 4 package test fakes + fanout spec; added "no escrow binding → skip" test.
  • Verify: api app + specs + equb specs typecheck 0; equb unit tests 44 pass.

W6 — gRPC circuit breakers + deadlines ✅​

  • apps/api/src/_infra/resilience/circuit-breaker.ts (open after N failures, half-open probe, recover); per-call deadlines on ledger + integration-gateway clients so a hung partner fails fast → transfer stays PENDING for the poller.
  • Verify: 4 breaker unit tests pass; api typecheck 0.

W8 — Broker decision ✅​

  • docs/adr/ADR-017-event-transport-postgres-outbox.md: Postgres outbox + table-poller is the spine; Kafka dormant; RabbitMQ not adopted. CLAUDE.md ADR range bumped to 017.

Both payout methods (RANDOM + FIXED) ✅​

  • Driven by the employer dashboard design (Fixed Order vs Random Auto-Assignment), which was a UI-only mock — backend was random-only.
  • Domain: EqubCycle.payoutMethod ('RANDOM' default | 'FIXED') + selectFixedWinner() (eligible member with lowest drawOrder; employer order applied via activate's drawOrderShuffle; no reveal).
  • AssignFixedWinnerUseCase (FIXED counterpart to RunEqubDraw; emits same equb.draw_completed.v1).
  • CreateEqubCycle + repo persist payoutMethod. Migration 20260618130000_equb_payout_method (column default RANDOM + CHECK) — applied to local DB. Column-add only → inherits existing tenant RLS (no RLS change).
  • API: POST /equb/cycles accepts payoutMethod; POST /equb/cycles/:id/draw branches RANDOM (seed+nonce) vs FIXED (drawOrder).
  • Verify: equb pkg + api + api specs typecheck 0; 4 FIXED-payout domain tests pass.
  • Finding: employer-web corporate-equb/create page is a UI mock (handleNext just navigates; no API call) — end-to-end UI wiring is a separate frontend task.

Test infra — fix the better-auth ESM transform failure ✅​

  • Root cause: better-auth/plugins is ESM-only (dist/plugins/index.mjs); ts-jest ignores node_modules → any spec transitively importing it ("Cannot use import statement outside a module") failed to load. Only createAccessControl is used (packages/shared/auth/src/lib/permissions.ts), at module load.
  • Fix: test/mocks/better-auth-plugins.cjs stub + moduleNameMapper in packages/equb/jest.config.cts and apps/api/jest.config.cts.
  • Result: equb suite fully green — 9 suites, 69 tests (was 4 suites / 48, with 5 dead). My apps/api specs (fanout, private-contribution, circuit-breaker) green — 19 tests. Fixed my own new fanout test (needed ≥2 members for roundCount≥2).
  • Honest note: fixing the transform revealed pre-existing api test-harness failures that were previously hidden behind "failed to load" — supertest specs (pension/platform/tax-rule-admin) fail on this.prisma.adminProfile.findFirst is not a function (incomplete prisma mock) and org-role.guard.spec on member.roleAssignments undefined (incomplete member mock). These predate this work, are unrelated to the Equb changes, and are separate test-harness debt. The api suite is also slow (supertest boots Nest per spec). NOT regressions — those suites couldn't run at all before.

W1.4 — Payout leg (escrow → winner) ✅​

  • EqubPayoutInput.status + postPayout honors it (PENDING mirror); CloseEqubRoundPayout accepts status.
  • New apps/api/src/products/equb/equb-escrow-payout.service.ts: resolves the current round's drawn winner, requires an ACTIVE escrow binding (source), resolves the winner's Employee.bankAccount (blocks EQUB_WINNER_NO_BANK_ACCOUNT if unset), posts the payout mirror PENDING via CloseEqubRoundPayout (compliance gate + round advance inside), transfers escrow → winner via EWA DisbursementPort, then confirm/fail the mirror (reusing confirmContribution/failContribution — settlement-generic flip-by-id).
  • Endpoint: POST /equb/escrow/cycles/:id/payout (@MoneyMoving, Idempotency-Key required). Works for BOTH RANDOM + FIXED winners (winner already chosen by /draw).
  • Wired in equb-api.module.ts.
  • Verify: equb pkg + api app + api specs typecheck 0; equb tests 48 pass (incl. 4 fixed-payout). The 5 fail-to-run suites are the pre-existing better-auth ESM transform issue (not assertions, not mine).
  • Note: round advances on payout even if the transfer is still PENDING (mirror confirmed async) — same shape as the contribution leg; full compensation on FAILED is future work.

W1.3 — async settlement confirm ✅​

  • Schema: equb_contribution.partnerReference; equb_round.payoutLedgerTransactionId + payoutPartnerReference (+ indexes). Migration 20260618140000_equb_settlement_tracking — applied to local DB. Nullable; sync-confirm path unaffected.
  • Persist providerRef (+ the real ledgerTransactionId, which the repo left null): fanout updates the contribution row after disburse; EqubEscrowPayoutService updates the round after disburse.
  • bank-settlement-applier.service.ts: new Equb branch — matches equb_contribution / equb_round by partnerReference (cross-tenant, RLS-bypass connection), then confirm/fail the PENDING mirror via the ledger (idempotent; webhook+poller race safe) + audit row. BankSettlementResult.domain + webhook response type + bumpApply gained EQUB/equb.
  • Fixed the payroll applier spec constructor (new prisma param) + fanout spec prisma mock (equbContribution.update).
  • Verify: api app + specs typecheck 0; equb 69 tests pass; integration applier 5 pass; fanout/private/applier-payroll specs 20 pass.

W1 core money flow is now complete for CORPORATE: contribute → escrow → draw (RANDOM|FIXED) → payout escrow→winner, with both synchronous and async (webhook/poller) settlement.

W1.6 — PRIVATE contribution onto escrow ✅​

  • PrivateEqubContributionService.contributeFromWallet refactored to the escrow path: requires an ACTIVE escrow binding; the member funds from their own Employee.bankAccount → escrow (real bank money, not a simulated wallet); posts the mirror PENDING; gateway transfer; confirm/fail; persists partnerReference (W1.3). Removed the dead poolAccount helper.
  • This retires the wallet-as-stored-value concern — PRIVATE money is a real member-bank → partner-escrow transfer; "wallet" remains only a ledger read-model label.
  • Spec rewritten (5 deps + binding/employee/disburse mocks) + 2 new gate tests (no binding, no member bank account). 11 tests pass; typecheck clean.

W1.5 — retire the prod-block ✅ (option A)​

  • Removed the legacy simulated-pool endpoints POST /api/equb/cycles/:id/contribute and :id/payout from equb.controller.ts (+ their use-case injections, DTOs RecordEqubContributionDto/CloseEqubPayoutDto, and the now-dead MoneyMoving/Headers imports + EqubContributionAmountMismatchError map branch). No external callers (no verify scripts, no controller spec). The use cases stay in the module (used by fanout / me-equb / escrow-payout).
  • Single money path now: contributions via fan-out (CORPORATE) / me-equb (PRIVATE), payouts via /equb/escrow/cycles/:id/payout — all require an ACTIVE escrow binding.
  • Retired the boot-time prod-block: removed the superRefine that forbade EQUB_MONEY_MOVEMENT_ENABLED=true in production. The per-cycle escrow binding is now the gate (not a boot block); updated the flag doc + .env.example.
  • Verify: shared-config 6, equb 69, api equb/resilience/applier 31 — all pass; typecheck clean across config + equb + api + specs.

✅ W1 COMPLETE. Equb is end-to-end partner-bank-escrow-backed, no custody, both payout methods, sync + async settlement, single money path, prod-enableable per-cycle.

W2 — reconciliation reader (ledger vs partner-bank) ✅​

  • apps/api/src/products/equb/bank-sandbox-statement-reader.ts — BankSandboxPartnerBankStatementReader reads the escrow account's balance from bank-sandbox over HTTP+HMAC (same wire shape as the identity-lookup adapter), returns bigint | null.
  • Bound in equb-api.module.ts via a config-driven factory: bank-sandbox reader when BANK_SANDBOX_BASE_URL + BANK_SANDBOX_SIGNING_KEY are set, else the stub (same swap-point as banking IdentityLookup). This unblocks EqubEscrowReconciliationWorker — it can now pull the partner-bank balance and compare to the equb:pool mirror (bank wins on drift). Manual POST /equb/escrow/bindings/:id/reconcile already exists as the operator path.
  • Verify: api app + specs typecheck 0; 5 reader unit tests pass (balance→bigint, HMAC headers, not-found→null, unreachable→null, malformed→null).
  • Note: bank-sandbox returns CURRENT balance (no historical statements) → same-day recon only in the sandbox; the real partner adapter honours the statement date.

W5 — leader election for non-idempotent schedulers ✅​

  • apps/api/src/_infra/scheduling/leader-election.service.ts — LeaderElectionService.tryRunAsLeader(lockKey, label, fn) using pg_try_advisory_xact_lock (transaction-scoped → auto-releases on settle/crash, non-blocking → non-leaders skip). LEADER_LOCK_KEYS for the 3 jobs. SchedulingModule (@Global) imported in AppModule.
  • Wired into all 3 non-idempotent cron workers (equb escrow recon, payroll auto-lock, court-order auto-submit): the scheduled tick runs under leader election (one pod per tick); every instance still reschedules so any can take over; drain() stays directly callable (admin/tests). Leader injected @Optional → positional test construction unaffected. Idempotent SKIP-LOCKED money pollers deliberately left multi-instance.
  • Verify: api app + specs typecheck 0; leader-election 3 tests + auto-lock + court-submit specs green (15 pass, 3 integration skipped).
  • Trade-off documented: the leader holds one transaction open for the job's duration (fine for these low-frequency crons).

W3 — notifications wired → live ✅ (code) / ops enablement remains​

  • The senders (SMS logger/http/smtp/ethio-telecom + email) + consumer + dispatcher already existed and worked — notifications only covered EWA settlement.
  • Added EqubPayoutCompletedHandler (equb.payout_completed.v1 → "you won the Equb round, ETB N is on its way" SMS to the winner), registered via the NOTIFICATION_HANDLERS multi-provider. So notifications now cover the Equb domain we built.
  • Verify: api app + specs typecheck 0; handler + dispatcher specs 12 tests pass.
  • Going truly "live" is an ops step (not code): set NOTIFICATIONS_CONSUMER_ENABLED=true + SMS_PROVIDER=ethio-telecom (+ SMSC creds) + sign off copy. Until then it logs via the dev sender.

W4 — DLQ + alerting on stuck events ✅ (no migration)​

  • Insight: retry-exhausted rows (status=FAILED, attemptCount >= max) in payroll_deduction_event_state + notification_event_state ARE the dead-letter queue — they already exist, just invisible. No new table.
  • dead-letter/dead-letter.service.ts — summary() (exact counts + recent items per consumer, refreshes the gauge) + requeue(consumer, eventId) (reset attemptCount=0 → poller re-claims). DeadLetterController (GET /api/admin/dead-letter, POST /api/admin/dead-letter/:consumer/:eventId/requeue, @RequirePlatformAdmin). DeadLetterModule wired in AppModule + the controller added to SessionMiddleware.forRoutes (the rake).
  • Alerting: demozpay_dead_letter_events_total{consumer} gauge (alert on > 0), refreshed on the admin GET.
  • Verify: api app + specs typecheck 0; 3 service tests pass.

W7 — ADR-011 cleanup (cross-domain DI → events) ✅ (decision; staged refactor to follow)​

  • Finding: the hard rule (packages/A ⇏ packages/B) already holds — no domain-to-domain imports exist. The cross-domain adapters all live in apps/api (app-layer composition, which ADR-011 already permits). The real gap was an unclassified read-path.
  • Amended ADR-011 with three shapes: (1) reaction → async event (unchanged); (2) synchronous regulatory gate (KYC/sanctions at disburse/payout) → REQUIRED synchronous + fail-closed, endorsed, must not be eventualised (kyc-read / kyc-sanctions-screen / equb-kyc / equb-sanctions adapters are correct as-is); (3) cross-domain calculation read → PREFER an event-fed projection (candidates: payroll-accrued-earnings, lending income, equb-behavior-signal adapters), with staged per-edge migration guidance.
  • Doc-only. The projection migrations are independently-shippable follow-ons the ADR now prioritises; none required for correctness today.

✅ ALL WORKSTREAMS COMPLETE (W1–W8)​

WSStatus
W1 Equb partner-bank escrow (1.1–1.6)✅
W2 Reconciliation reader✅
W3 Notifications (Equb handler; ops enablement remains)✅
W4 DLQ + alerting✅
W5 Leader election✅
W6 Circuit breakers✅
W7 ADR-011 amendment (projections = staged follow-on)✅
W8 Broker decision (ADR-017)✅

Remaining (see plan)​

  • ⬜ W1.4 payout leg (in progress) · W1.3 async settlement confirm · W1.6 PRIVATE onto escrow · W1.5 retire prod-block
  • ⬜ W2 reconciliation reader · W3 notifications · W4 DLQ · W5 leader election · W7 ADR-011 projections
  • ⬜ End-to-end UI wiring for the corporate-equb create flow (separate frontend task)