Skip to main content

Phase C — LookupAccount + Safe Disbursement Gate

Closes: GL-04 (the single highest-blast-radius money gap remaining at Phase B close-out). Shipped: 2026-05-29. Companion to: GO_LIVE_BLOCKERS.md, BANK_ORCHESTRATION.md, MONEY_FLOWS.md, PRODUCTION_READINESS.md.

What this phase does

Adds a fail-closed account-verification gate before every disbursement. The gate runs before any ledger PENDING row is posted and before any InitiateDisbursement is sent to the partner. If the partner cannot authoritatively confirm the destination account exists, the use case throws — no ledger entry, no outbox event, no partner submit, no settlement state. The aggregate stays APPROVED so the operator can correct the destination and retry.

Before C, a typo in a destination account number could result in:

  1. A partner ACCEPTED response for an account that doesn't exist (silently misrouted to an unintended holder if the partner's account-existence check is shallow), OR
  2. A partner REJECTED response handled at submit (recoverable) — but our ledger PENDING row got posted first and then needed reversal, generating noise.

After C, both outcomes are intercepted upstream: a fail-closed LookupAccount call is the disburse gate.

Status — by sub-deliverable

LIVE (runtime-proven this session)

ItemEvidence
Proto: LookupFailureReason enum + partner_reference + correlation_id + checked_at fields on the responsepackages/contracts/grpc/integration_gateway.proto; Go stubs regenerated and compiled clean.
Gateway gRPC handler LookupAccount realservices/integration-gateway/internal/server/lookup_and_health.go. 9 unit tests pass (happy + 6 reason mappings + unknown partner + adapter-without-lookup + missing fields).
Mock adapter LookupAccount with prefix-driven outcomesservices/integration-gateway/internal/adapters/mock/mock.go. 8 unit tests pass covering every typed reason.
Dashen adapter LookupAccount real HTTP+HMACservices/integration-gateway/internal/adapters/dashen/dashen.go. Compiles; exercised end-to-end against bank-sandbox by verify-c-lookup.sh.
Bank-sandbox GET /api/v1/accounts/:account endpoint with HMAC, prefix-driven outcomes, slow + malformed simulationservices/bank-sandbox/internal/handler/handler.go. 4 unit tests pass (prefix routing across 7 cases + malformed body + unsigned-rejected + slow→client-timeout).
Prometheus surface on gateway: /metrics endpoint + lookup_success_total{partner} + lookup_failure_total{partner,reason} + lookup_latency_seconds{partner}services/integration-gateway/internal/metrics/metrics.go, wired in cmd/.../main.go. Surfaced + asserted in verify-c-lookup.sh Test 7.
TS port: DisbursementPort.lookupAccount + AccountLookupInput + AccountLookupResult typed unionpackages/ewa/backend/application/ports/disbursement.port.ts + lending mirror.
TS gRPC adapter implementations in apps/apiapps/api/src/products/ewa/integration-gateway.grpc-client.ts + lending mirror.
EWA disburse use case calls lookupAccount first; throws EwaDestinationAccountInvalidError on failpackages/ewa/backend/application/disburse-ewa.usecase.ts. 5 unit tests pass covering happy + 3 reasons + correlation-id propagation.
Lending disburse use case calls lookupAccount first; throws LoanDestinationAccountInvalidError on failMirror. 3 unit tests pass.
Controller error mapping: 409 EWA_DESTINATION_ACCOUNT_INVALID / LOAN_DESTINATION_ACCOUNT_INVALID with reason + correlation id + partner_referenceBoth EWA + lending controllers.
End-to-end verify script verify-c-lookup.sh7/7 pass against the real docker-compose.test.yml stack (apps/api ◆ gateway ◆ bank-sandbox).
Idempotency contract: lookup is side-effect-free at the partner; safe to call multiple timesDocumented in port + adapter; no DB writes; no settlement row created on fail.
Append-only audit on failureEwaDestinationLookupFailed / LoanDestinationLookupFailed audit rows written before throw. Verified by unit specs.
Reconciliation safety: lookup failures do NOT create pending settlements / orphan ledger rows / fake obligationsSpecs assert ledger.posted.length === 0 + gateway.disburses.length === 0 + aggregate status === 'APPROVED' on every fail-closed branch.
Correlation IDs round-trip through the chain (apps/api → gateway → adapter → log lines → metric labels)Generated client-side if absent, echoed in response, present in error object, attached to outbound HTTP request headers, surfaced in audit row + slog WARN line.
S3 webhook regression unaffectedverify-s3.sh: 6/6 PASS.

PARTIAL — known limitations carried into pilot

ItemWhat's missingPilot impactMitigation
Name-matching policyAdapters return resolved_holder_name, but the use case does NOT compare it against an expected name. Reason: there is no "expected account holder name" field on the EWA/Lending request DTOs or the aggregates today.A bank account that exists but is registered to a DIFFERENT holder will pass the gate.Adding name-match requires threading expected-name through the DTO + aggregate. Phase C continuation. Until then: existence is the only check. Operators should reconcile bank-statement holder names manually for the pilot window.
Lookup result cachingEvery disburse triggers a fresh partner-side lookup (1 round-trip per disburse).At pilot scale (<100 disbursements/day) the cost is negligible.Add a Redis-backed TTL cache fronting the adapter for production scale. Phase C continuation.
Per-partner adapter health surfacingGetAdapterStatus still returns HEALTHY (stub). Lookup failures don't aggregate into a degraded-health signal that callers can branch on for fallback.Caller fails closed on partner-error anyway; correctness unaffected.Tracked separately as GL-12. Phase C continuation.

STUB / DEFERRED

ItemStatusNotes
Real Dashen partner integrationSTUBThe Dashen adapter HTTP+HMAC code runs end-to-end against services/bank-sandbox which mimics the wire shape. Swapping in real Dashen requires only env var changes (GATEWAY_DASHEN_BASE_URL, GATEWAY_DASHEN_SIGNING_KEY) — same client + signing code. Until Dashen sandbox onboarding completes, bank-sandbox IS the partner.
Other partner adapters (CBE, Awash, Telebirr, M-Birr)PLANNEDNo code today. Each adapter is independently ~2 weeks of work + sandbox onboarding.
Outbound lookup retry policy at the gatewayPLANNEDCurrently a single shot. A PARTNER_TIMEOUT results in fail-closed; the API caller can retry. Adding gateway-side exponential-backoff retry would shorten the user-visible failure window for transient partner blips. Phase C continuation.
Lookup distributed-tracing spansPLANNEDOTel SDK is wired but no exporter deployed (see PRODUCTION_READINESS.md §1). Correlation IDs partially fill this role; full trace spans land with the alerting backbone (GL-08).

Failure matrix

The proto enum LookupFailureReason taxonomy:

ReasonAdapter conditionsCaller (use case) decision
ACCOUNT_NOT_FOUNDPartner returns 404 / equivalent. Destination does not exist.Fail closed. Aggregate stays APPROVED; 409 to client with reason.
ACCOUNT_CLOSEDPartner returns 409 + reason: ACCOUNT_CLOSED. Account exists but is closed.Fail closed. Same shape.
ACCOUNT_BLOCKEDPartner returns 409 + reason: ACCOUNT_BLOCKED. Account exists but is on compliance hold / court order.Fail closed. Same shape. (Operator may treat differently — surface to compliance.)
PARTNER_TIMEOUTAdapter's HTTP request to the partner times out OR context cancelled.Fail closed. Account state is UNKNOWN; refusing is the safe answer.
PARTNER_UNAVAILABLEPartner returns 5xx OR transport error OR adapter not configured OR adapter doesn't implement AccountLookup.Fail closed. Same rationale.
INVALID_ACCOUNT_FORMATPartner returns 422 OR adapter pre-flight check fails on format (length, charset, checksum).Fail closed. This is a data-quality issue, surfaced for operator correction.

The use case treats every failure the same way: no settlement, no disburse, audit row, throw a typed error with reason + correlation_id. Caller distinguishes by reason for UI messaging; the safety model does NOT distinguish.

Idempotency guarantees

  • Lookup is a pure read at the partner. No DB writes at the gateway. No partner-side side effects. Calling LookupAccount N times for the same destination produces the same answer modulo partner state changes (account opened/closed between calls).
  • No caching today. Each disburse call triggers a fresh partner round-trip. Documented as a known scale concern; caching is a Phase C continuation item.
  • Failed lookup does NOT trigger ledger or gateway side effects. Specifically: ledger.postTransaction is never called; disbursement.disburse() is never called; the EWA / Loan aggregate stays APPROVED. Verified by unit spec.

Timeout handling

  • Client timeout (TS → gateway): the gRPC client's deadline applies. Default: none (set per-call). The lookupAccount call returns a typed PARTNER_UNAVAILABLE on transport error rather than throwing.
  • Gateway timeout (gateway → partner): the Dashen adapter uses a 10s HTTP client timeout by default (configurable via Config.HTTPClient). On timeout, the adapter returns Reason: PARTNER_TIMEOUT.
  • Bank-sandbox slow path: acct-slow-* accounts induce a 5s sleep before responding 200. The verify-c test asserts a 200ms client timeout against this path triggers PARTNER_TIMEOUT semantics correctly.

Retry semantics

  • The use case does NOT auto-retry on lookup failure. Fail-closed means the caller (admin / future payroll consumer) decides whether to retry.
  • A retry IS safe because lookup is side-effect-free at both the gateway and the partner.
  • A successful retry will proceed through the rest of the disburse flow normally (ledger PENDING + InitiateDisbursement + …).
  • Idempotency keys are NOT used at the lookup layer because there's nothing to dedup — the partner returns the same answer each time without booking anything.

Metrics surface

Three new metrics exposed on http://gateway:50053/metrics:

# success counter, partner-labelled (bounded cardinality: 4-5 partners)
demozpay_integration_gateway_lookup_success_total{partner="dashen"} 1

# failure counter labelled by partner + typed reason
demozpay_integration_gateway_lookup_failure_total{partner="dashen",reason="ACCOUNT_NOT_FOUND"} 1
demozpay_integration_gateway_lookup_failure_total{partner="dashen",reason="ACCOUNT_CLOSED"} 1
demozpay_integration_gateway_lookup_failure_total{partner="dashen",reason="PARTNER_TIMEOUT"} 0
...

# latency histogram, partner-labelled
demozpay_integration_gateway_lookup_latency_seconds_bucket{partner="dashen",le="0.25"} 5
demozpay_integration_gateway_lookup_latency_seconds_sum{partner="dashen"} 0.123
demozpay_integration_gateway_lookup_latency_seconds_count{partner="dashen"} 6

Cardinality: partner is bounded (4-5 active partners). Reason is bounded (the enum). No tenant_id label (would explode cardinality).

Suggested alert rules (Phase D / GL-08):

  • rate(demozpay_integration_gateway_lookup_failure_total{reason="PARTNER_UNAVAILABLE"}[5m]) > 1 — partner outage indicator.
  • rate(demozpay_integration_gateway_lookup_failure_total{reason="PARTNER_TIMEOUT"}[5m]) > 1 — partner latency degradation.
  • histogram_quantile(0.95, rate(demozpay_integration_gateway_lookup_latency_seconds_bucket[5m])) > 1.0 — p95 latency > 1s.

These rules are documented here as recommendations; the alerting backbone wiring is GL-08, not in scope for this phase.

What changes for real Dashen integration

When Dashen sandbox credentials arrive:

  1. Set GATEWAY_DASHEN_BASE_URL=https://api.dashenbank.com (or sandbox equivalent).
  2. Set GATEWAY_DASHEN_SIGNING_KEY=<the real shared secret>.
  3. Verify Dashen's actual GET /api/v1/accounts/:account shape matches what services/bank-sandbox emits:
    • Status code conventions (200 / 404 / 409 / 422 / 503).
    • Response body fields (exists, resolved_name, partner_reference, reason, message).
    • HMAC scheme: header names, canonical string format (currently X-Demoz-Timestamp\nX-Demoz-Nonce\nbody), clock-skew window (currently ±300s).
  4. If Dashen's shape differs: update services/integration-gateway/internal/adapters/dashen/dashen.go only. The gateway server, the TS port, the use case gate, the audit + metrics surface — none of those need changes.

The bank-sandbox's purpose is exactly this: prove the full chain works against a partner that adheres to our spec. The adapter swap is a Dockerfile env-var change.

Reconciliation implications

  • Failed lookup never creates a bank_statement_line reconciliation gap. Nothing was instructed to move; nothing needs to settle.
  • Failed lookup audit rows can be queried to surface "destination correction" workflow opportunities. audit_entry.action = 'EwaDestinationLookupFailed' OR 'LoanDestinationLookupFailed' filtered by tenantId shows operator-actionable rejections.
  • Successful lookup → successful disburse → reconciliation path is unchanged. The lookup is purely a gate; once it passes, the existing GAP-08 / GAP-09 / GAP-11 path takes over.

Known gaps (carried)

  1. Name-matching not enforced. See PARTIAL §1.
  2. No lookup result caching. See PARTIAL §2.
  3. No adapter health surfacing. Tracked as GL-12 separately.
  4. Real Dashen sandbox not yet onboarded. Operational item, not a code gap.
  5. Other partner adapters PLANNED. CBE, Awash, Telebirr, M-Birr.

Verification evidence — full sweep

GateOutcomeDetail
TS unit suite120/120 PASS+8 from C4 (5 EWA + 3 lending lookup-gate tests).
Gateway Go unit tests9 PASSLookupAccount handler routing + reason mapping + missing-field 400s + adapter-without-lookup → typed PARTNER_UNAVAILABLE.
Mock adapter unit tests8 PASSEvery prefix path → correct typed reason.
Bank-sandbox handler unit tests4 PASS7-case prefix routing + malformed body + unsigned-rejected + slow→timeout.
verify-c-lookup.sh E2E7/7 PASSFull chain: apps/api ◆ gateway ◆ bank-sandbox, valid + 5 failure reasons + metrics surface, against running docker-compose.test.yml stack.
verify-s3.sh regression6/6 PASSExisting webhook + applier + ledger path unaffected.
Telemedhin containers7/7 healthyUntouched.

Cross-references

  • The single highest-blast-radius operational gap pre-C → GO_LIVE_BLOCKERS.md#GL-04.
  • The bank-orchestrator contract this gate fits inside → BANK_ORCHESTRATION.md.
  • The disburse money flow with the gate inserted → MONEY_FLOWS.md Flow 1 & 3.
  • Where the metrics will be alerted from (Phase D) → GO_LIVE_BLOCKERS.md#GL-08.
  • The carried name-match work and other Phase C continuation items → see this file's PARTIAL section.