Phase C — LookupAccount + Safe Disbursement Gate
Closes: GL-04 (the single highest-blast-radius money gap remaining at Phase B close-out). Shipped: 2026-05-29. Companion to:
GO_LIVE_BLOCKERS.md,BANK_ORCHESTRATION.md,MONEY_FLOWS.md,PRODUCTION_READINESS.md.
What this phase does
Adds a fail-closed account-verification gate before every disbursement. The gate runs before any ledger PENDING row is posted and before any InitiateDisbursement is sent to the partner. If the partner cannot authoritatively confirm the destination account exists, the use case throws — no ledger entry, no outbox event, no partner submit, no settlement state. The aggregate stays APPROVED so the operator can correct the destination and retry.
Before C, a typo in a destination account number could result in:
- A partner ACCEPTED response for an account that doesn't exist (silently misrouted to an unintended holder if the partner's account-existence check is shallow), OR
- A partner REJECTED response handled at submit (recoverable) — but our ledger PENDING row got posted first and then needed reversal, generating noise.
After C, both outcomes are intercepted upstream: a fail-closed LookupAccount call is the disburse gate.
Status — by sub-deliverable
LIVE (runtime-proven this session)
| Item | Evidence |
|---|---|
Proto: LookupFailureReason enum + partner_reference + correlation_id + checked_at fields on the response | packages/contracts/grpc/integration_gateway.proto; Go stubs regenerated and compiled clean. |
Gateway gRPC handler LookupAccount real | services/integration-gateway/internal/server/lookup_and_health.go. 9 unit tests pass (happy + 6 reason mappings + unknown partner + adapter-without-lookup + missing fields). |
Mock adapter LookupAccount with prefix-driven outcomes | services/integration-gateway/internal/adapters/mock/mock.go. 8 unit tests pass covering every typed reason. |
Dashen adapter LookupAccount real HTTP+HMAC | services/integration-gateway/internal/adapters/dashen/dashen.go. Compiles; exercised end-to-end against bank-sandbox by verify-c-lookup.sh. |
Bank-sandbox GET /api/v1/accounts/:account endpoint with HMAC, prefix-driven outcomes, slow + malformed simulation | services/bank-sandbox/internal/handler/handler.go. 4 unit tests pass (prefix routing across 7 cases + malformed body + unsigned-rejected + slow→client-timeout). |
Prometheus surface on gateway: /metrics endpoint + lookup_success_total{partner} + lookup_failure_total{partner,reason} + lookup_latency_seconds{partner} | services/integration-gateway/internal/metrics/metrics.go, wired in cmd/.../main.go. Surfaced + asserted in verify-c-lookup.sh Test 7. |
TS port: DisbursementPort.lookupAccount + AccountLookupInput + AccountLookupResult typed union | packages/ewa/backend/application/ports/disbursement.port.ts + lending mirror. |
| TS gRPC adapter implementations in apps/api | apps/api/src/products/ewa/integration-gateway.grpc-client.ts + lending mirror. |
EWA disburse use case calls lookupAccount first; throws EwaDestinationAccountInvalidError on fail | packages/ewa/backend/application/disburse-ewa.usecase.ts. 5 unit tests pass covering happy + 3 reasons + correlation-id propagation. |
Lending disburse use case calls lookupAccount first; throws LoanDestinationAccountInvalidError on fail | Mirror. 3 unit tests pass. |
Controller error mapping: 409 EWA_DESTINATION_ACCOUNT_INVALID / LOAN_DESTINATION_ACCOUNT_INVALID with reason + correlation id + partner_reference | Both EWA + lending controllers. |
End-to-end verify script verify-c-lookup.sh | 7/7 pass against the real docker-compose.test.yml stack (apps/api ◆ gateway ◆ bank-sandbox). |
| Idempotency contract: lookup is side-effect-free at the partner; safe to call multiple times | Documented in port + adapter; no DB writes; no settlement row created on fail. |
| Append-only audit on failure | EwaDestinationLookupFailed / LoanDestinationLookupFailed audit rows written before throw. Verified by unit specs. |
| Reconciliation safety: lookup failures do NOT create pending settlements / orphan ledger rows / fake obligations | Specs assert ledger.posted.length === 0 + gateway.disburses.length === 0 + aggregate status === 'APPROVED' on every fail-closed branch. |
| Correlation IDs round-trip through the chain (apps/api → gateway → adapter → log lines → metric labels) | Generated client-side if absent, echoed in response, present in error object, attached to outbound HTTP request headers, surfaced in audit row + slog WARN line. |
| S3 webhook regression unaffected | verify-s3.sh: 6/6 PASS. |
PARTIAL — known limitations carried into pilot
| Item | What's missing | Pilot impact | Mitigation |
|---|---|---|---|
| Name-matching policy | Adapters return resolved_holder_name, but the use case does NOT compare it against an expected name. Reason: there is no "expected account holder name" field on the EWA/Lending request DTOs or the aggregates today. | A bank account that exists but is registered to a DIFFERENT holder will pass the gate. | Adding name-match requires threading expected-name through the DTO + aggregate. Phase C continuation. Until then: existence is the only check. Operators should reconcile bank-statement holder names manually for the pilot window. |
| Lookup result caching | Every disburse triggers a fresh partner-side lookup (1 round-trip per disburse). | At pilot scale (<100 disbursements/day) the cost is negligible. | Add a Redis-backed TTL cache fronting the adapter for production scale. Phase C continuation. |
| Per-partner adapter health surfacing | GetAdapterStatus still returns HEALTHY (stub). Lookup failures don't aggregate into a degraded-health signal that callers can branch on for fallback. | Caller fails closed on partner-error anyway; correctness unaffected. | Tracked separately as GL-12. Phase C continuation. |
STUB / DEFERRED
| Item | Status | Notes |
|---|---|---|
| Real Dashen partner integration | STUB | The Dashen adapter HTTP+HMAC code runs end-to-end against services/bank-sandbox which mimics the wire shape. Swapping in real Dashen requires only env var changes (GATEWAY_DASHEN_BASE_URL, GATEWAY_DASHEN_SIGNING_KEY) — same client + signing code. Until Dashen sandbox onboarding completes, bank-sandbox IS the partner. |
| Other partner adapters (CBE, Awash, Telebirr, M-Birr) | PLANNED | No code today. Each adapter is independently ~2 weeks of work + sandbox onboarding. |
| Outbound lookup retry policy at the gateway | PLANNED | Currently a single shot. A PARTNER_TIMEOUT results in fail-closed; the API caller can retry. Adding gateway-side exponential-backoff retry would shorten the user-visible failure window for transient partner blips. Phase C continuation. |
| Lookup distributed-tracing spans | PLANNED | OTel SDK is wired but no exporter deployed (see PRODUCTION_READINESS.md §1). Correlation IDs partially fill this role; full trace spans land with the alerting backbone (GL-08). |
Failure matrix
The proto enum LookupFailureReason taxonomy:
| Reason | Adapter conditions | Caller (use case) decision |
|---|---|---|
ACCOUNT_NOT_FOUND | Partner returns 404 / equivalent. Destination does not exist. | Fail closed. Aggregate stays APPROVED; 409 to client with reason. |
ACCOUNT_CLOSED | Partner returns 409 + reason: ACCOUNT_CLOSED. Account exists but is closed. | Fail closed. Same shape. |
ACCOUNT_BLOCKED | Partner returns 409 + reason: ACCOUNT_BLOCKED. Account exists but is on compliance hold / court order. | Fail closed. Same shape. (Operator may treat differently — surface to compliance.) |
PARTNER_TIMEOUT | Adapter's HTTP request to the partner times out OR context cancelled. | Fail closed. Account state is UNKNOWN; refusing is the safe answer. |
PARTNER_UNAVAILABLE | Partner returns 5xx OR transport error OR adapter not configured OR adapter doesn't implement AccountLookup. | Fail closed. Same rationale. |
INVALID_ACCOUNT_FORMAT | Partner returns 422 OR adapter pre-flight check fails on format (length, charset, checksum). | Fail closed. This is a data-quality issue, surfaced for operator correction. |
The use case treats every failure the same way: no settlement, no disburse, audit row, throw a typed error with reason + correlation_id. Caller distinguishes by reason for UI messaging; the safety model does NOT distinguish.
Idempotency guarantees
- Lookup is a pure read at the partner. No DB writes at the gateway. No partner-side side effects. Calling
LookupAccountN times for the same destination produces the same answer modulo partner state changes (account opened/closed between calls). - No caching today. Each disburse call triggers a fresh partner round-trip. Documented as a known scale concern; caching is a Phase C continuation item.
- Failed lookup does NOT trigger ledger or gateway side effects. Specifically:
ledger.postTransactionis never called;disbursement.disburse()is never called; the EWA / Loan aggregate stays APPROVED. Verified by unit spec.
Timeout handling
- Client timeout (TS → gateway): the gRPC client's deadline applies. Default: none (set per-call). The
lookupAccountcall returns a typedPARTNER_UNAVAILABLEon transport error rather than throwing. - Gateway timeout (gateway → partner): the Dashen adapter uses a 10s HTTP client timeout by default (configurable via
Config.HTTPClient). On timeout, the adapter returnsReason: PARTNER_TIMEOUT. - Bank-sandbox slow path:
acct-slow-*accounts induce a 5s sleep before responding 200. The verify-c test asserts a 200ms client timeout against this path triggersPARTNER_TIMEOUTsemantics correctly.
Retry semantics
- The use case does NOT auto-retry on lookup failure. Fail-closed means the caller (admin / future payroll consumer) decides whether to retry.
- A retry IS safe because lookup is side-effect-free at both the gateway and the partner.
- A successful retry will proceed through the rest of the disburse flow normally (ledger PENDING + InitiateDisbursement + …).
- Idempotency keys are NOT used at the lookup layer because there's nothing to dedup — the partner returns the same answer each time without booking anything.
Metrics surface
Three new metrics exposed on http://gateway:50053/metrics:
# success counter, partner-labelled (bounded cardinality: 4-5 partners)
demozpay_integration_gateway_lookup_success_total{partner="dashen"} 1
# failure counter labelled by partner + typed reason
demozpay_integration_gateway_lookup_failure_total{partner="dashen",reason="ACCOUNT_NOT_FOUND"} 1
demozpay_integration_gateway_lookup_failure_total{partner="dashen",reason="ACCOUNT_CLOSED"} 1
demozpay_integration_gateway_lookup_failure_total{partner="dashen",reason="PARTNER_TIMEOUT"} 0
...
# latency histogram, partner-labelled
demozpay_integration_gateway_lookup_latency_seconds_bucket{partner="dashen",le="0.25"} 5
demozpay_integration_gateway_lookup_latency_seconds_sum{partner="dashen"} 0.123
demozpay_integration_gateway_lookup_latency_seconds_count{partner="dashen"} 6
Cardinality: partner is bounded (4-5 active partners). Reason is bounded (the enum). No tenant_id label (would explode cardinality).
Suggested alert rules (Phase D / GL-08):
rate(demozpay_integration_gateway_lookup_failure_total{reason="PARTNER_UNAVAILABLE"}[5m]) > 1— partner outage indicator.rate(demozpay_integration_gateway_lookup_failure_total{reason="PARTNER_TIMEOUT"}[5m]) > 1— partner latency degradation.histogram_quantile(0.95, rate(demozpay_integration_gateway_lookup_latency_seconds_bucket[5m])) > 1.0— p95 latency > 1s.
These rules are documented here as recommendations; the alerting backbone wiring is GL-08, not in scope for this phase.
What changes for real Dashen integration
When Dashen sandbox credentials arrive:
- Set
GATEWAY_DASHEN_BASE_URL=https://api.dashenbank.com(or sandbox equivalent). - Set
GATEWAY_DASHEN_SIGNING_KEY=<the real shared secret>. - Verify Dashen's actual
GET /api/v1/accounts/:accountshape matches whatservices/bank-sandboxemits:- Status code conventions (200 / 404 / 409 / 422 / 503).
- Response body fields (
exists,resolved_name,partner_reference,reason,message). - HMAC scheme: header names, canonical string format (currently
X-Demoz-Timestamp\nX-Demoz-Nonce\nbody), clock-skew window (currently ±300s).
- If Dashen's shape differs: update
services/integration-gateway/internal/adapters/dashen/dashen.goonly. The gateway server, the TS port, the use case gate, the audit + metrics surface — none of those need changes.
The bank-sandbox's purpose is exactly this: prove the full chain works against a partner that adheres to our spec. The adapter swap is a Dockerfile env-var change.
Reconciliation implications
- Failed lookup never creates a
bank_statement_linereconciliation gap. Nothing was instructed to move; nothing needs to settle. - Failed lookup audit rows can be queried to surface "destination correction" workflow opportunities.
audit_entry.action = 'EwaDestinationLookupFailed' OR 'LoanDestinationLookupFailed'filtered bytenantIdshows operator-actionable rejections. - Successful lookup → successful disburse → reconciliation path is unchanged. The lookup is purely a gate; once it passes, the existing GAP-08 / GAP-09 / GAP-11 path takes over.
Known gaps (carried)
- Name-matching not enforced. See PARTIAL §1.
- No lookup result caching. See PARTIAL §2.
- No adapter health surfacing. Tracked as GL-12 separately.
- Real Dashen sandbox not yet onboarded. Operational item, not a code gap.
- Other partner adapters PLANNED. CBE, Awash, Telebirr, M-Birr.
Verification evidence — full sweep
| Gate | Outcome | Detail |
|---|---|---|
| TS unit suite | 120/120 PASS | +8 from C4 (5 EWA + 3 lending lookup-gate tests). |
| Gateway Go unit tests | 9 PASS | LookupAccount handler routing + reason mapping + missing-field 400s + adapter-without-lookup → typed PARTNER_UNAVAILABLE. |
| Mock adapter unit tests | 8 PASS | Every prefix path → correct typed reason. |
| Bank-sandbox handler unit tests | 4 PASS | 7-case prefix routing + malformed body + unsigned-rejected + slow→timeout. |
verify-c-lookup.sh E2E | 7/7 PASS | Full chain: apps/api ◆ gateway ◆ bank-sandbox, valid + 5 failure reasons + metrics surface, against running docker-compose.test.yml stack. |
verify-s3.sh regression | 6/6 PASS | Existing webhook + applier + ledger path unaffected. |
| Telemedhin containers | 7/7 healthy | Untouched. |
Cross-references
- The single highest-blast-radius operational gap pre-C →
GO_LIVE_BLOCKERS.md#GL-04. - The bank-orchestrator contract this gate fits inside →
BANK_ORCHESTRATION.md. - The disburse money flow with the gate inserted →
MONEY_FLOWS.mdFlow 1 & 3. - Where the metrics will be alerted from (Phase D) →
GO_LIVE_BLOCKERS.md#GL-08. - The carried name-match work and other Phase C continuation items → see this file's PARTIAL section.