RabbitMQ receipt processing runbook¶

This page covers setup, commands, recovery, and deployment. See RabbitMQ receipt processing design for the message lifecycle, reliability boundaries, idempotency model, and failure behavior.

All receipt work goes through RabbitMQ, including manual processing, full reprocessing, cache rebuilds, and scheduled discovery. Consumers reuse the OCR and persistence implementation. Database setup adds request tracking tables without rebuilding existing receipt state. There is no direct OCR fallback.

Contract and delivery¶

One UTF-8 JSON message describes one immutable source file:

{"version":1,"source_sha256":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","source_reference":"2026/receipt.pdf","attempt":1}

source_reference is a POSIX path relative to RECEIPT_SOURCE_ROOT. Publishers and workers need the same files beneath that root, though their absolute mount paths can differ. Absolute paths, parent traversal, escaping symlinks, malformed hashes, unsupported versions, and malformed messages are rejected. Workers check the actual hash before and after parsing. Put complete files in the inbox using an atomic rename; do not edit a file while it is being processed. The AMQP message_id is receipt.v1:<source_sha256> and remains stable across retries.

The publisher uses persistent messages, mandatory routing, and publisher confirmations. A failed or ambiguous publication makes the command fail; a later scan republishes unfinished sources. Discovery collapses duplicate hashes within one scan. Rediscovery and the commit/ACK failure window can produce duplicates. This is at-least-once delivery, with idempotency for the same SHA-256.

Each worker holds a PostgreSQL session advisory lock for the source hash, then checks completion again. Evidence, expense writes, and the final succeeded or review_required status commit in one transaction before ACK. A redelivery after that commit skips OCR and persistence. Different hashes can run concurrently. Local backlog processing uses the same per-source lock. A worker crash releases the database lock when its session closes; RabbitMQ requeues unacknowledged work. Use direct PostgreSQL connections or session pooling, not transaction pooling.

Each worker handles one receipt at a time (prefetch=1). OCR/DB work runs on a worker thread while the connection thread services heartbeats. SIGTERM stops new work and lets the current receipt finish. If Kubernetes kills it after the grace period, the unacknowledged delivery returns to the queue. Broker disconnection causes process exit, and the Deployment restarts the worker.

Retry and dead letters¶

The application declares three durable quorum queues, each bound to a direct exchange of the same name with routing key receipt:

Queue	Purpose
`receipts.v1.work`	Receipt processing
`receipts.v1.retry`	Wait 60 seconds, then dead-letter back to work
`receipts.v1.dead`	Invalid messages and exhausted work for inspection

Failures publish a new message to retry with an incremented attempt, then ACK the original only after confirmation. The third failed delivery goes to the DLQ. Malformed messages and changed source contents go there immediately. A failed retry/DLQ publication leaves the original unacknowledged. Broker-side transfers use quorum at-least-once dead lettering and reject-publish overflow, so the source retains a message until the destination accepts it. See the RabbitMQ quorum queue documentation.

The existing database attempt counter also limits each source to three processing starts, including interrupted starts. Scheduled discovery skips exhausted hashes; fresh publication cannot silently reset that counter. Connection failures before a database attempt have only the message's retry budget. The work queue's delivery limit of 20 additionally bounds repeated crashes without application settlement. Completed receipts requiring human review count as completed and are ACKed.

The DLQ keeps the original body, stable message ID when valid, and an error_type header. Worker logs include the hash, status or routing destination, and error type. Detailed processing errors remain in the existing database status table. Monitor work depth, unacknowledged work, retry depth, DLQ depth, and worker restarts. All queues reject new publications at 100,000 messages rather than dropping old work. Queue arguments are deployment constants; changing them requires draining and recreating the affected queues or introducing a new versioned prefix.

To recover an exhausted receipt, pause publication and workers and inspect the DLQ and ingest.receipt_processing_status. Fix the source/configuration, remove stale deliveries for that hash, then use the CLI to reset its attempt budget:

ledger receipts retry '2026-08-14/receipts_20260814_0001.pdf'
ledger receipts publish
ledger receipts consume

Use the exact path relative to the receipt source root. retry resets only the matching failed or interrupted receipt's processing status and attempt budget; it preserves source identity and canonical evidence. It refuses active processing locks, completed receipts (including review-required receipts), and ambiguous source references. A successful command reports retry_ready: true. It requires only database access and prepares the receipt for the next publish or process command; it does not publish or remove broker messages itself. Do not simply move an attempt-3 DLQ body back to work. For intentional OCR refresh of one receipt, use ledger receipts reprocess SOURCE_REFERENCE --verbose. It publishes a confirmed reprocess message to receipts.v1.work and returns queued. The consumer refreshes OCR and replaces the selected receipt's extraction results under the shared per-receipt lock. A request UUID makes completed-request redelivery a no-op; each new request gets its own three-attempt budget. If publication is uncertain, reuse the printed --request-id UUID. Deploy all consumers with v2 support before publishing reprocess messages. See receipt processing for K3s commands, rollout details, and RabbitMQ GUI monitoring. For a whole-inbox refresh, use receipts process --refresh-ocr-cache; it queues one v2 request per receipt. ocr-cache rebuild queues v3 requests that preserve saved extraction. Both commands return after confirmed publication and accept --request-id UUID for uncertain batch retries. Keep consumers running to execute the queued work. Stop all writers before rebuilding or switching the ingest schema; its completion state is the idempotency record.

Interpreting a cache-repair backlog¶

After introducing a new or empty OCR cache directory, the 15-minute publisher can repeatedly queue repairs for completed receipts whose caches are still missing. These publications reuse a request UUID for the same source hash, relative path, and previous completion time. RabbitMQ retains each delivery; the worker checks PostgreSQL request completion and ACKs completed copies with status=skipped. See scheduled cache discovery and duplicate backlog for the identity rules and tradeoff.

A queue that drains rapidly after a period of slow OCR can therefore be normal. Compare worker outcomes rather than interpreting every delivery as a new OCR run:

Worker log	Meaning
`Running OCR and parsing`	A processing attempt reached OCR/parsing.
`status=skipped`	Completion was already recorded; this delivery did not run OCR.
`status=succeeded` or `status=review_required`	Processing completed; review-required receipts still need review.
`routed=receipts.v1.dead`	The delivery left work for inspection, without successful completion.

For example, the retained worker logs on September 12, 2026, around 08:23–08:31 Winnipeg time contained 161 handled deliveries: 154 skipped, six processed (three succeeded and three required review), and one stale-source message sent to the dead-letter queue. This demonstrates how many deliveries can drain with few OCR runs; the outcome counts alone do not establish when each duplicate was originally published.

In the management UI, open Queues and Streams → receipts.v1.work and select last hour beside Queued messages. Total includes ready and unacknowledged deliveries. The configured x-max-length of 100,000 is a capacity limit, not an observed peak. Check retry and dead-letter queues separately.

RabbitMQ's management statistics are stored in memory and lost on broker restart, while durable messages can survive on the broker volume. A graph that shows zero before recovery is not evidence that the queue was empty. Its highest retained sample cannot establish the peak before a crash loop or restart. Persistent external monitoring is needed for history across restarts; see RabbitMQ management statistics.

To inspect retained worker activity since a specific time, use an explicit UTC timestamp and override the selector's default log tail. For example, 05:00 in Winnipeg on September 12, 2026, is 10:00 UTC:

kubectl --context brownrook-k3s1 -n home-budget logs \
  -l workload=receipt-worker \
  --since-time='2026-09-12T10:00:00Z' \
  --tail=-1 --timestamps=true --prefix=true

Add --previous to inspect the preceding container instance when available. Kubernetes log retention and container replacement can limit coverage; worker logs describe processing outcomes rather than historical queue depth.

Run and validate¶

Install the project with its database dependencies (python -m pip install -e '.[db,dev]'). Set DATABASE_URL, RABBITMQ_URL, and optionally RECEIPT_SOURCE_ROOT and HOME_BUDGET_OCR_CACHE. If these are stored in .env.dev, export the assignments so Python can read them (plain source does not export new shell variables):

set -a
source .env.dev
set +a

Broker credentials must have configure/write/read access to the queues and exchanges in a dedicated vhost. Then run:

ledger receipts publish /shared/receipts
ledger receipts consume /shared/receipts

The Kubernetes publisher and worker use the same ledger receipts commands. Add consumers to increase concurrency.

# Disposable local RabbitMQ; use only the separate test database.
docker compose -f compose.queue-test.yaml up -d --wait
TEST_RABBITMQ_URL='amqp://test:test@localhost:5673/%2F' \
  make test-db
docker compose -f compose.queue-test.yaml down -v

The disposable broker's management UI is available at http://localhost:15673 with the test username and password from compose.queue-test.yaml. For the Kubernetes overlay, use:

kubectl -n home-budget port-forward svc/rabbitmq 15672:15672

Then open http://localhost:15672 and sign in with the credentials stored in rabbitmq-secret.

The repository's test hook bootstraps the local home_budget_test database. When overriding TEST_DATABASE_URL, provision the disposable database and load the schema as in CI. PostgreSQL tests launch separate worker processes to verify same-source serialization, different-source concurrency, rollback, attempt limits, lock release, and commit-before-ACK replay. Broker tests cover confirmed publishing, unroutable messages, unacknowledged redelivery, delayed retries, DLQ routing, and consumer shutdown. CI supplies both services. Unit tests need no external services: make test.

Kubernetes rollout¶

deploy/rabbitmq is an opt-in Kustomize overlay over the existing k8s resources. It changes the existing receipt CronJob to discovery/publication, adds one OCR worker Deployment, and adds a single RabbitMQ StatefulSet with an 8 GiB local-path volume. The publisher and worker use the existing shared home-budget-data PVC. The worker image follows the immutable image selected by CI for the publisher. Use an application image built from this change.

Set RABBITMQ_DEFAULT_USER and RABBITMQ_DEFAULT_PASS in the uncommitted .env.k3s file (copy .env.k3s.example). Keep .env.dev for local Mac settings. Use receipts as the username and generate a password with openssl rand -hex 32. For an existing broker, use its current credentials. From the repository root, create the deployment secret:

set -a
source .env.k3s
set +a
.venv/bin/python - <<'PY' | kubectl --context "$KUBE_CONTEXT" -n "$KUBE_NAMESPACE" apply -f -
import json
import os
from urllib.parse import quote

user = os.environ["RABBITMQ_DEFAULT_USER"]
password = os.environ["RABBITMQ_DEFAULT_PASS"]
namespace = os.environ["KUBE_NAMESPACE"]
if not user or not password:
    raise SystemExit("Set RABBITMQ_DEFAULT_USER and RABBITMQ_DEFAULT_PASS in .env.k3s")
print(json.dumps({
    "apiVersion": "v1",
    "kind": "Secret",
    "metadata": {"name": "rabbitmq-secret", "namespace": namespace},
    "type": "Opaque",
    "stringData": {
        "RABBITMQ_DEFAULT_USER": user,
        "RABBITMQ_DEFAULT_PASS": password,
        "RABBITMQ_URL": f"amqp://{quote(user, safe='')}:{quote(password, safe='')}@rabbitmq.{namespace}.svc.cluster.local:5672/receipts",
    },
}))
PY

This derives the connection URL from the same credentials and URL-encodes them. Use the namespace-qualified service name: application pods run in home-budget, but the KEDA operator resolves the same URL from the keda namespace. The private rabbitmq.brownrook.net hostname exposes only the HTTPS management UI, not the AMQP port. Only the RabbitMQ settings are sent to Kubernetes. The namespace must already exist. k8s/rabbitmq-secret.example.yaml remains available as a manual template. The broker initializes the receipts vhost. Secrets are deliberately excluded from the overlay. Review the rendered configuration:

kubectl kustomize deploy/rabbitmq

During rollout, suspend the existing receipt CronJob, let its active local job finish, then switch the deployment target from k8s to deploy/rabbitmq. Confirm broker readiness and worker startup before allowing the publisher CronJob to resume. Set the Argo CD application's source path to deploy/rabbitmq and run make deploy-k3s. No cluster resources are changed by rendering the overlay.

The overlay includes a KEDA ScaledObject using the RabbitMQ queue scaler and HPA behavior controls. The brownrook-root GitOps application installs the pinned KEDA 2.20.2 chart from brownrook-infra/kubernetes/gitops/apps/keda-app.yaml. Verify the operator, CRD, and resulting HPA with:

kubectl --context brownrook-k3s1 get crd scaledobjects.keda.sh
kubectl --context brownrook-k3s1 -n keda get deployment
kubectl --context brownrook-k3s1 -n home-budget get scaledobject, hpa

KEDA reads RABBITMQ_URL from the worker environment and scales against receipts.v1.work at one ready message per worker. The receipt-worker pool is pinned to Arsene and scales from one to three workers. Its Deployment omits replicas, so Argo CD does not fight the HPA. The separate receipt-worker-longbow Deployment keeps one worker on Longbow, including when the queue is empty. Both pools share the same work queue and application configuration. Together they run two to four workers across the two agents when both nodes are available. Raising the maximum requires a reviewed Git change and enough additional capacity for 1 CPU and 8Gi memory per pod. At the four-worker ceiling, OCR reserves 4 CPUs and 32Gi memory and is capped at 20 CPUs and 32Gi memory: three Arsene workers can each use up to 6 CPU cores, and the Longbow worker can use up to 2. The memory request equals the limit so scheduling budgets for each worker's full memory allowance. Both pools retain a 1-CPU request; actual CPU usage depends on the OCR stage and its parallelism. Required node affinity excludes control-plane nodes, protecting k3s1's database, broker and cluster services.

The RabbitMQ overlays explicitly set receipt-processor.spec.suspend: false. Once synced, the receipt-processor CronJob runs ledger receipts publish every 15 minutes. The publisher checks inbox files, PostgreSQL completion status, and cache-file existence under HOME_BUDGET_OCR_CACHE. Completed receipts with missing OCR cache files are queued in RabbitMQ as full v2 reprocess requests. The receipt-worker consumer receives those requests, recreates the cache, and replaces saved extraction results. Completed receipts with current or legacy cache files are skipped. The publisher uses the same HOME_BUDGET_OCR_CACHE setting as consumers. Its JSON summary includes cache_missing, counted within published. The publisher never runs OCR; the consumer only processes delivered requests. See missing-cache detection for responsibilities and an immediate manual check.

For a maintenance pause, set suspend: true in deploy/rabbitmq/publisher-patch.yaml and sync through Argo CD, then restore false to resume. A direct cluster edit may be reverted by Argo CD self-healing.

Worker updates use maxSurge: 0 and maxUnavailable: 1, allowing an old worker to release its CPU reservation before its replacement starts. KEDA adds at most one worker per minute and waits ten minutes before scale-down; scale-down then removes at most one pod every five minutes. The 15-minute termination grace period lets a SIGTERM'd worker finish its current delivery before exiting, and RabbitMQ retains any delivery that is not acknowledged.

Additional receipt-worker nodes¶

The October 2026 cluster has three Ready amd64 nodes: k3s1 (server, 4 CPUs and approximately 11Gi allocatable memory), arsene (agent, 32 logical CPUs and approximately 91Gi), and longbow (agent, 4 CPUs and approximately 13Gi). All run K3s v1.36.4+k3s1 on RHEL 10.2. Workers use the same pinned amd64 application image, reserve 1 CPU and 8Gi memory, and have no GPU requirement. Only Arsene and Longbow are eligible for regular OCR workers, each through its own node-pinned Deployment. Longbow runs one 8Gi reservation; Arsene runs one to three, subject to its other pod reservations. This explicit split keeps a worker on each agent even when KEDA scales the Arsene pool down. Both pools have the workload=receipt-worker label for combined status and log commands.

The four-worker maximum is an initial capacity budget based on the October 4 snapshot: Arsene used approximately 41.5Gi memory, Longbow 1.9Gi, and k3s1 5.5Gi. These observations are not OCR peak measurements. Check node usage, reservations, OCR duration, OOM events, and shared-storage throughput before raising the ceiling again. Include temporary validation workers and workloads outside Kubernetes in the node budget.

For a reproducible replacement agent, use a unique hostname, an SSD-backed Linux installation, and the same K3s version as the server. On RHEL 10, install kernel-modules-extra and the K3s SELinux policy, retaining SELinux enforcement. Allow agent-to-server TCP 6443, inter-node VXLAN UDP 8472, and inter-node kubelet TCP 10250 on the private LAN. Allow SMB TCP 445 to the file server. Follow the K3s requirements for the host firewall and actual cluster pod/service CIDRs.

Provision the agent join token through the approved secrets mechanism into a root-owned 0600 file at /etc/rancher/k3s/agent-token. Do not put it in Git or shell history. Download and review the installer, then run it on the agent:

curl -fsSL https://get.k3s.io -o /tmp/install-k3s.sh
sudo env INSTALL_K3S_VERSION='v1.36.4+k3s1' sh /tmp/install-k3s.sh agent \
  --server https://k3s1.brownrook.net:6443 \
  --token-file /etc/rancher/k3s/agent-token --node-name arsene

This is the K3s agent join procedure with a pinned version and file-based token. Use the current server version and the replacement node's unique name when rebuilding later. Verify Ready, kubernetes.io/arch=amd64, sufficient allocatable resources, and no taint that prevents receipt-worker scheduling. The SMB CSI node DaemonSet must be Ready on the new node. A disposable pod using home-budget-data must resolve cluster DNS, connect to RabbitMQ, and read/write its own test directory on the RWX volume. Verify a held receipt lock prevents another node from acquiring it, then verify acquisition after release and valid cache checksums from both nodes. Never test by editing a production receipt or its cache.

To retire an agent, drain it while watching unfinished RabbitMQ deliveries, using kubectl --context brownrook-k3s1 drain NODE --ignore-daemonsets. Resolve any drain blocker rather than forcing production data loss. Once replacement workers are Ready and their unacknowledged work is recovered, stop the agent service, run /usr/local/bin/k3s-agent-uninstall.sh on the retired host, and delete its Node object. The uninstall procedure removes local K3s data; it does not remove the shared SMB receipt volume. For temporary maintenance, use drain followed by uncordon, without uninstalling the agent.

Autoscaling validation¶

Use a cache-only batch with at least four test receipts. In one terminal, watch the queue, scaler, and pods:

kubectl --context brownrook-k3s1 -n home-budget get scaledobject receipt-worker
kubectl --context brownrook-k3s1 -n home-budget get hpa
kubectl --context brownrook-k3s1 -n home-budget get \
  pod -l workload=receipt-worker -w

In another terminal, publish the disposable batch and follow request-correlated logs:

kubectl --context brownrook-k3s1 -n home-budget exec deploy/receipt-worker -- \
  ledger ocr-cache rebuild --missing-only --verbose
kubectl --context brownrook-k3s1 -n home-budget logs \
  -l workload=receipt-worker --prefix --follow

Record that two distinct pod names report status=active, batch log lines reach completed=TOTAL/TOTAL, and the Arsene HPA returns to one replica after the configured stabilization window while Longbow retains its one worker. Record pod-to-node placement with get pods -o wide: two pods on one node do not establish cross-node processing. Request logs include worker_host, worker_pid, worker_node, source reference, request identifier, and outcome. During a disposable run, delete one active worker pod; its unacknowledged request must be processed by the replacement pod. Confirm the cache checksum marker remains valid and canonical extraction rows are unchanged. Do not perform the interruption test against irreplaceable input or while unrelated production work is queued.

This broker is persistent but single-node, not highly available. Local storage loss can lose queued work; inbox rediscovery recovers unfinished sources within their attempt budget. A replicated broker is a separate operational slice. The management service is internal; no public ingress is included. The worker has a 15-minute shutdown grace period, and the broker ACK timeout is one hour.

For authenticated access to the management UI from the corporate LAN, use the deploy/rabbitmq-private overlay. It exposes only port 15672 through the private hostname at rabbitmq.brownrook.net; AMQP remains cluster-internal. RabbitMQ's own user and password are still required. The required private DNS, TLS secret, Traefik entry point, and firewall controls are documented in the network access guide.

To roll back, suspend publishing, drain work/retry queues, and stop workers before restoring the k8s deployment target. Preserve broker storage and DLQ contents until outstanding failures have been resolved. Use Argo CD's existing GitOps pruning workflow to remove resources no longer present in the selected overlay.