Building a production-ready async invoice pipeline: jobs, webhooks, retries, idempotency, and S3 delivery
Design a resilient async invoice pipeline with background jobs, HMAC-verified webhooks, idempotent handlers, exponential backoff for 429/503, and secure delivery to customer-owned S3 using Saliox.
Building a production-ready async invoice pipeline: jobs, webhooks, retries, idempotency, and S3 delivery
Synchronous HTML-to-PDF works for a single UI action; it breaks under month-end load, PDF/A or Factur‑X conversion, and durable delivery requirements. This guide shows platform engineers how to design and operate a resilient asynchronous invoice pipeline with background jobs, HMAC‑verified webhooks, idempotent handlers, safe retries, and delivery to customer‑owned S3.
1) Executive summary & why async pipelines for invoices
Why async for invoices:
- Heavy templates and assets (fonts, tables, watermarks) cost time and memory.
- Compliance modes (PDF/A-3; optional Factur‑X/EN16931) add validation and conversion.
- Batch cycles spike to thousands of documents in minutes.
- Finance requires durability and verifiability (typically in customer‑owned S3).
- Downstream email/AP/ERP systems are often slower than renderers.
Business goals:
- Reliability: never lose an invoice; reconcile and re-drive failures.
- Repeatability: idempotency across retries and replays.
- Observability: metrics, traces, alerts, and a runbook.
- Security: HMAC‑verified webhooks, server‑only API keys, least‑privilege S3.
- Cost control: backoff on 429/503; avoid retry storms.
Decision guide:
- Use synchronous only for small, low-latency single documents with simple templates.
- Use async jobs + webhooks for batches, heavier templates, PDF/A/Factur‑X, or S3 delivery with audit trails.
Saliox supports synchronous and asynchronous generation. According to the current pricing page, Medium and higher plans include async jobs, webhooks, and customer S3 delivery (re-check before publishing).
2) End-to-end architecture blueprint
Flow:
- Billing system submits generation request and persists a local “intent” record, then the provider returns a job ID.
- Provider enqueues and renders, stores the artifact (managed CDN or verified customer S3‑compatible bucket), and emits a webhook with status and location.
- Your webhook handler verifies HMAC, upserts processing state, and triggers side effects idempotently.
- A reconciliation worker polls job status and/or verifies S3 artifacts when webhooks are missed.
Queuing/retries:
- Prefer the provider’s background queue. Submit once, persist the job ID, and rely on webhooks plus a light poller.
- Back off on 429/503 at submission; surface permanent validation errors immediately.
Security boundaries:
- API keys: server‑side only (env vars/secrets manager).
- Webhooks: verify HMAC over the raw body using your webhook secret.
- S3: grant minimum required permissions to the provider, scoped to a prefix; use presigned URLs for consumers.
3) Submitting jobs: request design and delivery choices
Each generation request uses exactly one source: a URL, raw HTML (optionally with a base URL and structured data), or a hosted template with data.
For async:
- Select async webhook delivery and a storage target (managed CDN or a pre‑verified customer S3‑compatible bucket). Webhooks require stored output so the payload includes a URL.
- Choose stable, discoverable filenames and prefixes, e.g.
tenant_id/invoices/yyyy/mm/dd/invoice-<invoice_id>-<job_id>.pdf. - Set overwrite behavior to align with idempotency (often “overwrite allowed” for the same key).
- Attach a business reference and/or idempotency key via metadata if supported to simplify reconciliation.
4) Job lifecycle: 202 response, polling, retention
- Async submissions return HTTP 202 with a job ID while rendering continues. Persist the job ID with your invoice immediately.
- Prefer push via webhook, with periodic job-status polling for jobs that remain unresolved after a threshold (e.g., 15–30 minutes).
- According to current docs, async job records are accessible for seven days; design reconciliation to avoid dependence beyond that window.
- Keep your local submission intent until you confirm delivery.
5) Webhook verification and security
- Configure a webhook secret. Saliox signs the raw request body using HMAC‑SHA256 and sends
sha256=<hex digest>in theX‑Saliox‑Signatureheader. - Perform timing‑safe verification before parsing or acting on payloads. HMAC itself is defined in RFC 2104.
- On verification failure, return a non‑2xx and log the attempt. Saliox retries deliveries that time out or return non‑2xx responses.
- Rotate webhook secrets through your standard key hygiene process.
- If a timestamp or event ID is provided by the provider, verify freshness and/or reject replays within your window; otherwise, track seen signatures/body hashes locally to rate‑limit replays.
6) Designing idempotent webhook handlers
Webhooks may retry and arrive out of order.
Core approach:
- Use the provider job ID as the primary de‑duplication key.
- Upsert a job record (status, file URL, checksum/size if present) in a single transaction before side effects.
- Make side effects (email, ledger, events) conditional on the first transition to a terminal success state, guarded by unique constraints or idempotency keys.
Integrity checks:
- Prefer provider checksums if the payload includes one (verify field/format in docs).
- For customer‑owned S3, validate with HeadObject: ContentLength and ETag. Note: ETag may not equal MD5 for multipart uploads or with encryption; treat it as a hint. For cryptographic assurance, download and compute SHA‑256.
Finite‑state model:
- RECEIVED -> DOWNLOADING -> STORED -> PROCESSED -> ERRORED
- Transitions are idempotent; replays after PROCESSED are no‑ops. Persist terminal error reasons to aid re‑drives/manual fixes.
7) Handling rate limits and generation failures (429/503)
Submission errors:
- 429 Too Many Requests: honor
Retry-Afterwhen present; use exponential backoff with jitter; avoid synchronized retries. - 503 Service Unavailable: honor
Retry-Afterwhere present, retry with capped backoff and jitter, and move stubborn cases to a dead‑letter queue. Also follow the current Saliox rate-limit guidance.
Webhook redelivery:
- Failed deliveries are attempted three times (initial, ~5 min, ~30 min) per current docs. Design tolerance for delayed completion.
Error classes:
- Permanent: validation failures (e.g., missing mandatory invoice fields for Factur‑X). Surface immediately with context.
- Transient: network/capacity. Retry with caps and alert only after thresholds.
8) Delivery to customer‑owned S3 (best practices)
- Supported S3‑compatible services (per docs today) include AWS S3, DigitalOcean Spaces, and Cloudflare R2. Buckets must be configured and verified in the Saliox dashboard before use.
IAM/policies:
- Grant minimum permissions to the provider’s principal: PutObject (and PutObjectAcl only if required), limited to a path prefix like
tenant_id/invoices/*. - Add conditions where feasible (prefix constraints, object ownership/ACL expectations). Consult your cloud’s policy language. If you require SSE‑KMS or special headers, confirm provider support before enforcing bucket policies.
Naming and access:
- Use partitionable prefixes for performance/governance and to aid reconciliation.
- Choose overwrite behavior consistent with your idempotency plan.
- Prefer short‑lived presigned URLs for consumers; avoid public buckets.
Validation:
- Use
HeadObjectfor fast existence, size, and ETag checks. Remember that an S3 multipart-upload ETag is not necessarily an MD5 hash. - Optional GetObject + SHA‑256 for high assurance.
9) Monitoring, alerting, and operational runbook
Metrics:
- Job submission rate, success rate, end‑to‑end latency (submission → webhook).
- Webhook failure rate (4xx/5xx), signature failures.
- 429/503 counts and average backoff.
- S3 delivery and verification failures.
- Reconciliation backlog and any inferred queue depth.
Alerts:
- Spike in webhook failures or signature mismatches.
-
X unresolved jobs older than Y minutes.
- Sustained 429/503 on submissions beyond Z attempts.
- Completed jobs missing expected S3 objects.
Runbook:
- Locate by job ID or invoice reference. 2) Check job status at the provider. 3) If the file exists, re‑download and re‑run local post‑processing. 4) If missing/corrupted, re‑request using the same idempotency key (if supported). 5) Escalate permanent data/validation errors to business owners.
Periodic reconciliation:
- Scan job‑status and/or S3 for mismatches between expected and delivered invoices (hourly at scale; daily for lower volumes).
10) Testing, chaos, and end‑to‑end validation
- Unit tests: HMAC verification (happy/tampered), idempotent writes, state transitions.
- Integration: webhook replay/duplicates; network partition mid‑upload; provider 429/503; partial/corrupted files; Factur‑X success/validation‑error cases.
- Staging: use staging buckets and a test webhook endpoint (e.g., ngrok or a hosted tunnel) to validate end‑to‑end.
- Load testing: match production batch profiles to expose rate limits and queue behavior early.
11) Example patterns and code snippets
These are implementation patterns. Verify all endpoint paths, header casing, and signature encoding against the live docs before deployment.
Node.js (Express): Webhook verification + idempotent upsert (Postgres)
// app.js
// HMAC-verified, idempotent webhook handler.
// Current Saliox format: X-Saliox-Signature: sha256=<hex digest>.
const express = require('express');
const crypto = require('crypto');
const { Pool } = require('pg');
const app = express();
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
const WEBHOOK_SECRET = process.env.SALIOX_WEBHOOK_SECRET; // store in a secrets manager in prod
// Use raw body for HMAC; restrict size to limit DoS risk
app.post('/webhooks/saliox', express.raw({ type: 'application/json', limit: '256kb' }), async (req, res) => {
const signatureHeader = req.get('X-Saliox-Signature') || '';
// Saliox prefixes the lowercase hex HMAC with "sha256=".
const expectedSignature = 'sha256=' + crypto
.createHmac('sha256', Buffer.from(WEBHOOK_SECRET, 'utf8'))
.update(req.body)
.digest('hex');
const provided = Buffer.from(signatureHeader, 'utf8');
const expected = Buffer.from(expectedSignature, 'utf8');
if (provided.length !== expected.length || !crypto.timingSafeEqual(provided, expected)) {
return res.status(400).send('Signature verification failed'); // Non-2xx to trigger retry; confirm semantics in docs
}
// Parse JSON only after successful verification
let payload;
try {
payload = JSON.parse(req.body.toString('utf8'));
} catch (e) {
return res.status(400).send('Invalid JSON');
}
const jobId = payload.job_id;
const status = payload.status; // e.g., 'completed' | 'failed'
const fileUrl = payload.cdn_url; // platform CDN or customer S3-compatible URL
const checksum = payload.checksum; // optional; confirm presence/format
if (!jobId) return res.status(400).send('Missing jobId');
const client = await pool.connect();
try {
await client.query('BEGIN');
// Create/update job record before any side effects
const upsertSql = `
INSERT INTO invoice_jobs (job_id, status, file_url, checksum, updated_at)
VALUES ($1, $2, $3, $4, NOW())
ON CONFLICT (job_id)
DO UPDATE SET
status = EXCLUDED.status,
file_url = COALESCE(EXCLUDED.file_url, invoice_jobs.file_url),
checksum = COALESCE(EXCLUDED.checksum, invoice_jobs.checksum),
updated_at = NOW()
RETURNING job_id, status, file_url;
`;
const { rows } = await client.query(upsertSql, [jobId, status, fileUrl || null, checksum || null]);
const row = rows[0];
// Idempotent side-effect guard
if (row.status === 'completed') {
const processedCheck = await client.query('SELECT 1 FROM invoice_processing WHERE job_id = $1', [jobId]);
if (processedCheck.rowCount === 0) {
await client.query('INSERT INTO invoice_processing (job_id, processed_at) VALUES ($1, NOW())', [jobId]);
// TODO: enqueue email / bus event; design consumers to be idempotent
}
}
await client.query('COMMIT');
return res.status(200).send('OK');
} catch (err) {
await client.query('ROLLBACK');
console.error('Webhook handler error', err);
return res.status(500).send('Internal error');
} finally {
client.release();
}
});
app.listen(process.env.PORT || 3000, () => console.log('Webhook listener started'));
Python: HMAC verification utility
# hmac_verify.py
# Verify Saliox webhook signatures using the documented sha256=<hex digest> format.
import hashlib
import hmac
def verify_signature(raw_body: bytes, signature_header: str, secret: str) -> bool:
digest = hmac.new(secret.encode('utf-8'), raw_body, hashlib.sha256).hexdigest()
expected = f'sha256={digest}'
return hmac.compare_digest(expected, signature_header)
Polling/reconciliation worker (pseudocode)
Do not hardcode API paths. Check the job‑status endpoint path, auth, and response schema in the docs.
loop every 5 minutes:
jobs = db.query(
"SELECT job_id FROM invoice_jobs WHERE status NOT IN ('completed','failed') AND created_at < now() - interval '15 minutes' LIMIT 100"
)
for job in jobs:
try:
resp = http.get(
url = JOB_STATUS_ENDPOINT(job.job_id), # look up correct path in docs
headers = { 'Authorization': 'Bearer ' + API_KEY }
)
if resp.status == 429 or resp.status == 503:
backoff_ms = next_backoff(job.job_id)
schedule_after(backoff_ms, requeue_job_check(job.job_id))
continue
if resp.status >= 500:
requeue_job_check(job.job_id)
continue
if resp.status == 200:
state = resp.json().status # verify field name
url = resp.json().fileUrl # verify field name
db.upsert_job(job.job_id, state, url)
if state == 'completed' and url:
enqueue_verification(job.job_id, url)
elif resp.status == 404:
alert_missing_job(job.job_id)
else:
log_unexpected_status(job.job_id, resp.status)
except NetworkError:
requeue_job_check(job.job_id)
AWS SDK v3 (Node.js): Verify delivery to S3
// s3-verify.js
// Server-side verification of S3 object. Ensure IAM is scoped to the required bucket/prefix.
import { S3Client, HeadObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';
import { createHash } from 'crypto';
const s3 = new S3Client({ region: process.env.AWS_REGION });
export async function verifyS3Object({ bucket, key, expectedBytes, expectedEtag, computeSha256 = false }) {
const head = await s3.send(new HeadObjectCommand({ Bucket: bucket, Key: key }));
if (typeof expectedBytes === 'number' && head.ContentLength !== expectedBytes) {
throw new Error(`Size mismatch: expected ${expectedBytes}, got ${head.ContentLength}`);
}
if (expectedEtag && head.ETag && head.ETag.replaceAll('"','') !== expectedEtag.replaceAll('"','')) {
// ETag may not equal MD5 for multipart uploads or with encryption
throw new Error(`ETag mismatch: expected ${expectedEtag}, got ${head.ETag}`);
}
if (!computeSha256) return { ok: true };
const get = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
const hash = createHash('sha256');
await new Promise((resolve, reject) => {
get.Body.on('data', (chunk) => hash.update(chunk));
get.Body.on('end', resolve);
get.Body.on('error', reject);
});
return { ok: true, sha256: hash.digest('hex') };
}
12) Checklist, tradeoffs, and closing recommendations
Before shipping:
- Persist submission intent and provider job IDs.
- Configure and verify webhook secrets; perform timing‑safe HMAC checks on raw bodies.
- Keep the Saliox API key server‑side only.
- Choose storage (managed CDN vs verified customer S3‑compatible bucket) and set overwrite policy.
- Implement idempotent handlers with transactional upserts and a simple state machine.
- Implement exponential backoff and dead‑letter handling for stubborn 429/503 or processing failures.
- Add monitoring, alerts, and a periodic reconciliation job.
Recommended defaults:
- Prefer direct S3 delivery for auditability and long‑term control.
- Use webhooks for push‑driven workflows plus a reconciliation poller for missed events.
- Retain job IDs/statuses for at least the provider’s current async record window (7 days) plus buffer.
If you’re using Saliox:
- Use async jobs and webhooks (Medium plan and up; re‑check pricing) to receive HTTP 202 with a job ID while generation runs.
- Configure and verify a customer S3‑compatible bucket; reference the bucket and path in requests.
- Verify the
X‑Saliox‑SignatureHMAC‑SHA256 value before processing any webhook payload. - Expect three webhook attempts (initial, ~5 min, ~30 min) per the current documentation; use a polling and reconciliation worker for edge cases.
Further reading and references
Saliox documentation
- API documentation
- Async jobs, webhook payloads, signature verification, and retry behaviour
- Job-status polling
- Platform CDN storage
- Customer-owned S3 delivery
- Hosted templates
- Factur‑X / EN 16931 API
- Pricing and plan features
Technical references
- HMAC — RFC 2104
- 429 Too Many Requests — RFC 6585
- 503 Service Unavailable and Retry-After — RFC 9110
crypto.timingSafeEqual()— Node.js documentationHeadObject— Amazon S3 API- Multipart uploads and ETags — Amazon S3 User Guide
Try Saliox with a real document from your application. Start free, submit an asynchronous generation job, and test the full webhook and storage path before production rollout.