All systems operational·EU + US regions
13 September 2026

Building a production-ready async invoice pipeline: jobs, webhooks, retries, idempotency, and S3 delivery

Design a resilient async invoice pipeline with background jobs, HMAC-verified webhooks, idempotent handlers, exponential backoff for 429/503, and secure delivery to customer-owned S3 using Saliox.

Building a production-ready async invoice pipeline: jobs, webhooks, retries, idempotency, and S3 delivery

Building a production-ready async invoice pipeline: jobs, webhooks, retries, idempotency, and S3 delivery

Synchronous HTML-to-PDF works for a single UI action; it breaks under month-end load, PDF/A or Factur‑X conversion, and durable delivery requirements. This guide shows platform engineers how to design and operate a resilient asynchronous invoice pipeline with background jobs, HMAC‑verified webhooks, idempotent handlers, safe retries, and delivery to customer‑owned S3.

1) Executive summary & why async pipelines for invoices

Why async for invoices:

  • Heavy templates and assets (fonts, tables, watermarks) cost time and memory.
  • Compliance modes (PDF/A-3; optional Factur‑X/EN16931) add validation and conversion.
  • Batch cycles spike to thousands of documents in minutes.
  • Finance requires durability and verifiability (typically in customer‑owned S3).
  • Downstream email/AP/ERP systems are often slower than renderers.

Business goals:

  • Reliability: never lose an invoice; reconcile and re-drive failures.
  • Repeatability: idempotency across retries and replays.
  • Observability: metrics, traces, alerts, and a runbook.
  • Security: HMAC‑verified webhooks, server‑only API keys, least‑privilege S3.
  • Cost control: backoff on 429/503; avoid retry storms.

Decision guide:

  • Use synchronous only for small, low-latency single documents with simple templates.
  • Use async jobs + webhooks for batches, heavier templates, PDF/A/Factur‑X, or S3 delivery with audit trails.

Saliox supports synchronous and asynchronous generation. According to the current pricing page, Medium and higher plans include async jobs, webhooks, and customer S3 delivery (re-check before publishing).

2) End-to-end architecture blueprint

Flow:

  • Billing system submits generation request and persists a local “intent” record, then the provider returns a job ID.
  • Provider enqueues and renders, stores the artifact (managed CDN or verified customer S3‑compatible bucket), and emits a webhook with status and location.
  • Your webhook handler verifies HMAC, upserts processing state, and triggers side effects idempotently.
  • A reconciliation worker polls job status and/or verifies S3 artifacts when webhooks are missed.

Queuing/retries:

  • Prefer the provider’s background queue. Submit once, persist the job ID, and rely on webhooks plus a light poller.
  • Back off on 429/503 at submission; surface permanent validation errors immediately.

Security boundaries:

  • API keys: server‑side only (env vars/secrets manager).
  • Webhooks: verify HMAC over the raw body using your webhook secret.
  • S3: grant minimum required permissions to the provider, scoped to a prefix; use presigned URLs for consumers.

3) Submitting jobs: request design and delivery choices

Each generation request uses exactly one source: a URL, raw HTML (optionally with a base URL and structured data), or a hosted template with data.

For async:

  • Select async webhook delivery and a storage target (managed CDN or a pre‑verified customer S3‑compatible bucket). Webhooks require stored output so the payload includes a URL.
  • Choose stable, discoverable filenames and prefixes, e.g. tenant_id/invoices/yyyy/mm/dd/invoice-<invoice_id>-<job_id>.pdf.
  • Set overwrite behavior to align with idempotency (often “overwrite allowed” for the same key).
  • Attach a business reference and/or idempotency key via metadata if supported to simplify reconciliation.

4) Job lifecycle: 202 response, polling, retention

  • Async submissions return HTTP 202 with a job ID while rendering continues. Persist the job ID with your invoice immediately.
  • Prefer push via webhook, with periodic job-status polling for jobs that remain unresolved after a threshold (e.g., 15–30 minutes).
  • According to current docs, async job records are accessible for seven days; design reconciliation to avoid dependence beyond that window.
  • Keep your local submission intent until you confirm delivery.

5) Webhook verification and security

  • Configure a webhook secret. Saliox signs the raw request body using HMAC‑SHA256 and sends sha256=<hex digest> in the X‑Saliox‑Signature header.
  • Perform timing‑safe verification before parsing or acting on payloads. HMAC itself is defined in RFC 2104.
  • On verification failure, return a non‑2xx and log the attempt. Saliox retries deliveries that time out or return non‑2xx responses.
  • Rotate webhook secrets through your standard key hygiene process.
  • If a timestamp or event ID is provided by the provider, verify freshness and/or reject replays within your window; otherwise, track seen signatures/body hashes locally to rate‑limit replays.

6) Designing idempotent webhook handlers

Webhooks may retry and arrive out of order.

Core approach:

  • Use the provider job ID as the primary de‑duplication key.
  • Upsert a job record (status, file URL, checksum/size if present) in a single transaction before side effects.
  • Make side effects (email, ledger, events) conditional on the first transition to a terminal success state, guarded by unique constraints or idempotency keys.

Integrity checks:

  • Prefer provider checksums if the payload includes one (verify field/format in docs).
  • For customer‑owned S3, validate with HeadObject: ContentLength and ETag. Note: ETag may not equal MD5 for multipart uploads or with encryption; treat it as a hint. For cryptographic assurance, download and compute SHA‑256.

Finite‑state model:

  • RECEIVED -> DOWNLOADING -> STORED -> PROCESSED -> ERRORED
  • Transitions are idempotent; replays after PROCESSED are no‑ops. Persist terminal error reasons to aid re‑drives/manual fixes.

7) Handling rate limits and generation failures (429/503)

Submission errors:

Webhook redelivery:

  • Failed deliveries are attempted three times (initial, ~5 min, ~30 min) per current docs. Design tolerance for delayed completion.

Error classes:

  • Permanent: validation failures (e.g., missing mandatory invoice fields for Factur‑X). Surface immediately with context.
  • Transient: network/capacity. Retry with caps and alert only after thresholds.

8) Delivery to customer‑owned S3 (best practices)

IAM/policies:

  • Grant minimum permissions to the provider’s principal: PutObject (and PutObjectAcl only if required), limited to a path prefix like tenant_id/invoices/*.
  • Add conditions where feasible (prefix constraints, object ownership/ACL expectations). Consult your cloud’s policy language. If you require SSE‑KMS or special headers, confirm provider support before enforcing bucket policies.

Naming and access:

  • Use partitionable prefixes for performance/governance and to aid reconciliation.
  • Choose overwrite behavior consistent with your idempotency plan.
  • Prefer short‑lived presigned URLs for consumers; avoid public buckets.

Validation:

9) Monitoring, alerting, and operational runbook

Metrics:

  • Job submission rate, success rate, end‑to‑end latency (submission → webhook).
  • Webhook failure rate (4xx/5xx), signature failures.
  • 429/503 counts and average backoff.
  • S3 delivery and verification failures.
  • Reconciliation backlog and any inferred queue depth.

Alerts:

  • Spike in webhook failures or signature mismatches.
  • X unresolved jobs older than Y minutes.

  • Sustained 429/503 on submissions beyond Z attempts.
  • Completed jobs missing expected S3 objects.

Runbook:

  1. Locate by job ID or invoice reference. 2) Check job status at the provider. 3) If the file exists, re‑download and re‑run local post‑processing. 4) If missing/corrupted, re‑request using the same idempotency key (if supported). 5) Escalate permanent data/validation errors to business owners.

Periodic reconciliation:

  • Scan job‑status and/or S3 for mismatches between expected and delivered invoices (hourly at scale; daily for lower volumes).

10) Testing, chaos, and end‑to‑end validation

  • Unit tests: HMAC verification (happy/tampered), idempotent writes, state transitions.
  • Integration: webhook replay/duplicates; network partition mid‑upload; provider 429/503; partial/corrupted files; Factur‑X success/validation‑error cases.
  • Staging: use staging buckets and a test webhook endpoint (e.g., ngrok or a hosted tunnel) to validate end‑to‑end.
  • Load testing: match production batch profiles to expose rate limits and queue behavior early.

11) Example patterns and code snippets

These are implementation patterns. Verify all endpoint paths, header casing, and signature encoding against the live docs before deployment.

Node.js (Express): Webhook verification + idempotent upsert (Postgres)

// app.js
// HMAC-verified, idempotent webhook handler.
// Current Saliox format: X-Saliox-Signature: sha256=<hex digest>.

const express = require('express');
const crypto = require('crypto');
const { Pool } = require('pg');

const app = express();
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
const WEBHOOK_SECRET = process.env.SALIOX_WEBHOOK_SECRET; // store in a secrets manager in prod

// Use raw body for HMAC; restrict size to limit DoS risk
app.post('/webhooks/saliox', express.raw({ type: 'application/json', limit: '256kb' }), async (req, res) => {
  const signatureHeader = req.get('X-Saliox-Signature') || '';

  // Saliox prefixes the lowercase hex HMAC with "sha256=".
  const expectedSignature = 'sha256=' + crypto
    .createHmac('sha256', Buffer.from(WEBHOOK_SECRET, 'utf8'))
    .update(req.body)
    .digest('hex');

  const provided = Buffer.from(signatureHeader, 'utf8');
  const expected = Buffer.from(expectedSignature, 'utf8');

  if (provided.length !== expected.length || !crypto.timingSafeEqual(provided, expected)) {
    return res.status(400).send('Signature verification failed'); // Non-2xx to trigger retry; confirm semantics in docs
  }

  // Parse JSON only after successful verification
  let payload;
  try {
    payload = JSON.parse(req.body.toString('utf8'));
  } catch (e) {
    return res.status(400).send('Invalid JSON');
  }

  const jobId = payload.job_id;
  const status = payload.status;    // e.g., 'completed' | 'failed'
  const fileUrl = payload.cdn_url;  // platform CDN or customer S3-compatible URL
  const checksum = payload.checksum; // optional; confirm presence/format

  if (!jobId) return res.status(400).send('Missing jobId');

  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    // Create/update job record before any side effects
    const upsertSql = `
      INSERT INTO invoice_jobs (job_id, status, file_url, checksum, updated_at)
      VALUES ($1, $2, $3, $4, NOW())
      ON CONFLICT (job_id)
      DO UPDATE SET
        status = EXCLUDED.status,
        file_url = COALESCE(EXCLUDED.file_url, invoice_jobs.file_url),
        checksum = COALESCE(EXCLUDED.checksum, invoice_jobs.checksum),
        updated_at = NOW()
      RETURNING job_id, status, file_url;
    `;
    const { rows } = await client.query(upsertSql, [jobId, status, fileUrl || null, checksum || null]);
    const row = rows[0];

    // Idempotent side-effect guard
    if (row.status === 'completed') {
      const processedCheck = await client.query('SELECT 1 FROM invoice_processing WHERE job_id = $1', [jobId]);
      if (processedCheck.rowCount === 0) {
        await client.query('INSERT INTO invoice_processing (job_id, processed_at) VALUES ($1, NOW())', [jobId]);
        // TODO: enqueue email / bus event; design consumers to be idempotent
      }
    }

    await client.query('COMMIT');
    return res.status(200).send('OK');
  } catch (err) {
    await client.query('ROLLBACK');
    console.error('Webhook handler error', err);
    return res.status(500).send('Internal error');
  } finally {
    client.release();
  }
});

app.listen(process.env.PORT || 3000, () => console.log('Webhook listener started'));

Python: HMAC verification utility

# hmac_verify.py
# Verify Saliox webhook signatures using the documented sha256=<hex digest> format.

import hashlib
import hmac


def verify_signature(raw_body: bytes, signature_header: str, secret: str) -> bool:
    digest = hmac.new(secret.encode('utf-8'), raw_body, hashlib.sha256).hexdigest()
    expected = f'sha256={digest}'
    return hmac.compare_digest(expected, signature_header)

Polling/reconciliation worker (pseudocode)

Do not hardcode API paths. Check the job‑status endpoint path, auth, and response schema in the docs.

loop every 5 minutes:
  jobs = db.query(
    "SELECT job_id FROM invoice_jobs WHERE status NOT IN ('completed','failed') AND created_at < now() - interval '15 minutes' LIMIT 100"
  )

  for job in jobs:
    try:
      resp = http.get(
        url = JOB_STATUS_ENDPOINT(job.job_id),  # look up correct path in docs
        headers = { 'Authorization': 'Bearer ' + API_KEY }
      )

      if resp.status == 429 or resp.status == 503:
        backoff_ms = next_backoff(job.job_id)
        schedule_after(backoff_ms, requeue_job_check(job.job_id))
        continue

      if resp.status >= 500:
        requeue_job_check(job.job_id)
        continue

      if resp.status == 200:
        state = resp.json().status    # verify field name
        url   = resp.json().fileUrl   # verify field name
        db.upsert_job(job.job_id, state, url)
        if state == 'completed' and url:
          enqueue_verification(job.job_id, url)
      elif resp.status == 404:
        alert_missing_job(job.job_id)
      else:
        log_unexpected_status(job.job_id, resp.status)

    except NetworkError:
      requeue_job_check(job.job_id)

AWS SDK v3 (Node.js): Verify delivery to S3

// s3-verify.js
// Server-side verification of S3 object. Ensure IAM is scoped to the required bucket/prefix.

import { S3Client, HeadObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';
import { createHash } from 'crypto';

const s3 = new S3Client({ region: process.env.AWS_REGION });

export async function verifyS3Object({ bucket, key, expectedBytes, expectedEtag, computeSha256 = false }) {
  const head = await s3.send(new HeadObjectCommand({ Bucket: bucket, Key: key }));

  if (typeof expectedBytes === 'number' && head.ContentLength !== expectedBytes) {
    throw new Error(`Size mismatch: expected ${expectedBytes}, got ${head.ContentLength}`);
  }

  if (expectedEtag && head.ETag && head.ETag.replaceAll('"','') !== expectedEtag.replaceAll('"','')) {
    // ETag may not equal MD5 for multipart uploads or with encryption
    throw new Error(`ETag mismatch: expected ${expectedEtag}, got ${head.ETag}`);
  }

  if (!computeSha256) return { ok: true };

  const get = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
  const hash = createHash('sha256');
  await new Promise((resolve, reject) => {
    get.Body.on('data', (chunk) => hash.update(chunk));
    get.Body.on('end', resolve);
    get.Body.on('error', reject);
  });
  return { ok: true, sha256: hash.digest('hex') };
}

12) Checklist, tradeoffs, and closing recommendations

Before shipping:

  • Persist submission intent and provider job IDs.
  • Configure and verify webhook secrets; perform timing‑safe HMAC checks on raw bodies.
  • Keep the Saliox API key server‑side only.
  • Choose storage (managed CDN vs verified customer S3‑compatible bucket) and set overwrite policy.
  • Implement idempotent handlers with transactional upserts and a simple state machine.
  • Implement exponential backoff and dead‑letter handling for stubborn 429/503 or processing failures.
  • Add monitoring, alerts, and a periodic reconciliation job.

Recommended defaults:

  • Prefer direct S3 delivery for auditability and long‑term control.
  • Use webhooks for push‑driven workflows plus a reconciliation poller for missed events.
  • Retain job IDs/statuses for at least the provider’s current async record window (7 days) plus buffer.

If you’re using Saliox:

Further reading and references

Saliox documentation

Technical references

Try Saliox with a real document from your application. Start free, submit an asynchronous generation job, and test the full webhook and storage path before production rollout.