Intermediate · Lesson 6 of 6 · 18 min
Reliable workflow architecture
Design automations that survive real-world failures: idempotency, retries with backoff, error handling, logging and human review.
Assume every step fails sometimes
In production, APIs return 429s and 5xx errors, the Claude API occasionally returns 529 when it is overloaded, webhooks arrive twice, credentials expire and models return output that fails validation. None of this is rare at volume.
For every step, answer three questions before launch. What happens when it fails? Who finds out, and how quickly? Can the run be repeated safely? If you cannot answer the third, you have an idempotency problem waiting to happen.
Idempotency
An idempotent step gives the same result whether it runs once or five times. Without it, a retried webhook creates two opportunities, sends two welcome SMS messages or charges a card twice.
The fix is a stable key. Use the event's own ID, such as HighLevel's webhookId, or a key derived from the business fact, such as form ID plus submission ID. Record it in a table with a unique constraint before you act, and skip the work if the insert fails because the key exists. Prefer update-or-create operations keyed on an external ID over blind creates.
Retries with backoff and jitter
Retry only errors that can succeed later: 429s with a retry-after header, 500 and 529 from the Claude API, gateway errors and network timeouts. Do not retry 400, 401, 403 or 404, because the same request will fail the same way. On the Claude API, a 429 without retry-after signals a spend cap, which retrying cannot fix.
Wait longer after each failure (exponential backoff), add randomness (jitter) so many clients do not retry in lockstep, and cap the number of attempts. Retrying a call that has side effects, such as creating a contact, is only safe when that step is idempotent.
const RETRYABLE = new Set([429, 500, 502, 503, 504, 529]);
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
function backoffMs(attempt: number): number {
const ceiling = Math.min(30_000, 1_000 * 2 ** (attempt - 1));
return ceiling / 2 + Math.random() * (ceiling / 2); // jitter
}
export async function fetchWithRetry(url: string, init: RequestInit, maxAttempts = 5): Promise<Response> {
for (let attempt = 1; ; attempt++) {
try {
const response = await fetch(url, { ...init, signal: AbortSignal.timeout(60_000) });
if (response.ok || !RETRYABLE.has(response.status) || attempt >= maxAttempts) return response;
const retryAfter = Number(response.headers.get("retry-after"));
// On the Claude API a 429 without retry-after is a spend cap; retrying won't help.
if (response.status === 429 && !retryAfter) return response;
await sleep(retryAfter > 0 ? retryAfter * 1_000 : backoffMs(attempt));
} catch (error) {
// Network failure or timeout: retry until attempts run out.
if (attempt >= maxAttempts) throw error;
await sleep(backoffMs(attempt));
}
}
}Error handling and logging
When retries run out, the failure must go somewhere a person will see it. In n8n, assign an error workflow that starts with an Error Trigger node and alerts the owner. Make offers error handlers on modules. In custom code, write failed items to an error queue or table that can be replayed after the fix.
Log the inputs, the outcome and the identifiers you need to investigate: your own run ID, the event ID and the provider's request ID. Every Claude API response carries a request-id header for this. Do not log API keys, tokens or personal data you do not need; log a contact ID, not the contact's email and phone.
Human-in-the-loop by design
Put a person in front of any action that is hard to undo or reaches a customer: sending messages, moving deals, issuing refunds, deleting records. Route model outputs with low confidence or failed validation to the same review queue.
A review queue needs an owner, a check schedule and a fallback when it backs up. Log each reviewer's decision next to the model's suggestion. Those records tell you how often the model is right, which is the evidence you need before letting any category of action skip review.
Key points
- ◆Decide for every step what happens on failure, who is told and whether it can safely re-run.
- ◆Make side-effecting steps idempotent with a stable key and a unique constraint.
- ◆Retry only transient errors, with exponential backoff, jitter and a cap.
- ◆Send exhausted failures to an alert or replayable error queue, and log IDs rather than personal data.
- ◆Review risky or low-confidence actions, and use logged decisions to earn automation.
Terms in this lesson
Try it · exercise
Harden one workflow against duplicates and failures, then prove it with two deliberate tests.
- Take a webhook-triggered workflow you built, or the n8n webhook from the webhooks lesson.
- Write a failure table: one row per step with its failure mode, how you detect it and the response (retry, queue or person).
- Add a duplicate check keyed on webhookId, for example a Remove Duplicates node in n8n or a lookup in a sheet or table before the action step.
- Create an error workflow with an Error Trigger node that sends you an email or Slack message, and assign it in the workflow settings.
- Publish the workflow so these tests run as real executions on the production URL rather than in the editor.
- Send the same payload twice, then break a credential used by a later step and send a new payload with a different webhookId.
You're done when: The duplicate send produces only one record or action, and the broken credential produces an alert in your email or Slack instead of a silent failure.