Workflow invariants, replay, idempotency, and races

Find security consequences in workflow ordering and concurrency, with serial controls and durable-state evidence.

Used in these playbooks#

Setup and invariant

Choose one reversible workflow in a local or nonfinancial sandbox: accepting an invitation, creating one test resource, applying a test credit, approving a draft, or consuming a one-time action. Write the exact invariant, for example: one invitation grants at most one membership, to its intended recipient and role. Record preconditions, state transitions, actor permissions, durable result, retries, and cleanup. Logic findings depend on intended rules, not whether the request is technically valid. Business logic.

Numbered test procedure

1. Draw the normal transition table. List draft → submitted → approved → completed, or the product's actual equivalent. For each edge, record actor, required prior state, immutable fields, and transaction identifier. Perform one normal run with synthetic objects and save before/after state, not just HTTP responses.
2. Test skipped and reordered transitions. On a fresh disposable object, attempt the observed completion request before approval. Then change only a value that should be frozen after approval, such as recipient or test quantity. Compare with a fully approved control and an invalid object ID. If a transition is intentionally optional, adjust the invariant rather than treating missing UI steps as a bug.
3. Test serial replay. Repeat the same completed operation once after it settles. For documented idempotency keys, compare the same key/same body with the same key/different body, using sandbox data. Determine whether keys are scoped by actor, tenant, endpoint, and expiry. Returning the first result again can be correct idempotent behavior.
4. Test boundary values selectively. Use zero, a single negative value, or a precision boundary only where the sandbox permits it. Record server-calculated totals and immutable values. Avoid real billing, inventory, refunds, transfers, or customer notifications. A strange displayed number without a committed consequence is not enough.
5. Test concurrency only in an authorized setting. First demonstrate expected serial behavior. Reset a test object, then send exactly two simultaneous requests as a small experiment. The local example below uses a barrier; it does not guarantee server arrival within the vulnerable interval. If there is no overlap, record inconclusive timing rather than absence of a race. More aggressive synchronization belongs in the isolated lab. Race conditions.
6. Inspect after completion. Wait for both requests and relevant workers. Count durable memberships/resources or inspect the ledger of synthetic events. Two success responses may correspond to one commit. Repeat only to resolve a specific uncertainty, then restore the object and stop at the first demonstrated invariant failure.

Command card

This sends exactly two requests to a disposable local invitation endpoint. Put a valid local test bearer token in A.token, with restrictive permissions, and replace the route with the observed one. Create a fresh invitation for each serial/concurrent comparison. Run against localhost only.

from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from threading import Barrier
from urllib.request import Request, build_opener, ProxyHandler, HTTPRedirectHandler
from urllib.error import HTTPError

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None

gate = Barrier(2)
token = Path('A.token').read_text().strip()
def attempt(index):
    req = Request('http://127.0.0.1:8000/api/invitations/TEST-INVITE/accept',
                  data=b'{}', method='POST',
                  headers={'Authorization': 'Bearer ' + token,
                           'Content-Type': 'application/json'})
    gate.wait()
    try:
        with build_opener(ProxyHandler({}), NoRedirect()).open(req, timeout=10) as response:
            return index, response.status, response.read(1024).decode(errors='replace')
    except HTTPError as error:
        return index, error.code, error.read(1024).decode(errors='replace')

with ThreadPoolExecutor(max_workers=2) as pool:
    print(list(pool.map(attempt, range(2))))

Report threshold and checklist

Report a durable violation of the stated rule, with serial controls, initial conditions, minimal request sequence, and final state. Duplicate responses, transient UI counters, expected retries, and intentionally cumulative operations are common false positives. Describe the demonstrated consequence; do not infer financial loss from a sandbox counter alone.

• Business rule and expected actor written down.
• Normal and serial-replay controls completed.
• Fresh fixture used for each experiment.
• Only bounded authorized concurrency used.
• Durable state and asynchronous effects verified.

The Python example disables environment proxies and redirect following. A redirect is captured by the HTTPError branch; inspect its destination before any separate request.

Events that arrive after the request

Apply the same invariant to workers and integration events, not only the HTTP endpoint. Record a deduplication key's scope and when it becomes durable relative to the protected effect. A claim written before a worker fails can lose work; a claim written after the effect can permit duplicates. Use an instrumented fault point in the replica to distinguish these cases, and inspect committed state after retry. Then run the webhook event integrity procedure for signature, event ownership and out-of-order delivery checks. Do not assume provider delivery time is a reliable object version or that an HTTP acknowledgement proves processing completed.

Vulnerability reverse references#

• Race condition in a security-sensitive transition — Concurrent operations violate a rule that holds when the same actions run serially.
• Workflow order or approval bypass — An actor can complete a protected workflow without its required prior approval or immutable conditions.
• Single-use action or idempotency replay failure — A replay creates another protected effect where the operation must be unique or idempotent.
• Agent retry commits a protected single-use action twice — A response-loss or resume path duplicates an operation whose application contract requires at-most-once effect.
• Sensitive business flow exceeds its permitted frequency or allocation — An actor exceeds an intended recipient, reservation, workflow or tenant allocation through repeated otherwise valid operations.

Nearby shared procedures#

Object, function, and property authorization · Authentication, recovery, federation, and session lifecycle · Trace input into queries, commands, and templates · Server-side fetching and network trust boundaries · Browser execution, cross-origin reads, and forged actions · Uploads, downloads, paths, archives, and XML · REST, GraphQL, gRPC, WebSocket, and SSE permission parity · Proxy normalization, cache isolation, and request boundaries