AWS Teams Building with Strands Agents: Making Tool Calls Recoverable in Production

A customer opens a billing dispute. Your Strands agent pulls the last three invoices from the billing service, checks the account tier in the CRM, and decides the customer was double-charged for an add-on. It calls the issue_goodwill_credit tool. The billing API returns 200 and a $40 credit lands on the account.

Then the ECS scheduler replaces the task. Maybe a deployment rolled out, or Fargate Spot capacity was reclaimed. The agent never reached the next step: opening a ticket and replying to the customer.

When the message’s visibility timeout expires, the queue redelivers it. A fresh task starts the agent from the top, reaches the same conclusion, and calls issue_goodwill_credit again. Now the customer has $80, finance has a reconciliation problem, and nobody can say with confidence which tool calls actually completed the first time around.

For teams building on AWS with the open-source Strands Agents SDK, this is what happens by default when an agent loop with side effects runs on infrastructure that is allowed to disappear.

Why the whole invocation gets replayed

A Strands agent runs a loop: send the conversation to the model, receive either a final answer or a request to call one or more tools, execute those tools, append the results, and go around again. All of that state lives in the memory of a single Python process.

That design is pleasant to develop against, but it makes the entire invocation the unit of failure. When the process dies, the conversation history and the record of tool calls already made go with it. The only safe recovery your platform knows about is “run the whole thing again.”

On AWS, the process dies more often than many teams expect:

• ECS and EKS replace tasks and pods during deployments, node drains, scaling events, and failed health checks.
• Spot capacity gets reclaimed with only a short warning.
• Lambda and other time-limited compute cap how long an invocation can run. A slow model response plus a few retried tool calls can push an agent past the limit.
• Transient errors from downstream services (a CRM rate limit, a ticketing API timeout) often get handled by raising an exception that unwinds the whole loop.

Combined with tools that write to billing, CRM, and ticketing systems, routine infrastructure behavior turns into duplicate credits and duplicate tickets.

Why the usual workarounds fall short

Retrying only the failed tool call. Wrapping individual tools in retry logic handles a flaky HTTP call, but it does nothing when the process itself is gone. The retry state is in the same memory that just vanished.

Saving the conversation to DynamoDB or S3 after each turn. Closer, but you now own a checkpointing protocol, including what to do about a tool call that executed but whose result was never saved. That case is exactly the duplicate credit, and it tends to show up only under load.

Making every tool idempotent and calling it done. Necessary, but on its own it does not save the wasted model calls or tell you where the agent was.

Raising timeouts and avoiding Spot. This lowers the failure rate and raises the bill. Deployments still interrupt in-flight agents.

The common thread is that none of these give the agent loop a durable memory of its own progress that survives the process.

The general approach: durable steps plus idempotent side effects

The pattern that holds up in production has two halves, and neither depends on a particular vendor.

First, make each step of the agent loop a durable unit. Every model call and every tool call should be recorded as a discrete step with its input and output persisted before the loop moves on. This is the core idea behind durable execution: an append-only history plus a runtime that rebuilds state by replaying it after a crash. When a new process picks up the work, completed steps return their recorded results instead of executing again. The agent resumes at the first step that did not finish.

Second, make side-effecting tools idempotent at the edge. Durable execution narrows the duplicate window to the single step that was running when the process died. That step may run again, because the runtime cannot know whether the external call landed. So each side-effecting tool should send an idempotency key derived from stable inputs, for example the session ID plus the tool name plus the invoice ID. If the billing API supports idempotency keys, pass it through. If it does not, keep a small table of keys you have already acted on and check it before writing.

Read-only steps may repeat harmlessly, write tools are protected by keys, and the recorded history answers “which tool calls completed?”

A few design principles follow from this:

• Keep the orchestration logic deterministic and push all I/O into the recorded steps.
• Give every agent run a stable, business-meaningful ID (a dispute ID, a ticket number) so retries attach to the same history instead of starting a new one.

Implementing it with Strands and Dapr Workflow

One practical route is Dapr Workflow, the workflow engine in the open-source Dapr project, a CNCF graduated runtime that runs as a sidecar next to your app. Dapr Workflow keeps an event-sourced history in a state store, replays it after a crash, and does not re-execute completed activities. Orchestrator code must be deterministic, with I/O in activities, which lines up exactly with the principles above.

For example, Diagrid, started by the original creators of Dapr, publishes a Python package named diagrid with a Strands integration. Its DaprWorkflowAgentRunner wraps an existing Strands Agent so that each LLM call and each tool call becomes a separate durable activity. Your agent definition stays the same.

from strands import Agent, tool

from strands.models.openai import OpenAIModel

from diagrid.agent.strands import DaprWorkflowAgentRunner

@tool

def lookup_invoice(invoice_id: str) -> dict:

    “””Fetch an invoice from the billing service.”””

    …

agent = Agent(model=OpenAIModel(model_id=”gpt-4o-mini”), tools=[lookup_invoice], system_prompt=”…”)

runner = DaprWorkflowAgentRunner(agent=agent, name=”billing-agent”)

runner.start()

async for event in runner.run_async(task=”…”, session_id=”s-001″): …

The example uses OpenAIModel, but Strands supports multiple model providers, and the runner wraps the agent, not the model. In practice you would use a stable, business-meaningful session_id such as the dispute ID, and confirm in testing how the runner handles a redelivered message for an existing session.

Install it with pip install “diagrid[strands]”. For local development, run dapr init, configure a state store component with actorStateStore: “true”, and start the app with dapr run –app-id <id> –resources-path ./resources — python3 agent.py. Kill the process mid-run and restart it to watch the agent resume rather than repeat.

In the opening scenario, the invoice lookups, the CRM check, the model’s decision, and the credit call are each recorded. When the ECS task is replaced and a new one picks up the run, those steps return their stored results, and the agent continues at the ticketing step. If the task died during the credit call itself, the activity runs again, and your idempotency key ensures the billing API returns the original credit instead of issuing another.

Where to run it on AWS

On EKS, you install the Dapr control plane into the cluster, and the sidecar injector adds a Dapr sidecar to each annotated pod. The workflow history lives in whichever state store component you configure. Dapr state stores are pluggable (PostgreSQL and DynamoDB are among the options), so confirm that the one you pick works as an actor state store. Use HA mode for the control plane (three replicas) in production, and add Dapr resiliency policies for timeouts and retries on downstream calls.

Outside Kubernetes, Dapr can also run in self-hosted mode next to your process, though on ECS you should expect to do more of the wiring yourself.

If you would rather not operate the runtime, a managed service is an option. Diagrid’s Catalyst is one example of a durable execution platform for AI agents built on Dapr. It can be vendor-hosted, including on a dedicated cluster on AWS, or deployed bring-your-own-cloud in your own AWS account. As with any vendor, verify the recovery behavior, data residency, and operational fit against your own workloads before committing.

A rollout checklist

• ☐  Inventory every tool your agent can call and label it read-only or side-effecting.
• ☐  Add an idempotency key to every side-effecting tool, derived from stable inputs such as session ID plus business object ID.
• ☐  For downstream APIs without idempotency support, record processed keys in your own store and check before writing.
• ☐  Pick a stable, business-meaningful session ID for each agent run and use it on every retry.
• ☐  Wrap the agent so each model call and tool call is a durable, recorded step.
• ☐  Choose a state store for workflow history and confirm its backups.
• ☐  Run a kill test: terminate the container mid-run, restart, and verify that no side effect is repeated.
• ☐  Wire step history into your tracing so on-call engineers can see which tools completed.
• ☐  Add retry policies with backoff for flaky downstream services instead of letting exceptions unwind the loop.

Conclusion

The duplicate goodwill credit is not a model problem or a prompt problem. It is a state problem: the agent’s progress lives in a process that AWS is allowed to replace at any time. Treat each model call and tool call as a durable step, protect every write with an idempotency key, and give every run a stable ID. Then container replacements and Spot reclaims become events your agents resume from, not incidents finance discovers a week later.