AWS Step Functions Cookbook

01 Nov 2024

This is my AWS Step Functions Mapping and Patterns and cookbook covering data manipulation, concurrency patterns, error handling, and advanced workflows.

Basic Concepts
Path Processing Fundamentals
Concurrency Patterns
Advanced State Patterns
Complex Workflows
Error Handling & Resilience
Integration Patterns
Best Practices & Tips

Basic Concepts

Processing Order Visualization

Input → InputPath → Parameters → Task → ResultSelector → ResultPath → OutputPath → Output

JSONPath Quick Reference

// First book's title
$.store.book[0].title

// All book titles
$.store.book[*].title

// Books under $10
$.store.book[?(@.price < 10)]

// All authors
$..author

// Number of books
$.store.book.length()

Path Processing Fundamentals

AWS Step Functions allows you to manipulate the data passing through your state machine at various stages using InputPath, Parameters, ResultSelector, ResultPath, and OutputPath.

1. InputPath with Complex Filtering

Example: Filter Nested Arrays

Initial State Input:

{
  "orders": [
    { "id": 1, "status": "PENDING", "amount": 100 },
    { "id": 2, "status": "SHIPPED", "amount": 200 },
    { "id": 3, "status": "PENDING", "amount": 150 }
  ],
  "metadata": {
    "requestId": "abc-123"
  }
}

State Definition with InputPath:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessPendingOrders",
  "InputPath": "$.orders[?(@.status == 'PENDING')]",
  "ResultPath": "$.processedOrders",
  "End": true
}

Lambda Receives:

[
  { "id": 1, "status": "PENDING", "amount": 100 },
  { "id": 3, "status": "PENDING", "amount": 150 }
]

If Lambda Returns:

{
  "processedCount": 2,
  "totalAmount": 250
}

Resulting State Output:

{
  "orders": [
    { "id": 1, "status": "PENDING", "amount": 100 },
    { "id": 2, "status": "SHIPPED", "amount": 200 },
    { "id": 3, "status": "PENDING", "amount": 150 }
  ],
  "metadata": {
    "requestId": "abc-123"
  },
  "processedOrders": {
    "processedCount": 2,
    "totalAmount": 250
  }
}

Example: Multiple Field Selection

Initial State Input:

{
  "orderId": "order-456",
  "items": [
    { "productId": "prod-1", "quantity": 2 },
    { "productId": "prod-2", "quantity": 1 }
  ],
  "address": {
    "street": "123 Main St",
    "city": "Anytown",
    "zip": "12345"
  },
  "customerInfo": {
    "name": "John Doe",
    "email": "john@example.com"
  }
}

State Definition with Parameters:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessOrderDetails",
  "Parameters": {
    "orderDetails": {
      "id.$": "$.orderId",
      "items.$": "$.items[*].productId",
      "shipping.$": "$.address"
    },
    "customerEmail.$": "$.customerInfo.email"
  },
  "ResultPath": "$.orderProcessingResult",
  "End": true
}

Lambda Receives:

{
  "orderDetails": {
    "id": "order-456",
    "items": ["prod-1", "prod-2"],
    "shipping": {
      "street": "123 Main St",
      "city": "Anytown",
      "zip": "12345"
    }
  },
  "customerEmail": "john@example.com"
}

If Lambda Returns:

{
  "confirmationNumber": "CONF-789",
  "estimatedDelivery": "2023-12-01"
}

Resulting State Output:

{
  "orderId": "order-456",
  "items": [
    { "productId": "prod-1", "quantity": 2 },
    { "productId": "prod-2", "quantity": 1 }
  ],
  "address": {
    "street": "123 Main St",
    "city": "Anytown",
    "zip": "12345"
  },
  "customerInfo": {
    "name": "John Doe",
    "email": "john@example.com"
  },
  "orderProcessingResult": {
    "confirmationNumber": "CONF-789",
    "estimatedDelivery": "2023-12-01"
  }
}

2. Advanced Parameters Usage

Example: Dynamic Array Processing and Intrinsic Functions

Initial State Input:

{
  "item": "product-123",
  "items": ["product-123", "product-456"],
  "stockStatus": "true",
  "date": "2023-11-15",
  "time": "14:30:00",
  "metadata": {
    "sessionId": "sess-001"
  }
}

State Definition with Parameters:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessData",
  "Parameters": {
    "itemsArray.$": "States.Array($.item)",
    "itemsCount.$": "States.ArrayLength($.items)",
    "hasStock.$": "States.StringToBoolean($.stockStatus)",
    "processTime": {
      "startTime.$": "$$.State.EnteredTime",
      "formatted.$": "States.Format('{}T{}Z', $.date, $.time)"
    },
    "metadata": {
      "executionStartTime.$": "$$.Execution.StartTime",
      "retryCount.$": "$$.State.RetryCount",
      "sessionId.$": "$.metadata.sessionId"
    }
  },
  "ResultPath": "$.processingResult",
  "End": true
}

Lambda Receives:

{
  "itemsArray": ["product-123"],
  "itemsCount": 2,
  "hasStock": true,
  "processTime": {
    "startTime": "2023-11-15T14:30:00Z",
    "formatted": "2023-11-15T14:30:00Z"
  },
  "metadata": {
    "executionStartTime": "2023-11-15T14:29:50Z",
    "retryCount": 0,
    "sessionId": "sess-001"
  }
}

If Lambda Returns:

{
  "status": "Processed",
  "processedAt": "2023-11-15T14:30:10Z"
}

Resulting State Output:

{
  "item": "product-123",
  "items": ["product-123", "product-456"],
  "stockStatus": "true",
  "date": "2023-11-15",
  "time": "14:30:00",
  "metadata": {
    "sessionId": "sess-001"
  },
  "processingResult": {
    "status": "Processed",
    "processedAt": "2023-11-15T14:30:10Z"
  }
}

3. Replacing or Overwriting Data in State Output

Example 1: Overwriting State Data Using ResultPath

Initial State Input:

{
  "orderId": "order-456",
  "status": "NEW",
  "customerInfo": {
    "name": "Jane Doe",
    "email": "jane@example.com"
  }
}

State Definition with ResultPath Overwrite:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessOrderStatus",
  "Parameters": {
    "orderDetails": {
      "id.$": "$.orderId",
      "status": "PROCESSED"
    }
  },
  "ResultPath": "$",
  "End": true
}

Lambda Receives:

{
  "orderDetails": {
    "id": "order-456",
    "status": "PROCESSED"
  }
}

If Lambda Returns:

{
  "status": "SUCCESS",
  "processedAt": "2023-12-01T10:00:00Z"
}

Resulting State Output:

{
  "status": "SUCCESS",
  "processedAt": "2023-12-01T10:00:00Z"
}

Using ResultPath with the "$" path overwrites the entire state input, replacing it with the result from the Lambda function.

Example 2: Partially Replacing Data with ResultPath

Initial State Input:

{
  "orderId": "order-123",
  "items": [
    { "productId": "prod-1", "quantity": 1 },
    { "productId": "prod-2", "quantity": 3 }
  ],
  "customer": {
    "name": "Alice",
    "email": "alice@example.com"
  },
  "metadata": {
    "requestId": "req-789"
  }
}

State Definition with Partial Overwrite Using ResultPath:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:UpdateOrder",
  "Parameters": {
    "orderId.$": "$.orderId",
    "items.$": "$.items[*].productId"
  },
  "ResultPath": "$.updatedOrder",
  "End": true
}

Lambda Receives:

{
  "orderId": "order-123",
  "items": ["prod-1", "prod-2"]
}

If Lambda Returns:

{
  "status": "UPDATED",
  "timestamp": "2023-12-01T12:30:00Z"
}

Resulting State Output:

{
  "orderId": "order-123",
  "items": [
    { "productId": "prod-1", "quantity": 1 },
    { "productId": "prod-2", "quantity": 3 }
  ],
  "customer": {
    "name": "Alice",
    "email": "alice@example.com"
  },
  "metadata": {
    "requestId": "req-789"
  },
  "updatedOrder": {
    "status": "UPDATED",
    "timestamp": "2023-12-01T12:30:00Z"
  }
}

In this example, the ResultPath directs the result to the updatedOrder field in the existing state, allowing a partial overwrite.

Concurrency Patterns

1. Parallel State Processing

Example: Basic Parallel Processing

Initial State Input:

{
  "orderId": "order-789",
  "customerId": "cust-123",
  "items": ["item-1", "item-2"]
}

State Definition with Parallel:

{
  "Type": "Parallel",
  "Branches": [
    {
      "StartAt": "ProcessOrders",
      "States": {
        "ProcessOrders": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:region:account-id:function:ProcessOrders",
          "End": true
        }
      }
    },
    {
      "StartAt": "UpdateInventory",
      "States": {
        "UpdateInventory": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:region:account-id:function:UpdateInventory",
          "End": true
        }
      }
    }
  ],
  "ResultPath": "$.parallelResults",
  "End": true
}

Tasks Receive:

ProcessOrders Lambda Receives:

{
  "orderId": "order-789",
  "customerId": "cust-123",
  "items": ["item-1", "item-2"]
}

UpdateInventory Lambda Receives:

{
  "orderId": "order-789",
  "customerId": "cust-123",
  "items": ["item-1", "item-2"]
}

If Lambdas Return:

ProcessOrders Returns:
```
{
  "orderStatus": "Processed"
}
```
UpdateInventory Returns:
```
{
  "inventoryStatus": "Updated"
}
```

Resulting State Output:

{
  "orderId": "order-789",
  "customerId": "cust-123",
  "items": ["item-1", "item-2"],
  "parallelResults": [
    {
      "orderStatus": "Processed"
    },
    {
      "inventoryStatus": "Updated"
    }
  ]
}

Example: Parallel with Shared Context

Initial State Input:

{
  "userId": "user-456",
  "sessionData": {
    "sessionId": "sess-789",
    "timestamp": "2023-11-15T15:00:00Z"
  },
  "preferences": {
    "theme": "dark",
    "notifications": true
  }
}

State Definition with Shared Parameters:

{
  "Type": "Parallel",
  "Parameters": {
    "context.$": "$.sessionData",
    "userId.$": "$.userId"
  },
  "Branches": [
    {
      "StartAt": "Branch1",
      "States": {
        "Branch1": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:region:account-id:function:BranchFunction1",
          "End": true
        }
      }
    },
    {
      "StartAt": "Branch2",
      "States": {
        "Branch2": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:region:account-id:function:BranchFunction2",
          "End": true
        }
      }
    }
  ],
  "ResultSelector": {
    "branch1Result.$": "$[0]",
    "branch2Result.$": "$[1]"
  },
  "ResultPath": "$.branchResults",
  "End": true
}

Tasks Receive:

Both Lambdas Receive:

{
  "context": {
    "sessionId": "sess-789",
    "timestamp": "2023-11-15T15:00:00Z"
  },
  "userId": "user-456"
}

If Lambdas Return:

BranchFunction1 Returns:

{
  "activityLog": "Activity recorded"
}

BranchFunction2 Returns:

{
  "preferenceUpdate": "Preferences saved"
}

Resulting State Output:

{
  "userId": "user-456",
  "sessionData": {
    "sessionId": "sess-789",
    "timestamp": "2023-11-15T15:00:00Z"
  },
  "preferences": {
    "theme": "dark",
    "notifications": true
  },
  "branchResults": {
    "branch1Result": {
      "activityLog": "Activity recorded"
    },
    "branch2Result": {
      "preferenceUpdate": "Preferences saved"
    }
  }
}

2. Map State with Concurrency Control

Example: Basic Map with Concurrency

Initial State Input:

{
  "orders": [
    { "orderId": "order-1", "amount": 100 },
    { "orderId": "order-2", "amount": 150 },
    { "orderId": "order-3", "amount": 200 },
    { "orderId": "order-4", "amount": 250 },
    { "orderId": "order-5", "amount": 300 }
  ]
}

State Definition with Map:

{
  "Type": "Map",
  "ItemsPath": "$.orders",
  "MaxConcurrency": 2,
  "Iterator": {
    "StartAt": "ProcessSingleOrder",
    "States": {
      "ProcessSingleOrder": {
        "Type": "Task",
        "Resource": "arn:aws:lambda:region:account-id:function:ProcessSingleOrder",
        "End": true
      }
    }
  },
  "ResultPath": "$.processedOrders",
  "End": true
}

Lambda Receives (per iteration):

First Iteration:
```
{ "orderId": "order-1", "amount": 100 }
```
Second Iteration:
```
{ "orderId": "order-2", "amount": 150 }
```

(Up to MaxConcurrency of 2 at a time.)

If Lambdas Return:

{
  "orderId": "order-X",
  "status": "Processed"
}

Resulting State Output:

{
  "orders": [
    { "orderId": "order-1", "amount": 100 },
    { "orderId": "order-2", "amount": 150 },
    { "orderId": "order-3", "amount": 200 },
    { "orderId": "order-4", "amount": 250 },
    { "orderId": "order-5", "amount": 300 }
  ],
  "processedOrders": [
    { "orderId": "order-1", "status": "Processed" },
    { "orderId": "order-2", "status": "Processed" },
    { "orderId": "order-3", "status": "Processed" },
    { "orderId": "order-4", "status": "Processed" },
    { "orderId": "order-5", "status": "Processed" }
  ]
}

Example: Map with `ItemSelector`

Initial State Input:

{
  "array": [1, 4, 9, 16, 25]
}

State Definition with Map and ItemSelector:

{
  "Type": "Map",
  "ItemsPath": "$.array",
  "ItemSelector": {
    "index.$": "$$.Map.Item.Index",
    "value.$": "$$.Map.Item.Value",
    "sqrtValue.$": "States.MathSqrt($$.Map.Item.Value)"
  },
  "Iterator": {
    "StartAt": "ProcessItem",
    "States": {
      "ProcessItem": {
        "Type": "Pass",
        "ResultPath": "$",
        "End": true
      }
    }
  },
  "ResultPath": "$.results",
  "End": true
}

Data Processed in Each Iteration:

First Iteration:

{
  "index": 0,
  "value": 1,
  "sqrtValue": 1.0
}

Second Iteration:

{
  "index": 1,
  "value": 4,
  "sqrtValue": 2.0
}

Resulting State Output:

{
  "array": [1, 4, 9, 16, 25],
  "results": [
    { "index": 0, "value": 1, "sqrtValue": 1.0 },
    { "index": 1, "value": 4, "sqrtValue": 2.0 },
    { "index": 2, "value": 9, "sqrtValue": 3.0 },
    { "index": 3, "value": 16, "sqrtValue": 4.0 },
    { "index": 4, "value": 25, "sqrtValue": 5.0 }
  ]
}

Advanced State Patterns

1. Dynamic Branching with Choice State

Initial State Input:

{
  "order": {
    "total": 1200,
    "priority": "HIGH",
    "items": [
      { "productId": "prod-1", "category": "STANDARD" },
      { "productId": "prod-2", "category": "FRAGILE" }
    ],
    "shipping": {
      "express": true
    }
  }
}

State Definition with Choice:

{
  "Type": "Choice",
  "Choices": [
    {
      "And": [
        {
          "Variable": "$.order.total",
          "NumericGreaterThan": 1000
        },
        {
          "Variable": "$.order.priority",
          "StringEquals": "HIGH"
        }
      ],
      "Next": "PremiumProcessing"
    },
    {
      "Or": [
        {
          "Variable": "$.order.items[?(@.category == 'FRAGILE')]",
          "IsPresent": true
        },
        {
          "Variable": "$.order.shipping.express",
          "BooleanEquals": true
        }
      ],
      "Next": "SpecialHandling"
    }
  ],
  "Default": "StandardProcessing"
}

Execution Path:

Since $.order.total > 1000 and $.order.priority == "HIGH", the execution moves to PremiumProcessing.

2. Wait States with Dynamic Duration

Initial State Input:

{
  "task": "WaitForApproval",
  "delay": 30 // Wait for 30 seconds
}

State Definition with Wait:

{
  "Type": "Wait",
  "SecondsPath": "$.delay",
  "Next": "CheckApprovalStatus"
}

Execution Flow:

The state machine waits for 30 seconds (as specified in $.delay) before proceeding to CheckApprovalStatus.

Complex Workflows

1. Orchestration Pattern

Initial State Input:

{
  "orderId": "order-999",
  "customerId": "cust-789",
  "items": ["item-5", "item-6"]
}

State Machine Definition:

{
  "StartAt": "InitializeOrder",
  "States": {
    "InitializeOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:InitializeOrder",
      "Next": "ParallelProcessing"
    },
    "ParallelProcessing": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "ValidateInventory",
          "States": {
            "ValidateInventory": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:region:account-id:function:ValidateInventory",
              "End": true
            }
          }
        },
        {
          "StartAt": "ProcessPayment",
          "States": {
            "ProcessPayment": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:region:account-id:function:ProcessPayment",
              "End": true
            }
          }
        }
      ],
      "ResultPath": "$.processingResults",
      "Next": "EvaluateResults"
    },
    "EvaluateResults": {
      "Type": "Choice",
      "Choices": [
        {
          "And": [
            {
              "Variable": "$.processingResults[0].inventoryAvailable",
              "BooleanEquals": true
            },
            {
              "Variable": "$.processingResults[1].paymentSuccessful",
              "BooleanEquals": true
            }
          ],
          "Next": "FulfillOrder"
        }
      ],
      "Default": "HandleFailure"
    },
    "FulfillOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:FulfillOrder",
      "End": true
    },
    "HandleFailure": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:HandleFailure",
      "End": true
    }
  }
}

Execution Flow:

InitializeOrder Lambda receives the initial input and performs setup tasks.
ParallelProcessing runs ValidateInventory and ProcessPayment in parallel.
EvaluateResults checks if both inventory is available and payment was successful.
If both are true, it proceeds to FulfillOrder; otherwise, it moves to HandleFailure.

Resulting State Output:

Depends on the execution path taken. If successful, the order is fulfilled; if not, failure handling is executed.

2. Saga Pattern with Compensation

Initial State Input:

{
  "reservationDetails": {
    "flightId": "FL-123",
    "hotelId": "HT-456",
    "carRentalId": "CR-789"
  }
}

State Machine Definition:

{
  "StartAt": "ReserveFlight",
  "States": {
    "ReserveFlight": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:ReserveFlight",
      "Next": "ReserveHotel",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.errorInfo",
          "Next": "CleanupState"
        }
      ]
    },
    "ReserveHotel": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:ReserveHotel",
      "Next": "ReserveCar",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.errorInfo",
          "Next": "CompensateFlight"
        }
      ]
    },
    "ReserveCar": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:ReserveCar",
      "Next": "CompleteBooking",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.errorInfo",
          "Next": "CompensateHotel"
        }
      ]
    },
    "CompensateHotel": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:CancelHotel",
      "Next": "CompensateFlight"
    },
    "CompensateFlight": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:CancelFlight",
      "Next": "CleanupState"
    },
    "CompleteBooking": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:CompleteBooking",
      "End": true
    },
    "CleanupState": {
      "Type": "Pass",
      "End": true
    }
  }
}

Execution Flow:

If any reservation step fails, the state machine invokes compensation steps to undo previous actions.

Resulting State Output:

Depends on the success or failure of each reservation step and the execution of compensation actions.

Error Handling & Resilience

1. Comprehensive Error Handling

Example: Retry Configuration with Catch

State Definition with Retry and Catch:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessData",
  "Retry": [
    {
      "ErrorEquals": ["States.Timeout"],
      "IntervalSeconds": 2,
      "MaxAttempts": 3,
      "BackoffRate": 2.0
    },
    {
      "ErrorEquals": ["CustomError"],
      "IntervalSeconds": 1,
      "MaxAttempts": 2
    }
  ],
  "Catch": [
    {
      "ErrorEquals": ["States.ALL"],
      "ResultPath": "$.errorInfo",
      "Next": "ErrorHandler"
    }
  ],
  "End": true
}

Execution Flow:

The task retries on States.Timeout and CustomError according to specified settings.
If all retries fail or another error occurs, it catches the error and transitions to ErrorHandler.

2. Circuit Breaker Implementation

State Definition with Circuit Breaker Logic:

{
  "Type": "Choice",
  "Choices": [
    {
      "Variable": "$.serviceHealth.errorCount",
      "NumericGreaterThan": 5,
      "Next": "CircuitOpen"
    }
  ],
  "Default": "ProcessNormally"
}

Execution Flow:

If $.serviceHealth.errorCount exceeds 5, the workflow transitions to CircuitOpen, preventing further processing.
Otherwise, it proceeds to ProcessNormally.

Integration Patterns

1. Service Integration Pattern

Example: SQS `SendMessage` Integration

Initial State Input:

{
  "orderId": "order-1010",
  "message": "Order placed successfully."
}

State Definition with SQS Integration:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::sqs:sendMessage",
  "Parameters": {
    "QueueUrl": "https://sqs.region.amazonaws.com/account-id/queue-name",
    "MessageBody.$": "$.message",
    "MessageAttributes": {
      "OrderId": {
        "DataType": "String",
        "StringValue.$": "$.orderId"
      }
    }
  },
  "ResultSelector": {
    "MessageId.$": "$.MessageId",
    "Timestamp.$": "$$.State.EnteredTime"
  },
  "ResultPath": "$.sqsResult",
  "End": true
}

Resulting State Output:

{
  "orderId": "order-1010",
  "message": "Order placed successfully.",
  "sqsResult": {
    "MessageId": "abcd1234-5678-90ef-ghij-klmnopqrstuv",
    "Timestamp": "2023-11-15T16:00:00Z"
  }
}

2. API Gateway Integration

Initial State Input:

{
  "orderData": {
    "orderId": "order-2020",
    "items": ["item-A", "item-B"]
  }
}

State Definition with API Gateway Integration:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::apigateway:invoke",
  "Parameters": {
    "ApiEndpoint": "https://myapi.execute-api.region.amazonaws.com",
    "Method": "POST",
    "Headers": {
      "Content-Type": "application/json"
    },
    "Path": "/orders",
    "RequestBody.$": "$.orderData"
  },
  "ResultSelector": {
    "StatusCode.$": "$.StatusCode",
    "ResponseBody.$": "$.ResponseBody"
  },
  "ResultPath": "$.apiResponse",
  "End": true
}

Resulting State Output:

{
  "orderData": {
    "orderId": "order-2020",
    "items": ["item-A", "item-B"]
  },
  "apiResponse": {
    "StatusCode": 200,
    "ResponseBody": {
      "message": "Order received",
      "orderId": "order-2020"
    }
  }
}

Best Practices & Tips

1. Use Intrinsic Functions Effectively

Example: Utilizing Intrinsic Functions in Parameters

Initial State Input:

{
  "date": "2023-11-15",
  "time": "17:00:00",
  "item": "item-XYZ",
  "items": ["item-XYZ", "item-ABC"],
  "status": "true"
}

State Definition with Intrinsic Functions:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessIntrinsic",
  "Parameters": {
    "timestamp.$": "States.Format('{}T{}Z', $.date, $.time)",
    "itemsArray.$": "States.Array($.item)",
    "itemsCount.$": "States.ArrayLength($.items)",
    "isValid.$": "States.StringToBoolean($.status)"
  },
  "ResultPath": "$.intrinsicResult",
  "End": true
}

Lambda Receives:

{
  "timestamp": "2023-11-15T17:00:00Z",
  "itemsArray": ["item-XYZ"],
  "itemsCount": 2,
  "isValid": true
}

2. Implement Idempotency for Reliability

State Definition with Idempotency Key:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessWithIdempotency",
  "Parameters": {
    "idempotencyKey.$": "States.Format('{}-{}', $.orderId, $$.State.Name)",
    "payload.$": "$"
  },
  "ResultPath": "$.idempotentResult",
  "End": true
}

Ensures that each execution can be uniquely identified and avoids duplicate processing.

3. Monitor Execution Progress with CloudWatch

State Definition with CloudWatch Integration:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::cloudwatch:putMetricData",
  "Parameters": {
    "Namespace": "StepFunctions/Processing",
    "MetricData": [
      {
        "MetricName": "ProcessingTime",
        "Value.$": "$.duration",
        "Unit": "Milliseconds",
        "Dimensions": [
          {
            "Name": "ExecutionId",
            "Value.$": "$$.Execution.Id"
          }
        ]
      }
    ]
  },
  "ResultPath": "$.cloudWatchResult",
  "End": true
}

Resulting State Output:

Includes metrics data sent to CloudWatch for monitoring purposes.

4. Replace Data Effectively with OutputPath

Using OutputPath to selectively replace or retain data in the output state can help keep workflow data clean and relevant.

Example 3: Replacing Data with OutputPath

Initial State Input:

{
  "userId": "user-789",
  "preferences": {
    "notifications": true,
    "theme": "light"
  },
  "settings": {
    "email": "user@example.com",
    "subscription": "basic"
  }
}

State Definition with OutputPath Replacement:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:UpdatePreferences",
  "Parameters": {
    "userId.$": "$.userId",
    "preferences.$": "$.preferences"
  },
  "OutputPath": "$.preferences",
  "End": true
}

Lambda Receives:

{
  "userId": "user-789",
  "preferences": {
    "notifications": true,
    "theme": "light"
  }
}

If Lambda Returns:

{
  "notifications": false,
  "theme": "dark"
}

Resulting State Output:

{
  "notifications": false,
  "theme": "dark"
}

By setting the OutputPath to $.preferences, the final state output retains only the updated preferences data, replacing the initial input with the result of this task.

AWS Step Functions Cookbook

Table of Contents

Basic Concepts

Processing Order Visualization

JSONPath Quick Reference

Path Processing Fundamentals

1. InputPath with Complex Filtering

Example: Filter Nested Arrays

Example: Multiple Field Selection

2. Advanced Parameters Usage

Example: Dynamic Array Processing and Intrinsic Functions

3. Replacing or Overwriting Data in State Output

Example 1: Overwriting State Data Using ResultPath

Example 2: Partially Replacing Data with ResultPath

Concurrency Patterns

1. Parallel State Processing

Example: Basic Parallel Processing

Example: Parallel with Shared Context

2. Map State with Concurrency Control

Example: Basic Map with Concurrency

Example: Map with ItemSelector

Advanced State Patterns

1. Dynamic Branching with Choice State

2. Wait States with Dynamic Duration

Complex Workflows

1. Orchestration Pattern

2. Saga Pattern with Compensation

Error Handling & Resilience

1. Comprehensive Error Handling

Example: Retry Configuration with Catch

2. Circuit Breaker Implementation

Integration Patterns

1. Service Integration Pattern

Example: SQS SendMessage Integration

2. API Gateway Integration

Best Practices & Tips

1. Use Intrinsic Functions Effectively

2. Implement Idempotency for Reliability

3. Monitor Execution Progress with CloudWatch

4. Replace Data Effectively with OutputPath

Example 3: Replacing Data with OutputPath

Example: Map with `ItemSelector`

Example: SQS `SendMessage` Integration