Feature: Implement Enterprise Integration Pattern - Splitter Skill

[giuseppe-trisciuoglio]

Feature Request: Implement Enterprise Integration Pattern - Splitter Skill

This issue proposes the creation of a new skill to implement the Enterprise Integration Patterns – Splitter using TypeScript, NestJS, and AWS Lambda, along with other relevant AWS services like SQS, DynamoDB/Aurora, and API Gateway.

Introduction to the Splitter Pattern

The Splitter pattern is a fundamental strategy in distributed systems architecture, designed to break down a single composite message into multiple independent messages. This decomposition allows for the parallel processing of individual components by different consumers, thereby enhancing system scalability, resilience, and efficiency.

In the context of an order management system, an initial message containing a list of orders can be split into individual messages, each representing a specific order. These individual orders can then be processed independently and concurrently, for example, by different operators or microservices, without mutual dependencies.

Proposed Architecture with AWS

To implement the Splitter pattern on AWS, a combination of serverless and managed services can be leveraged, ensuring scalability, high availability, and optimized costs. The proposed architecture includes:

• Amazon API Gateway: Used as an HTTP/S endpoint to receive initial requests containing lists of orders. It can also integrate directly with AWS Lambda.
• AWS Lambda: Serverless functions that execute business logic code. They will be used to implement the Splitter logic and worker functions for order processing.
• Amazon SQS (Simple Queue Service): A fully managed message queuing service that facilitates decoupling and scalability of microservices. It will be used to distribute individual orders after the split.
• Amazon DynamoDB or Amazon Aurora: NoSQL (DynamoDB) or relational (Aurora) databases to persist the state of orders and other relevant information.

Detailed Logical Flow

The processing flow follows a series of well-defined steps, ensuring that each order is handled efficiently and robustly.

1. Request Reception: A client system sends an HTTP/S request containing a list of orders to an endpoint exposed via Amazon API Gateway. API Gateway forwards the request to an AWS Lambda function designated as OrderListProcessor.
2. Splitter Implementation (AWS Lambda with NestJS): The OrderListProcessor function (implemented in TypeScript with NestJS) acts as the Splitter. It reads the message, separates each order, and publishes each individual order as a new message to a dedicated Amazon SQS queue (e.g., OrderProcessingQueue).
3. Work Distribution (Amazon SQS): Each individual order message is placed into the Amazon SQS OrderProcessingQueue, acting as a buffer and distribution mechanism for asynchronous, scalable, and parallel processing.
4. Parallel Processing (AWS Lambda Worker with NestJS): One or more AWS Lambda functions, configured as consumers of the OrderProcessingQueue, act as workers. Each worker retrieves a single order message from the queue and processes it independently.
5. Order Status Update: Once a worker completes processing an order, it updates its status in Amazon DynamoDB or Amazon Aurora.

Advantages of this Architecture

• Work Parallelization: Simultaneous processing of orders, reducing total execution time.
• Automatic Scalability: AWS Lambda and Amazon SQS scale automatically.
• Component Decoupling: Sender and processors are fully decoupled.
• Resilience: Messages can be reprocessed in case of worker failure.
• Flexibility with NestJS: Clean architecture, dependency injection, and testability.

Pattern Extensions: Aggregator

The architecture can be extended with the Aggregator Pattern to collect and combine results of individual order processing into a single final message or state.

Technical Considerations (TypeScript/NestJS)

• NestJS Project Structure for AWS Lambda: Use Webpack for tree-shaking and bundling to minimize cold starts.
• Handling SQS Events in NestJS: Deserialization of SQS message bodies and proper error handling (e.g., Dead-Letter Queues).
• Dependency Injection: Leverage NestJS's DI system for AWS SDK clients and custom services.
• Message Serialization/Deserialization: Define clear message contracts (TypeScript interfaces) for data exchanged via SQS.