add asynchronous job infrastructure (#45)

*Issue #, if available:*

*Description of changes:*

* Implement aynschronous job infrastructure using Lambda
* Add AppSync Events API for realtime notification (e.g. on job
completion)
* Update docs

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.

Author: tmokmss

.github/workflows/build.yml

@@ -1,5 +1,10 @@
 name: Build
-on: [push, workflow_dispatch]
+on: 
+  push:
+    branches:
+    - main
+  workflow_dispatch:
+  pull_request:
 jobs:
   Build-and-Test-CDK:
     runs-on: ubuntu-latest

README.md

@@ -1,39 +1,35 @@
 # Serverless Full Stack WebApp Starter Kit
 [![Build](https://github.com/aws-samples/serverless-full-stack-webapp-starter-kit/actions/workflows/build.yml/badge.svg)](https://github.com/aws-samples/serverless-full-stack-webapp-starter-kit/actions/workflows/build.yml)
 
-[日本語の解説記事はこちら](https://tmokmss.hatenablog.com/entry/20220611/1654931458)。 (Additional article about this kit in Japanese.)
-
 This is a full stack webapp kit for starters who want to leverage the power of AWS serverless services!
 
 Features include:
 
-* Express API endpoint (both with and without authentication)
-* React.js frontend (assets are delivered via CDN)
-* E-mail authentication
+* Next.js App Router on AWS Lambda
+* CloudFront + Lambda function URL with response stream support
+* End to end type safety from client to server
+* Cognito authentication
 * Asynchronous job queue
-* Scheduled job runner
-* Instant deployment of the entire app
-
-**NOTE:** You can also refer to [aws-samples/trpc-nextjs-ssr-prisma-lambda](https://github.com/aws-samples/trpc-nextjs-ssr-prisma-lambda) for end-to-end type safety, server side rendering, or relational database examples.
+* Instant deployment of the entire app with a single command
 
 ## Overview
 Here is the architecture of this kit. We use:
 
-* [Amazon DynamoDB](https://aws.amazon.com/dynamodb/), a serverless scalable NoSQL database
-* [Amazon API Gateway HTTP API](https://aws.amazon.com/api-gateway/) + [AWS Lambda](https://aws.amazon.com/lambda/) to build serverless API endpoint ([`serverless-express`](https://github.com/vendia/serverless-express))
-* [Amazon CloudFront](https://aws.amazon.com/cloudfront/) + [S3](https://aws.amazon.com/s3/) to distribute frontend assets (React.js, Amplify libraries, MUI)
+* [Amazon Aurora PostgreSQL Serverless v2](https://aws.amazon.com/rds/aurora/serverless/), a serverless relational database with Prisma ORM
+* [Next.js App Router](https://nextjs.org/docs/app) on [AWS Lambda](https://aws.amazon.com/lambda/) for a unified frontend and backend solution
+* [Amazon CloudFront](https://aws.amazon.com/cloudfront/) + Lambda Function URL with response streaming support for efficient content delivery
 * [Amazon Cognito](https://aws.amazon.com/cognito/) for authentication. By default, you can sign in/up by email, but you can federate with other OIDC providers such as Google, Facebook, and more with a little modification.
-* [Amazon SQS](https://aws.amazon.com/sqs/) + AWS Lambda for asynchronous job queue. 
-* [Amazon EventBridge](https://aws.amazon.com/eventbridge/) to run scheduled jobs. 
+* [AWS AppSync Events](https://docs.aws.amazon.com/appsync/latest/eventapi/event-api-welcome.html) + AWS Lambda for asynchronous job and realtime notification.
+* [Amazon EventBridge](https://aws.amazon.com/eventbridge/) to run scheduled jobs.
 * [Amazon CloudWatch](https://aws.amazon.com/cloudwatch/) + S3 for access logging.
 * [AWS CDK](https://aws.amazon.com/cdk/) for Infrastructure as Code. It enables you to deploy the entire application with the simplest commands.
 
-![architecture](imgs/architecture.svg)
+![architecture](imgs/architecture.png)
 
 Since it fully leverages AWS serverless services, you can use it with high cost efficiency, scalability, and almost no heavy lifting of managing servers! In terms of cost, we further discuss how much it costs in the below [#Cost](#cost) section.
 
 ### About the sample app
-To show how this kit works, we include a sample web app to write and store your memos.
+To show how this kit works, we include a sample web app to manage your todo list.
 With this sample, you can easily understand how each component works with other ones, and what the overall experience will be like.
 
 <img align="right" width="300" src="./imgs/signin.png">
@@ -51,9 +47,19 @@ You can further improve this sample or remove all the specific code and write yo
 ## Deploy
 You need the following tools to deploy this sample:
 
-* [Node.js](https://nodejs.org/en/download/) (>= v16)
+* [Node.js](https://nodejs.org/en/download/) (>= v20)
 * [Docker](https://docs.docker.com/get-docker/)
 * [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and a configured IAM profile
+* A public domain name configured as a Hosted Zone in Amazon Route53
+
+Before deployment, you need to update the domain name in [`bin/cdk.ts`](cdk/bin/cdk.ts) to use your own domain that is configured as a Hosted Zone in Route53:
+
+```typescript
+const props: EnvironmentProps = {
+  // ...
+  domainName: 'your-domain.example.com', // Replace with your domain name
+};
+```
 
 Then run the following commands:
 
@@ -64,53 +70,45 @@ npx cdk bootstrap
 npx cdk deploy
 ```
 
-Initial deployment usually takes about 10 minutes. You can also use `npx cdk deploy` command to deploy when you modified your CDK templates in the future.
+Initial deployment usually takes about 20 minutes. You can also use `npx cdk deploy` command to deploy when you modified your CDK templates in the future.
 
 After a successful deployment, you will get a CLI output like the below:
 
 ```
- ✅  ServerlessFullstackWebappStarterKitStack
+ ✅  ServerlessWebappStarterKitStack
 
-✨  Deployment time: 235.21s
+✨  Deployment time: 407.12s
 
 Outputs:
-ServerlessFullstackWebappStarterKitStack.AuthUserPoolClientId8216BF9A = xxxxxxxxxxxx
-ServerlessFullstackWebappStarterKitStack.AuthUserPoolIdC0605E59 = ap-northeast-1_xxxxxxx
-ServerlessFullstackWebappStarterKitStack.BackendApiBackendApiUrl4A0A7879 = https://xxxxxx.execute-api.ap-northeast-1.amazonaws.com
-ServerlessFullstackWebappStarterKitStack.FrontendDomainName = https://xxxxxxxxxx.cloudfront.net
-Stack ARN:
-arn:aws:cloudformation:ap-northeast-1:123456789012:stack/ServerlessFullstackWebappStarterKitStack/e47a02d0-e40a-11ec-8ea7-0ed955f95f17
+ServerlessWebappStarterKitStack.AuthUserPoolClientId8216BF9A = xxxxxxxxxxxxxxxxxx
+ServerlessWebappStarterKitStack.AuthUserPoolIdC0605E59 = us-west-2_xxxxxx
+ServerlessWebappStarterKitStack.DatabaseDatabaseSecretsCommandF4A622EB = aws secretsmanager get-secret-value --secret-id DatabaseClusterSecretD1FB63-xxxxxxx --region us-west-2
+ServerlessWebappStarterKitStack.DatabasePortForwardCommandC3718B89 = aws ssm start-session --region us-west-2 --target i-xxxxxxxxxx --document-name AWS-StartPortForwardingSessionToRemoteHost --parameters '{"portNumber":["5432"], "localPortNumber":["5432"], "host": ["foo.cluster-bar.us-west-2.rds.amazonaws.com"]}'
+ServerlessWebappStarterKitStack.FrontendDomainName = https://web.mtomooka.people.aws.dev
 ```
 
 Opening the URL in `FrontendDomainName` output, you can now try the sample app on your browser.
 
 ## Add your own features
-To implement your own features, you may want to add frontend pages, backend API endpoints, or async jobs. The frontend is an ordinary React.js application, so you can follow the conventional ways to add pages to it. As for backend, there are step-by-step guides to add features in [backend/README.md](backend/README.md), so please follow the guide.
+To implement your own features, you may want to add frontend pages, API routes, or async jobs. The project uses Next.js App Router, which provides a unified approach for both frontend and backend development. You can follow the conventional Next.js patterns to add pages and API routes. For more details, please refer to the [`webapp/README.md`](./webapp/README.md) guide.
 
 If you want to add another authentication method such as Google or Facebook federation, you can follow this document: [Add social sign-in to a user pool](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-configuring-federation-with-social-idp.html).
 
 ## Local development
-To develop frontend or backend locally, please refer to each `README.md` in the subdirectories:
+To develop the webapp locally, please refer to the [`webapp/README.md`](./webapp/README.md).
 
-* [Frontend](./frontend/README.md)
-* [Backend](./backend/README.md)
-
-Instead of running the backend API on your local environment, you can use `cdk watch` feature for development by just running the following command:
+Before running webapp locally, you have to populate .env.local by the following steps:
 
 ```sh
-cd cdk
-npx cdk watch
+cp .env.local.example .env.local
+# edit .env.local using the stack outputs shown after cdk deploy 
 ```
 
-`cdk watch` allows you to instantly deploy your backend as soon as you change the code and "tail" logs from your Lambda functions,  enabling rapid iteration cycles.　See [this blog for more details](https://aws.amazon.com/blogs/developer/increasing-development-speed-with-cdk-watch/).
-
-Using `cdk watch`, you will access the deployed AWS resources from your local frontend. You have to configure environment variables in [frontend/.env](./frontend/.env) file to properly access these resources. Please refer to [Frontend README](./frontend/README.md) for more details.
-
 ## Cost
-API Gateway, Lambda, SQS, CloudWatch, CloudFront, and S3 offer free tier plans, which allows you to use those services almost freely for small businesses.
+Lambda, SQS, CloudWatch, CloudFront, and S3 offer free tier plans, which allows you to use those services almost freely for small businesses.
 Up to one million requests per month, most of the costs related to those services are free. See [this page for more details](https://aws.amazon.com/free/).
 
-DynamoDB is billed basically by how many read and write counts processed. See [this page for the current prices](https://aws.amazon.com/dynamodb/pricing/on-demand/). DynamoDB provisioned capacity mode also offers free tier plans, so if you want to pay the minimal cost, you can switch the billing mode (see [database.ts](cdk/lib/constructs/database.ts)).
+Aurora PostgreSQL Serverless v2 is billed based on compute and storage usage. The database automatically scales up and down based on workload, and you only pay for the resources you use. See [this page for the current prices](https://aws.amazon.com/rds/aurora/pricing/). Aurora Serverless v2 can scale down to almost zero during periods of inactivity, helping to minimize costs (see [database.ts](cdk/lib/constructs/database.ts) for configuration details).
 
 Other costs will be derived from data transfer and Elastic Container Repository (used for Docker Lambda). Although it usually does not cost much compared to other services, you may want to continuously monitor the billing metrics. Please refer to [the document to set CloudWatch alarm for AWS charges](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html).
 

…rverless-fullstack-webapp-starter-kit.ts cdk/bin/cdk.tscdk/bin/serverless-fullstack-webapp-starter-kit.ts renamed to cdk/bin/cdk.ts cdk/bin/serverless-fullstack-webapp-starter-kit.ts renamed to cdk/bin/cdk.ts

@@ -1,5 +1,5 @@
 {
-  "app": "npx ts-node --prefer-ts-exts bin/serverless-fullstack-webapp-starter-kit.ts",
+  "app": "npx ts-node --prefer-ts-exts bin/cdk.ts",
   "watch": {
     "include": [
       "**"

cdk/cdk.json

@@ -1,45 +1,44 @@
 import { Construct } from 'constructs';
-import { Duration } from 'aws-cdk-lib';
-import { ITable } from 'aws-cdk-lib/aws-dynamodb';
-import { DockerImageCode, DockerImageFunction } from 'aws-cdk-lib/aws-lambda';
-import { Queue, QueueEncryption } from 'aws-cdk-lib/aws-sqs';
-import { SqsEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';
+import { CfnOutput, Duration } from 'aws-cdk-lib';
+import { Architecture, DockerImageCode, DockerImageFunction, IFunction } from 'aws-cdk-lib/aws-lambda';
 import { Platform } from 'aws-cdk-lib/aws-ecr-assets';
+import { Database } from './database';
+import { EventBus } from './event-bus';
 
 export interface AsyncJobProps {
-  readonly database: ITable;
+  readonly database: Database;
+  readonly eventBus: EventBus;
 }
 
 export class AsyncJob extends Construct {
-  readonly queue: Queue;
+  readonly handler: IFunction;
+
   constructor(scope: Construct, id: string, props: AsyncJobProps) {
     super(scope, id);
-    const { database } = props;
-
-    const visibilityTimeout = Duration.minutes(10);
-
-    const queue = new Queue(this, 'Queue', {
-      visibilityTimeout,
-      encryption: QueueEncryption.KMS_MANAGED,
-    });
+    const { database, eventBus } = props;
 
     const handler = new DockerImageFunction(this, 'Handler', {
-      code: DockerImageCode.fromImageAsset('../backend', {
-        cmd: ['handler-job.handler'],
-        platform: Platform.LINUX_AMD64,
+      code: DockerImageCode.fromImageAsset('../webapp', {
+        cmd: ['async-job-runner.handler'],
+        platform: Platform.LINUX_ARM64,
+        file: 'job.Dockerfile',
       }),
       memorySize: 256,
-      timeout: visibilityTimeout,
+      timeout: Duration.minutes(10),
+      architecture: Architecture.ARM_64,
       environment: {
-        TABLE_NAME: database.tableName,
+        ...database.getLambdaEnvironment('main'),
+        EVENT_HTTP_ENDPOINT: eventBus.httpEndpoint,
       },
-      // limit concurrency to mitigate any possible EDoS attacks 
+      vpc: database.cluster.vpc,
+      // limit concurrency to mitigate any possible EDoS attacks
       reservedConcurrentExecutions: 1,
     });
 
-    database.grantReadWriteData(handler);
-    handler.addEventSource(new SqsEventSource(queue, { maxBatchingWindow: Duration.seconds(5) }));
+    handler.connections.allowToDefaultPort(database);
+    eventBus.api.grantPublish(handler);
 
-    this.queue = queue;
+    new CfnOutput(this, 'HandlerArn', { value: handler.functionArn });
+    this.handler = handler;
   }
 }

cdk/lib/constructs/async-job.ts

@@ -168,6 +168,5 @@ export class CloudFrontLambdaFunctionUrlService extends Construct {
     });
 
     this.url = `https://${domainName}`;
-    new CfnOutput(this, 'CloudFrontUrl', { value: this.url });
   }
 }

cdk/lib/constructs/cf-lambda-furl-service/service.ts

@@ -0,0 +1,18 @@
+import { util } from '@aws-appsync/utils';
+
+/**
+ * Allow subscription only for the channels that
+ * 1. begin with /public
+ * 2. begin with /user/<userId>
+ * https://docs.aws.amazon.com/appsync/latest/eventapi/channel-namespace-handlers.html
+ */
+export function onSubscribe(ctx) {
+  if (ctx.info.channel.path.startsWith(`/event-bus/public`)) {
+    return;
+  }
+  if (ctx.info.channel.path.startsWith(`/event-bus/user/${ctx.identity.username}`)) {
+    return;
+  }
+  console.log(`user ${ctx.identity.username} tried connecting to wrong channel: ${ctx.channel}`);
+  util.unauthorized();
+}

cdk/lib/constructs/event-bus/handler.mjs

@@ -0,0 +1,63 @@
+import { Construct } from 'constructs';
+import * as appsync from 'aws-cdk-lib/aws-appsync';
+import { CfnOutput, CfnResource, Names, Stack } from 'aws-cdk-lib';
+import { IUserPool } from 'aws-cdk-lib/aws-cognito';
+import { join } from 'path';
+
+export interface EventBusProps {}
+
+export class EventBus extends Construct {
+  public readonly httpEndpoint: string;
+  public readonly api: appsync.EventApi;
+
+  private userPoolCount = 0;
+
+  constructor(scope: Construct, id: string, props: EventBusProps) {
+    super(scope, id);
+
+    const api = new appsync.EventApi(this, 'Api', {
+      apiName: Names.uniqueResourceName(this, { maxLength: 30 }),
+      authorizationConfig: {
+        authProviders: [
+          {
+            authorizationType: appsync.AppSyncAuthorizationType.IAM,
+          },
+        ],
+        connectionAuthModeTypes: [appsync.AppSyncAuthorizationType.IAM],
+        defaultPublishAuthModeTypes: [appsync.AppSyncAuthorizationType.IAM],
+        defaultSubscribeAuthModeTypes: [appsync.AppSyncAuthorizationType.IAM],
+      },
+    });
+
+    new appsync.ChannelNamespace(this, 'Namespace', {
+      api,
+      channelNamespaceName: 'event-bus',
+      code: appsync.Code.fromAsset(join(__dirname, 'handler.mjs')),
+    });
+
+    this.httpEndpoint = `https://${api.httpDns}`;
+    this.api = api;
+
+    new CfnOutput(this, 'HttpEndpoint', { value: this.httpEndpoint });
+  }
+
+  public addUserPoolProvider(userPool: IUserPool) {
+    if (this.userPoolCount == 0) {
+      (this.api.node.defaultChild as CfnResource).addPropertyOverride('EventConfig.ConnectionAuthModes.1', {
+        AuthType: 'AMAZON_COGNITO_USER_POOLS',
+      });
+      (this.api.node.defaultChild as CfnResource).addPropertyOverride('EventConfig.DefaultSubscribeAuthModes.1', {
+        AuthType: 'AMAZON_COGNITO_USER_POOLS',
+      });
+    }
+
+    this.userPoolCount += 1;
+    (this.api.node.defaultChild as CfnResource).addPropertyOverride(`EventConfig.AuthProviders.${this.userPoolCount}`, {
+      AuthType: 'AMAZON_COGNITO_USER_POOLS',
+      CognitoConfig: {
+        AwsRegion: Stack.of(this).region,
+        UserPoolId: userPool.userPoolId,
+      },
+    });
+  }
+}