Skip to content

Commit ca0d07e

Browse files
glitchedmobglitchedmob
committed
feat: update documentation and allow running locally
1 parent be3abbd commit ca0d07e

20 files changed

Lines changed: 421 additions & 96 deletions

File tree

‎.env.example‎

Lines changed: 3 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,7 @@
11
APP_ENV=
2+
APP_DOMAIN_NAME=localhost
23
DYNAMODB_PORT=8000
34
DYNAMODB_ENDPOINT="http://localhost:${DYNAMODB_PORT}"
45
DYNAMODB_REGION=us-east-2
5-
AWS_ACCESS_KEY=test
6-
AWS_SECRET_ACCESS_KEY=test
7-
MEETUP_TOKEN_FUNCTION_NAME=
8-
EVENTS_TABLE_NAME=
9-
IMPORTER_LOG_TABLE_NAME=
10-
MEETUP_GROUP_NAMES=
11-
# Only Used for meetupproxy lambda
12-
MEETUP_PRIVATE_KEY=
13-
MEETUP_USER_ID=
14-
MEETUP_CLIENT_KEY=
15-
MEETUP_SIGNING_KEY_ID=
16-
MEETUP_AUTH_URL=
6+
DYNAMODB_AWS_ACCESS_KEY=test
7+
DYNAMODB_AWS_SECRET_ACCESS_KEY=test

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,5 @@ cdk.out
2525

2626

2727
.env
28+
.lambda-env.json
29+

‎.lambda-env.json.example‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
{
2+
"Parameters": {
3+
"DYNAMODB_ENDPOINT": "http://dynamodb-local:8000",
4+
"DYNAMODB_REGION": "us-east-2",
5+
"DYNAMODB_AWS_ACCESS_KEY": "test",
6+
"DYNAMODB_AWS_SECRET_ACCESS_KEY": "test",
7+
8+
"MEETUP_GROUP_NAMES": "open-sgf,sgfdevs",
9+
"MEETUP_PROXY_FUNCTION_NAME": "Staging-MeetupProxy",
10+
"EVENTS_TABLE_NAME": "MeetupEvents",
11+
"GROUP_ID_DATE_TIME_INDEX_NAME": "GroupIdDateTimeIndex",
12+
"ARCHIVED_EVENTS_TABLE_NAME": "MeetupArchivedEvents",
13+
"API_USERS_TABLE_NAME": "MeetupApiUsers",
14+
"APP_URL": "http://localhost:3000",
15+
"JWT_ISSUER": "localhost:3000",
16+
"JWT_SECRET": "some-super-secret-value",
17+
18+
"MEETUP_PRIVATE_KEY": "",
19+
"MEETUP_USER_ID": "",
20+
"MEETUP_CLIENT_KEY": "",
21+
"MEETUP_SIGNING_KEY_ID": "",
22+
"MEETUP_AUTH_URL": ""
23+
}
24+
}

‎README.md‎

Lines changed: 46 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -4,38 +4,51 @@ The SGF Meetup API lists Meetup event details for local tech groups in the Sprin
44

55
## Table of Contents
66
- [How to Use This API](#how-to-use-this-api)
7-
- [Requesting an API Key](#requesting-an-api-key)
8-
- [Authentication](#authentication)
9-
- [For Contributors](#for-contributors)
10-
- [Prerequisites](#prerequisites)
7+
- [URLs](#urls)
8+
- [Documentation](#documentation)
9+
- [Requesting Credentials](#requesting-credentials)
10+
- [Architecture](#architecture)
11+
- [Contributing](#contributing)
1112
- [First Time Setup](#first-time-setup)
1213
- [Running the Project](#running-the-project)
1314
- [Shutting Down](#shutting-down)
1415
- [Troubleshooting](#troubleshooting)
16+
- [Project Structure](#project-structure)
1517

1618
## How to Use This API
1719

18-
### Requesting an API Key
20+
### URLs
1921

20-
Request an API key by opening a GitHub Issue in the [SGF Meetup API repo](https://github.com/Open-SGF/sgf-meetup-api/issues/).
22+
- Production: https://sgf-meetup-api.opensgf.org
23+
- Staging: https://staging-sgf-meetup-api.opensgf.org
24+
25+
### Documentation
26+
27+
- Swagger playground: [/swagger/index.html](https://staging-sgf-meetup-api.opensgf.org/swagger/index.html)
28+
- OpenAPI document: [/swagger/doc.json](https://staging-sgf-meetup-api.opensgf.org/swagger/doc.json)
29+
30+
### Requesting Credentials
31+
32+
Request API credentials by opening a GitHub Issue in the [SGF Meetup API repo](https://github.com/Open-SGF/sgf-meetup-api/issues/).
2133

2234
In the GitHub Issue, submit a username for the API and contact information. We will assign a password to the username and send it to the contact information listed.
2335

24-
### Authentication
36+
## Architecture
2537

38+
See [docs/architecture.md](./docs/architecture.md)
2639

40+
## Contributing
2741

28-
## For Contributors
42+
### First Time Setup
2943

30-
### Prerequisites
31-
- [Node 18.x](https://nodejs.org) (Ideally using [nvm](https://github.com/nvm-sh/nvm))
44+
#### Required Tools
45+
- [Go 1.24](https://go.dev/dl/)
46+
- [Node 22.x](https://nodejs.org) (Ideally using [nvm](https://github.com/nvm-sh/nvm))
3247
- [Docker Desktop](https://www.docker.com/products/docker-desktop/)
3348
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
3449
- [AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html)
3550
- An Open SGF AWS Account
36-
- Message a project organizer to get one set up
37-
38-
### First Time Setup
51+
- Message a project organizer to get one set up
3952

4053
#### Sign in to the AWS CLI
4154
- `aws configure sso`
@@ -62,35 +75,43 @@ Including the commands in npm scripts.
6275
```bash
6376
nvm install # if using nvm
6477
npm install
78+
go mod download
6579
```
6680

67-
#### Create `.env`
81+
#### Create `.env` and `.lambda-env.json`
82+
83+
`.lambda-env.json` is used when running the application with the SAM cli.
84+
85+
`.env` is used for other instances where programs run directly on a developers machine.
86+
6887
```bash
69-
cp lambdas/.env.example lambdas/.env
88+
cp .env.example .env
89+
cp .lambda-env.json.example .lambda-env.json
7090
```
7191

7292
#### Populate additional env variables
7393
- `MEETUP_GROUP_NAMES` should be a comma seperated list of meetup group names to import events from
7494
- This value can be pulled from the url of a Meetup groups page e.g. with meetup.com/sgfdevs, sgfdevs is the group name
75-
- `API_KEYS` should be a comma seperated list of strings you'll use to locally call the API
7695

77-
#### Setup initial database
96+
#### Database/User Setup
7897
- `docker compose up -d`
79-
- If you are on linux or WSL there will likely be permission issues with a folder used by docker
80-
- You can fix those with `sudo chown -R 1000:1000 docker/dynamodb`
81-
- `npm run dev:sync-dynamodb`
98+
- `go run ./cmd/syncdynamodb`
99+
- `go run ./cmd/upsertuser -clientId <ID> -clientSecret <SECRET>`
82100

83101
### Running the project
84102
- `docker compose up -d` (if not already running)
85-
- `nvm use` (if using nvm)
86103
- Run importer script
87-
- `npm run dev:importer`
104+
- `go run ./cmd/localsamrunner importer`
88105
- Run API
89-
- `npm run dev:api`
90-
- Use an API client (like [Postman](https://www.postman.com/)) to send requests to `http://localhost/events`
91-
- You'll need to make sure the `Authorization` header is set to one of your `API_KEYS` from your `.env`
106+
- `go run ./cmd/localsamrunner api`
107+
- Open Swagger docs
108+
- [http://localhost:3000/swagger/index.html](http://localhost:3000/swagger/index.html)
109+
110+
> **Note:** Valid AWS creds must be present to run either of the above commands.
111+
The easiest way to handle this would be to have a valid aws profile and add the `--profile <profile name>` to the above commands
92112
93113
### Shutting down
114+
- CTRL + C to shut down API
94115
- `docker compose down`
95116

96117
### Troubleshooting
@@ -100,15 +121,3 @@ If it's been awhile since you've last run the project, your SSO session in the A
100121
To fix it:
101122
- `aws sso login`
102123
- Open the link in a browser and follow the prompts
103-
104-
#### `npm run dev:sync-dynamodb` Hangs in Linux Environments (Including WSL)
105-
This can be caused by permissions errors with the `./docker` folder that docker compose creates.
106-
To fix it change the permissions of that folder to your local user
107-
```bash
108-
sudo chown $USER ./docker -R
109-
```
110-
111-
### `npm run dev:sync-dynamodb` Causes "UnrecognizedClientException: The security token included in the request is invalid."
112-
Specify a DYNAMODB_ENDPOINT environment variable pointing to localhost.
113-
114-
Example: `DYNAMODB_ENDPOINT=http://localhost:8000 npm run dev:sync-dynamodb`

‎cmd/localsamrunner/main.go‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
package main
2+
3+
import (
4+
"context"
5+
"flag"
6+
"log"
7+
"os"
8+
"os/exec"
9+
"path/filepath"
10+
"sgf-meetup-api/pkg/shared/appconfig"
11+
"sgf-meetup-api/pkg/shared/resource"
12+
)
13+
14+
func main() {
15+
config, err := appconfig.NewCommonConfig(context.Background())
16+
17+
if err != nil {
18+
log.Fatal("unable to create config")
19+
}
20+
21+
cmds := []*exec.Cmd{
22+
exec.Command("npm", "run", "synth"),
23+
}
24+
25+
flag.Parse()
26+
args := flag.Args()
27+
if len(args) < 1 {
28+
log.Fatal("Usage: go run main.go [api|importer]")
29+
}
30+
31+
subcommand := args[0]
32+
samArgs := args[1:]
33+
34+
switch subcommand {
35+
case "api":
36+
cmds = append(cmds, startAPI(config.AppEnv, samArgs...))
37+
case "importer":
38+
cmds = append(cmds, invokeImporter(config.AppEnv, samArgs...))
39+
default:
40+
log.Fatal("Unknown command. Use 'api' or 'importer'")
41+
}
42+
43+
for _, cmd := range cmds {
44+
cmd.Stdout = os.Stdout
45+
cmd.Stderr = os.Stderr
46+
if err := cmd.Run(); err != nil {
47+
log.Fatalf("Command failed: %v", err)
48+
}
49+
}
50+
}
51+
52+
func startAPI(appEnv string, samArgs ...string) *exec.Cmd {
53+
templateNamer := resource.NewNamer(appEnv, "SgfMeetupApiGo.template.json")
54+
55+
templatePath := filepath.Join(
56+
"./cdk.out",
57+
templateNamer.FullName(),
58+
)
59+
60+
args := []string{
61+
"local", "start-api",
62+
"-t", templatePath,
63+
"--docker-network", "sgf-meetup-api",
64+
"--env-vars", ".lambda-env.json",
65+
}
66+
67+
args = append(args, samArgs...)
68+
69+
return exec.Command("sam", args...)
70+
}
71+
72+
func invokeImporter(appEnv string, samArgs ...string) *exec.Cmd {
73+
templateNamer := resource.NewNamer(appEnv, "SgfMeetupApiGo.template.json")
74+
75+
templatePath := filepath.Join(
76+
"./cdk.out",
77+
templateNamer.FullName(),
78+
)
79+
80+
functionNamer := resource.NewNamer(appEnv, "Importer")
81+
82+
args := []string{
83+
"local", "invoke",
84+
"-t", templatePath,
85+
"--docker-network", "sgf-meetup-api",
86+
"--env-vars", ".lambda-env.json",
87+
functionNamer.FullName(),
88+
}
89+
90+
args = append(args, samArgs...)
91+
92+
return exec.Command("sam", args...)
93+
}

‎docs/.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
.$*.bkp

‎docs/architecture.drawio‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
<mxfile host="Electron" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/26.2.15 Chrome/134.0.6998.205 Electron/35.2.1 Safari/537.36" version="26.2.15">
2+
<diagram name="Page-1" id="h7a4USh2CO4sRyCAYE8R">
3+
<mxGraphModel dx="2066" dy="1219" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" math="0" shadow="0">
4+
<root>
5+
<mxCell id="0" />
6+
<mxCell id="1" parent="0" />
7+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-2" value="Meetup API" style="ellipse;shape=cloud;whiteSpace=wrap;html=1;" vertex="1" parent="1">
8+
<mxGeometry x="261" y="110" width="120" height="80" as="geometry" />
9+
</mxCell>
10+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-14" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" source="v0BFZ6OObwXRyGVg_hsL-3" target="v0BFZ6OObwXRyGVg_hsL-13">
11+
<mxGeometry relative="1" as="geometry" />
12+
</mxCell>
13+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-15" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=none;startFill=0;" edge="1" parent="1" source="v0BFZ6OObwXRyGVg_hsL-3" target="v0BFZ6OObwXRyGVg_hsL-4">
14+
<mxGeometry relative="1" as="geometry" />
15+
</mxCell>
16+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-17" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;" edge="1" parent="1" source="v0BFZ6OObwXRyGVg_hsL-3" target="v0BFZ6OObwXRyGVg_hsL-16">
17+
<mxGeometry relative="1" as="geometry" />
18+
</mxCell>
19+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-3" value="Importer Lambda&lt;div&gt;&lt;br&gt;&lt;/div&gt;" style="outlineConnect=0;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;shape=mxgraph.aws3.lambda;fillColor=#F58534;gradientColor=none;" vertex="1" parent="1">
20+
<mxGeometry x="288" y="430" width="76.5" height="93" as="geometry" />
21+
</mxCell>
22+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-4" value="MeetupEvents&lt;div&gt;Dynamo Table&lt;/div&gt;" style="outlineConnect=0;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;shape=mxgraph.aws3.dynamo_db;fillColor=#2E73B8;gradientColor=none;" vertex="1" parent="1">
23+
<mxGeometry x="469" y="560" width="72" height="81" as="geometry" />
24+
</mxCell>
25+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-26" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=none;startFill=0;" edge="1" parent="1" source="v0BFZ6OObwXRyGVg_hsL-6" target="v0BFZ6OObwXRyGVg_hsL-20">
26+
<mxGeometry relative="1" as="geometry" />
27+
</mxCell>
28+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-6" value="API Gateway" style="outlineConnect=0;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;shape=mxgraph.aws3.api_gateway;fillColor=#D9A741;gradientColor=none;" vertex="1" parent="1">
29+
<mxGeometry x="697" y="780" width="76.5" height="93" as="geometry" />
30+
</mxCell>
31+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-8" value="" style="group" vertex="1" connectable="0" parent="1">
32+
<mxGeometry x="77" y="436" width="110" height="124" as="geometry" />
33+
</mxCell>
34+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-5" value="" style="points=[];aspect=fixed;html=1;align=center;shadow=0;dashed=0;fillColor=#FF6A00;strokeColor=none;shape=mxgraph.alibaba_cloud.eventbridge;" vertex="1" parent="v0BFZ6OObwXRyGVg_hsL-8">
35+
<mxGeometry x="15" width="80" height="80" as="geometry" />
36+
</mxCell>
37+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-7" value="EventBridge&lt;div&gt;Schedule Rule)&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;resizable=0;points=[];autosize=1;strokeColor=none;fillColor=none;" vertex="1" parent="v0BFZ6OObwXRyGVg_hsL-8">
38+
<mxGeometry y="84" width="110" height="40" as="geometry" />
39+
</mxCell>
40+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-9" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="1" source="v0BFZ6OObwXRyGVg_hsL-5" target="v0BFZ6OObwXRyGVg_hsL-3">
41+
<mxGeometry relative="1" as="geometry" />
42+
</mxCell>
43+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-13" value="ArchivedMeetupEvents&lt;div&gt;Dynamo Table&lt;/div&gt;" style="outlineConnect=0;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;shape=mxgraph.aws3.dynamo_db;fillColor=#2E73B8;gradientColor=none;" vertex="1" parent="1">
44+
<mxGeometry x="469" y="300" width="72" height="81" as="geometry" />
45+
</mxCell>
46+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-16" value="&lt;div&gt;MeetupProxy Lambda&lt;/div&gt;" style="outlineConnect=0;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;shape=mxgraph.aws3.lambda;fillColor=#F58534;gradientColor=none;" vertex="1" parent="1">
47+
<mxGeometry x="288" y="240" width="76.5" height="93" as="geometry" />
48+
</mxCell>
49+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-18" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=0.55;entryY=0.95;entryDx=0;entryDy=0;entryPerimeter=0;endArrow=none;startFill=0;" edge="1" parent="1" source="v0BFZ6OObwXRyGVg_hsL-16" target="v0BFZ6OObwXRyGVg_hsL-2">
50+
<mxGeometry relative="1" as="geometry" />
51+
</mxCell>
52+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-27" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;endArrow=none;startFill=0;" edge="1" parent="1" source="v0BFZ6OObwXRyGVg_hsL-20" target="v0BFZ6OObwXRyGVg_hsL-4">
53+
<mxGeometry relative="1" as="geometry" />
54+
</mxCell>
55+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-20" value="Api Lambda&lt;div&gt;&lt;br&gt;&lt;/div&gt;" style="outlineConnect=0;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;shape=mxgraph.aws3.lambda;fillColor=#F58534;gradientColor=none;" vertex="1" parent="1">
56+
<mxGeometry x="466.75" y="780" width="76.5" height="93" as="geometry" />
57+
</mxCell>
58+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-21" value="ApiUsers&lt;div&gt;Dynamo Table&lt;/div&gt;" style="outlineConnect=0;dashed=0;verticalLabelPosition=bottom;verticalAlign=top;align=center;html=1;shape=mxgraph.aws3.dynamo_db;fillColor=#2E73B8;gradientColor=none;" vertex="1" parent="1">
59+
<mxGeometry x="261" y="786" width="72" height="81" as="geometry" />
60+
</mxCell>
61+
<mxCell id="v0BFZ6OObwXRyGVg_hsL-28" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;entryX=1;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;endArrow=none;startFill=0;" edge="1" parent="1" source="v0BFZ6OObwXRyGVg_hsL-20" target="v0BFZ6OObwXRyGVg_hsL-21">
62+
<mxGeometry relative="1" as="geometry" />
63+
</mxCell>
64+
</root>
65+
</mxGraphModel>
66+
</diagram>
67+
</mxfile>

‎docs/architecture.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Architecture
2+
3+
[Architecture Diagram](./architecture.drawio)
4+
5+
A primary goal of this application was for it to be hosted entirely on the AWS free tier.
6+
This would allow the app to be used indefinitely without needing major infrastructure changes.
7+
8+
With that in mind the app uses the following AWS Services:
9+
- [CDK](https://aws.amazon.com/cdk/)
10+
- Handles all infrastructure creation and updates
11+
- [Lambda](https://aws.amazon.com/lambda/)
12+
- The main compute for the app
13+
- [DynamoDB](https://aws.amazon.com/dynamodb/)
14+
- Handles all data storage
15+
- [EventBridge Scheduler](https://aws.amazon.com/eventbridge/scheduler/)
16+
- Triggers certain lambdas on a set interval
17+
- [API Gateway](https://aws.amazon.com/api-gateway/)
18+
- Exposes lambdas to the public internet
19+
20+
21+
Each of the lambdas is writting using [Go](https://go.dev/). TGo was picked almost entirely because one of the authors, [glitchedmob](https://github.com/glitchedmob/), was interested in learning how to use it.
22+
23+
The project makes signicant use of these libraries:
24+
- [spf13/viper](https://github.com/spf13/viper)
25+
- Library for loading configuration from multiple sources
26+
- [gin-gonic/gin](https://gin-gonic.com/)
27+
- A web framework used in the API Lambda
28+
- [google/wire](https://github.com/google/wire)
29+
- A tool for generating dependency injection boilerplate
30+
- [swaggo/swag](https://github.com/google/wire)
31+
- For generating OpenAPI/Swagger documentation
32+
33+
Of course, there are more libraries, but these are the main big ones.
34+
35+
The project also users this libraries in tests:
36+
- [testcontainers/testcontainers-go](https://github.com/testcontainers/testcontainers-go)
37+
- Creates databases in docker containers for testing
38+
- [stretchr/testify](https://github.com/stretchr/testify)
39+
- Mocking and assertion utilities
40+
- [brianvoe/gofakeit](https://github.com/brianvoe/gofakeit)
41+
- Used for generating fake test data

0 commit comments

Comments
 (0)