Set up lambda local dev environment#78
Conversation
| RUN uv export \ | ||
| --no-emit-workspace \ | ||
| --package lambda \ | ||
| --package did_lambda \ |
There was a problem hiding this comment.
was the package rename necessary for some reason?
There was a problem hiding this comment.
Yeah, I ran into this issue: https://docs.astral.sh/ruff/rules/invalid-module-name/#why-is-this-bad
Particularly this part:
Further, in order for Python modules to be importable, they must be valid identifiers. As such, they cannot start with a digit, or collide with hard keywords, like import or class.
In our case, lambda is a reserved Python keyword, so having the module named lambda was causing issues with ty and importing sibling modules under /packages/lambda/src/lambda.
DanielSass
left a comment
DanielSass
left a comment
There was a problem hiding this comment.
curious about the package rename, but otherwise this looks dope! thanks Kevin
| description = "" | ||
| dependencies = [ | ||
| "lxml>=6.0.2", | ||
| "pydantic>=2.13.4", |
There was a problem hiding this comment.
Move pydantic to be an explicit dependency of core and did_lambda after running into module resolution issues with the lambda container.
| def lambda_handler(event: SQSEvent, _context: LambdaContext) -> dict: | ||
| """Download manifests and the XML objects they reference.""" | ||
| for sqs_record in event.records: | ||
| for s3_record in sqs_record.decoded_nested_s3_event.records: | ||
| if s3_record.event_name != "ObjectCreated:Put": | ||
| continue | ||
|
|
||
| bucket = s3_record.s3.bucket.name | ||
| manifest_key = s3_record.s3.get_object.key | ||
|
|
||
| # fetch manifest | ||
| try: | ||
| manifest = _get_manifest(bucket, manifest_key) | ||
| for file in manifest.files: | ||
| logger.info(file) | ||
| except ValidationError: | ||
| logger.error("Invalid manifest file") | ||
| raise | ||
|
|
||
| return {"statusCode": 200, "message": "OK"} |
There was a problem hiding this comment.
Implemented some basic functionality in the lambda for us to refine in #76
Right now all it does is:
- Listen for SQSEvents, specifically events from S3 with the
ObjectCreated:Putevent name - Fetches the manifest.json from the event
- Parses it using a pydantic model, and logs the data
| """Models for S3 manifest files.""" | ||
|
|
||
| from pydantic import BaseModel, Field | ||
|
|
||
|
|
||
| class ManifestRecord(BaseModel): | ||
| """An eICR and its reportability response.""" | ||
|
|
||
| eicr: str | ||
| rr: str | None = None | ||
| setId: str | ||
| versionNumber: int | ||
|
|
||
|
|
||
| class Manifest(BaseModel): | ||
| """A manifest containing files to process.""" | ||
|
|
||
| files: list[ManifestRecord] = Field(alias="Files") |
There was a problem hiding this comment.
I would suspect these models to change with further discussions with APHL.
I've left the RR as "optional" to allow us to test eICRs locally without having to also upload RRs.
|
|
||
| typecheck: | ||
| glob: "*.py" | ||
| glob: "packages/**/*.py" |
There was a problem hiding this comment.
Minor change to lefthook to only lint/typecheck Python in our packages.
This was causing an issue with docker/sqs-poller.py; since it is a self-contained script, ty was reporting that it could not resolve its dependencies (boto3 and httpx)
| @@ -0,0 +1,95 @@ | |||
| #!/usr/bin/env -S uv run | |||
There was a problem hiding this comment.
This is a generic script that is run as a separate service in compose.yml.
Since we are not running the lambda as a service under localstack, we need another way for SQS events to invoke the lambda.
| path: ./uv.lock | ||
|
|
||
| localstack: | ||
| image: docker.io/localstack/localstack:community-archive |
There was a problem hiding this comment.
I mentioned this during share time, but I briefly tested using floci by changing the image here, and it worked seamlessly without having to change any of the init scripts or env variables.
I haven't evaluated other AWS emulator alternatives yet, so this is something we could explore in the future.
2 participants
Related Issue
#79
Changes Proposed
lambdaworkspace member todid_lambda(see reasoning)pydantica explicit dependency ofcoreanddid_lambda.Additional Information
For the localstack version, I've pinned it to
localstack:community-archivewhich is the tag that points to the last version of localstack before they began requiring licenses for the free version.The lambda is run as a separate service (not from within localstack).
This is based on information we've gathered during our last tech sync with Geo and the
DiD Engineering Sync — Pipeline Integration & Infradocument.Testing
docker compose up --env-file .env.local -dlocalhost:8081to see the uploader. Try uploading an eICR and/or an RR.localhost:8080to see stackport. Navigate to the S3 bucket under the S3 resource to find the manifest.json underDIDInput/and your eICR/RR undereICRMessageV2/and/orRRMessageV2, namespaced with the same persistenceId UUID.docker compose logs -f did_lambdato follow lambda logs. You should have seen an event with your upload similar to:docker compose downto spin down containersTest watch mode
docker compose up --env-file .env.local --watchto watch for changes.lambda_function.pythat will show up in the logs.did_lambdacontainer restart. On next eICR upload, your log changes should be reflected.Checklist for Primary Reviewer