2349 docker compose dev env

[fyliu]

• Fixes Setup docker-compose for dev environment #2349

What changes did you make?

• Updated Docker Development Environment wiki page with documentation
• Set up development environment in docker-compose environment
  • docker-compose.yml description of the 6 service containers
  • Made some decisions that are more or less negotiable but I thought for the best. I tried to present the reasons and tradeoffs but it might still not be complete/clear. We can unmake decisions if some of them are very undesirable.

Why did you make the changes (we will use this info to test)?

1. I wanted to have an easily runable environment for the public. Anyone coming across the repo will be able to setup.
2. I wanted to preserve the way current frontend developers in the team uses the software, which is running the software locally but connecting to the shared backend that's in Azure. The current local host setup should still work.
3. I wanted to have a way for backend developers to switch to a local database for development. The mock sendgrid service is needed to capture what's being sent to sendgrid. This is more or less the same as the public setup but can be switched to frontend-only by commenting out some config lines.

Testing

1. Please note any missing documentation (from link above) that will need to be added.

2. There's no changes to the existing dotenv file. All the new settings are in override files that gets applied.

3. See the documentation for how to run the 4 configurations. They all can be run using the same npm run env -- command and passing in the configuration names.

   • I mainly tested user creation for local database and user login for shared database.

4. npm run test from the server/ directory should still work for the "shared-db" configuration.

5. npm run flyway:migrate from server/ directory should still work for the "shared-db" configuration.

6. Maybe have developers on different platforms try it to find any issues.

Remarks

• It's difficult to find the wiki page from the main one. I think we need to put a more prominent link somewhere in "Contributing".
• "Contributing" will need updating where it provides docker instructions.

[fyliu]

Looks like the latest commit broke something in the CI. I'll fix that.

[fyliu]

I rebased the branch to develop.

[fyliu]

We decided in the 2025-06-04 meeting to:

• keep the .env files with the individual projects, like server would have its own .env file like before, and other directories that need .env files will have their own.
• add scripts in the root package.json like npm run compose to hide the more complicated command we need to load multiple .env files.

[fyliu]

I moved the settings back into the server/ directory and added scripts to automate applying the extra settings for docker, which now exist as override files. So I can leave the old dotenv file alone. This is ready for review.

[fyliu]

I had some extra "fixes" I didn't intend to push since they're not in the scope of this issue.

[fyliu]

• reverted the client install of dotenvx since it's only done to be consistent with the server, but it installs more packages than dotenv. dotenvx includes dotenv.
• added the EMAIL_SENDER config to sendgrid override file so that the public server/.env.example file will work. Even an empty server/.env file will work

[entrotech]

@fyliu
When trying to run the env script:
Option 1: app running on host machine, shared database - this works.
Option 2 Running the app in docker, I get this:

$ npm run env -- app up -d

> root@0.2.59 env
> bash scripts/start-env.sh app up -d

+ ./scripts/copy-env.sh app
+ rm -f .env
+ rm -f env/docker/node
+ rm -f env/docker/vite
+ rm -f env/docker/urls
+ rm -f env/docker/db
+ rm -f env/docker/sendgrid
+ rm -f env/host/db
+ rm -f env/host/sendgrid
+ cp .env.example .env
+ cp env/docker/node.example env/docker/node
+ cp env/docker/vite.example env/docker/vite
+ cp env/docker/urls.example env/docker/urls
+ call_start app up -d
+ docker compose --env-file .env --env-file ./server/.env --env-file env/docker/node --env-file env/docker/vite --env-file env/docker/urls up -d
invalid interpolation format for services.db-migrate.command.[]: "required variable DOCKER_DB_USER_NAME is missing a value: error". You may need to escape any $ with another $.

johnc@John-ThinkPad MINGW64 /c/git/hackforla/tdm-calculator (2349-docker-compose-dev-env)
$

Option 3: Running everything in docker, I get this:

$ npm run env -- all up -d

> root@0.2.59 env
> bash scripts/start-env.sh all up -d

+ ./scripts/copy-env.sh all
+ rm -f .env
+ rm -f env/docker/node
+ rm -f env/docker/vite
+ rm -f env/docker/urls
+ rm -f env/docker/db
+ rm -f env/docker/sendgrid
+ rm -f env/host/db
+ rm -f env/host/sendgrid
+ cp .env.example .env
+ cp env/docker/node.example env/docker/node
+ cp env/docker/vite.example env/docker/vite
+ cp env/docker/urls.example env/docker/urls
+ cp env/docker/db.example env/docker/db
+ cp env/docker/sendgrid.example env/docker/sendgrid
+ call_start all up -d
+ docker compose --env-file .env --env-file env/docker/node --env-file env/docker/vite --env-file env/docker/urls --env-file env/docker/db --env-file env/docker/sendgrid up -d
time="2025-06-20T17:35:27-07:00" level=warning msg="The \"CLIENT_PORT\" variable is not set. Defaulting to a blank string."
time="2025-06-20T17:35:27-07:00" level=warning msg="The \"CLIENT_EXPOSED_PORT\" variable is not set. Defaulting to a blank string."
time="2025-06-20T17:35:27-07:00" level=warning msg="The \"CLIENT_PORT\" variable is not set. Defaulting to a blank string."
time="2025-06-20T17:35:27-07:00" level=warning msg="The \"PORT\" variable is not set. Defaulting to a blank string."
invalid interpolation format for services.db-migrate.command.[]: "required variable DOCKER_DB_USER_NAME is missing a value: error". You may need to escape any $ with another $.

johnc@John-ThinkPad MINGW64 /c/git/hackforla/tdm-calculator (2349-docker-compose-dev-env)      
$

Option 4: Running database in docker, app on host machine:

$ npm run env -- local-db up -d

> root@0.2.59 env
> bash scripts/start-env.sh local-db up -d

+ ./scripts/copy-env.sh local-db
+ rm -f .env
+ rm -f env/docker/node
+ rm -f env/docker/vite
+ rm -f env/docker/urls
+ rm -f env/docker/db
+ rm -f env/docker/sendgrid
+ rm -f env/host/db
+ rm -f env/host/sendgrid
+ cp .env.example .env
+ cp env/docker/db.example env/docker/db
+ cp env/host/db.example env/host/db
+ cp env/host/sendgrid.example env/host/sendgrid
+ call_start local-db up -d
+ docker compose --env-file .env --env-file env/docker/db --env-file env/host/sendgrid up db-migrate sendgrid -d
time="2025-06-20T17:37:16-07:00" level=warning msg="The \"CLIENT_EXPOSED_PORT\" variable is not set. Defaulting to a blank string."
time="2025-06-20T17:37:16-07:00" level=warning msg="The \"CLIENT_PORT\" variable is not set. Defaulting to a blank string."
time="2025-06-20T17:37:16-07:00" level=warning msg="The \"CLIENT_PORT\" variable is not set. Defaulting to a blank string."
time="2025-06-20T17:37:16-07:00" level=warning msg="The \"PORT\" variable is not set. Defaulting to a blank string."
invalid interpolation format for services.db-migrate.command.[]: "required variable DOCKER_DB_USER_NAME is missing a value: error". You may need to escape any $ with another $.

johnc@John-ThinkPad MINGW64 /c/git/hackforla/tdm-calculator (2349-docker-compose-dev-env)    
$

It looks like maybe these missing env variables are in the .env.example file, but the start-env.sh script is not looking for them there? I am running Windows and Git Bash, if that matters.

[fyliu]

@entrotech I set the scripts to echo every command it runs. Looks like the docker compose command is passing the env files correctly. I haven't tried to use git bash for this. I'll try to fix it to work.

Update 2025-06-21: I had trouble setting up npm on my win11 yesterday. I'll keep trying.

I did all the work in linux and I generally push windows users to use WSL since it's the easiest way to install all the dependencies: git, npm, python, etc.

[fyliu]

@entrotech I was able to set up and run the PR using git bash on windows. I didn't get the missing value errors. I'm wondering if Docker just needs to be updated. It's one thing people say some older Dockers do.

I also just removed dotenv from the client. It doesn't really need it.

[entrotech]

@fyliu
I was on version 2.1.1 of docker compose. Re-installed Docker Desktop for Windows, now I get:

$ docker compose version
Docker Compose version v2.35.1-desktop.1

Now when I try to compose I get:

$ npm run env -- app up -d

> root@0.2.59 env
> bash scripts/start-env.sh app up -d

+ ./scripts/copy-env.sh app
+ rm -f .env
+ rm -f env/docker/node
+ rm -f env/docker/vite
+ rm -f env/docker/node
+ rm -f env/docker/vite
+ rm -f env/docker/urls
+ rm -f env/docker/db
+ rm -f env/docker/sendgrid
+ rm -f env/host/db
+ rm -f env/host/sendgrid
+ cp .env.example .env
+ cp env/docker/node.example env/docker/node
+ cp env/docker/vite.example env/docker/vite
+ cp env/docker/urls.example env/docker/urls
+ call_start app up -d
+ docker compose --env-file .env --env-file ./server/.env --env-file env/docker/node --env-file env/docker/vite --env-file env/docker/urls up -d
unable to get image 'tdm-calculator-client': request returned 500 Internal Server Error for API route and version http://%2F%2F.%2Fpipe%2FdockerDesktopLinuxEngine/v1.49/images/tdm-calculator-client/json, check if the server supports the requested API version

and if I check for any images that might have been built:

$ docker image ls
REPOSITORY                       TAG                      IMAGE ID       CREATED         SIZE
tdm                              latest                   065cfec7e2fc   2 months ago    360MB
<none>                           <none>                   44d294a56da5   2 months ago    370MB
<none>                           <none>                   665dbfe48b16   2 months ago    441MB
testcontainers/ryuk              0.11.0                   0cd5951d043f   8 months ago    15.3MB
mcr.microsoft.com/mssql/server   2017-latest              837b47c7b5e8   9 months ago    1.33GB
mcr.microsoft.com/mssql/server   2022-CU13-ubuntu-22.04   994123c66ce4   13 months ago   1.58GB
mcr.microsoft.com/mssql/server   2019-latest              376667ae3d77   14 months ago   1.48GB
mcr.microsoft.com/mssql/server   2022-latest              a702c27e4969   15 months ago   1.58GB
mcr.microsoft.com/mssql/server   latest                   a702c27e4969   15 months ago   1.58GB
testcontainers/ryuk              0.5.1                    ec913eeff75a   2 years ago     12.7MB

These are all old, so it doesn't look like any images were built.

[entrotech]

I also tried running just the db in docker:

$ npm run env -- local-db up -d

> root@0.2.59 env
> bash scripts/start-env.sh local-db up -d

+ ./scripts/copy-env.sh local-db
+ rm -f .env
+ rm -f env/docker/node
+ rm -f env/docker/vite
+ rm -f env/docker/urls
+ rm -f env/docker/db
+ rm -f env/docker/sendgrid
+ rm -f env/host/db
+ rm -f env/host/sendgrid
+ cp .env.example .env
+ cp env/docker/db.example env/docker/db
+ cp env/host/db.example env/host/db
+ cp env/host/sendgrid.example env/host/sendgrid
+ call_start local-db up -d
+ docker compose --env-file .env --env-file env/docker/db --env-file env/host/sendgrid up db-migrate sendgrid -d
[+] Running 23/23
 ✔ sendgrid Pulled                                                                                                                                   40.4s 
 ✔ db-migrate Pulled                                                                                                                                163.7s 
[+] Running 6/6
 ✔ Network tdm-calculator_default         Created                                                                                                     0.1s 
 ✔ Volume "tdm-calculator_sqlvolume"      Created                                                                                                     0.0s 
 ✘ Container tdm-calculator-db-1          Error                                                                                                     106.6s 
 ✔ Container tdm-calculator-sendgrid-1    Started                                                                                                     5.0s 
 ✔ Container tdm-calculator-db-init-1     Created                                                                                                     0.1s 
 ✔ Container tdm-calculator-db-migrate-1  Created                                                                                                     0.4s 
dependency failed to start: container tdm-calculator-db-1 is unhealthy

This seems to have removed all my other images, but did create one new one:

$ docker ps -all
CONTAINER ID   IMAGE            COMMAND                  CREATED         STATUS    PORTS     NAMES
8fb6e9326f50   redgate/flyway   "flyway -user=sa -pa…"   9 minutes ago   Created             tdm-calculator-db-migrate-1

[fyliu]

None of the scripts should magically remove image. I'm guessing, but maybe the newer docker is incompatible with the older one and all the images are now invalid? Ah, I found this SO answer about newer Dockers on Windows using the WSL flag which makes the old images unrecognized by the Docker system. I'm not sure if there's data in the old containers that you want to save. I see another SO answer with steps to migrate images and volumes although I don't understand much after a quick glance.

You might try to rebuild the tdm images first with npm run env -- all build --pull --no-cache. The flags will force docker pull all the images and also not use docker's build cache stuff. It's basically a clean build.

The flyway and sqlserver images are big downloads.

[fyliu]

If you don't care about the containers you had, you can clean them up. Here's a list of artifacts you can remove in docker. The last command includes everything.

# Omit the -f to see what's being deleted before giving confirmation

# remove build cache
docker buildx prune -f

# remove unused containers
docker container prune -f

# remove dangling images not associated with a container
docker image prune -f

# remove unused images
docker image prune -f -a

# remove all unused containers, images, network, and build cache
docker system prune -f

Reference used: How to clear Docker cache and free up space on your system

[ExperimentsInHonesty]

@Tony-Villa @entrotech @fyliu what is the next step for this PR? It looks like there are conflicts to resolve. Is Fang responsible for doing that. Are there any other blockers to approving this?

[ExperimentsInHonesty]

@fyliu Tony is able to get this working on his Mac, but John can't get it working on his PC. Given that this issue is supposed to make it easier to work on the code base, the fact that john is having isssues making it work, is concerning. What do you think we should do next?

[fyliu]

I'm seeing this just now.

@entrotech What is failing? Is there any helpful messages in the terminal we can work with?

I think you're using git bash on windows and have the code checked out on C:\...?

One thing to note is to run it using the npm script rather than using docker compose up directly. The npm script adds some environment options.