Merge pull request #1 from CrazyVito11/development

README.md

@@ -1,8 +1,9 @@
 # Docker AUR Cache
-Docker image that automatically builds AUR packages and hosts them for faster installation on client devices.
 
+Docker image that automatically builds AUR packages and hosts them for faster installation on client devices.
 
 ## Why?
+
 > "Why not just use an AUR helper to build the packages for you?"
 >
 > _- Probably you_
@@ -14,29 +15,24 @@ Some AUR packages can also be quite complex, taking a while to compile, which yo
 This Docker container was made to compile these AUR packages beforehand, and then provide them in such a way so that pacman can install those AUR packages directly, without needing AUR helpers or other build tools on your machine.
 This can also be extra beneficial if you have multiple computers running Arch Linux, as then you only have to compile the packages once for all those machines.
 
-
 ## Setup (Container)
-1. Make sure you meet the follow prerequisites
-    - Docker + Docker Compose have been installed
-    - The [Traefik reverse proxy container](https://github.com/CrazyVito11/traefik-reverse-proxy) has been configured
-    - A registered domain name
-        - You can use something like [PiHole](https://github.com/pi-hole/pi-hole) to register local domains if you aren't going to host it publicly
-> [!TIP]
-> While this container was designed to work with Traefik and a domain name, with a couple of tweaks you can bind to a TCP port instead.
-> Allowing you to skip the requirement of needing both Traefik and a domain name to host this application.
->
-> See the section **Bind NGINX to port instead of Traefik** for instructions.
-2. Clone this repository to your server
-3. Make a copy of `.env.example` and call it `.env`
+
+1. Make sure you meet the follow prerequisites - Docker + Docker Compose have been installed - The [Traefik reverse proxy container](https://github.com/CrazyVito11/traefik-reverse-proxy) has been configured - A registered domain name - You can use something like [PiHole](https://github.com/pi-hole/pi-hole) to register local domains if you aren't going to host it publicly
+
+   > [!TIP]
+   > While this container was designed to work with Traefik and a domain name, the default production compose file should work without.
+   > Allowing you to skip the requirement of needing both Traefik and a domain name to host this application.
+   >
+   > See the section **Bind NGINX to port instead of Traefik** for instructions.
+
+2. Create a folder, preferably one called `docker-aur-cache`
+3. Run `curl -fsSL https://raw.githubusercontent.com/CrazyVito11/docker-aur-cache/refs/heads/master/setup.sh | sh`
+   - This script wil create the following files and folders: `./repository/`, `./assets/nginx-default.conf`, `.env` & `packagelist.json`.
 4. Make changes to the `.env` file if needed
-5. Make a copy of `packagelist.json.example` and call it `packagelist.json`
-6. Add the packages you want to provide to the `packagelist.json` file
-    - You can find documentation about this file at section **Configure packagelist**.
-7. Update the repository permissions with `chmod 777 ./repository`
-    - **Note:** We are aware that this isn't very secure, this will be improved in the future.
-8. Build the container with `docker compose build`
-9. Start the container with `docker compose up -d`
-10. Set the right permissions with `docker compose exec build-manager bash -c 'chown builder /package-staging; chown builder /aur-package-list'`
+5. Add the packages you want to provide to the `packagelist.json` file
+   - You can find documentation about this file at section **Configure packagelist**.
+6. Run the container with `docker compose up -d`
+   - If you don't have a prebuild image, don't forget to build it first using `docker compose build`
 
 The container should now be ready, try visiting your domain and you should see a index page!
 This page will also show packages that are available for downloading.
@@ -45,8 +41,8 @@ This page will also show packages that are available for downloading.
 > Normally you would have to wait until Monday @ 01:00 for it to start building, but instructions are available in case you want to start the build immediately.
 > See the section **Force the build immediately** for more information.
 
-
 ## Setup (Client)
+
 To configure Pacman to use your self-hosted repository, open your `/etc/pacman.conf` file and add the following repository:
 
 ```
@@ -61,30 +57,31 @@ Server = http://docker-aur-cache.localhost/
 And that should be it!
 Simply perform a full system update and you should be able to install packages that are hosted by your instance.
 
-
 ## Configure packagelist
+
 The `packagelist.config.json` file is a JSON formatted file that will be used to configure all sorts of things, like which packages you want to build, the tweaks these packages need, limit system resources given to the builder, etc.
 
 ### Packagelist config description
+
 This is the root of the `packagelist.config.json` file.
 
 | **Field**      | **Required** | **Type**                   | **Description**                                                                   |
-|----------------|--------------|----------------------------|-----------------------------------------------------------------------------------|
+| -------------- | ------------ | -------------------------- | --------------------------------------------------------------------------------- |
 | `builderLimit` | Yes          | `builder limit object`     | Defines how many system resources the builder instance is allowed to use.         |
 | `packages`     | Yes          | `array of package objects` | An array of packages that you want to be build each time the builder is executed. |
 
-
 ### Builder limit object description
+
 This object is used to limit how many system resources the builder instance is allowed to use.
 
 | **Field**      | **Required** | **Type** | **Description**                                                                                                                                                                    |
-|----------------|--------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| -------------- | ------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `maxBuildTime` | Yes          | `string` | Defines how long the build process is allowed to run for 1 package before it's terminated. The format is `1234X`, where X determines the scale. Allowed scales are `s`, `m`, `h` . |
 | `cpusetCpus`   | Yes          | `string` | Defines which CPU cores it's allowed to use. It can be a list _(`0,1,2,3`)_ of cores or a range of cores _(`0-3`)_.                                                                |
 | `memory`       | Yes          | `string` | Defines how much memory it's allowed to use before it's terminated. The format is `1234X`, where X determines the scale. Allowed scales are `b`, `k`, `m`, `g`.                    |
 
-
 ### Package object description
+
 This object is used to define the settings in order to build one specific package.
 
 | **Field**                | **Required** | **Type**               | **Description**                                                                                                            |
@@ -98,26 +95,27 @@ This object is used to define the settings in order to build one specific packag
 > The build process is executed in a separate container for each AUR package, which is destroyed after the build is complete.
 > If you for example run a command for `package-a` to import a key that `package-b` will also need, you will have to add that command also to the configuration of `package-b`.
 
-
 ## Tips
+
 ### Force the build immediately
+
 If you want the container to start building right now, you can run the following command:
 
 `docker compose exec --user builder build-manager bash /repository-builder/build-packages.sh`
 
 It should now start building your packages, wait for this process to finish and then you can install the packages you configured.
 
-
 ### Download older version of AUR package
+
 If you need to downgrade a specific package for some reason, you can find older version in the `archive` directory of the webserver that is hosting the packages.
 
 Simply download the package from there and then install it manually with `pacman -U some-package-1.0.0-x86_64.pkg.tar.zst`.
 
 > [!WARNING]
 > Older versions of packages are automatically cleaned up after 30 days to preserve storage space on your server.
 
-
 ### Package build fails with a SIGKILL
+
 Chances are that the build process consumed too much memory and was killed by the Docker engine or the host operating system.
 The builder process has been given a high OOM score to make sure it is killed first, instead of other _(potentially more crucial)_ applications.
 
@@ -126,17 +124,17 @@ To resolve this, you will have to raise the builder memory limit, free up more R
 > [!TIP]
 > Sometimes the AUR also provides `-bin` _(precompiled binary)_ variants of a package, this could also be a solution to work around the issue.
 
-
 ### Access the build reports
+
 The application generates a report every time it runs the build process, allowing you to more easily troubleshoot troublesome packages.
 
 These reports can be found in the `build-reports` directory of the webserver that is hosting the packages.
 
 > [!TIP]
 > For now these are only available in JSON format, but these will become easier to read in a future commit.
 
-
 ### Bind NGINX to port instead of Traefik
+
 While it's intended to be used with my Traefik container, it does increase the steps needed to get this application up and running, and it might not even be possible in certain situations.
 
 With a couple of modifications to the `docker-compose.yml` file you can skip the need for Traefik and a domain name, and just connect directly to a TCP port instead.
@@ -145,6 +143,6 @@ With a couple of modifications to the `docker-compose.yml` file you can skip the
 2. Remove the `labels` section from the `nginx` container.
 3. Remove the `traefik-reverse-proxy` network from the `nginx` container.
 4. Add a `ports` section to the `nginx` container and bind the port you want to use to `80` on the container.
-    - **Example:** `- 8080:80` to bind it to port `8080` on your host machine.
+   - **Example:** `- 8080:80` to bind it to port `8080` on your host machine.
 
 It should now be accessible on your desired TCP port without having to set up my Traefik configuration as well.

build-manager/docker/Dockerfile

@@ -1,54 +1,71 @@
-FROM --platform=linux/amd64 archlinux:multilib-devel
+# Multi-stage build for build-manager
+FROM --platform=linux/amd64 archlinux:multilib-devel AS base
 
 # Update the image and install dependencies
 RUN pacman-key --init
 RUN pacman -Sy --noconfirm archlinux-keyring
 RUN pacman -Syu --noconfirm fcron nodejs-lts-iron npm docker
 
-# Create the builder user
+# Create the builder user with default values
 RUN useradd -u 1000 -ms /bin/bash builder
 RUN usermod -a -G fcron builder
 RUN chmod o+x /usr/sbin/fcron
-USER builder
-RUN mkdir -p /home/builder
-WORKDIR /home/builder
 
-# Prepare Docker access
-USER root
-ARG DOCKER_GID
-RUN groupadd -g $DOCKER_GID dockerindocker
-RUN usermod -a -G dockerindocker builder
-USER builder
-
-# Build build-manager application
-USER builder
-COPY ./build-manager /build-manager
-USER root
-RUN chown -R builder /build-manager
-USER builder
+ARG DOCKER_GID=964
+ENV DOCKER_GID=$DOCKER_GID
+
+# Create docker group with default GID and add builder to it
+RUN groupadd -g $DOCKER_GID docker || true
+RUN usermod -a -G docker builder
+
+# Set up Docker socket permissions
+RUN echo '#!/bin/bash' > /usr/local/bin/setup-docker-permissions.sh && \
+    echo 'chmod 666 /var/run/docker.sock' >> /usr/local/bin/setup-docker-permissions.sh && \
+    chmod +x /usr/local/bin/setup-docker-permissions.sh
+
+# Build stage for build-manager
+FROM base AS build-manager-builder
 WORKDIR /build-manager
-RUN npm install && npm run build
+COPY ./build-manager/package*.json ./
+RUN npm ci
+COPY ./build-manager/src ./src
+COPY ./build-manager/tsconfig.json ./
+RUN npm run build
 
-# Copy builder application
-USER builder
-COPY ./builder /builder
-USER root
-RUN chown -R builder /builder
+# Build stage for builder
+FROM base AS builder-builder
+WORKDIR /builder
+COPY ./builder/package*.json ./
+RUN npm ci
+COPY ./builder/src ./src
+COPY ./builder/tsconfig.json ./
+COPY ./builder/.dockerignore ./
+COPY ./builder/Dockerfile ./
+RUN npm run build
 
-# Copy script
-USER root
-COPY /build-manager/docker/build-packages.sh /repository-builder/
-COPY /build-manager/docker/clean-repository-archive.sh /repository-builder/
-RUN chmod 777 /repository-builder/build-packages.sh
-RUN chmod 777 /repository-builder/clean-repository-archive.sh
-USER builder
+# Final stage
+FROM base AS final
+
+# Copy built applications and dependencies
+COPY --from=build-manager-builder --chown=builder:builder /build-manager/dist /build-manager/dist
+COPY --from=build-manager-builder --chown=builder:builder /build-manager/package*.json /build-manager/
+COPY --from=build-manager-builder --chown=builder:builder /build-manager/node_modules /build-manager/node_modules
+COPY --from=builder-builder --chown=builder:builder /builder /builder
+
+# Copy scripts
+COPY ./build-manager/docker/build-packages.sh /repository-builder/
+COPY ./build-manager/docker/clean-repository-archive.sh /repository-builder/
+RUN chown builder:builder /repository-builder/build-packages.sh /repository-builder/clean-repository-archive.sh
+RUN chmod 755 /repository-builder/build-packages.sh /repository-builder/clean-repository-archive.sh
 
 # Configure fcron
-USER root
-COPY /build-manager/docker/fcron_builder.tab /tmp/
-RUN fcrontab -u builder /tmp/fcron_builder.tab
+COPY ./build-manager/docker/fcron_builder.tab /tmp/
+RUN chown builder:builder /tmp/fcron_builder.tab
+RUN runuser -u builder -- fcrontab /tmp/fcron_builder.tab
 
 WORKDIR /repository-builder
-RUN chmod 777 /repository-builder
+RUN chown builder:builder /repository-builder
+RUN chmod 755 /repository-builder
 
-CMD ["fcron", "--foreground", "--firstsleep", "0"]
+USER root
+CMD ["fcron", "--foreground", "--firstsleep", "0"]

build-manager/docker/build-packages.sh

@@ -2,7 +2,6 @@
 
 set -e
 
-
 # First, ensure that required directories are made
 mkdir -p /repository/archive
 mkdir -p /repository/build-reports
@@ -17,4 +16,4 @@ node ./dist/build-manager.js \
     --build_report_dir=/repository/build-reports \
     --repository_archive_dir=/repository/archive \
     --repository_dir=/repository \
-    --repository_name=docker-aur-cache
+    --repository_name=$COMPOSE_PROJECT_NAME

build-manager/package.json

@@ -1,8 +1,9 @@
 {
   "name": "build-manager",
-  "main": "./src/build-manager.ts",
+  "main": "./dist/build-manager.js",
   "scripts": {
-    "build": "npx tsc"
+    "build": "npx tsc",
+    "start": "node ./dist/build-manager.js"
   },
   "devDependencies": {
     "@types/dockerode": "^3.3.31",

build-manager/src/Helpers/BuilderHelper.ts

@@ -8,17 +8,17 @@ export default class BuilderHelper {
         return `cd /builder; node ./dist/builder.js --package_configuration_encoded="${packageConfigurationEncoded}" --build_dir="/repository-builder" --package_staging_dir="/package-staging" --aur_package_list_path="${aurPackageListPath}"`;
     }
 
-    public static getBuilderMounts(): MountConfig {
+    public static getBuilderMounts(repositoryName: string): MountConfig {
         return [
             {
                 Target: '/package-staging',
-                Source: 'docker-aur-cache_package-staging',
+                Source: `${repositoryName}_package-staging`,
                 Type: 'volume',
                 ReadOnly: false,
             },
             {
                 Target: '/aur-package-list',
-                Source: 'docker-aur-cache_aur-package-list',
+                Source: `${repositoryName}_aur-package-list`,
                 Type: 'volume',
                 ReadOnly: true,
             }

build-manager/src/build-manager.ts

@@ -180,7 +180,7 @@ const publishBuildPackages = async () => {
 
     execSync(`mv ${params.package_staging_dir}/*.pkg.tar.zst ${params.repository_dir}/`);
 
-    console.log("[build-manager] Removing old database (if it exists)");
+    console.log("[build-manager] Removing old database (if iparams.repository_namet exists)");
     execSync(`rm -f ${params.repository_dir}/${params.repository_name}.db* ${params.repository_dir}/${params.repository_name}.files*`);
 
     console.log("[build-manager] Adding packages to database");
@@ -205,7 +205,7 @@ const handlePackageList = async (aurPackageListPath: string) => {
             logs: [],
         };
 
-        if (packageConfiguration.enabled) {
+        if (packageConfiguration.enabled) {params.repository_name
             try {
                 const command = BuilderHelper.getBuilderStartCommand(aurPackageListPath, packageConfiguration);
 
@@ -219,7 +219,7 @@ const handlePackageList = async (aurPackageListPath: string) => {
                     Cmd: ['/bin/bash', '-c', command],
                     HostConfig: {
                         OomScoreAdj: 1000, // Make it more likely the builder will be killed in low RAM situations instead of (potentially more crucial) applications
-                        Mounts: BuilderHelper.getBuilderMounts(),
+                        Mounts: BuilderHelper.getBuilderMounts(params.repository_name),
                         CpusetCpus: packageListConfiguration.builderLimit.cpusetCpus,
                         Memory: FilesystemHelper.stringifiedSizeToBytes(packageListConfiguration.builderLimit.memory)
                     }

builder/Dockerfile

@@ -26,7 +26,8 @@ RUN npm install && npm run build
 
 WORKDIR /repository-builder
 USER root
-RUN chmod 777 /repository-builder
+RUN chown builder:builder /repository-builder
+RUN chmod 755 /repository-builder
 USER builder
 
 CMD ["fcron", "--foreground", "--firstsleep", "0"]

builder/package.json

@@ -1,8 +1,9 @@
 {
   "name": "builder",
-  "main": "./src/builder.ts",
+  "main": "./dist/builder.js",
   "scripts": {
-    "build": "npx tsc"
+    "build": "npx tsc",
+    "start": "node ./dist/builder.js"
   },
   "devDependencies": {
     "@types/node": "^20.14.11",

docker-compose.yml → docker-compose.dev.yml

@@ -5,6 +5,8 @@ services:
       dockerfile: ./build-manager/docker/Dockerfile
       args:
         - DOCKER_GID=${DOCKER_GID:-964}
+    environment:
+      - REPOSITORY_NAME=${REPOSITORY_NAME:-docker-aur-cache}
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
       - ./repository:/repository:cached