|
| 1 | +# AnvilOps Contributors' Guide |
| 2 | + |
| 3 | +This document contains a list of things you'll want to know before contributing to AnvilOps. Before you read this document, check out [README.md](./README.md), and if you still have questions after reading this document, they may be addressed in the individual subproject READMEs (e.g. [backend/README.md](./backend/README.md)). |
| 4 | + |
| 5 | +## Philosophy |
| 6 | + |
| 7 | +### Goals |
| 8 | + |
| 9 | +- Ease-of-use: Make software really easy to deploy on Kubernetes. |
| 10 | +- Abstraction: Abstract away Kubernetes-specific details and concepts so that the user doesn't need any Kubernetes background knowledge. |
| 11 | +- Portability: People should be able to install AnvilOps on any Kubernetes cluster, without depending on Purdue-specific or proprietary components. |
| 12 | + - This means users should easily be able to leave AnvilOps, and AnvilOps shouldn't store any configurations in the user's repositories. |
| 13 | + - AnvilOps should also be quick and easy to install, which is why it's distributed as a Helm chart with most of the necessary dependencies included. |
| 14 | +- Security: AnvilOps should defend against accidental misuse, but because we value portability and simplicity, it probably can't protect against adversarial users. It's intended for private use among trusted users. However, this does not mean security isn't a priority. Things like tenant isolation and best-effort protections like rate limits and build timeouts are still very important. |
| 15 | + - AnvilOps's container images should be minimal, containing only the necessary tools to run the service. In production, we use distroless images to minimize attack surface because they don't contain things like shells or debugging tools. |
| 16 | + - Run user code in separate namespaces in isolated containers |
| 17 | + - Use network policies to limit data exfiltration |
| 18 | +- Developer-friendliness: the AnvilOps development environment should be very easy to set up, and as much of the setup as possible should be automated. Currently, once Tilt is installed and a local K8s cluster is running, it's just `tilt up` plus a few required environment variables. |
| 19 | + |
| 20 | +### Non-goals |
| 21 | + |
| 22 | +- Lock users into the platform with nonstandard configurations or special features that don't map to standard Kubernetes configurations |
| 23 | +- Make everyone's use case work on the platform - AnvilOps is designed to fit _most_ use cases, which are typically quite simple |
| 24 | + |
| 25 | +## Common Tasks |
| 26 | + |
| 27 | +As a maintainer, you should periodically perform the following tasks: |
| 28 | + |
| 29 | +### Update Railpack |
| 30 | + |
| 31 | +Railpack is the system that AnvilOps uses to build users' applications without Dockerfiles. It chooses a build template based on the files in the repo and generates build instructions that BuildKit can use to build an image. |
| 32 | + |
| 33 | +As it improves support for languages and frameworks, it should be updated to give users those benefits. |
| 34 | + |
| 35 | +You can check the current version by [visiting the project page on GitHub](https://github.com/railwayapp/railpack/releases) or via the GitHub CLI: |
| 36 | + |
| 37 | +```sh |
| 38 | +gh release view --repo railwayapp/railpack --json tagName --jq '.tagName' |
| 39 | +``` |
| 40 | + |
| 41 | +Copy the current version number to the first line of `builders/railpack/Dockerfile`. Before pushing the change to `main`, make sure a Railpack build works as expected from the UI. |
| 42 | + |
| 43 | +Note: Railpack requires a few container images to execute the builds that it configures. AnvilOps vendors these images during a release, so the version number in that Dockerfile is used to pull the latest Railpack images from GitHub Packages and push them to the same place that the AnvilOps images live. See the `copy_railpack_images` function in `.github/workflows/release/release.sh`. If Railpack pushes a change that modifies the way these images are used, the logic there or in `builders/railpack/docker-entrypoint.sh` may need to be adjusted. |
| 44 | + |
| 45 | +### Address Security Alerts |
| 46 | + |
| 47 | +On every push, we have Trivy check the repository for security issues in a GitHub Actions workflow. If any are found, the check will fail. |
| 48 | + |
| 49 | +Most of these issues come from Node.js packages, and they usually can be addressed via `npm audit fix`: |
| 50 | + |
| 51 | +```sh |
| 52 | +cd frontend |
| 53 | +npm audit fix |
| 54 | + |
| 55 | +cd ../backend |
| 56 | +npm audit fix |
| 57 | + |
| 58 | +cd ./regclient-napi |
| 59 | +npm audit fix |
| 60 | + |
| 61 | +cd ../../filebrowser |
| 62 | +npm audit fix |
| 63 | + |
| 64 | +# These last three are just used to generate TypeScript types and HTML, so it's highly unlikely that detected vulnerabilities can actually be exploited |
| 65 | +cd ../openapi |
| 66 | +npm audit fix |
| 67 | + |
| 68 | +cd ../swagger-ui |
| 69 | +npm audit fix |
| 70 | + |
| 71 | +cd ../docs |
| 72 | +npm audit fix |
| 73 | +``` |
| 74 | + |
| 75 | +Of course, most of these security issues are in code paths that AnvilOps doesn't exercise, but it's still a good practice to find 0 CVEs on a security scan. |
| 76 | + |
| 77 | +## Updates We're Waiting On |
| 78 | + |
| 79 | +Here are a few things that we'd like to do, but we're waiting on external packages to update first: |
| 80 | + |
| 81 | +### Node.js 26 |
| 82 | + |
| 83 | +Node 26 will become LTS (long-term support) in October 2026. When this happens, you can: |
| 84 | + |
| 85 | +- Remove the `es-arraybuffer-base64` package (which is a polyfill for `Uint8Array.prototype.toBase64`) and the relevant portion of `backend/patches/@fishballpkg+acme+0.15.0.patch`. This patch was created with `patch-package`, so the best way to edit it is by editing the package source code and running `npx patch-package @fishballpkg/acme`. |
| 86 | +- Consider adopting the new `node:ffi` module instead of using Node-API for the `regclient-napi` package. This could make the integration a lot simpler and remove the reliance on a C++ compiler (although, [Go's ABI](https://go.dev/src/cmd/compile/abi-internal) is unstable, so CGo will probably still be required). |
| 87 | + |
| 88 | +### Oxlint and Oxfmt |
| 89 | + |
| 90 | +We currently use ESLint and Prettier for linting and formatting. These tools are standard, but Oxlint and Oxfmt (see their [website](https://oxc.rs/)) have recently become generally available. Switching to Oxlint should improve CI pipeline times, and switching to Oxfmt should make auto-formatting a bit faster. |
| 91 | + |
| 92 | +Oxlint should be easy to adopt except the `eslint-plugin-boundaries`, which doesn't natively support it yet. |
| 93 | + |
| 94 | +### TypeScript 7 |
| 95 | + |
| 96 | +Be sure to upgrade to TypeScript 7 when it's released. The team rewrote the compiler in Go for a ~10x speedup, which will shave a few seconds off image builds. |
| 97 | + |
| 98 | +### Rancher >2.12 |
| 99 | + |
| 100 | +Since version 2.12, Rancher can be used as a standalone OIDC provider, so the logic mapping OAuth users to Rancher users can be removed (see `getPrincipalIdValue` in `src/service/auth.ts`). Rancher 2.12 was released in July 2025, but Purdue's Anvil and Geddes clusters hasn't adopted it yet, so we still need to maintain support for a separate auth provider. When Rancher is upgraded on the Purdue clusters, everything related to CILogon can be removed, and Rancher can be the exclusive auth provider for AnvilOps. |
| 101 | + |
| 102 | +### Kubernetes 1.36 |
| 103 | + |
| 104 | +Kubernetes 1.36 enables user namespaces by default (see [this article](https://kubernetes.io/blog/2026/04/23/kubernetes-v1-36-userns-ga/) for more info). When it becomes feasible (i.e. when the Linux kernel and Kubernetes are updated and the host machines are configured properly), user namespaces should be enabled by default for the BuildKit deployment that AnvilOps ships with. This should significantly reduce the attack surface after a container escape vulnerability is exploited. |
| 105 | + |
| 106 | +Kubernetes 1.36 also enables image volumes by default. This means the log shipper can be loaded from an image volume instead of using an `initContainer` to copy the binary into a shared volume with the user's main container. See `log-shipper/README.md` and `backend/src/service/common/cluster/resources/logs.ts`. |
| 107 | + |
| 108 | +## Testing and Static Analysis |
| 109 | + |
| 110 | +On every push, a script runs in a GitHub Actions workflow to lint all of the subprojects in the repository (`.github/workflows/lint/lint.sh`). This check should pass on a pull request before it is merged, and any new linters should be added to this script. It runs `shellcheck` for shell script, `golangci-lint` for the log shipper and regclient-napi packages, `hadolint` for Dockerfiles, and ESLint for the frontend and backend. |
| 111 | + |
| 112 | +On the backend, we have `vitest` set up for unit tests; however, we're at basically 0% coverage. As you add new features or fix bugs, it would be wise to add tests that would have failed before your change and succeed after implementing your change. These tests can use a PGlite database (an in-memory version of Postgres). Every call to `backend/test/util/db.ts` creates a new instance of the common `Database` interface that's backed by a new, empty database, perfect for testing. |
| 113 | + |
| 114 | +The Dockerfiles also run `tsc`, the TypeScript type checker. On the backend, we started with a pretty loose TypeScript configuration that allowed many things that should have caused type errors. Eventually, `strictNullChecks` in `backend/tsconfig.json` should be set to `true`, and the type errors it causes should be fixed. |
| 115 | + |
| 116 | +If you want to set up end-to-end tests, consider running them in Tilt with [`tilt ci`](https://docs.tilt.dev/ci.html). You can rely on Tilt to set up the environment, and then create a K8s `Job` that runs a custom image that uses `vitest` to run some tests (make sure the `Job` is only created when the user wants to run the tests, and that test dependencies never make it into the main image). Then, you can create a GitHub Actions workflow to run `tilt ci` on pushes and pull requests. This is a great way to smoke-test the whole system before deploying a change. If any of the resources fails (including the testing Job), then `tilt ci` will exit with a nonzero status code, which can be used to mark a CI check as failed. |
| 117 | + |
| 118 | +I (@FluxCapacitor2) started this process in [#7](https://github.com/PurdueRCAC/AnvilOps/pull/7/commits), but it was abandoned because other feature work was more important at the time. |
| 119 | + |
| 120 | +## OpenAPI |
| 121 | + |
| 122 | +AnvilOps uses an OpenAPI specification to define the expected request and response types for every API endpoint. It's located at `openapi/openapi.yaml`. |
| 123 | + |
| 124 | +Whenever you modify it, you need to run `npm run generate` in the `openapi` directory (you may need to run `npm install` first). This will regenerate the TypeScript types based on the new contents of the specification. |
| 125 | + |
| 126 | +## Environments |
| 127 | + |
| 128 | +### Local (Tilt) |
| 129 | + |
| 130 | +When you're developing AnvilOps, you'll be running it in Tilt. Tilt is a tool that makes it easy to sync local changes to a Kubernetes cluster while handling things like image builds and configuration changes. Follow the instructions in [tilt/README.md](./tilt/README.md) to get started. Once your Tilt environment is ready, making a change to a source file should automatically trigger a rebuild and restart. |
| 131 | + |
| 132 | +### Releases |
| 133 | + |
| 134 | +This repository has a script to create a release of AnvilOps. A release includes all the container images and Helm charts required to deploy AnvilOps, including a vendored version of Railpack. To run these scripts, you'll need to set some environment variables, which are documented in a comment at the top of the file. After the script finishes publishing the release, it'll print out some installation instructions, which are also added to the GitHub release if one is created. |
| 135 | + |
| 136 | +## How do I... |
| 137 | + |
| 138 | +### ...Add a new API endpoint? |
| 139 | + |
| 140 | +Follow [these instructions](./backend/README.md#adding-a-new-api-handler) in backend/README.md. |
| 141 | + |
| 142 | +### ...Add a new page on the frontend? |
| 143 | + |
| 144 | +We're using React Router. The routes are registered in `frontend/src/App.tsx`. |
| 145 | + |
| 146 | +When you add a new page, you'll probably need to fetch some data. Check out the query client (the `api` constant) in `frontend/src/lib/api.ts`. It provides a type-safe way to access the API based on the generated TypeScript types that come from the OpenAPI spec. |
| 147 | + |
| 148 | +### ...Modify the K8s manifest generation? |
| 149 | + |
| 150 | +Look at: |
| 151 | + |
| 152 | +- `backend/src/service/common/cluster/resources.ts`: Generating the manifests |
| 153 | +- `backend/src/service/common/deployment.ts`: Invoking the manifest generation, invoking the K8s client to apply them to the cluster |
| 154 | +- `backend/src/service/common/cluster/kubernetes.ts`: The Kubernetes API client |
| 155 | + |
| 156 | +### ...View API docs? |
| 157 | + |
| 158 | +Visit `/openapi` on a production deployment of AnvilOps, or start up the Tilt environment and run `npm run dev` from the `swagger-ui` directory. |
| 159 | + |
| 160 | +This is Swagger UI, which is auto-generated API documentation from our OpenAPI spec. The specification itself is located at `openapi/openapi.yaml`. |
| 161 | + |
| 162 | +## Backend: Architectural Boundaries |
| 163 | + |
| 164 | +On the backend, we've drawn some boundaries to keep the codebase less tangled and easier to test. Check out [the relevant section in backend/README.md](./backend/README.md#project-structure) for more details. |
| 165 | + |
| 166 | +These boundaries are enforced using the `eslint-plugin-boundaries` ESLint plugin. This is checked on every push with the lint GitHub Actions workflow. You may need to modify these boundaries for specific edge cases. You can do so from `backend/eslint.config.js`. |
| 167 | + |
| 168 | +## Areas for Improvement |
| 169 | + |
| 170 | +Here are a few things that I view as flaws with AnvilOps. They are good starting points for new development. |
| 171 | + |
| 172 | +1. AnvilOps can only provide hostnames for services that use HTTP. It doesn't support TCP or UDP services. |
| 173 | + |
| 174 | + - Users may want this for things like databases, cache services, file servers, or other specialized apps. |
| 175 | + - Currently, users need to work around this by creating a LoadBalancer Service manually. |
| 176 | + |
| 177 | +2. AnvilOps pushes all user images to the same Harbor project. |
| 178 | + |
| 179 | + - This means an AnvilOps user could run an image created by another AnvilOps user if they knew the full tag name. |
| 180 | + - We allowed this because we wanted to avoid a dependency on a particular image registry (e.g. Purdue uses Harbor). However, the increase in security is probably worth the decrease in portability. |
| 181 | + |
| 182 | +3. AnvilOps does not encrypt in-cluster traffic between the API server and the pods it creates. |
| 183 | + |
| 184 | + - We did this for simplicity; generating and rotating certificates adds extra moving parts to a deployment, and we didn't want to rely on an external operator like cert-manager. This should probably be an opt-in feature. |
| 185 | + - Traffic between the ingress controller and the end user (e.g. someone using the dashboard) _is_ encrypted; this is referring to connections between the API server and the log exporter (`log-shipper/main.go`) and builders (`builder/*/docker-entrypoint.sh`) |
| 186 | + |
| 187 | +4. AnvilOps does not allow users to change the startup command or attach to their containers to execute commands |
| 188 | + |
| 189 | + - These features would make it much easier for users to debug their apps and perform tasks like running database migrations. Users can work around these by building a custom image with a specific startup command and by running `kubectl exec` manually, respectively. |
| 190 | + |
| 191 | +5. AnvilOps does not create preview environments from GitHub pull requests |
| 192 | + |
| 193 | + - Many PaaS providers watch for pull requests and create a copy of the production environment that deploys the pull request's contents. Then, when the pull request is closed or merged, the associated infrastructure is deleted. This is a nice convenience. Users can work around this by creating a new app for each branch and deleting it manually. |
| 194 | + |
| 195 | +6. AnvilOps does not allow users to deploy from a tarball or set of files that aren't in a GitHub repository |
| 196 | + |
| 197 | + - Users with more advanced needs may want to upload files directly to AnvilOps and then have the platform handle the image build, push, and deployment. This use-case currently isn't supported. They can work around this by building the image themselves, but they need access to a container registry to host their images. |
| 198 | + |
| 199 | +7. AnvilOps does not allow users to edit volume configurations after an app has been created |
| 200 | + |
| 201 | + - If a user needs to change the size of their volumes or add new ones, they need to create a new app and handle the data migration manually. |
| 202 | + - Editing volumes is tricky in Kubernetes. Decreasing a volume's size seems to be impossible, and increasing it [is supported with limitations](https://kubernetes.io/blog/2022/05/05/volume-expansion-ga/). |
0 commit comments