Documentation on how to setup the Build-VM (via chef on GCP) to Development.md

.bazelrc

@@ -1,5 +1,9 @@
 # Global bazelrc file, see https://docs.bazel.build/versions/master/guide.html#bazelrc.
 
+# Use local Cache directory if building on a VM:
+# On Chef VM, create a directory and comment in the following line:
+# build --disk_cache=</tmp/bazel/> # Optional for multi-user cache: Make this directory owned by a group name e.g. "bazelcache"
+
 # Use strict action env to prevent leaks of env vars.
 build --incompatible_strict_action_env
 

DEVELOPMENT.md

@@ -10,6 +10,104 @@ This document outlines the process for setting up the development environment fo
 
 ## Setting up the Environment
 
+Decide first if you'd like a full buildsystem (on a VM) or a containerized dev environment.
+
+### VM as buildsystem
+Uses a Ubuntu 24.04 as base to run `chef` to setup all dependencies.
+The initial compilation is CPU intense and `16vcpu` are recommended.
+This was tested on GCP: a balanced disk of 500 GB and a VM type that supports nested virtualization should be chosen, as of writing (May 2025)
+`n2-standard-16` works well. 
+
+> [!Warning]
+>  The first build takes several hours and at least 160 Gb of space
+
+Turn on nested virtualization and avoid the use of `spot` VMs for the first build to avoid the very long first
+build interrupting. If you create the VMs as templates from an image, you can later switch to more cost-effective `spot` instances.
+
+
+
+```yaml
+advancedMachineFeatures:
+  enableNestedVirtualization: true
+```
+
+1) Install chef and some dependencies
+
+```
+sudo apt update sudo apt install -y git coreutils mkcert libnss3-tools screen libvirt-daemon-system libvirt-clients qemu-kvm virt-manager
+curl -L https://chefdownload-community.chef.io/install.sh | sudo bash
+```
+Now, on this VM, clone pixie (or your fork of it)
+
+```
+git clone https://github.com/pixie-io/pixie.git
+cd pixie/tools/chef
+sudo chef-solo -c solo.rb -j node_workstation.json
+sudo usermod -aG libvirt $USER
+```
+
+Make permanent the env loading via your bashrc
+```sh
+echo "source /opt/px_dev/pxenv.inc " >> ~/.bashrc
+```
+
+Put the baselrc into your homedir:
+```sh
+cp .bazelrc ~/.
+```
+In order to very significantly speed up your work, you may opt for a local cache directory. This can be shared between users of the VM, if both are part of the same group.
+Create a cache dir under <directory-path> like /tmp/bazel
+```sh
+sudo groupadd bazelcache
+sudo usermod -aG bazelcache $USER
+sudo mkdir -p <directory-path>
+sudo chown :bazelcache <directory-path>
+sudo chmod 2775 <directory-path>
+```
+
+2) Create/Use a registry you control and login
+
+```sh
+docker login ghcr.io/<myregistry>
+```
+
+3) Make Minikube run and deploy a vanilla pixie
+
+If you added your user to the libvirt group (`sudo usermod -aG libvirt $USER`), this will now work (if you did this interactively: you need to refresh your group membership, e.g. by logout/login)
+```sh
+make dev-env-start
+```
+Deploy vanilla pixie (remote cloud)
+```sh
+sudo bash -c "$(curl -fsSL https://getcosmic.ai/install.sh)"
+export PX_CLOUD_ADDR=getcosmic.ai
+px auth
+px deploy -p=1Gi
+```
+For reference and further information https://docs.px.dev/installing-pixie/install-guides/hosted-pixie/cosmic-cloud
+
+4) Once you make changes to the source code, or switch to another source code version, use Skaffold to deploy (after you have the vanilla setup working on minikube)
+
+Check, that your docker login token is still valid:
+
+```sh
+> skaffold run -f skaffold/skaffold_vizier.yaml -p x86_64_sysroot --default-repo=ghcr.io/<myregistry>
+```
+
+Optional: you can set default-repo on config, so that you don't need to pass it as an argument everytime
+```sh
+> skaffold config set default-repo ghcr.io/<myregistry> 
+> skaffold run -f skaffold/skaffold_vizier.yaml -p x86_64_sysroot
+```
+
+5) Golden image
+
+Once all the above is working and the first cache has been build, bake an image of your VM for safekeeping.
+
+
+
+
+### Containerized Devenv
 To set up the developer environment required to start building Pixie's components, run the `run_docker.sh` script. The following script will run the Docker container and dump you out inside the docker container console from which you can run all the necessary tools to build, test, and deploy Pixie in development mode.
 
 1. Since this script runs a Docker container, you must have Docker installed. To install it follow these instructions [here](https://docs.docker.com/get-docker/).

skaffold/skaffold_vizier.yaml

@@ -144,3 +144,4 @@ profiles:
     path: /build/artifacts/context=./bazel/args
     value:
     - --config=x86_64_sysroot
+    - --compilation_mode=opt