[nodejs] Aligns development and CI flows to avoid issues (#67)

Fixes ES-492.

This PR aligns the development an CI flows when it comes to where they
expect the forked dependencies to be. If they are in different places
between development and CI flows, there might be some issues with the CI
build.

This happens because even though the forked libraries might be put
wherever we want at development time, in the CI they are expected to be
inside the root folder of the repository because of limitations in the
CI runners.

The problem starts because the location of the libraries during
development will influence the contents of the `package.json` files,
which will be wrong from the perspective of the CI.

**tl;dr:** all the forked repos have to be cloned **inside** the root
folder of this repository. This should **not** be changed.

Additionally the following was done:

- Forked libraries added to `.gitignore`.
- `.envrc` (auto env-var loader) added to `.gitignore`.
- `mise.toml` (polyglot version manager configuration file) added to
`.gitignore`.
- CI runner now uses the `setup-nodejs` action.
- CI runner nodejs was upgraded to version 20 to clear up CI warnings at
build time.
- CI runner OS upgraded to Ubuntu 24.04.
- The layer build script runs an `npm cache clean --force` before doing
anything to ensure a clean starting state.
- The README has been updated with better development instructions
according to recent changes and experiences.
- `opentelemetry-propagator-aws-xray` and
`opentelemetry-propagator-aws-xray-lambda` are manually compiled, as
they are new dependencies of `opentelemetry-instrumentation-aws-lambda`.

Author: douglascamata

.github/workflows/publish-nodejs.yml

@@ -12,7 +12,7 @@ on:
 jobs:
 
   package:
-    runs-on: ubuntu-22.04
+    runs-on: ubuntu-24.04
     if: ${{ github.ref == 'refs/heads/coralogix-nodejs-autoinstrumentation' || inputs.environment == 'test' }}
     environment: ${{ inputs.environment }}
     env:
@@ -39,6 +39,9 @@ jobs:
         repository: coralogix/import-in-the-middle
         ref: refs/heads/coralogix-autoinstrumentation
         ssh-key: ${{ secrets.OPENTELEMETRY_CI_IITM_GITHUB_KEY }}
+    - uses: actions/setup-node@v4
+      with:
+        node-version: 20
     - name: Build
       run: ./ci-scripts/build_nodejs_layer.sh
     - name: Check layer size
@@ -102,4 +105,4 @@ jobs:
       env:
         LAYER_NAME: "coralogix-opentelemetry-nodejs-wrapper"
         COMPATIBLE_RUNTIMES: "nodejs16.x nodejs18.x nodejs20.x"
-      run: ./ci-scripts/publish_${{ inputs.environment }}.sh
+      run: ./ci-scripts/publish_${{ inputs.environment }}.sh

.github/workflows/release-layer-nodejs.yml

@@ -20,7 +20,7 @@ jobs:
 
       - uses: actions/setup-node@v4
         with:
-          node-version: 18
+          node-version: 20
 
       - name: Build
         run: |

.gitignore

@@ -22,3 +22,10 @@ build.toml
 tsconfig.tsbuildinfo
 
 collector/VERSION
+
+opentelemetry-js/
+opentelemetry-js-contrib-cx/
+import-in-the-middle/
+
+.envrc
+mise.toml

ci-scripts/build_nodejs_layer.sh

@@ -27,6 +27,8 @@ echo "OPENTELEMETRY_JS_PATH=$OPENTELEMETRY_JS_PATH"
 echo "IITM_PATH=$IITM_PATH"
 echo "CWD=$CWD"
 
+npm cache clean --force
+
 pushd "$OPENTELEMETRY_JS_CONTRIB_PATH" > /dev/null
 # Generate version files in opentelemetry-js-contrib
 npx lerna@6.6.2 run version:update # Newer versions have trouble with our lerna.json which contains `useWorkspaces`
@@ -40,9 +42,14 @@ npm install && npm run compile
 popd > /dev/null
 
 # Build opentelemetry-propagator-aws-xray
-# pushd $OPENTELEMETRY_JS_CONTRIB_PATH/propagators/opentelemetry-propagator-aws-xray
-# npm install && npm run compile
-# popd > /dev/null
+pushd "$OPENTELEMETRY_JS_CONTRIB_PATH/propagators/propagator-aws-xray" > /dev/null
+npm install --ignore-scripts && npm run compile
+popd > /dev/null
+
+# Build opentelemetry-propagator-aws-xray-lambda
+pushd "$OPENTELEMETRY_JS_CONTRIB_PATH/propagators/propagator-aws-xray-lambda" > /dev/null
+npm install && npm run compile
+popd > /dev/null
 
 # Build opentelemetry-propagation-utils
 pushd "$OPENTELEMETRY_JS_CONTRIB_PATH/packages/opentelemetry-propagation-utils" > /dev/null
@@ -81,6 +88,7 @@ popd > /dev/null
 pushd "$OPENTELEMETRY_JS_PATH/experimental/packages/opentelemetry-instrumentation" > /dev/null
 rm -f opentelemetry-instrumentation-*.tgz
 npm install && npm run compile && npm pack
+ls -lah opentelemetry-instrumentation-*.tgz
 popd > /dev/null
 
 # Build opentelemetry-sdk-trace-base
@@ -98,11 +106,11 @@ popd > /dev/null
 # Install forked libraries in cx-wrapper
 pushd "./nodejs/packages/cx-wrapper" > /dev/null
 npm install \
-    "${OPENTELEMETRY_JS_CONTRIB_PATH}/plugins/node/opentelemetry-instrumentation-aws-lambda/opentelemetry-instrumentation-aws-lambda-"*.tgz \
-    "${OPENTELEMETRY_JS_CONTRIB_PATH}/plugins/node/opentelemetry-instrumentation-mongodb/opentelemetry-instrumentation-mongodb-"*.tgz \
-    "${OPENTELEMETRY_JS_CONTRIB_PATH}/plugins/node/opentelemetry-instrumentation-aws-sdk/opentelemetry-instrumentation-aws-sdk-"*.tgz \
-    "${OPENTELEMETRY_JS_PATH}/experimental/packages/opentelemetry-instrumentation/opentelemetry-instrumentation-"*.tgz \
-    "${IITM_PATH}/import-in-the-middle-"*.tgz
+    "${OPENTELEMETRY_JS_CONTRIB_PATH}"/plugins/node/opentelemetry-instrumentation-aws-lambda/opentelemetry-instrumentation-aws-lambda-*.tgz \
+    "${OPENTELEMETRY_JS_CONTRIB_PATH}"/plugins/node/opentelemetry-instrumentation-mongodb/opentelemetry-instrumentation-mongodb-*.tgz \
+    "${OPENTELEMETRY_JS_CONTRIB_PATH}"/plugins/node/opentelemetry-instrumentation-aws-sdk/opentelemetry-instrumentation-aws-sdk-*.tgz \
+    "${OPENTELEMETRY_JS_PATH}"/experimental/packages/opentelemetry-instrumentation/opentelemetry-instrumentation-*.tgz \
+    "${IITM_PATH}"/import-in-the-middle-*.tgz
 popd > /dev/null
 
 # Build cx-wrapper
@@ -114,12 +122,12 @@ popd > /dev/null
 # Install libraries in layer
 pushd "./nodejs/packages/layer" > /dev/null
 npm install \
-    "${OPENTELEMETRY_JS_CONTRIB_PATH}/plugins/node/opentelemetry-instrumentation-aws-lambda/opentelemetry-instrumentation-aws-lambda-"*.tgz \
-    "${OPENTELEMETRY_JS_CONTRIB_PATH}/plugins/node/opentelemetry-instrumentation-mongodb/opentelemetry-instrumentation-mongodb-"*.tgz \
-    "${OPENTELEMETRY_JS_CONTRIB_PATH}/plugins/node/opentelemetry-instrumentation-aws-sdk/opentelemetry-instrumentation-aws-sdk-"*.tgz \
-    "${OPENTELEMETRY_JS_PATH}/experimental/packages/opentelemetry-instrumentation/opentelemetry-instrumentation-"*.tgz \
-    "${IITM_PATH}/import-in-the-middle-"*.tgz \
-    "${CWD}/nodejs/packages/cx-wrapper/cx-wrapper-"*.tgz
+    "${OPENTELEMETRY_JS_CONTRIB_PATH}"/plugins/node/opentelemetry-instrumentation-aws-lambda/opentelemetry-instrumentation-aws-lambda-*.tgz \
+    "${OPENTELEMETRY_JS_CONTRIB_PATH}"/plugins/node/opentelemetry-instrumentation-mongodb/opentelemetry-instrumentation-mongodb-*.tgz \
+    "${OPENTELEMETRY_JS_CONTRIB_PATH}"/plugins/node/opentelemetry-instrumentation-aws-sdk/opentelemetry-instrumentation-aws-sdk-*.tgz \
+    "${OPENTELEMETRY_JS_PATH}"/experimental/packages/opentelemetry-instrumentation/opentelemetry-instrumentation-*.tgz \
+    "${IITM_PATH}"/import-in-the-middle-*.tgz \
+    "${CWD}"/nodejs/packages/cx-wrapper/cx-wrapper-*.tgz
 popd > /dev/null
 
 # Install copyfiles and bestzip # used by `npm run clean/compile`
@@ -128,4 +136,5 @@ npm install -g copyfiles bestzip rimraf
 # Build layer
 pushd "./nodejs/packages/layer" > /dev/null
 npm run clean && npm install
-popd > /dev/null
+ls -lah build/layer.zip
+popd > /dev/null

dev/build-nodejs.sh

@@ -10,32 +10,32 @@ ROOT_DIR=$(git rev-parse --show-toplevel)
 
 # Expected by build_nodejs_layer.sh
 if [ -z "$OPENTELEMETRY_JS_CONTRIB_PATH" ]; then
-    export OPENTELEMETRY_JS_CONTRIB_PATH="$ROOT_DIR/../opentelemetry-js-contrib-cx"
+	export OPENTELEMETRY_JS_CONTRIB_PATH="$ROOT_DIR/opentelemetry-js-contrib-cx"
 fi
 
 if [ -z "$OPENTELEMETRY_JS_PATH" ]; then
-    export OPENTELEMETRY_JS_PATH="$ROOT_DIR/../oss/opentelemetry-js"
+	export OPENTELEMETRY_JS_PATH="$ROOT_DIR/opentelemetry-js"
 fi
 
 if [ -z "$IITM_PATH" ]; then
-    export IITM_PATH="$ROOT_DIR/../import-in-the-middle"
+	export IITM_PATH="$ROOT_DIR/import-in-the-middle"
 fi
 
 if [ ! -d "$OPENTELEMETRY_JS_CONTRIB_PATH" ]; then
-    git clone git@github.com:coralogix/opentelemetry-js-contrib.git "$OPENTELEMETRY_JS_CONTRIB_PATH" -b coralogix-autoinstrumentation
+	git clone git@github.com:coralogix/opentelemetry-js-contrib.git "$OPENTELEMETRY_JS_CONTRIB_PATH" -b coralogix-autoinstrumentation
 fi
 
 if [ ! -d "$OPENTELEMETRY_JS_PATH" ]; then
-    git clone git@github.com:coralogix/opentelemetry-js.git "$OPENTELEMETRY_JS_PATH" -b coralogix-autoinstrumentation
+	git clone git@github.com:coralogix/opentelemetry-js.git "$OPENTELEMETRY_JS_PATH" -b coralogix-autoinstrumentation
 fi
 
 if [ ! -d "$IITM_PATH" ]; then
-    git clone git@github.com:coralogix/import-in-the-middle.git "$IITM_PATH" -b coralogix-autoinstrumentation
+	git clone git@github.com:coralogix/import-in-the-middle.git "$IITM_PATH" -b coralogix-autoinstrumentation
 fi
 
 "$ROOT_DIR/ci-scripts/build_nodejs_layer.sh"
 
 # Useful for using the layer locally
-pushd "$ROOT_DIR/nodejs/packages/layer" > /dev/null
+pushd "$ROOT_DIR/nodejs/packages/layer" >/dev/null
 rm -rf ./build/layer && unzip -q ./build/layer.zip -d ./build/layer
-popd > /dev/null
+popd >/dev/null

nodejs/README.md

@@ -10,18 +10,49 @@ included and loaded automatically if you use the AWS SDK v2.
 
 ## Building
 
-To build the layer and sample applications, in this `nodejs` folder, run
+### Requirements
 
+You will need to provide the path to 3 forked dependencies through environment
+variables for the libraries below:
+
+- opentelemetry-js: `OPENTELEMETRY_JS_PATH`.
+- opentelemetry-js-contrib: `OPENTELEMETRY_JS_CONTRIB_PATH`.
+- import-in-the-middle: `IITM_PATH`.
+
+Note that these paths are very important, because they are will impact the
+relative paths in some `package.json` files in ways that could potentially
+break CI scripts.
+
+To avoid this issue we recommend setting them like so:
+
+```sh
+export OPENTELEMETRY_JS_CONTRIB_PATH=./opentelemetry-js-contrib-cx
+export OPENTELEMETRY_JS_PATH=./opentelemetry-js
+export IITM_PATH=./import-in-the-middle
 ```
-npm install
+
+This project's `.gitignore` is already configured with these folders
+to ensure your git index stays clean.
+
+### The layer
+
+To build the layer and sample applications run the command below from
+the root of the application:
+
+```sh
+./dev/build-nodejs.sh
 ```
 
-This will download all dependencies and compile all code. The layer zip file will be present at `./packages/layer/build/layer.zip`.
+This is a thin wrapper over `./ci-scripts/build_nodejs_layer.sh` that
+will clone the forked dependencies if the paths indicated by the
+previously mentioned environment variables are empty, then use
+that script to download all dependencies and compile all code.
+The layer zip file will be present at `./packages/layer/build/layer.zip`.
 
 ## Sample applications
 
 Sample applications are provided to show usage of the above layer.
 
 - Application using AWS SDK - shows using the wrapper with an application using AWS SDK without code change.
   - [Using layer built from source](./integration-tests/aws-sdk)
-  - [WIP] [Using OTel Public Layer](./sample-apps/aws-sdk) 
+  - [WIP] [Using OTel Public Layer](./sample-apps/aws-sdk)