feat(observability): add @sourceloop/observability with OTEL-first bootstrap, profiles, and pluggable instrumentation

package.json

@@ -76,6 +76,7 @@
     "packages/cli/",
     "packages/cache/",
     "packages/feature-toggle/",
+    "packages/observability/",
     "packages/file-utils/",
     "packages/custom-sf-changelog/",
     "services/*",

packages/observability/.eslintignore

@@ -0,0 +1,2 @@
+test-enterprise-features.js
+test-vendor-agnostic.js

packages/observability/.eslintrc.js

@@ -0,0 +1,26 @@
+module.exports = {
+  extends: '@loopback/eslint-config',
+  rules: {
+    'no-extra-boolean-cast': 'off',
+    '@typescript-eslint/interface-name-prefix': 'off',
+    'no-prototype-builtins': 'off',
+    // Explicitly enable the no-console rule as an error
+    'no-console': 'error',
+    '@typescript-eslint/no-misused-promises': [
+      'error',
+      {
+        // Allows promises to be used in void return contexts (e.g., event handlers)
+        // Set to true if you want stricter checks.
+        checksVoidReturn: false,
+      },
+    ],
+  },
+  parserOptions: {
+    // Point to your tsconfig.json for typed linting
+    project: './tsconfig.json',
+    // Define the root directory for the tsconfig.json path
+    tsconfigRootDir: __dirname,
+  },
+  // Add '.eslintrc.js' to prevent ESLint from linting itself with typed rules
+  ignorePatterns: ['dist', '.eslintrc.js', 'templates'],
+};

packages/observability/.gitignore

@@ -0,0 +1,69 @@
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# Runtime data
+pids
+*.pid
+*.seed
+*.pid.lock
+
+# Directory for instrumented libs generated by jscoverage/JSCover
+lib-cov
+
+# Coverage directory used by tools like istanbul
+coverage
+
+# nyc test coverage
+.nyc_output
+
+# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
+.grunt
+
+# Bower dependency directory (https://bower.io/)
+bower_components
+
+# node-waf configuration
+.lock-wscript
+
+# Compiled binary addons (http://nodejs.org/api/addons.html)
+build/Release
+
+# Dependency directories
+node_modules/
+jspm_packages/
+
+# Typescript v1 declaration files
+typings/
+
+# Optional npm cache directory
+.npm
+
+# Optional eslint cache
+.eslintcache
+
+# Optional REPL history
+.node_repl_history
+
+# Output of 'npm pack'
+*.tgz
+
+# Yarn Integrity file
+.yarn-integrity
+
+# dotenv environment variables file
+.env
+
+# Transpiled JavaScript files from Typescript
+/dist
+
+# Cache used by TypeScript's incremental build
+*.tsbuildinfo
+mochawesome-report
+
+
+#Private key pem file
+*.pem

packages/observability/.prettierignore

@@ -0,0 +1,4 @@
+dist
+*.json
+node_modules/
+mochawesome-report/

packages/observability/.prettierrc

@@ -0,0 +1,7 @@
+{
+  "bracketSpacing": false,
+  "singleQuote": true,
+  "printWidth": 80,
+  "trailingComma": "all",
+  "arrowParens": "avoid"
+}

packages/observability/README.md

@@ -0,0 +1,249 @@
+# @sourceloop/observability
+
+OpenTelemetry-first observability bootstrap and optional LoopBack integration for Sourceloop services.
+
+## Design
+
+This package follows a hybrid model:
+
+1. **Bootstrap is the required startup path**
+   Use bootstrap before your application imports go too far, so tracing can start early.
+2. **The LoopBack component is optional**
+   Use `ObservabilityComponent` only when you want DI-bound config, framework bindings, and profile extension-point integration.
+
+That means this package should be thought of as:
+
+- `bootstrap.ts`: core runtime startup
+- `component.ts`: LoopBack adapter
+
+## What It Provides
+
+- Early env-led or override-led bootstrap for observability startup
+- DI-bound config via `ObservabilityBindings.config`
+- Built-in profiles for `default`, `newrelic`, `signoz`, and `datadog`
+- LoopBack extension-point support for custom profiles
+- Generic helpers for spans, exception recording, and propagation headers
+
+## Startup Model
+
+### Required: bootstrap
+
+Call bootstrap in your service entrypoint before application construction.
+
+```ts
+import {bootstrapObservabilityFromEnv} from '@sourceloop/observability';
+
+bootstrapObservabilityFromEnv();
+```
+
+You can also pass explicit startup config for tests or advanced bootstrap flows:
+
+```ts
+import {bootstrapObservability} from '@sourceloop/observability';
+
+bootstrapObservability({
+  enabled: true,
+  profile: 'default',
+  serviceName: 'audit-service',
+});
+```
+
+Startup-only instrumentation toggles are also configured here:
+
+```ts
+bootstrapObservability({
+  enabled: true,
+  profile: 'default',
+  serviceName: 'audit-service',
+  instrumentations: {
+    http: true,
+    express: true,
+    pg: false,
+    mysql: true,
+    redis: false,
+    kafka: false,
+  },
+});
+```
+
+### Optional: LoopBack component
+
+Use the component only if you want framework-native bindings and DI config.
+
+```ts
+import {
+  ObservabilityBindings,
+  ObservabilityComponent,
+} from '@sourceloop/observability';
+
+this.bind(ObservabilityBindings.config).to({
+  serviceVersion: '1.0.0',
+  environment: process.env.NODE_ENV,
+  resourceAttributes: {
+    'deployment.color': 'blue',
+  },
+});
+
+this.component(ObservabilityComponent);
+```
+
+Important:
+
+- Bootstrap controls startup-critical config.
+- DI config is meant for component-level enrichment, not for replacing early bootstrap.
+- Instrumentation toggles and custom instrumentations are bootstrap concerns, not DI concerns.
+- If you skip bootstrap and rely only on DI config, early instrumentation will not be enabled.
+
+## Config Resolution
+
+The package resolves config in two stages.
+
+### Bootstrap config
+
+Resolved by `resolveBootstrapConfig()`.
+
+Used for startup-critical fields such as:
+
+- `enabled`
+- `profile`
+- `serviceName`
+- exporter endpoint/protocol/headers
+- sampler
+- instrumentation toggles
+
+Precedence:
+
+1. bootstrap overrides
+2. environment variables
+3. defaults
+
+Instrumentation toggles can be set either in `instrumentations` during bootstrap or through environment variables:
+
+- `OBSERVABILITY_INSTRUMENT_HTTP`
+- `OBSERVABILITY_INSTRUMENT_EXPRESS`
+- `OBSERVABILITY_INSTRUMENT_PG`
+- `OBSERVABILITY_INSTRUMENT_MYSQL`
+- `OBSERVABILITY_INSTRUMENT_REDIS`
+- `OBSERVABILITY_INSTRUMENT_KAFKA`
+
+Set each one to `true` or `false` to enable or disable that instrumentation. `express` requires `http` to remain enabled.
+
+The same toggles can be set programmatically through `bootstrapObservability({...})` using the `instrumentations` object shown above. That is the preferred non-env path.
+
+You can also attach your own instrumentation instances during bootstrap:
+
+```ts
+import {
+  bootstrapObservability,
+  ObservabilityInstrumentation,
+} from '@sourceloop/observability';
+
+const customInstrumentation: ObservabilityInstrumentation = {
+  enable() {
+    // initialize your instrumentation
+  },
+  disable() {
+    // clean up your instrumentation
+  },
+  setTracerProvider(tracerProvider) {
+    // optional: keep the tracer provider if your instrumentation needs it
+  },
+};
+
+bootstrapObservability({
+  enabled: true,
+  profile: 'default',
+  serviceName: 'audit-service',
+  customInstrumentations: [customInstrumentation],
+});
+```
+
+Use `customInstrumentations` when you need a package-specific or in-house instrumentation that is not built into `@sourceloop/observability`.
+
+### Component config
+
+Resolved by `resolveComponentConfig()`.
+
+Used for LoopBack/application enrichment such as:
+
+- `serviceVersion`
+- `environment`
+- extra `resourceAttributes`
+
+Precedence:
+
+1. DI config
+2. bootstrap-resolved config
+
+## Profiles
+
+Built-in profiles:
+
+- `default`
+- `newrelic`
+- `signoz`
+- `datadog`
+
+`default` means plain OTLP behavior with no vendor-specific profile layered on top.
+
+These profiles are implemented by profile classes inside the package. Custom profiles can be added using the LoopBack extension point exposed by the package.
+
+## Dependencies
+
+OTLP exporters are bundled as package dependencies because OTLP is the built-in transport path for all current profiles.
+
+There are currently no vendor-native SDK dependencies in the package. The built-in `newrelic`, `signoz`, and `datadog` profiles are OTLP-oriented presets, not native agent integrations.
+
+OpenTelemetry instrumentation libraries are optional peer dependencies so consumers can cherry-pick only the instrumentations they want to use:
+
+- `@opentelemetry/instrumentation-http`
+- `@opentelemetry/instrumentation-express`
+- `@opentelemetry/instrumentation-pg`
+- `@opentelemetry/instrumentation-redis`
+- `@opentelemetry/instrumentation-kafkajs`
+- `@opentelemetry/instrumentation-mysql`
+- `@opentelemetry/instrumentation-mysql2`
+
+If an instrumentation is enabled in config but its package is not installed, bootstrap will fail fast with a clear startup error.
+
+## Public API
+
+Bootstrap/runtime:
+
+- `bootstrapObservability()`
+  Starts observability using explicit startup overrides. Use this in tests or when startup config needs to be supplied programmatically before the app is constructed.
+- `bootstrapObservabilityFromEnv()`
+  Starts observability from environment-driven configuration only. This is the standard entrypoint for service bootstrapping.
+- `shutdownObservability()`
+  Shuts down the active tracer runtime and clears internal singleton state. Use this for graceful shutdown or test cleanup.
+- `isObservabilityEnabled()`
+  Returns whether observability is currently active in the bootstrapped runtime.
+
+LoopBack:
+
+- `ObservabilityBindings`
+  Provides LoopBack binding keys for DI-bound observability config, runtime, and resolved config.
+- `ObservabilityComponent`
+  Registers the optional LoopBack integration layer, including config bindings and built-in profile bindings.
+
+Tracing helpers:
+
+- `withSpan()`
+  Runs an async function inside a new active span and closes the span automatically when the function completes.
+- `addSpanAttributes()`
+  Adds attributes to the current active span when one exists.
+- `recordException()`
+  Records an error on the current span and marks the span status as failed.
+- `createPropagationHeaders()`
+  Injects the current trace context into a plain header carrier for outbound calls.
+- `getTraceContext()`
+  Returns the current trace identifiers and propagation headers for correlation or logging.
+
+Config helpers:
+
+- `resolveBootstrapConfig()`
+  Resolves startup-critical config from explicit overrides, environment variables, and defaults.
+- `resolveComponentConfig()`
+  Resolves the LoopBack/component-level config by layering DI enrichment on top of the bootstrapped config.
+- `validateObservabilityConfig()`
+  Validates that the resolved observability config is coherent before runtime startup.