Skip to content

fix(ci): harden prod Jenkins deployment pipelines#752

Merged
yashkrishan merged 3 commits into
mainfrom
clean/backend-jenkins-p1
May 27, 2026
Merged

fix(ci): harden prod Jenkins deployment pipelines#752
yashkrishan merged 3 commits into
mainfrom
clean/backend-jenkins-p1

Conversation

@yashkrishan
Copy link
Copy Markdown
Collaborator

@yashkrishan yashkrishan commented Apr 27, 2026

Fix environment variable typos, remove hardcoded workspace paths, add preflight checks, and ensure rollback commands correctly interpolate namespace values.

Summary by CodeRabbit

  • Bug Fixes

    • Corrected environment variable handling in production pipelines
    • Fixed parameter interpolation in rollback commands
    • Improved branch validation with clearer error messaging
  • Chores

    • Added preflight checks to fail early on missing workspace or Dockerfile
    • Standardized Docker build context handling across production pipelines
  • Tests

    • Added quality gates: dependency sync, linting, and unit/integration test stages

Fix environment variable typos, remove hardcoded workspace paths, add preflight checks, and ensure rollback commands correctly interpolate namespace values.

Made-with: Cursor
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 27, 2026

Walkthrough

Three production Jenkins pipelines were updated to fix branch-to-environment mappings, add early preflight checks for workspace/Dockerfile presence, change Docker build context to ${env.WORKSPACE}, and ensure ${params.namespace} interpolates in rollback commands.

Changes

Cohort / File(s) Summary
Production Jenkinsfiles
deployment/prod/celery/Jenkinsfile_CELERY_Prod, deployment/prod/convo-server/Jenkinsfile_Convo_Prod, deployment/prod/mom-api/Jenkinsfile_API_Prod
Fixed misspelled env assignment for origin/main, removed/adjusted branch mappings, tightened unknown-branch error text to list allowed refs, added Quality Gates and Preflight Checks stages (validate workspace + Dockerfile, run linters/tests), changed Docker build context to ${env.WORKSPACE}, and switched kubectl rollout undo quoting to enable ${params.namespace} interpolation.

Sequence Diagram(s)

sequenceDiagram
  participant Dev as Developer/Branch
  participant Jenkins as Jenkins Pipeline
  participant FS as Workspace (env.WORKSPACE)
  participant Docker as Docker Build
  participant Registry as Image Registry
  participant K8s as Kubernetes (kubectl)

  Dev->>Jenkins: push (origin/main / origin/dir-restruct)
  Jenkins->>Jenkins: Quality Gates (uv, ruff, tests)
  Jenkins->>FS: Preflight Checks (workspace & Dockerfile)
  alt preflight OK
    Jenkins->>Docker: docker build context=${env.WORKSPACE}
    Docker->>Registry: push image
    Jenkins->>K8s: kubectl apply / rollout restart
  else preflight fails
    Jenkins->>Jenkins: fail pipeline early
  end
  Note over Jenkins,K8s: rollback uses interpolated "${params.namespace}"
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Suggested reviewers

  • dhirenmathur
  • ASCE-D

Poem

🐰 I nudged the pipeline with a twitch and hop,

Checked workspace bounds before the build would stop,
Context now points where the workspace gleams,
Rollbacks whisper namespaces in their dreams,
CI carrots lined up—hop, the checks won't flop.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly and concisely describes the main objective of the changeset: hardening production Jenkins deployment pipelines through bug fixes, validation improvements, and parameter interpolation corrections.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch clean/backend-jenkins-p1

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
Review rate limit: 7/8 reviews remaining, refill in 7 minutes and 30 seconds.

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
deployment/prod/mom-api/Jenkinsfile_API_Prod (1)

50-59: Optional: test -d "${env.WORKSPACE}" is largely a no-op.

Jenkins guarantees the workspace directory exists before any sh step can run (the shell itself is launched with WORKSPACE as cwd), so this assertion will essentially never fire. The test -f against api.Dockerfile is the valuable check. If you want the preflight to catch real issues, consider also asserting on other files the build relies on (e.g. dependency manifests copied by the Dockerfile) so a partial/dirty workspace fails fast here rather than several minutes into the Docker build.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@deployment/prod/mom-api/Jenkinsfile_API_Prod` around lines 50 - 59, The
Preflight Checks stage currently runs a redundant test -d "${env.WORKSPACE}" and
only verifies api.Dockerfile; remove the no-op workspace existence check and
expand the check in the 'Preflight Checks' stage to assert presence of other
build-critical files referenced by the Dockerfile (e.g., dependency manifests,
package.json, go.mod, requirements.txt or any files copied in the Dockerfile) by
replacing or augmenting the test -f
"${env.WORKSPACE}/deployment/prod/mom-api/api.Dockerfile" line with additional
test -f checks for those specific files so a missing/partial workspace fails
fast before Docker build.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@deployment/prod/convo-server/Jenkinsfile_Convo_Prod`:
- Around line 27-30: The Jenkinsfile currently permits branch == "origin/devops"
and sets env.ENVIRONMENT = 'devops' even though other prod pipelines
(Jenkinsfile_API_Prod, Jenkinsfile_CELERY_Prod) do not and env.ENVIRONMENT is
never used; remove the devops branch allowance to match the other prod pipelines
by deleting the else if (branch == "origin/devops") block and the corresponding
env.ENVIRONMENT assignment, leaving the error(...) fallback intact;
alternatively, if allowing devops was intentional, add the same branch check and
env.ENVIRONMENT handling to Jenkinsfile_API_Prod and Jenkinsfile_CELERY_Prod and
ensure downstream usage of env.ENVIRONMENT or document why prod deploy access
from origin/devops is required.

---

Nitpick comments:
In `@deployment/prod/mom-api/Jenkinsfile_API_Prod`:
- Around line 50-59: The Preflight Checks stage currently runs a redundant test
-d "${env.WORKSPACE}" and only verifies api.Dockerfile; remove the no-op
workspace existence check and expand the check in the 'Preflight Checks' stage
to assert presence of other build-critical files referenced by the Dockerfile
(e.g., dependency manifests, package.json, go.mod, requirements.txt or any files
copied in the Dockerfile) by replacing or augmenting the test -f
"${env.WORKSPACE}/deployment/prod/mom-api/api.Dockerfile" line with additional
test -f checks for those specific files so a missing/partial workspace fails
fast before Docker build.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: eaa589fc-e304-4a9e-bc88-5d0dae40b70a

📥 Commits

Reviewing files that changed from the base of the PR and between 438636f and 7ad6044.

📒 Files selected for processing (3)
  • deployment/prod/celery/Jenkinsfile_CELERY_Prod
  • deployment/prod/convo-server/Jenkinsfile_Convo_Prod
  • deployment/prod/mom-api/Jenkinsfile_API_Prod

Comment thread deployment/prod/convo-server/Jenkinsfile_Convo_Prod Outdated
Remove origin/devops branch allowance from convo prod Jenkinsfile to match prod branch policy used by API and Celery pipelines.

Made-with: Cursor
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
deployment/prod/convo-server/Jenkinsfile_Convo_Prod (1)

50-59: Preflight stage is fine; consider trimming the redundant workspace check and adding a clearer failure message.

test -d "${env.WORKSPACE}" is effectively a tautology — Jenkins won't reach this step if the workspace doesn't exist. The meaningful check is the Dockerfile presence. With the current bare test -f, a failure surfaces only as a non‑zero exit code in the console; an explicit error message will make triage faster.

♻️ Optional refinement
         stage('Preflight Checks') {
             steps {
                 script {
-                    sh """
-                    test -d "${env.WORKSPACE}"
-                    test -f "${env.WORKSPACE}/deployment/prod/convo-server/convo.Dockerfile"
-                    """
+                    sh """
+                    set -eu
+                    DOCKERFILE="${env.WORKSPACE}/deployment/prod/convo-server/convo.Dockerfile"
+                    if [ ! -f "\$DOCKERFILE" ]; then
+                        echo "Preflight failed: Dockerfile not found at \$DOCKERFILE" >&2
+                        exit 1
+                    fi
+                    """
                 }
             }
         }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@deployment/prod/convo-server/Jenkinsfile_Convo_Prod` around lines 50 - 59,
Remove the redundant workspace existence test in the Preflight Checks stage and
replace the bare Dockerfile test with an explicit check that fails with a clear
error message: inside the script block where sh is run (stage 'Preflight Checks'
in Jenkinsfile_Convo_Prod), remove or omit the line testing "${env.WORKSPACE}"
and change the Dockerfile check for
"${env.WORKSPACE}/deployment/prod/convo-server/convo.Dockerfile" to a
conditional that prints a descriptive error (e.g., "ERROR: convo.Dockerfile not
found at <path>") and exits non‑zero so the failure is obvious in logs; keep the
check focused on the file path and ensure messages reference convo.Dockerfile
and env.WORKSPACE so it's easy to locate.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@deployment/prod/convo-server/Jenkinsfile_Convo_Prod`:
- Around line 50-59: Remove the redundant workspace existence test in the
Preflight Checks stage and replace the bare Dockerfile test with an explicit
check that fails with a clear error message: inside the script block where sh is
run (stage 'Preflight Checks' in Jenkinsfile_Convo_Prod), remove or omit the
line testing "${env.WORKSPACE}" and change the Dockerfile check for
"${env.WORKSPACE}/deployment/prod/convo-server/convo.Dockerfile" to a
conditional that prints a descriptive error (e.g., "ERROR: convo.Dockerfile not
found at <path>") and exits non‑zero so the failure is obvious in logs; keep the
check focused on the file path and ensure messages reference convo.Dockerfile
and env.WORKSPACE so it's easy to locate.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d3a1e5c5-db65-411e-8131-de8f8f318820

📥 Commits

Reviewing files that changed from the base of the PR and between 7ad6044 and d39a401.

📒 Files selected for processing (1)
  • deployment/prod/convo-server/Jenkinsfile_Convo_Prod

Run dependency sync, ruff reporting, and unit/integration test gates before prod image builds across backend deployment pipelines.

Made-with: Cursor
@sonarqubecloud
Copy link
Copy Markdown

@yashkrishan
Copy link
Copy Markdown
Collaborator Author

CI/CD hardening update:

  • Added Jenkins Quality Gates stage before prod image build/push for API, conversation server, and celery prod pipelines.
  • Blocking gates now run dependency sync plus unit and integration tests with SKIP_REAL_PARSE=1.
  • Ruff currently runs in report-only mode because the existing backend baseline has many pre-existing ruff findings; making it blocking would break prod deploys until a separate ruff cleanup lands.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@deployment/prod/mom-api/Jenkinsfile_API_Prod`:
- Line 155: The shell invocation using params.namespace in the sh call (sh
"kubectl rollout undo deployment/mom-api-deployment -n ${params.namespace}") is
vulnerable to command injection; add server-side validation/sanitization before
interpolation: validate params.namespace against a strict Kubernetes namespace
regex/whitelist (e.g. /^[a-z0-9]([-a-z0-9]*[a-z0-9])?$/), fail the build or set
a safe default if it doesn't match, assign the validated value to a local
variable (e.g., validatedNamespace) and use that variable in the sh invocation
instead of raw params.namespace to ensure only safe namespace names are passed
to kubectl.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0fc37e4a-99c6-4eac-bf54-c6701b8d8d55

📥 Commits

Reviewing files that changed from the base of the PR and between d39a401 and b0a3ba2.

📒 Files selected for processing (3)
  • deployment/prod/celery/Jenkinsfile_CELERY_Prod
  • deployment/prod/convo-server/Jenkinsfile_Convo_Prod
  • deployment/prod/mom-api/Jenkinsfile_API_Prod
🚧 Files skipped from review as they are similar to previous changes (2)
  • deployment/prod/celery/Jenkinsfile_CELERY_Prod
  • deployment/prod/convo-server/Jenkinsfile_Convo_Prod

Comment thread deployment/prod/mom-api/Jenkinsfile_API_Prod
Comment thread deployment/prod/convo-server/Jenkinsfile_Convo_Prod
Comment thread deployment/prod/convo-server/Jenkinsfile_Convo_Prod
@yashkrishan yashkrishan merged commit 0c5c7ec into main May 27, 2026
3 checks passed
@yashkrishan yashkrishan deleted the clean/backend-jenkins-p1 branch May 27, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants