Adapt Release and RelEng documentation to reworked release automation

Update the main preparation PR to include all remaining steps and to
include further instructions. That PR in this repository then also acts
as replacement for the issue tracking the preparation.

Author: HannesWell

JenkinsJobs/Builds/dockerImages.jenkinsfile

@@ -1,4 +1,4 @@
-// Constants read by 
+// Constants read by job creation
 private static final _JOB_DESCRIPTION = 'Build and publish custom Docker images'
 
 pipeline {

JenkinsJobs/Builds/markBuild.jenkinsfile

@@ -1,4 +1,4 @@
-// Constants read by 
+// Constants read by job creation
 private static final _JOB_DESCRIPTION = 'Mark a build as stable/unstable or to (not to) be kept indefinitely.'
 
 pipeline {

JenkinsJobs/Releng/prepareNextDevCycle.jenkinsfile

@@ -348,10 +348,24 @@ pipeline {
 					def prBranch = "prepare-R${NEXT_RELEASE_VERSION}"
 					def aggregatorPreparationPR = githubAPI.createPullRequest('eclipse-platform/eclipse.platform.releng.aggregator', prHeadline, """\
 						Prepare development of Eclipse ${NEXT_RELEASE_VERSION}.
-						This includes:
+						
+						This includes, but is not limited to:
 						- Updating the version of the Maven parent, all references to it and the Eclipse products to `${NEXT_RELEASE_VERSION}`
 						- Updating the release version to `${NEXT_RELEASE_VERSION}` across build scripts
 						- Updating the previous release version to the current Release-Candidate: `${PREVIOUS_RELEASE_CANDIDATE_ID}`
+						
+						Further tasks:
+						- [ ] Review and submit all PRs created in the submodules to prepare them for the new release cycle.
+						  These PRs are all linked to this one.
+						- [ ] Review and submit the new N&N pages created for the [eclipse website](https://github.com/eclipse-platform/www.eclipse.org-eclipse/pulls) repository.
+						- [ ] Submit this PR.
+						  Before the preparation PRs in all submodules have been submitted the build of this change cannot complete (this can still submit it before).
+						- [ ] Update build calendar for platform ${NEXT_RELEASE_VERSION} release cycle.
+						
+						After this and all submodule PRs are submitted:
+						- [ ] Run the [`Create Jobs`](https://ci.eclipse.org/releng/job/CreateJobs/) seed job to create the Jenkins jobs of the new stream
+						- [ ] Start the new [`I-build-${NEXT_RELEASE_VERSION}`](https://ci.eclipse.org/releng/job/Builds/job/I-build-${NEXT_RELEASE_VERSION}/) job for the first I-build of the new stream
+						  - In case the new I-build job is not sufficiently stable, disable it temporarily until it's ready for the usual schedule
 						""".stripIndent(), prBranch)
 
 					utilities.forEachGitSubmodule{ submodulePath ->

JenkinsJobs/seedJob.jenkinsfile

@@ -89,7 +89,7 @@ pipeline {
 									if (jobDef.parameters != null && !jobDef.parameters.isEmpty()) {
 										parameters {
 											for (parameter in jobDef.parameters) {
-												switch(parameter.type) {
+												switch (parameter.type) {
 													case 'string':
 														stringParam {
 															name(parameter.name)

RELEASE.md

@@ -1,54 +1,75 @@
 # Jenkins
 
-## **Job Creation with Job DSL**
+## Job Creation with Job DSL
 
-**Create Jobs**
+### Create Jobs
 
-The [Create Jobs](https://ci.eclipse.org/releng/job/CreateJobs/) job is used to populate the jenkins subfolders with the jobs defined in [JenkinsJobs](JenkinsJobs) groovy files.
-There are 2 Process Job DSLs steps, the first looks for FOLDER.groovy files and creates the folders, the second creates the jobs themselves.
+The [Create Jobs](https://ci.eclipse.org/releng/job/CreateJobs/) job is used to populate the Jenkins subfolders with the jobs defined in [JenkinsJobs](JenkinsJobs).
+It is defined in [`JenkinsJobs/seedJob.jenkinsfile`](JenkinsJobs/seedJob.jenkinsfile).
 
-Create Jobs *must be run manually*. Unfortunately JobDSL needs to be run by a specific user, so the build cannot be automatically started by a timer or when it detects jenkins changes without installing an additional plugin like [Authorize Project](https://plugins.jenkins.io/authorize-project/)) which supposedly still works but is abandoned and I (Sam) have not had time to investigate further or find alternatives. This means that while any committer can make changes to the Jenkins Jobs in git, someone with Jenkins rights will have to start the build to implement those changes.
+Create Jobs *must be run manually*.
+Unfortunately JobDSL needs to be run by a specific user, so the build cannot be automatically started by a timer or when it detects jenkins changes without installing an additional Plug-in (like [Authorize Project](https://plugins.jenkins.io/authorize-project/)) which supposedly still works but is abandoned.
+This means that while any committer can make changes to the Jenkins Jobs in git, someone with Jenkins permissions will have to start the build to implement those changes.
 
 Obsolete jobs have to be deleted manually.
 They are not deleted automatically as some may be are still in use for a short time (e.g. Y-builds if a java-release is imminent) or to serve as reference in case the new jobs have problems.
 
-**The JenkinsJobs Folder**
+### The JenkinsJobs Folder
 
 As a general rule, the [JenkinsJobs](./JenkinsJobs) folder should match the layout of Jenkins itself.
-Every subfolder should contain one `.jenkinsfile` per individual pipeline. 
-
-Note: JobDSL does also support the creation of Views, so those can be added at some point but someone will either have to manually keep them up to date with the desired jobs or write a script to keep them up-to-date.
-
-Many jobs have the release version in the job name because it is required for results parsing (i.e. the [automated tests](https://ci.eclipse.org/releng/job/AutomatedTests/)).
-In order to minimize manual labor the currently active streams are listed in the [buildConfigurations.json](JenkinsJobs/buildConfigurations.json) file.
-Version dependent jobs then parse this file during creation and make a job for each stream.
+Every subfolder contains a `.jenkinsfile` file for each individual job.
+The folders containing the individual job definitions are replicated in the Jenkins instance. 
+Labels and descriptions of the folders implied by that structure can be specified in the [Seed Job](JenkinsJobs/seedJob.jenkinsfile):
+https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/9a0ef15799eb3420d748a54d8719ca08c3c7b5af/JenkinsJobs/seedJob.jenkinsfile#L19-L26
+
+Multiple build and test jobs have the release version in the job name because it is required for result parsing, e.g. the [automated tests](https://ci.eclipse.org/releng/job/AutomatedTests/).
+In order to minimize constant adjustments of the job definitions the active streams are configured through the [buildConfigurations.json](JenkinsJobs/buildConfigurations.json) file.
+This file is used to create the versioned build and test jobs and is also read by these jobs during their execution for detailed configuration.
+
+### New Jobs
+
+Adding a job to Jenkins should be as easy as adding a new `.jenkinsfile` file to git, but here are some general points to consider.
+
+- Use **only CamelCase in filenames**, no spaces or dashes (it breaks JobDSL).
+- The job Names/IDs (used in the job URL) are the same as the names of the `.jenkinsfile` files, without `.jenkinsfile` extension.
+- The display names are derived from that filenames by inserting spaces before each uppercase letter.
+  - The generated display name can be overwritten by a static `_JOB_DISPLAY_NAME` field.
+- Similarly the job's description can be defined using a static `_JOB_DESCRIPTION` field.
+  
+* **Script Formatting** You can use `'''`triple apostrophes`'''` to define multiline scripts. It's best to append a back-slash to prevent a blank line at the top of the script step in Jenkins.
+* To see what plugins are installed and what methods are available in the Releng Jenkins you can consult the [api viewer](https://ci.eclipse.org/releng/plugin/job-dsl/api-viewer/index.html).
 
-**New Jobs**
+### Testing new Jobs or trying out changes before production
 
-Adding a job to Jenkins should be as easy as adding a new groovy file to git, but here are some general pointers.
+Just like regular code, new changes or changes on jobs should be tested in a 'testing' environment before being used in production.
+As we don't have a dedicated testing environment, the regular [RelEng Jeninks](https://ci.eclipse.org/releng/) has to be used,
+but one can use a 'testing' copy of the final job that doesn't have any effect on the production environment.
+A testing job has to be created manually in the Jenkins UI and will only exist temporarily for the time of testing.
+It's recommended to put your name in the job's name to help others identifying the responsible person, in case you forgot to clean it up later.
+In the testing job you can use another source for the Jenkins file, for example your fork of this repo and another branch.
 
-* **No dashes in filenames**, it breaks JobDSL. If there was a `-` in the job name I changed it to `_` in the file name.
-* **Job Names**
-  - No spaces, this is just for the good of scripts etc. The job NAME is used in the url, but you can always add a `displayName` field if you want the job to appear with a more natural looking name on the page. See (Releng/deployToMaven)[JenkinsJobs/Releng/FOLDER.groovy] as an example.
-  - The folder needs to be part of the job name, otherwise Jenkins will just create the job at the dashboard level.
-* **job vs pipelineJob**
-  - Anything that can build on a regular jenkins node or via an available lable uses the [job](https://jenkinsci.github.io/job-dsl-plugin/#path/job) template.
-  - Anything that defines its own kubernetes pod needs to be a [pipelineJob](https://jenkinsci.github.io/job-dsl-plugin/#path/pipelineJob)
-* **Script Formatting** You can use `'''`triple apostrophes`'''` to define multiline scripts. I've found it's best to start the script on the same line and not the next, otherwise you end up with blank space at the top of the script step in Jenkins which can break the build (like the arm64 smoke tests).
-* To see what plugins are installed and what methods are available in the Releng Jenkins you can consult the [api viewer](https://ci.eclipse.org/releng/plugin/job-dsl/api-viewer/index.html).
+**Make sure your testing job does not have an effect on the production environment**
 
+'Disable' all steps that have any effect on the production environment or redirect them to a testing environment too
+- Disable `git push`. Use `--dry-run` or comment out
+- Disable modification at the download server via `ssh` or `scp` or redirect changes to a 'try-out' area, which is hidden from the public
+  - Some jobs already have a `DRY_RUN` parameter that deploy to `https://download.eclipse.org/eclipse/try-outs/`.
+- Don't send mails to mailing lists via the `emailext` step, instead just send a mail to your private mail directly.
+- Disable triggers for other jobs that may modify the production environment
 
 # Updating Java Versions
 
-Every 6 months there is a new Java release which requires additional builds and testing. The java release usually corresponds to odd numbered streams so a new java version would be tested in additional builds run during the 4.25 stream and then included in the 4.26 release I-builds. 
+Every 6 months there is a new Java release which requires additional builds and testing.
+The Java release usually corresponds to odd numbered streams so a new Java version would be tested in separated builds run during the 4.41 stream and will then be included in the main line of the 4.42 release.
 
-## **Y and P Builds**
-The **Y-build** is a full sdk build with the new java version for testing. 
+## Y and P Builds
+The **Y-build** is a full Eclipse-SDK build with the support of the new Java version included for testing.
 
-The **P-build** is a patch build that contains modified plugins designed to be installed on top of the current I-build to test the new java version.
+The **P-build** is a patch build that contains modified plugins designed to be installed on top of the current I-build to use the new java version support as early as possible.
 They are now managed by the JDT-project itself in [org.eclipse.jdt.releng](https://github.com/eclipse-jdt/eclipse.jdt/tree/master/org.eclipse.jdt.releng).
 
-The builds themselves and their unit tests are in the [build.jenkinsfile](./JenkinsJobs/Builds/build.jenkinsfile) in GitHub and the [Y Builds](https://ci.eclipse.org/releng/job/YBuilds/) folder in jenkins.
+The Y-build themselves and their unit tests are, just like the regular I-builds, defined in the [build.jenkinsfile](./JenkinsJobs/Builds/build.jenkinsfile) respectively the [integrationTests.jenkinsfile](./JenkinsJobs/AutomatedTests/integrationTests.jenkinsfile) in git
+and used for the [Y Builds](https://ci.eclipse.org/releng/job/YBuilds/) folder in Jenkins.
 
 ## Setting Up New Builds
 

RELENG.md

@@ -18,25 +18,25 @@ while [[ "$#" -gt 0 ]]; do
   esac
 done
 
-MAJOR="${STREAM%.*}"
-MINOR="${STREAM#*.}"
-
 TITLE="Release ${STREAM}"
 
 BODY="Umbrella bug to track release activities for ${STREAM}
 For previous bug please refer to eclipse-platform/eclipse.platform.releng.aggregator#${PREV_ISSUE}.
 
-
 - [ ] Readme file for ${STREAM}
-- [ ] ${STREAM} Acknowledgements
 - [ ] Migration Guide
-- [ ] SWT Javadoc bash for ${STREAM}
+- [ ] Ensure the build is cleared of comparator errors in doc bundles before RC2 build
+- [ ] Preparation of the subsequent development cycle
+  - After RC2 was promoted, run the [`Prepare Next Development Cycle`](https://ci.eclipse.org/releng/job/Releng/job/prepareNextDevCycle/) Jenkins job and complete the tasks listed in the PR created by it in this repository.
+  - [ ] Run it with final parameters as `dry-run` (ideally shortly) before and verify correctnes.
+
 - [ ] Publish ${STREAM} to Maven central
+  - The final publication should be done one or two days before the release to ensure everything is mirrored at the time of release.
 - [ ] Contribute ${STREAM} to SimRel
-- [ ] Clean up intermediate artifacts (milestones, I-builds and old releases)
-
-@notifications:
-@SDawley, @lshanmug, @SarikaSinha, @ktatavarthi, @niraj-modi
+  - Should be done one day before the release to ensure the SimRel aggregation does not contain the RC I-build repo, which is deleted on release day.
+- [ ] Submit the pending PRs to update build scripts to ${STREAM} GA on master and the ${STREAM}-maintenance
+  - Should be done one day before the release
+- [ ] Delete the old [`I-build-${STREAM}`](https://ci.eclipse.org/releng/job/Builds/) job and its associated [Test jobs](https://ci.eclipse.org/releng/job/AutomatedTests/).
 "
 
 echo "Creating Issue $TITLE"