Skip to content

Commit 253c025

Browse files
authored
Preprocessing v3.0.0 release
- module updates - faster bclconvert output handling - modified output structure, featuring qc reports per library and enhanced run QC - fastq output when aligner equals false, with falco report - per sample analysis definition for more flexibility
2 parents e3021ca + 6981bdc commit 253c025

308 files changed

Lines changed: 15006 additions & 9084 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.devcontainer/devcontainer.json

Lines changed: 14 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,28 +1,20 @@
11
{
22
"name": "nfcore",
3-
"image": "nfcore/gitpod:latest",
4-
"remoteUser": "gitpod",
5-
"runArgs": ["--privileged"],
3+
"image": "nfcore/devcontainer:latest",
64

7-
// Configure tool-specific properties.
8-
"customizations": {
9-
// Configure properties specific to VS Code.
10-
"vscode": {
11-
// Set *default* container specific settings.json values on container create.
12-
"settings": {
13-
"python.defaultInterpreterPath": "/opt/conda/bin/python",
14-
"python.linting.enabled": true,
15-
"python.linting.pylintEnabled": true,
16-
"python.formatting.autopep8Path": "/opt/conda/bin/autopep8",
17-
"python.formatting.yapfPath": "/opt/conda/bin/yapf",
18-
"python.linting.flake8Path": "/opt/conda/bin/flake8",
19-
"python.linting.pycodestylePath": "/opt/conda/bin/pycodestyle",
20-
"python.linting.pydocstylePath": "/opt/conda/bin/pydocstyle",
21-
"python.linting.pylintPath": "/opt/conda/bin/pylint"
22-
},
5+
"remoteUser": "root",
6+
"privileged": true,
237

24-
// Add the IDs of extensions you want installed when the container is created.
25-
"extensions": ["ms-python.python", "ms-python.vscode-pylance", "nf-core.nf-core-extensionpack"]
26-
}
8+
"remoteEnv": {
9+
// Workspace path on the host for mounting with docker-outside-of-docker
10+
"LOCAL_WORKSPACE_FOLDER": "${localWorkspaceFolder}"
11+
},
12+
13+
"onCreateCommand": "./.devcontainer/setup.sh",
14+
15+
"hostRequirements": {
16+
"cpus": 4,
17+
"memory": "16gb",
18+
"storage": "32gb"
2719
}
2820
}

.devcontainer/setup.sh

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#!/usr/bin/env bash
2+
3+
# Customise the terminal command prompt
4+
echo "export PROMPT_DIRTRIM=2" >> $HOME/.bashrc
5+
echo "export PS1='\[\e[3;36m\]\w ->\[\e[0m\\] '" >> $HOME/.bashrc
6+
export PROMPT_DIRTRIM=2
7+
export PS1='\[\e[3;36m\]\w ->\[\e[0m\\] '
8+
9+
# Update Nextflow
10+
nextflow self-update
11+
12+
# Update welcome message
13+
echo "Welcome to the nf-cmgg/preprocessing devcontainer!" > /usr/local/etc/vscode-dev-containers/first-run-notice.txt

.github/CONTRIBUTING.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ If you wish to contribute a new step, please use the following coding standards:
7171
5. Add any new parameters to `nextflow_schema.json` with help text (via the `nf-core pipelines schema build` tool).
7272
6. Add sanity checks and validation for all relevant parameters.
7373
7. Perform local tests to validate that the new code works as expected.
74-
8. If applicable, add a new test command in `.github/workflow/ci.yml`.
74+
8. If applicable, add a new test in the `tests` directory.
7575
9. Update MultiQC config `assets/multiqc_config.yml` so relevant suffixes, file name clean up and module plots are in the appropriate order. If applicable, add a [MultiQC](https://https://multiqc.info/) module.
7676
10. Add a description of the output files and if relevant any appropriate images from the MultiQC report to `docs/output.md`.
7777

.github/ISSUE_TEMPLATE/bug_report.yml

Lines changed: 9 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -13,42 +13,29 @@ body:
1313
id: command_used
1414
attributes:
1515
label: Command used and terminal output
16-
description: Steps to reproduce the behaviour. Please paste the command you used
17-
to launch the pipeline and the output from your terminal.
16+
description: Steps to reproduce the behaviour. Please paste the command you used to launch the pipeline and the output from your terminal.
1817
render: console
19-
placeholder: "$ nextflow run ...
20-
18+
placeholder: |
19+
$ nextflow run ...
2120
2221
Some output where something broke
2322
24-
"
2523
- type: textarea
2624
id: files
2725
attributes:
2826
label: Relevant files
29-
description: "Please drag and drop the relevant files here. Create a `.zip` archive
30-
if the extension is not allowed.
31-
32-
Your verbose log file `.nextflow.log` is often useful _(this is a hidden file
33-
in the directory where you launched the pipeline)_ as well as custom Nextflow
34-
configuration files.
27+
description: |
28+
Please drag and drop the relevant files here. Create a `.zip` archive if the extension is not allowed.
29+
Your verbose log file `.nextflow.log` is often useful _(this is a hidden file in the directory where you launched the pipeline)_ as well as custom Nextflow configuration files.
3530
36-
"
3731
- type: textarea
3832
id: system
3933
attributes:
4034
label: System information
41-
description: "* Nextflow version _(eg. 23.04.0)_
42-
35+
description: |
36+
* Nextflow version _(eg. 23.04.0)_
4337
* Hardware _(eg. HPC, Desktop, Cloud)_
44-
4538
* Executor _(eg. slurm, local, awsbatch)_
46-
47-
* Container engine: _(e.g. Docker, Singularity, Conda, Podman, Shifter, Charliecloud,
48-
or Apptainer)_
49-
39+
* Container engine: _(e.g. Docker, Singularity, Podman, Shifter, Charliecloud, or Apptainer)_
5040
* OS _(eg. CentOS Linux, macOS, Linux Mint)_
51-
5241
* Version of nf-cmgg/preprocessing _(eg. 1.1, 1.5, 1.8.2)_
53-
54-
"

.github/PULL_REQUEST_TEMPLATE.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,14 @@ These are the most common things requested on pull requests (PRs).
88
99
Remember that PRs should be made against the dev branch, unless you're preparing a pipeline release.
1010
11-
Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-cmgg/preprocessing/tree/master/.github/CONTRIBUTING.md)
11+
Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-cmgg/preprocessing/tree/main/.github/CONTRIBUTING.md)
1212
-->
1313

1414
## PR checklist
1515

1616
- [ ] This comment contains a description of changes (with reason).
1717
- [ ] If you've fixed a bug or added code that should be tested, add tests!
18-
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-cmgg/preprocessing/tree/master/.github/CONTRIBUTING.md)
18+
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-cmgg/preprocessing/tree/main/.github/CONTRIBUTING.md)
1919
- [ ] Make sure your code lints (`nf-core pipelines lint`).
2020
- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
2121
- [ ] Check for unexpected warnings in debug mode (`nextflow run . -profile debug,test,docker --outdir <OUTDIR>`).
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
name: "Get number of shards"
2+
description: "Get the number of nf-test shards for the current CI job"
3+
inputs:
4+
max_shards:
5+
description: "Maximum number of shards allowed"
6+
required: true
7+
paths:
8+
description: "Component paths to test"
9+
required: false
10+
tags:
11+
description: "Tags to pass as argument for nf-test --tag parameter"
12+
required: false
13+
outputs:
14+
shard:
15+
description: "Array of shard numbers"
16+
value: ${{ steps.shards.outputs.shard }}
17+
total_shards:
18+
description: "Total number of shards"
19+
value: ${{ steps.shards.outputs.total_shards }}
20+
runs:
21+
using: "composite"
22+
steps:
23+
- name: Install nf-test
24+
uses: nf-core/setup-nf-test@v1
25+
with:
26+
version: ${{ env.NFT_VER }}
27+
- name: Get number of shards
28+
id: shards
29+
shell: bash
30+
run: |
31+
# Run nf-test with dynamic parameter
32+
nftest_output=$(nf-test test \
33+
--profile +docker \
34+
$(if [ -n "${{ inputs.tags }}" ]; then echo "--tag ${{ inputs.tags }}"; fi) \
35+
--dry-run \
36+
--ci \
37+
--changed-since HEAD^) || {
38+
echo "nf-test command failed with exit code $?"
39+
echo "Full output: $nftest_output"
40+
exit 1
41+
}
42+
echo "nf-test dry-run output: $nftest_output"
43+
44+
# Default values for shard and total_shards
45+
shard="[]"
46+
total_shards=0
47+
48+
# Check if there are related tests
49+
if echo "$nftest_output" | grep -q 'No tests to execute'; then
50+
echo "No related tests found."
51+
else
52+
# Extract the number of related tests
53+
number_of_shards=$(echo "$nftest_output" | sed -n 's|.*Executed \([0-9]*\) tests.*|\1|p')
54+
if [[ -n "$number_of_shards" && "$number_of_shards" -gt 0 ]]; then
55+
shards_to_run=$(( $number_of_shards < ${{ inputs.max_shards }} ? $number_of_shards : ${{ inputs.max_shards }} ))
56+
shard=$(seq 1 "$shards_to_run" | jq -R . | jq -c -s .)
57+
total_shards="$shards_to_run"
58+
else
59+
echo "Unexpected output format. Falling back to default values."
60+
fi
61+
fi
62+
63+
# Write to GitHub Actions outputs
64+
echo "shard=$shard" >> $GITHUB_OUTPUT
65+
echo "total_shards=$total_shards" >> $GITHUB_OUTPUT
66+
67+
# Debugging output
68+
echo "Final shard array: $shard"
69+
echo "Total number of shards: $total_shards"

.github/actions/nf-test/action.yml

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
name: "nf-test Action"
2+
description: "Runs nf-test with common setup steps"
3+
inputs:
4+
profile:
5+
description: "Profile to use"
6+
required: true
7+
shard:
8+
description: "Shard number for this CI job"
9+
required: true
10+
total_shards:
11+
description: "Total number of test shards(NOT the total number of matrix jobs)"
12+
required: true
13+
paths:
14+
description: "Test paths"
15+
required: true
16+
tags:
17+
description: "Tags to pass as argument for nf-test --tag parameter"
18+
required: false
19+
runs:
20+
using: "composite"
21+
steps:
22+
- name: Setup Nextflow
23+
uses: nf-core/setup-nextflow@v2
24+
with:
25+
version: "${{ env.NXF_VERSION }}"
26+
27+
- name: Set up Python
28+
uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6
29+
with:
30+
python-version: "3.14"
31+
32+
- name: Install nf-test
33+
uses: nf-core/setup-nf-test@v1
34+
with:
35+
version: "${{ env.NFT_VER }}"
36+
install-pdiff: true
37+
38+
- name: Setup apptainer
39+
if: contains(inputs.profile, 'singularity')
40+
uses: eWaterCycle/setup-apptainer@main
41+
42+
- name: Set up Singularity
43+
if: contains(inputs.profile, 'singularity')
44+
shell: bash
45+
run: |
46+
mkdir -p $NXF_SINGULARITY_CACHEDIR
47+
mkdir -p $NXF_SINGULARITY_LIBRARYDIR
48+
49+
- name: Run nf-test
50+
shell: bash
51+
env:
52+
NFT_WORKDIR: ${{ env.NFT_WORKDIR }}
53+
run: |
54+
nf-test test \
55+
--profile=+${{ inputs.profile }} \
56+
$(if [ -n "${{ inputs.tags }}" ]; then echo "--tag ${{ inputs.tags }}"; fi) \
57+
--ci \
58+
--changed-since HEAD^ \
59+
--verbose \
60+
--tap=test.tap \
61+
--shard ${{ inputs.shard }}/${{ inputs.total_shards }}
62+
63+
# Save the absolute path of the test.tap file to the output
64+
echo "tap_file_path=$(realpath test.tap)" >> $GITHUB_OUTPUT
65+
66+
- name: Generate test summary
67+
if: always()
68+
shell: bash
69+
run: |
70+
# Add header if it doesn't exist (using a token file to track this)
71+
if [ ! -f ".summary_header" ]; then
72+
echo "# 🚀 nf-test results" >> $GITHUB_STEP_SUMMARY
73+
echo "" >> $GITHUB_STEP_SUMMARY
74+
echo "| Status | Test Name | Profile | Shard |" >> $GITHUB_STEP_SUMMARY
75+
echo "|:------:|-----------|---------|-------|" >> $GITHUB_STEP_SUMMARY
76+
touch .summary_header
77+
fi
78+
79+
if [ -f test.tap ]; then
80+
while IFS= read -r line; do
81+
if [[ $line =~ ^ok ]]; then
82+
test_name="${line#ok }"
83+
# Remove the test number from the beginning
84+
test_name="${test_name#* }"
85+
echo "| ✅ | ${test_name} | ${{ inputs.profile }} | ${{ inputs.shard }}/${{ inputs.total_shards }} |" >> $GITHUB_STEP_SUMMARY
86+
elif [[ $line =~ ^not\ ok ]]; then
87+
test_name="${line#not ok }"
88+
# Remove the test number from the beginning
89+
test_name="${test_name#* }"
90+
echo "| ❌ | ${test_name} | ${{ inputs.profile }} | ${{ inputs.shard }}/${{ inputs.total_shards }} |" >> $GITHUB_STEP_SUMMARY
91+
fi
92+
done < test.tap
93+
else
94+
echo "| ⚠️ | No test results found | ${{ inputs.profile }} | ${{ inputs.shard }}/${{ inputs.total_shards }} |" >> $GITHUB_STEP_SUMMARY
95+
fi
96+
97+
- name: Clean up
98+
if: always()
99+
shell: bash
100+
run: |
101+
sudo rm -rf /home/ubuntu/tests/

0 commit comments

Comments
 (0)