Skip to content

Commit 5ed26d3

Browse files
chore(ci): log basic details of downloaded ncbi datasets (#5513)
- Doesn't harm to have md5, file size, number of sequences/avg length etc: - Also schedule the prio 2 at 1hr offset from the regular 2hr prio 1 for more spread out load for ncbi. <img width="730" height="475" alt="image" src="https://github.com/user-attachments/assets/93620e4d-371d-4b9f-9130-b7b0b9a0fe84" /> Tested manually with one taxon id, see screenshot
1 parent b19b92e commit 5ed26d3

2 files changed

Lines changed: 6 additions & 3 deletions

File tree

.github/workflows/datasets-mirror-priority-2.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ name: Mirror NCBI Datasets (Priority 2 - Daily)
22
on:
33
workflow_dispatch: # Allows manual triggering to re-run all priority-2 taxa
44
schedule:
5-
- cron: '12 2 * * *' # Runs daily at 02:12 UTC
5+
- cron: '12 3 * * *' # Runs daily at 03:12 UTC
66
jobs:
77
mirror:
88
strategy:

.github/workflows/datasets-mirror.yml

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,16 @@ jobs:
2424
- uses: mamba-org/setup-micromamba@v2
2525
with:
2626
environment-name: datasets
27-
create-args: ncbi-datasets-cli s3cmd
28-
- name: Download NCBI Dataset
27+
create-args: -c conda-forge -c bioconda ncbi-datasets-cli s3cmd seqkit
28+
- name: Download NCBI Dataset and create tzst archive
2929
shell: bash -l {0}
3030
run: |
3131
datasets download virus genome taxon ${{ inputs.taxon_id }} --no-progressbar --filename ${{ inputs.taxon_id }}.zip
3232
unzip -o ${{ inputs.taxon_id }}.zip
3333
tar -I 'zstd -T0 -18' -cvf ${{ inputs.taxon_id }}.tar.zst ncbi_dataset
34+
cat md5sum.txt
35+
ls -lh ${{ inputs.taxon_id }}.* ncbi_dataset/data/*
36+
seqkit stats ncbi_dataset/data/genomic.fna
3437
- name: Create S3cmd config
3538
run: |
3639
cat <<EOF > ~/.s3cfg

0 commit comments

Comments
 (0)