Skip to content

F3 · Real pipeline corpus, CI workflows, versioning and staging

Done 2026-10-26 · commits b2687d8 and 0e5b97a (6 GB limit) · nf-core/demo 1.2.0 passes with Docker in a cloud session

What changed

  • tests/pipelines/real/corpus.yaml: 14 nf-core pipelines and 9 Snakemake Workflow Catalog workflows, in PR, nightly and manual tiers. Each entry records its scenario class; nf-core entries also record their minimum Nextflow version. Every entry is pinned to a release tag except snakemake-workflows/chipseq, which has no release and tracks main.

  • tests/pipelines/real/run_corpus.py:

    • clones the entry's tag with git;
    • sets process.resourceLimits for the runner (private, public, or auto for the machine);
    • runs the pipeline on its own test profile.

  • Workflows:

    • pipelines-synthetic.yml (three engine legs);
    • pipelines-real.yml (matrix from the corpus);
    • release-please.yml;
    • staging.yml: deploys over the tailnet, checks the public Funnel URL, and posts a scoped share link on the PR.

  • Specs: dev environment §10–§13 (corpus, CI table, versioning, instance verification and access).

Why

Thomas wanted real cases as well as generated ones, using the pipelines' own test datasets on GitHub CI. He also wanted versions that are easy to follow, and a public staging instance opened by a magic link.

Decision: decision log, 2026-10-26, real pipelines, versioning, staging access.

Evidence

$ python3 tests/pipelines/real/run_corpus.py --engine nextflow --name demo --limits auto --nextflow nextflow-26.10.3
PASS  nextflow  demo                                104s
$ cat limits.config
process.resourceLimits = [cpus: 2, memory: 5.GB, time: 6.h]

8 tasks completed, 0 failed (FASTQC ×3, SEQTK_TRIM ×3, COWPY, MULTIQC).

Figures and media

The corpus as run_corpus.py --list prints it:

Follow-up: a sketch of the staging mechanism (deploy over the tailnet, check through the Funnel URL, share link posted on the PR) is part of step D5's prompt.

Findings and open questions

  • nextflow run nf-core/x -r v failed through the session proxy. Nextflow reads the manifest through the GitHub API, which answered "Forbidden". Cloning with git avoids it.
  • Limits must stay below the free memory. With limits at 7 GB on an 8 GB machine, the local executor refused a 6 GB task ("avail: 5.8 GB").
  • The private-runner limit had the same problem. corpus.yaml set 7 GB for 8 GB runners. It is now 6 GB, and the T7 prompt asks to measure the memory GitHub runners really leave free [to be validated].
  • Not proven yet: the CI workflows have not run on GitHub; the session has no push access. Step T7 makes them green.