Remember that I have been posting about the #SnakemakeHackathon2025 ?
I never really finished that series. But now, we have two late contributions by Ward Deboutte and @johanneskoester . One describing the polishing of the multiple extension handling of #Snakemake for named inputs (https://zenodo.org/records/17121446) and stabilizing the JSON validator (https://zenodo.org/records/17121551).
Cool!
The #isc25 is over and I half-recovered from the weekend, too. Time to continue my thread summing up the #SnakemakeHackathon2025 !
To me, an important contribution was from Michael Jahn from the Charpentier Lab: A complete re-design of the workflow catalogue. Have a look: https://snakemake.github.io/snakemake-workflow-catalog/ - findability of ready-to-use workflows has greatly improved! Also, the description on how to contribute is now easy to find.
A detailed description has been published in the #researchequals collection https://www.researchequals.com/collections/hm1w-cg under https://doi.org/10.5281/zenodo.15574642
#Snakemake #ReproducibleComputing #ReproducibleResearch #OpenScience
Returning from the #isc25 I will continue this thread with something applicable everywhere, not just on #HPC clusters:
Workflow runs can crash. There are a number of possible reasons. Snakemake offers a `--rerun-incomple` flag (or short `--ri`) which lets a user resume a workflow.
This contribution from Filipe G. Viera describes a small fix to stabilize the feature. Not only will incomplete files be removed after a crash, now it is ensured that all metadata with them are deleted too, before resuming: https://zenodo.org/records/15490098
#Snakemake #SnakemakeHackathon2025 #ReproducibleComputing #OpenScience
Today tooting from the #ISC25 - the International Supercomputing Conference. What better opportunity to brag about something I've done to facilitate using GPUs with Snakemake?
Here is my contribution, simpler job configuration for GPU jobs:
https://doi.org/10.5281/zenodo.15551797
Not alone though: Without valuable input of @dryak . Without him, I would have overlooked something crucial.
And when we talk about reproducible AI, my take is that we ought to consider workflow managers, too. Something which protocols what you have done with little effort.
#SnakemakeHackathon2025 #Snakemake #ReproducibleComputing #OpenScience
Before I continue uploading - and I do have a couple of more contributions to add to the #ResearchEquals collection - first another contribution by Johanna Elena Schmitz and Jens Zentgraf made at the #SnakemakeHackathon2025
One difficulty when dealing with a different scientific question: Do I need to re-invent the wheel (read: write a Workflow from scratch?) just to address my slightly different question?
Snakemake already allowed to incorporate "alien" workflows, even #Nextflow workflows, into desired workflows. The new contribution allows for a more dynamic contribution - with very little changes.
Check it out: https://zenodo.org/records/15489694
Let's take a look at another contribution of Johanna Elena Schmitz and Jens Zentgraf from the #SnakemakeHackathon2025
Snakemake users probably know that
`$ snakemake [args] --report`
will generate a self-contained HTML report. Including all plots and #metadata a researcher's heart longs for.
Now, why trigger this manually? If the workflow runs successfully, now we can write (or configure):
`$ snakemake [args] --report-after-run`
and Snakemake will autogenerate the same report.
For details see https://doi.org/10.5281/zenodo.15489764
One important feature implemented in the #SnakemakeHackathon2025 : Snakemake will calculate file checksums to detect changes. If a file changes, the rule producing it needs to be re-executed when a workflow it re-triggered. But what if a file is too big for reasonable checksum calculation? You do not what to wait forever, after all.
This contribution describes the implementation of a threshold users may set: https://doi.org/10.5281/zenodo.15489401
One important bug fix during the #SnakemakeHackathon2025 : the config replacement. Now, users can overwrite existing configurations entirely with `--replace-workflow-config`.
Details: https://zenodo.org/records/15479268
Did you know? During the #SnakemakeHackathon2025 we had a staggering 194 work items!
It took a while, but now we are gathering contribution reports and present them online as a ResearchEquals (https://fediscience.org/@ResearchEquals@akademienl.social) collection:
https://www.researchequals.com/collections/hm1w-cg
The first 10 are online and I will post some highlights in the coming weeks.
#Snakemake #ReproducibleComputing #ReproducibleResearch #OpenScience
Busy year:
- Workflow programming for Data Analysis on #HPC Systems (Course in Mainz in January): ✅
- Same Course in Dresden (February) ✅
- #SnakemakeHackathon2025 at the CERN in March: ✅
- upcoming: #OpenScience Retreat (no hashtag, yet?) in April
- International Supercomputing Conference in June (so, @boegel, I will be there, after all and hope to meet people from @irods, too ; will you be there folks from #iRODS ?)
- German Conference for #Bioinformatics and NHR Conference in September
And I do not know whether this will be all. I have a nagging feeling there is more to come 😉
The #SnakemakeHackathon2025 just induced another collaboration for me!
It is always good to work along with like-minded people, interested in #ReproducibleComputing, #HPC and #Bioinformatics
Today, the #SnakemakeHackathon2025 at @CERN has ended with the release of Snakemake 9.0! Huge thanks to the amazing participants and all organizers! In total there were 19 new releases within the ecosystem, including hundreds of new features and bug fixes for Snakemake and its plugins!
Today at the #SnakemakeHackathon2025:
We did a tour around the CERN!
But, of course, work was done, too! On cvmfs support for software deployment, many core features, improved metadata reporting, more stable SLURM executor, etc. etc. etc. Soon you will read more. 😉
The third day of the #SnakemakeHackathon2025 at @CERN is over. 11 fantastic pull requests have been merged, fixing various bugs, enabling custom logging, improving the docs, and the expressiveness of the language. https://snakemake.github.io
Today is the day of closed pull request for #Snakemake. The #SnakemakeHackathon2025 participants worked at full speed!
We decided to write a white-paper summarizing our achievements rather than posting individual things. Suffice to say, that also the documentation made a great leap towards better readability!
The second day of the #SnakemakeHackathon2025 is over. Today I have merged 11 fantastic pull requests in the in main snakemake repo alone. And we have our first #Snakemake release during the hackathon, adding integration with xonsh for scripting and helpers for extracting params from input files.
#SnakemakeHackathon2025 ! We started!
At the CERN for better #ReproducibleComputing and #ReproducibleResearch .
