Possibly cover qbatch about hpc-novice

[psteinb]

I am trying to revive this repo, so let me reply to this. I would vote NOT to use higher level abstractions like qbatch, snakemake and others in this lesson. While they offer a great productivity boost for those they can use them, I think they might be too overwhelming for the target audience of this lesson.

[gdevenyi]

Well, snakemake and qbatch are pretty different.

qbatch is designed explicitly as an abstraction layer to take a list of command-line calls and push them onto a cluster without knowing much about the cluster. You can write a for loop, make it echo, and pipe it to qbatch and off to the cluster it goes. snakemake looks more like a make system with cluster integration and targets and other things.

[psteinb]

Gabriel, don't get me wrong on this, from a broad perspective, I think snakemake and qbatch offer an abstraction to the batch system. therefor, I don't see them becoming an integral part of this lesson. Feel free to disagree. IMO, both should go into a part of this repo dedicated to "where to go from here". I am simply quite careful to expose novice learners to a tool which they appreciate only after they know the batch system and the involved workflow to submit a lot of similar jobs or to automate submission.

[ChristinaLK]

I'll echo @psteinb here: it may seem straightforward to hand people a tool that seamlessly parallelizes their work, but I think the value of a workshop like this is to actually introduce people to some of the basics under the hood: how a cluster is structured, the paradigm of job submission, what parallel submission means, how to interact with a scheduler, etc.

I think there's a fine line between "removing unnecessary complexity" and "introducing a different version of abstraction." :P

It's a cool tool though! Here at UW Madison we actually have a dedicated system for independently parallel jobs (the situation where you might use qbatch) and it's amenable to many workflows.

[gdevenyi]

Okay.