Is there a Desktop Job Scheduler for MacOS

New to AiiDA
1

Hello everyone,

I have written a workchain that uses software on my local computer(Apple M1, macOS Ventura 13.5.2). I intend to use this to run workchains on multiple files in parallel, which would lead to many running instances of the local software, so I would like to use a scheduler that isn’t direct.

I have been having trouble finding a scheduler available to submit these calculations to. Does anyone have any advice on schedulers for Mac (with available plugins ideally)

Thanks,

2

The direct scheduler does not work?

3

I think the issue is that once you start running a bunch of work chains in parallel on your local system, the CPU’s get overwhelmed quickly? As far as I know you can’t limit the number of active processes on a computer.

It seems you can install slurm with brew, but that might be overkill and a lot of work. It might also be an interesting use case of the HyperQueue scheduler, for which there is a plugin, but I’d have to try that out myself to see how (or if) it works.

If you can run one or a couple work chains together without issue, you could also consider writing a submission controller, so you could e.g. always keep two active work chains. Let me know if you need help with that, the documentation is still lacking.

4

Seems like the submission controller will be a good option for me to explore. I’ll see about getting that implemented, thanks!

1 Like
5

Great, let me know if you need any help! A few notes:

  1. I’ve made some more recent changes locally (see here). I’ll try to integrate these quickly and do a release if you think they’re useful.
  2. I haven’t updated the QE example yet since moving to pydantic, apologies. Below is a rough outline of what a submission controller with the changes in [1] should look like (pseudo-code):
from aiida import orm
from aiida_submission_controller import FromGroupSubmissionController

from your_module import YourWorkChain


class YourWorkChainController(FromGroupSubmissionController):
    """A SubmissionController for submitting `YourWorkChain`s."""
    code: str
    other_parameter: int

    def get_inputs_and_processclass_from_extras(self, extras_values, dry_run=False):
        """Return inputs and process class for the submission of this specific process."""
        parent_node = self.get_parent_node_from_extras(extras_values)

        builder = YourWorkChain.get_builder()

        # Provide the inputs for the builder based on the `parent_node` and controller settings.

        return builder

You can then submit a batch of work chains using:

your_controller = YourWorkChainController(
    unique_extra_keys=('unique_id'),
    code=orm.load_code('your_code@localhost'),
    group_label='your/workchain/group',  # label of the group you want to put your work chains in
    max_concurrent=max_concurrent, 
    parent_group_label='parent/group/label',  # label of the parent group with e.g. `StructureData`
)
supercon_controller.submit_new_batch(verbose=True)
6

For the lightweight scheduler that easy to run and setup for local machine, as mention, hyperqueue can be a nice solution. Check Implementing the Flux scheduler from LLNL - #4 by jusong.yu for details.