# Is the scheduler error out of walltime actually working?

**URL:** <https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233>\
**Category:** General Usage\
**Created:** [January 30, 2024, 3:23pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233 "2024-01-30T15:23:40Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [January 30, 2024, 3:23pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/1 "2024-01-30T15:23:40Z")

</div>

Hi everyone,  
I have an implementation of a Parser that should be able to catch the “built-in” `CalcJob.exit_codes.ERROR_SCHEDULER_OUT_OF_WALLTIME `and just return that error. My pytests are showing that my implementation works, provided the `ERROR_SCHEDULER_OUT_OF_WALLTIME` is detected.  
Nevertheless, on a real test CalcJob, running out of walltime on purpose, it is not catching the scheduler OOW error.

This is the report of the CalcJob:

```auto
*** 3952595: None
*** Scheduler output:
[...]

*** Scheduler errors:
[...]
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: ***JOB 5476661 ON gcn2101 CANCELLED AT 2024-01-30T13:34:08 DUE TO TIME LIMIT***
slurmstepd: error: ***STEP 5476661.2 ON gcn2101 CANCELLED AT 2024-01-30T13:34:08 DUE TO TIME LIMIT***
[...]

```

Any idea why it is happening? I am using aiida-core 2.4.0post0 (a branch from Sebastiaan P. Huber).

---

<div class="post-metadata">

**Author:** ![danielhollas](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/danielhollas/32/4_2.png) [@danielhollas](https://aiida.discourse.group/u/danielhollas)\
**Post date:** [January 30, 2024, 3:54pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/2 "2024-01-30T15:54:17Z")

</div>

It would be helpful to see the code of the parser.

The scheduler errors in the report seems like they are expected. How exactly are you getting the report? What is the exit code of the Calcjob?

---

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [January 30, 2024, 4:42pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/3 "2024-01-30T16:42:18Z")

</div>

This is the `parse` method of the Parser:

```auto
    def parse(self, **kwargs):
        """Parse the contents of the output files retrieved in the `FolderData`."""
        logs = get_logging_container()

        _, parsed_data, logs = self.parse_stdout_from_retrieved(logs)

        base_exit_code = self.check_base_errors(logs)
        if base_exit_code:
            return self.exit(base_exit_code, logs)

        self.add_units(parsed_data)
        self.out('parameters', orm.Dict(parsed_data))

        # First check whether the scheduler already reported an exit code.
        if self.node.exit_status is not None:

            # The following scheduler errors should correspond to cases where we can simply restart the calculation
            # and have a chance that the calculation will succeed as the error can be transient.
            recoverable_scheduler_error = self.node.exit_status in [
                FlareCalculation.exit_codes.ERROR_SCHEDULER_OUT_OF_WALLTIME.status,
                FlareCalculation.exit_codes.ERROR_SCHEDULER_NODE_FAILURE.status,
            ]

            if recoverable_scheduler_error:
                # Now it is unlikely we can provide a more specific exit code so we keep the scheduler one.
                return ExitCode(self.node.exit_status, self.node.exit_message)

        if 'retrieved_temporary_folder' in kwargs: # might not be there?
            exit_code = self.parse_model(kwargs['retrieved_temporary_folder'])
            if exit_code in logs.error:
                return self.exit(self.exit_codes.get(exit_code), logs)

            if not self.is_fake:
                settings = {}
                if 'settings' in self.node.inputs:
                    settings = _lowercase_dict(self.node.inputs.settings.get_dict(), dict_name='settings')
                units = settings.get('units', FlareCalculation._default_units)
                self.parse_dft_trajectory(kwargs['retrieved_temporary_folder'], units)

        for exit_code in list(self.get_error_map().values()) + ['ERROR_OUTPUT_STDOUT_INCOMPLETE']:
            if exit_code in logs.error:
                return self.exit(self.exit_codes.get(exit_code), logs)

        return self.exit(logs=logs)

```

This is the full log:

```auto
*** 3952595: None
*** Scheduler output:
================================================================================
JobID = 5476661
User = hbiblore, Account = hbiblore
Partition = standard96, Nodelist = gcn2101
================================================================================
============ Job Information ===================================================
Submitted: 2024-01-30T12:33:38
Started: 2024-01-30T12:33:39
Ended: 2024-01-30T13:34:11
Elapsed: 61 min, Limit: 60 min, Difference: -1 min
CPUs: 192, Nodes: 1
Estimated NPL: 14.1087
================================================================================

*** Scheduler errors:
Module for Anaconda3 2020.11 loaded.
Module for Intel Compilers and Libraries 2018.5 loaded.
Module for Intel(R) MPI Library 2018.4 loaded.
Module for hdf5 (Version 1.10.5) loaded.
Module for GCC (version 9.3.0) loaded.
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: ***JOB 5476661 ON gcn2101 CANCELLED AT 2024-01-30T13:34:08 DUE TO TIME LIMIT***
slurmstepd: error: ***STEP 5476661.2 ON gcn2101 CANCELLED AT 2024-01-30T13:34:08 DUE TO TIME LIMIT***
/scratch/usr/hbiblore/.conda/envs/flare/lib/python3.8/site-packages/requests/ __init__.py:89: RequestsDependencyWarning: urllib3 (2.0.4) or chardet (3.0.4) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
/scratch/usr/hbiblore/.conda/envs/flare/lib/python3.8/site-packages/ase/md/md.py:48: FutureWarning: Specify the temperature in K using the 'temperature_K' argument
  warnings.warn(FutureWarning(w))

*** 3 LOG MESSAGES:
+-> ERROR at 2024-01-30 13:36:34.332489+01:00
 | ERROR_OUTPUT_STDOUT_INCOMPLETE
+-> ERROR at 2024-01-30 13:36:34.336965+01:00
 | The stdout output file was incomplete probably because the calculation got interrupted.
+-> WARNING at 2024-01-30 13:36:34.339480+01:00
 | output parser returned exit code<312>: The stdout output file was incomplete probably because the calculation got interrupted.

```

---

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [January 30, 2024, 4:45pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/4 "2024-01-30T16:45:45Z")

</div>

Ah, sorry, my bad, I was testing with a “not-interrupted” output… Just noticed I will emit a `base_exit_code` in first place if detected, and the interrupted output is catched first. Sorry for wasting your time.

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [January 30, 2024, 7:02pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/5 "2024-01-30T19:02:05Z")

</div>

This is indeed a bit of a pit-fall. The exit code set by the scheduler parser, if any, will be overwritten by any exit code returned by the “main” `Parser`. So the `Parser` implementation has to _explicitly_ check if an exit code is already set on the node and decide whether it wants to keep that, or override it with another one. I agree that this is not immediately obvious but I wasn’t sure how else to do it, apart from starting to allow multiple exit codes, but I rejected that since that would have had a lot of consequences in other parts of the code.

---

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [January 30, 2024, 10:56pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/6 "2024-01-30T22:56:09Z")

</div>

Thanks for the reply. I actually recalled that `self.check_base_errors(logs)` doesn’t check for the interrupted stdout, hence, it seems indeed that `self.node.exit_status` is simply `None`, although the scheduler was OOW. So, it is not clear to me where’s the pitfall in the code.

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [January 31, 2024, 8:17am UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/7 "2024-01-31T08:17:01Z")

</div>

Did the scheduler _actually_ detect the OOW error and set the exit code? The `SlurmScheduler` plugin checks the `State` key of the detailed job info and if it is `TIMEOUT` it will return the `ERROR_SCHEDULER_OUT_OF_WALLTIME`:

```python
if data['State'] == 'TIMEOUT':
    return CalcJob.exit_codes.ERROR_SCHEDULER_OUT_OF_WALLTIME

```

Could you check the attributes of the `CalcJobNode` that ran out of walltime and report the contents here? I would like to see if the state attribute contains `TIMEOUT` or not.

---

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [January 31, 2024, 10:18am UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/8 "2024-01-31T10:18:21Z")

</div>

This is the result of `verdi node attributes <PK>`

```auto
{
    "append_text": "",
    "custom_scheduler_commands": "",
    "detailed_job_info": {
        "retval": 1,
        "stderr": "sacct: error: Invalid field requested: \"Reserved\"\n",
        "stdout": ""
    },
    "environment_variables": {},
    "environment_variables_double_quotes": false,
    "exit_message": "The stdout output file was incomplete probably because the calculation got interrupted.",
    "exit_status": 312,
    "import_sys_environment": true,
    "input_filename": "aiida.yaml",
    "job_id": "5477677",
    "last_job_info": {
        "allocated_machines_raw": "gcn2841",
        "annotation": "None",
        "dispatch_time": {
            "date": "2024-01-30T19:08:06.000000",
            "timezone": null
        },
        "job_id": "5477677",
        "job_owner": "hbiblore",
        "job_state": "running",
        "num_machines": 1,
        "num_mpiprocs": 192,
        "queue_name": "standard96",
        "raw_data": [
            "5477677",
            "R",
            "None",
            "gcn2841",
            "hbiblore",
            "1",
            "192",
            "gcn2841",
            "standard96",
            "1:00:00",
            "59:59",
            "2024-01-30T19:08:06",
            "aiida-3958716",
            "2024-01-30T19:08:05"
        ],
        "requested_wallclock_time_seconds": 3600,
        "submission_time": {
            "date": "2024-01-30T19:08:05.000000",
            "timezone": null
        },
        "title": "aiida-3958716",
        "wallclock_time_seconds": 3599
    },
    "max_wallclock_seconds": 3600,
    "metadata_inputs": {
        "metadata": {
            "call_link_label": "iteration_01",
            "dry_run": false,
            "options": {
                "append_text": "",
                "custom_scheduler_commands": "",
                "environment_variables_double_quotes": false,
                "import_sys_environment": true,
                "input_filename": "aiida.yaml",
                "max_wallclock_seconds": 3600,
                "mpirun_extra_params": [],
                "output_filename": "aiida.out",
                "parser_name": "learn.flare",
                "prepend_text": "",
                "resources": {
                    "num_cores_per_mpiproc": 96,
                    "num_machines": 1,
                    "num_mpiprocs_per_machine": 1
                },
                "scheduler_stderr": "_scheduler-stderr.txt",
                "scheduler_stdout": "_scheduler-stdout.txt",
                "submit_script_filename": "_aiidasubmit.sh",
                "withmpi": false
            },
            "store_provenance": true
        }
    },
    "mpirun_extra_params": [],
    "output_filename": "aiida.out",
    "parser_name": "learn.flare",
    "prepend_text": "",
    "process_label": "FlareCalculation",
    "process_state": "finished",
    "remote_workdir": "/scratch-emmy/projects/hbi00059/aiida_fs/7c/cb/0a13-e9cd-4d05-8910-ab9570898a20",
    "resources": {
        "num_cores_per_mpiproc": 96,
        "num_machines": 1,
        "num_mpiprocs_per_machine": 1
    },
    "retrieve_list": [
        "aiida.out",
        "_scheduler-stdout.txt",
        "_scheduler-stderr.txt"
    ],
    "retrieve_temporary_list": [
        "aiida_flare.json",
        "aiida_dft.xyz"
    ],
    "scheduler_lastchecktime": "2024-01-30T20:10:14.148966+01:00",
    "scheduler_state": "done",
    "scheduler_stderr": "_scheduler-stderr.txt",
    "scheduler_stdout": "_scheduler-stdout.txt",
    "sealed": true,
    "submit_script_filename": "_aiidasubmit.sh",
    "version": {
        "core": "2.4.0.post0",
        "plugin": "0.1.0a0"
    },
    "withmpi": false
}

```

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [January 31, 2024, 10:45am UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/9 "2024-01-31T10:45:03Z")

</div>

> [@bastonero](#):
>
> ```auto
> "detailed_job_info": {
> "retval": 1,
> "stderr": "sacct: error: Invalid field requested: \"Reserved\"\n",
> "stdout": ""
> },
> 
> ```

There is the problem. Apparently, the call to `sacct` to get the detailed job info failed. So the parser never set the `ERROR_SCHEDULER_OUT_OF_WALLTIME` exit code. Not sure why your `sacct` call is failing. What version of SLURM is installed?

---

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [January 31, 2024, 11:08am UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/10 "2024-01-31T11:08:01Z")

</div>

Annn I see, thanks! This is what is installed on the cluster I use: `slurm 23.02.7`

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [January 31, 2024, 11:30am UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/11 "2024-01-31T11:30:54Z")

</div>

It seems the `Reserved` field is no longer supported by `sacct` in the latest version: [Slurm Workload Manager - sacct](https://slurm.schedmd.com/sacct.html#OPT_helpformat)

Unfortunately, the `SlurmScheduler` hardcodes the list of fields it requests. So you would have to manually up date the source code to fix it. Or subclass the plugin and create a new entry point and use that. You should remove the `Reserved` key in the `_detailed_job_info_fields` list of `SlurmScheduler` and it should work (unless more fields have been removed).

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [January 31, 2024, 11:33am UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/12 "2024-01-31T11:33:19Z")

</div>

I think this change is the culprit:

> <https://github.com/SchedMD/slurm/blob/863ead570d450e25022f04cc5c9cfb379aa8ae4d/RELEASE_NOTES#L181C1-L182C40>

So instead of removing `Reserved` you could also rename it to `Planned`.

Ideally, the plugin should not fail if an unsupported field is requested though. Or try to determine it dynamically, but this would involve a second call to `sacct` at least.

---

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [January 31, 2024, 11:49am UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/13 "2024-01-31T11:49:06Z")

</div>

Or should it be possible to implement version-dependent fields?

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [January 31, 2024, 11:56am UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/14 "2024-01-31T11:56:25Z")

</div>

Sure, but that would still require a call to the scheduler to determine the version. But on top of that, you would still have to actively update the plugin whenever a new version changes something. The dynamic solution would hopefully be able to work for all versions without requiring updates.

---

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [January 31, 2024, 12:26pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/15 "2024-01-31T12:26:52Z")

</div>

Looking into the [`parse_output`](https://aiida.readthedocs.io/projects/aiida-core/en/latest/_modules/aiida/schedulers/plugins/slurm.html#SlurmScheduler.parse_output), the `detailed_job_info` had the `"stdout": ""`, meaning it will catch the first try/except, hence, even if the `_scheduler-stderr.txt` contains useful info, it will be ignored.

Could it be a solution to try, before raising error, to look into the `stderr` and, if nothing is found, to raise the error? At least one removes this version incompatibility.

Moreover, as a general question, why all the fileds for format are employed, when eventually only the `State` field is used?

It would be good to either implement the first or the second solutions, so that plugins building up the OOW of Slurm do not depend on the Slurm versioning, and do not rely on custom aiida-core versions.

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [January 31, 2024, 3:27pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/16 "2024-01-31T15:27:17Z")

</div>

> [@bastonero](#):
>
> Looking into the [`parse_output`](https://aiida.readthedocs.io/projects/aiida-core/en/latest/_modules/aiida/schedulers/plugins/slurm.html#SlurmScheduler.parse_output), the `detailed_job_info` had the `"stdout": ""`, meaning it will catch the first try/except, hence, even if the `_scheduler-stderr.txt` contains useful info, it will be ignored.

The problem is that I don’t want to be parsing the content of the stderr file, because that would be text parsing. I intentionally want to parse the output of `sacct` as that is structured.

> [@bastonero](#):
>
> Could it be a solution to try, before raising error, to look into the `stderr` and, if nothing is found, to raise the error? At least one removes this version incompatibility.

The problem is that you would have to match the text to an exit code, and here the text could easily vary between versions for example. It is just not as robust.

> [@bastonero](#):
>
> Moreover, as a general question, why all the fileds for format are employed, when eventually only the `State` field is used?

This is because the `get_detailed_job_info` was not added explicitly for parsing errors of failed jobs. It existed before to give a user more information about a completed job. It was just repurposed later on to parse the `State` key from to determine job failures. We assumed that these fields would not change, but that was clearly an incorrect assumption ^^

> [@bastonero](#):
>
> It would be good to either implement the first or the second solutions, so that plugins building up the OOW of Slurm do not depend on the Slurm versioning, and do not rely on custom aiida-core versions.

I agree that this would be ideal, but I don’t think the proposed solutions are the correct ones. I think we should just find a way to determine the fields dynamically. They can be determined with `sacct --helpformat`, which returns something like:

```auto
$ sacct --helpformat
Account AdminComment AllocCPUS AllocNodes         
AllocTRES AssocID AveCPU AveCPUFreq         
AveDiskRead AveDiskWrite AvePages AveRSS             
AveVMSize BlockID Cluster Comment            
Constraints ConsumedEnergy ConsumedEnergyRaw Container          
CPUTime CPUTimeRAW DBIndex DerivedExitCode    
Elapsed ElapsedRaw Eligible End                
ExitCode Flags GID Group              
JobID JobIDRaw JobName Layout    

```

The scheduler plugin provides the command to call to get the detailed job info as follows:

```python
    def _get_detailed_job_info_command(self, job_id):
        fields = ','.join(self._detailed_job_info_fields)
        return f'sacct --format={fields} --parsable --jobs={job_id}'

```

So the fields are specified by concatenating the `_detailed_job_info_fields` list items with a comma. I can replicate this with

```auto
$ sacct --helpformat | tr -s '\n' ' ' | tr ' ' ','
Account,AdminComment,AllocCPUS,AllocNodes,AllocTRES,AssocID,AveCPU,AveCPUFreq,AveDiskRead,AveDiskWrite,AvePages,AveRSS,AveVMSize,BlockID,Cluster,Comment,Constraints,ConsumedEnergy,ConsumedEnergyRaw,Container,CPUTime,CPUTimeRAW,DBIndex,DerivedExitCode,Elapsed,ElapsedRaw,Eligible,End,ExitCode,Flags,GID,Group,JobID,JobIDRaw,JobName,Layout,MaxDiskRead,MaxDiskReadNode,MaxDiskReadTask,MaxDiskWrite,MaxDiskWriteNode,MaxDiskWriteTask,MaxPages,MaxPagesNode,MaxPagesTask,MaxRSS,MaxRSSNode,MaxRSSTask,MaxVMSize,MaxVMSizeNode,MaxVMSizeTask,McsLabel,MinCPU,MinCPUNode,MinCPUTask,NCPUS,NNodes,NodeList,NTasks,Partition,Priority,QOS,QOSRAW,Reason,ReqCPUFreq,ReqCPUFreqGov,ReqCPUFreqMax,ReqCPUFreqMin,ReqCPUS,ReqMem,ReqNodes,ReqTRES,Reservation,ReservationId,Reserved,ResvCPU,ResvCPURAW,Start,State,Submit,SubmitLine,Suspended,SystemComment,SystemCPU,Timelimit,TimelimitRaw,TotalCPU,TRESUsageInAve,TRESUsageInMax,TRESUsageInMaxNode,TRESUsageInMaxTask,TRESUsageInMin,TRESUsageInMinNode,TRESUsageInMinTask,TRESUsageInTot,TRESUsageOutAve,TRESUsageOutMax,TRESUsageOutMaxNode,TRESUsageOutMaxTask,TRESUsageOutMin,TRESUsageOutMinNode,TRESUsageOutMinTask,TRESUsageOutTot,UID,User,UserCPU,WCKey,WCKeyID,WorkDir,

```

So now I am thinking we can change `_get_detailed_job_info_command` to the following:

```python
    def _get_detailed_job_info_command(self, job_id):
        return f"sacct --format=$(sacct --helpformat | tr -s '\n' ' ' | tr ' ' ',') --parsable --jobs={job_id}"

```

I just tested this for SLURM 22.05.3 and it seems to work. Maybe you could try updating your SlurmScheduler plugin and run a test calculation? It should always run this code, even when the job finishes just fine, so you don’t even need to force it to fail.

---

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [January 31, 2024, 6:06pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/17 "2024-01-31T18:06:57Z")

</div>

Thanks Sebastian, very smart solution!

I think we are almost there. The only issue now is that the `parse_output` logic still relies on the definition of `self._detailed_job_info_fields`, which would match wrongly the fields. I assume it’s easy now to just parse directly the fields from the stdout, and zip them together.

Do you see any flaw?

For example:

```auto
[...]

lines = detailed_stdout.splitlines()

try:
    master = lines[1]
except IndexError:
    raise ValueError('the `detailed_job_info.stdout` contained less than two lines.')

fields = lines[0].split('|')
attributes = master.split('|')

if len(fields) != len(attributes):
    raise ValueError(
        'second line in `detailed_job_info.stdout` differs in length with the `_detailed_job_info_fields '
        'attribute of the scheduler.'
    )

data = dict(zip(fields, attributes))

[...]

```

Here the detailed info which may useful for testing.

```auto
    "detailed_job_info": {
        "retval": 0,
        "stderr": "",
        "stdout": "Account|AdminComment|AllocCPUS|AllocNodes|AllocTRES|AssocID|AveCPU|AveCPUFreq|AveDiskRead|AveDiskWrite|AvePages|AveRSS|AveVMSize|BlockID|Cluster|Comment|Constraints|ConsumedEnergy|ConsumedEnergyRaw|Container|CPUTime|CPUTimeRAW|DBIndex|DerivedExitCode|Elapsed|ElapsedRaw|Eligible|End|ExitCode|Extra|FailedNode|Flags|GID|Group|JobID|JobIDRaw|JobName|Layout|Licenses|MaxDiskRead|MaxDiskReadNode|MaxDiskReadTask|MaxDiskWrite|MaxDiskWriteNode|MaxDiskWriteTask|MaxPages|MaxPagesNode|MaxPagesTask|MaxRSS|MaxRSSNode|MaxRSSTask|MaxVMSize|MaxVMSizeNode|MaxVMSizeTask|McsLabel|MinCPU|MinCPUNode|MinCPUTask|NCPUS|NNodes|NodeList|NTasks|Partition|Planned|PlannedCPU|PlannedCPURAW|Priority|QOS|QOSRAW|Reason|ReqCPUFreq|ReqCPUFreqGov|ReqCPUFreqMax|ReqCPUFreqMin|ReqCPUS|ReqMem|ReqNodes|ReqTRES|Reservation|ReservationId|Start|State|Submit|SubmitLine|Suspended|SystemComment|SystemCPU|Timelimit|TimelimitRaw|TotalCPU|TRESUsageInAve|TRESUsageInMax|TRESUsageInMaxNode|TRESUsageInMaxTask|TRESUsageInMin|TRESUsageInMinNode|TRESUsageInMinTask|TRESUsageInTot|TRESUsageOutAve|TRESUsageOutMax|TRESUsageOutMaxNode|TRESUsageOutMaxTask|TRESUsageOutMin|TRESUsageOutMinNode|TRESUsageOutMinTask|TRESUsageOutTot|UID|User|UserCPU|WCKey|WCKeyID|WorkDir|\nhbiblore||192|1|billing=192,cpu=192,mem=362000M,node=1|5388|||||||||ghlrn4||||||20:16:00|72960|15072155|0:0|00:06:20|380|2024-01-31T18:14:27|2024-01-31T18:20:51|0:0|||SchedMain,StartRecieved|31606|hbiblore|5484315|5484315|aiida-3970071||||||||||||||||||||||192|1|gcn2020||standard96|00:00:04|00:06:24|384|100023|normal|1|None|Unknown|Unknown|Unknown|Unknown|96|362000M|1|billing=96,cpu=96,mem=362000M,node=1|||2024-01-31T18:14:31|TIMEOUT|2024-01-31T18:14:27|sbatch _aiidasubmit.sh|00:00:00||18:13.054|00:06:00|6|05:36:12|||||||||||||||||31115|hbiblore|05:17:59|*ingenieurwissenschaften|2265|/scratch-emmy/projects/hbi00059/aiida_fs/93/62/f952-d0a0-48f0-beee-d217efb1f44b|\nhbiblore||192|1|cpu=192,mem=362000M,node=1|5388|00:00:00|2.30M|32.03M|0.16M|9|179420K|1387944K||ghlrn4|||218.84K|218839||20:19:12|73152|15072155||00:06:21|381|2024-01-31T18:14:31|2024-01-31T18:20:52|0:15||||||5484315.batch|5484315.batch|batch|Unknown||32.03M|gcn2020|0|0.16M|gcn2020|0|9|gcn2020|0|179420K|gcn2020|0|1387944K|gcn2020|0||00:00:04|gcn2020|0|192|1|gcn2020|1|||||||||0|0|0|0|192||1||||2024-01-31T18:14:31|CANCELLED|2024-01-31T18:14:31||00:00:00||00:00.145|||00:00.535|cpu=00:00:00,energy=218839,fs/disk=33585254,mem=179420K,pages=9,vmem=1387944K|cpu=00:00:04,energy=218839,fs/disk=33585254,mem=179420K,pages=9,vmem=1387944K|cpu=gcn2020,energy=gcn2020,fs/disk=gcn2020,mem=gcn2020,pages=gcn2020,vmem=gcn2020|cpu=0,fs/disk=0,mem=0,pages=0,vmem=0|cpu=00:00:04,energy=218839,fs/disk=33585254,mem=179420K,pages=9,vmem=1387944K|cpu=gcn2020,energy=gcn2020,fs/disk=gcn2020,mem=gcn2020,pages=gcn2020,vmem=gcn2020|cpu=0,fs/disk=0,mem=0,pages=0,vmem=0|cpu=00:00:00,energy=218839,fs/disk=33585254,mem=179420K,pages=9,vmem=1387944K|energy=84,fs/disk=171967|energy=936,fs/disk=171967|energy=gcn2020,fs/disk=gcn2020|fs/disk=0|energy=936,fs/disk=171967|energy=gcn2020,fs/disk=gcn2020|fs/disk=0|energy=84,fs/disk=171967|||00:00.389||||\nhbiblore||192|1|billing=192,cpu=192,mem=362000M,node=1|5388|00:00:00|2.30M|0.00M|0.00M|0|3364K|178828K||ghlrn4|||219.38K|219377||20:16:00|72960|15072155||00:06:20|380|2024-01-31T18:14:31|2024-01-31T18:20:51|0:0||||||5484315.extern|5484315.extern|extern|Unknown||0.00M|gcn2020|0|0.00M|gcn2020|0|0|gcn2020|0|3364K|gcn2020|0|178828K|gcn2020|0||00:00:00|gcn2020|0|192|1|gcn2020|1|||||||||0|0|0|0|192||1||||2024-01-31T18:14:31|COMPLETED|2024-01-31T18:14:31||00:00:00||00:00:00|||00:00.001|cpu=00:00:00,energy=219377,fs/disk=2012,mem=3364K,pages=0,vmem=178828K|cpu=00:00:00,energy=219377,fs/disk=2012,mem=3364K,pages=0,vmem=178828K|cpu=gcn2020,energy=gcn2020,fs/disk=gcn2020,mem=gcn2020,pages=gcn2020,vmem=gcn2020|cpu=0,fs/disk=0,mem=0,pages=0,vmem=0|cpu=00:00:00,energy=219377,fs/disk=2012,mem=3364K,pages=0,vmem=178828K|cpu=gcn2020,energy=gcn2020,fs/disk=gcn2020,mem=gcn2020,pages=gcn2020,vmem=gcn2020|cpu=0,fs/disk=0,mem=0,pages=0,vmem=0|cpu=00:00:00,energy=219377,fs/disk=2012,mem=3364K,pages=0,vmem=178828K|energy=84,fs/disk=1|energy=936,fs/disk=1|energy=gcn2020,fs/disk=gcn2020|fs/disk=0|energy=936,fs/disk=1|energy=gcn2020,fs/disk=gcn2020|fs/disk=0|energy=84,fs/disk=1|||00:00:00||||\nhbiblore||192|1|cpu=192,mem=362000M,node=1|5388|05:33:53|156K|2745.53M|2.33M|916|59603200K|229345388K||ghlrn4|||164.59K|164595||11:21:36|40896|15072155||00:03:33|213|2024-01-31T18:17:18|2024-01-31T18:20:51|0:0||||||5484315.0|5484315.0|pmi_proxy|Cyclic||2745.53M|gcn2020|0|2.33M|gcn2020|0|916|gcn2020|0|59603200K|gcn2020|0|229345388K|gcn2020|0||05:33:53|gcn2020|0|192|1|gcn2020|1|||||||||Unknown|Unknown|Unknown|Unknown|192||1||||2024-01-31T18:17:18|COMPLETED|2024-01-31T18:17:18|/usr/local/slurm/slurm/current/install/bin/srun --nodelist gcn2020 -N 1 -n 1 --input none /sw/comm/impi/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin/pmi_proxy --control-port gcn2020.usr.hlrn.de:43167 --pmi-connect alltoall --pmi-aggregate -s 0 --rmk slurm --launcher slurm --demux poll --pgid 0 --enable-stdin 1 --retries 10 --control-code 317447439 --usize -2 --proxy-id -1|00:00:00||18:12.907|||05:36:11|cpu=05:33:53,energy=164595,fs/disk=2878896622,mem=59603200K,pages=916,vmem=229345388K|cpu=05:33:53,energy=164595,fs/disk=2878896622,mem=59603200K,pages=916,vmem=229345388K|cpu=gcn2020,energy=gcn2020,fs/disk=gcn2020,mem=gcn2020,pages=gcn2020,vmem=gcn2020|cpu=0,fs/disk=0,mem=0,pages=0,vmem=0|cpu=05:33:53,energy=164595,fs/disk=2878896622,mem=59603200K,pages=916,vmem=229345388K|cpu=gcn2020,energy=gcn2020,fs/disk=gcn2020,mem=gcn2020,pages=gcn2020,vmem=gcn2020|cpu=0,fs/disk=0,mem=0,pages=0,vmem=0|cpu=05:33:53,energy=164595,fs/disk=2878896622,mem=59603200K,pages=916,vmem=229345388K|energy=84,fs/disk=2446026|energy=853,fs/disk=2446026|energy=gcn2020,fs/disk=gcn2020|fs/disk=0|energy=853,fs/disk=2446026|energy=gcn2020,fs/disk=gcn2020|fs/disk=0|energy=84,fs/disk=2446026|||05:17:58||||\n"
    },

```

---

<div class="post-metadata">

**Author:** ![sphuber](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/sphuber/32/6_2.png) [@sphuber](https://aiida.discourse.group/u/sphuber)\
**Post date:** [January 31, 2024, 6:43pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/18 "2024-01-31T18:43:32Z")

</div>

That makes perfect sense. I made a PR: [`SlurmScheduler`: Make detailed job info fields dynamic by sphuber · Pull Request #6270 · aiidateam/aiida-core · GitHub](https://github.com/aiidateam/aiida-core/pull/6270)

Could you check that out and give it a shot? There are tests, but only for `parse_output`, so it won’t actually test the new command string.

---

<div class="post-metadata">

**Author:** ![bastonero](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/bastonero/32/32_2.png) [@bastonero](https://aiida.discourse.group/u/bastonero)\
**Post date:** [January 31, 2024, 7:07pm UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/19 "2024-01-31T19:07:15Z")

</div>

Awesome. thanks a lot Sebastiaan. The implementation works perfectly now! The stdout is complete, and the TIMEOUT is catched nicely.

---

<div class="post-metadata">

**Author:** ![giovannipizzi](https://yyz2.discourse-cdn.com/free1/user_avatar/aiida.discourse.group/giovannipizzi/32/17_2.png) [@giovannipizzi](https://aiida.discourse.group/u/giovannipizzi)\
**Post date:** [February 1, 2024, 7:59am UTC](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233/20 "2024-02-01T07:59:14Z")

</div>

Thanks a lot to both! I added some comments in the PR to make it possibly even easier/more robust.

[Next page](https://aiida.discourse.group/t/is-the-scheduler-error-out-of-walltime-actually-working/233.md?page=2)