Unexpected Dameon Crash

General Usage
1

Dear AiiDA developers,

I am experiencing an issue with running AiiDA v2.7.3 on my macOS computer.

I initially installed AiiDA with PostgreSQL v14.21 (using Homebrew) and RabbitMQ v4.2.4 (using via the Generic Unix Binary Build*).

When I start the server, everything appears to work correctly, and the daemon successfully submits jobs, sometimes running for several minutes. However, after some time, the daemons, which are surprisingly slow, crash. The Circus log file contains the following error:

2026-02-26 16:01:35 circus[18649] [INFO] Starting master on pid 18649
2026-02-26 16:01:35 circus[18649] [INFO] Arbiter now waiting for commands
2026-02-26 16:01:35 circus[18649] [INFO] aiida-aiida started
2026-02-26 16:01:35 circus[18649] [INFO] circusd-stats started
2026-02-26 16:01:35 circus[18651] [INFO] Starting the stats streamer
2026-02-26 16:04:33 circus[18649] [INFO] Arbiter exiting
2026-02-26 16:04:33 circus[18651] [INFO] Stats streamer stopped
2026-02-26 16:04:33 tornado.general[18651] [WARNING] Got events for stream <zmq.eventloop.zmqstream.ZMQStream object at 0x104766ea0> attached to closed socket: Socket operation on non-socket
2026-02-26 16:04:33 circus[18651] [INFO] Stats streamer stopped
2026-02-26 16:04:33 circus[18651] [INFO] Stats streamer stopped
2026-02-26 16:04:33 circus[18649] [INFO] circusd-stats stopped
2026-02-26 16:04:34 circus[18649] [INFO] aiida-aiida stopped
2026-02-26 16:04:35 circus[19070] [INFO] Starting master on pid 19070
2026-02-26 16:04:35 circus[19070] [INFO] Arbiter now waiting for commands
2026-02-26 16:04:35 circus[19070] [INFO] aiida-aiida started
2026-02-26 16:04:35 circus[19070] [INFO] circusd-stats started
2026-02-26 16:04:35 circus[19072] [INFO] Starting the stats streamer
2026-02-26 16:05:00 circus[19070] [INFO] Arbiter exiting
2026-02-26 16:05:00 circus[19072] [INFO] Stats streamer stopped
2026-02-26 16:05:00 tornado.general[19072] [WARNING] Got events for stream <zmq.eventloop.zmqstream.ZMQStream object at 0x111b33770> attached to closed socket: Socket operation on non-socket
2026-02-26 16:05:00 circus[19072] [INFO] Stats streamer stopped
2026-02-26 16:05:00 circus[19072] [INFO] Stats streamer stopped
2026-02-26 16:05:00 circus[19070] [INFO] circusd-stats stopped
2026-02-26 16:05:01 tornado.application[19070] [ERROR] Exception in callback <bound method Arbiter.manage_watchers of <circus.arbiter.Arbiter object at 0x114500b00>>
Traceback (most recent call last):
File “/Users/gjoalland/Library/Python/3.12/lib/python/site-packages/tornado/ioloop.py”, line 945, in _run
val = self.callback()
^^^^^^^^^^^^^^^
…
File “/Users/gjoalland/Library/Python/3.12/lib/python/site-packages/circus/util.py”, line 1038, in wrapper
raise ConflictError(“arbiter is already running %s command”
circus.exc.ConflictError: arbiter is already running watcher_decr command
2026-02-27 14:14:00 tornado.application[20783] [ERROR] Exception in callback <bound method Arbiter.manage_watchers of <circus.arbiter.Arbiter object at 0x10b45d430>>
Traceback (most recent call last):
File “/Users/gjoalland/Library/Python/3.12/lib/python/site-packages/tornado/ioloop.py”, line 945, in _run
val = self.callback()
^^^^^^^^^^^^^^^
File “/Users/gjoalland/Library/Python/3.12/lib/python/site-packages/circus/util.py”, line 1038, in wrapper
raise ConflictError(“arbiter is already running %s command”
circus.exc.ConflictError: arbiter is already running arbiter_stop command
2026-03-02 14:40:42 circus[20783] [INFO] aiida-aiida stopped

Could someone please help me find a solution to this error?

Thank you in advance.

*I also attempted to install RabbitMQ directly via Homebrew, but encountered Erlang compatibility issues (as mentioned here). This led me to switch to the Generic Unix Binary Build.

2

Hi,

I’m also on macOS so I can relate to the RabbitMQ struggles! :grinning_face_with_smiling_eyes:

Looking at your Circus logs, the error:
ConflictError: arbiter is already running watcher_decr command

This usually happens when RabbitMQ drops the connection
unexpectedly and Circus receives conflicting stop/restart
commands at the same time it’s not really an AiiDA bug
itself but more of a RabbitMQ stability issue on macOS.

A few things that might help:

  1. First check if RabbitMQ is actually stable:
    rabbitmqctl status

    Check the RabbitMQ logs around the same timestamps
    as your Circus crashes — you’ll likely see connection
    drops there.

  2. Try adding a heartbeat to your rabbitmq.conf:
    heartbeat = 60

    This helped stabilize my setup on macOS.

  3. Also worth trying:
    verdi config set daemon.timeout 60

The Generic Unix Binary Build for RabbitMQ can sometimes
be a bit flaky on macOS compared to a native install.
I know you mentioned Erlang compatibility issues with
Homebrew — did you try the specific Erlang version
mentioned in the RabbitMQ compatibility matrix?
That fixed the Homebrew install for me personally.

Hope this helps, let me know if you find the root cause
curious to know what ends up being the fix on your end too!