Hello,
I have been getting workflows that are getting stuck in the created status for the initial workflow. It seems similar to a question posed by Calculations get stuck in "created" state. Here at LLNL they recently forced us to update to RabbitMQ v3.13.7 and I believe that to possibly be the source of the issue. Back when this happened I had emailed them to see if they had configured it with the settings suggested in the documentation and pointed out in the other link. They had told me they did do this. Is there a way to check that this is the case? If it is not the case, what else might be causing this issue? Thanks.
Nathan
Hi @nkeilbart, sorry to hear that you have this problem.
There are two things you can try.
- Calculations get stuck in "created" state this seems like that there are too workchains/calcjobs in workers, usually increase the worker number or
verdi config set daemon.worker_process_slots 800 to increase the worker’s slots from default 200 to 800.
- Since you also mentioned this problem appears after the rabbitmq is updated, maybe there are tasks in the queue that not able to be consumed. I’d suggest to clean the queue. First you can list the queues by
rabbitmqadmin list queues name and then try rabbitmqadmin purge queue name=<queue_name>. This can work only if you have admin permission to rabbitmq.
Let me know if you have further questions.