# Simulation works when run directly on a compute node but fails silently under srun and under sbatch?

**URL:** <https://discuss.flower.ai/t/simulation-works-when-run-directly-on-a-compute-node-but-fails-silently-under-srun-and-under-sbatch/1248>\
**Category:** General\
**Created:** [June 4, 2026, 2:54pm UTC](https://discuss.flower.ai/t/simulation-works-when-run-directly-on-a-compute-node-but-fails-silently-under-srun-and-under-sbatch/1248 "2026-06-04T14:54:51Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mariza](https://avatars.discourse-cdn.com/v4/letter/m/b3f665/32.png) [@mariza](https://discuss.flower.ai/u/mariza)\
**Post date:** [June 4, 2026, 2:54pm UTC](https://discuss.flower.ai/t/simulation-works-when-run-directly-on-a-compute-node-but-fails-silently-under-srun-and-under-sbatch/1248/1 "2026-06-04T14:54:51Z")

</div>

I’m running a Flower simulation on a SLURM HPC cluster and getting inconsistent behavior depending on how I launch it. Hoping someone with SLURM experience can help.

**Setup:**

- Flower [1.28.0], Ray [2.51.1], Python 3.12, conda environment

- Plain FedAvg, 5 clients, small PyTorch MLP on a tabular dataset

- Simulation backend (Ray), `flwr run . --stream`

**What works:**

- ssh directly onto a compute node, activate my conda env, `flwr run . --stream` → completes all 3 rounds in ~2 min.

- Also works on the login node

**What fails:**

- Inside an interactive `srun --pty` and with sbatch allocation, the run reaches:

[ROUND 1]  
configure\_fit: strategy sampled 5 clients (out of 5) and stops

No error message

The content of the sbatch file

#!/bin/bash

#SBATCH --job-name=fl\_run

#SBATCH --cpus-per-task=32

#SBATCH --mem=64G

#SBATCH --partition=long

#SBATCH --time=04:00:00

#SBATCH --output=logs/fl\_%j.out

#SBATCH --account=2001786

# Activate your environment

spack load miniconda3

source activate Masterthesis

export RAY\_DISABLE\_RUNTIME\_ENV\_FILE\_DEPTH\_LIMIT=1

flwr federation simulation-config --num-supernodes 5 --client-resources-num-cpus 5 --client-resources-num-gpus 0.0

# Run

flwr run . --stream

echo “\>\>\> Done. DB saved as mlflow\_${SLURM\_JOB\_ID}.db”

output

✅ Updated simulation configuration.  
🎊 Successfully started run 2901953363831888993  
e[92mINFO e[0m: Starting logstream for run\_id `2901953363831888993`  
e[92mINFO e[0m: Starting Flower Simulation  
DEBUG:flwr:Initialising: RayBackend  
DEBUG:flwr:Backend config: {‘client\_resources’: {‘num\_cpus’: 5, ‘num\_gpus’: 0.0}, ‘init\_args’: {‘logging\_level’: ‘WARNING’, ‘log\_to\_driver’: True}, ‘actor’: {‘tensorflow’: 0}}  
e[92mINFO e[0m: Starting Flower ServerApp, config: num\_rounds=3, no round\_timeout  
INFO:flwr:Starting Flower ServerApp, config: num\_rounds=3, no round\_timeout  
e[92mINFO e[0m:  
INFO:flwr:  
e[92mINFO e[0m: [INIT]  
INFO:flwr:[INIT]  
e[92mINFO e[0m: Using initial global parameters provided by strategy  
INFO:flwr:Using initial global parameters provided by strategy  
e[92mINFO e[0m: Starting evaluation of initial global parameters  
INFO:flwr:Starting evaluation of initial global parameters  
e[92mINFO e[0m: Evaluation returned no results (`None`)  
INFO:flwr:Evaluation returned no results (`None`)  
e[92mINFO e[0m:  
INFO:flwr:  
e[92mINFO e[0m: [ROUND 1]  
INFO:flwr:[ROUND 1]  
e[92mINFO e[0m: configure\_fit: strategy sampled 5 clients (out of 5)  
INFO:flwr:configure\_fit: strategy sampled 5 clients (out of 5)  
Done. DB saved as mlflow\_745334.db

---

<div class="post-metadata">

**Author:** ![mohammad](https://dub1.discourse-cdn.com/flex013/user_avatar/discuss.flower.ai/mohammad/32/35_2.png) [@mohammad](https://discuss.flower.ai/u/mohammad)\
**Post date:** [June 12, 2026, 7:32pm UTC](https://discuss.flower.ai/t/simulation-works-when-run-directly-on-a-compute-node-but-fails-silently-under-srun-and-under-sbatch/1248/2 "2026-06-12T19:32:01Z")

</div>

Hi @mariza, thanks for the question.  
I would say this is almost a ray and SLURM launch problem, not FedAvg.

In here:  
`configure_fit: strategy sampled 5 clients (out of 5)`  
`Done. DB saved ...`

Flower should next log something like `aggregate_fit: received...` and since it does not, the run is dying while dispatching work to Ray `ClientApp` actors. Your script then prints `Done` regardless of whether `flwr run` failed.

I’d first make the script fail with:

`set -euo pipefail`  
`flwr run . --stream`  
`echo ">>> Done"`

and then check the SLURM exit code plus Ray logs.