# Server Automatically shutdown during implement Flower on 3 devices **URL:** \ **Category:** Flower Framework\ **Created:** [June 6, 2024, 8:32pm UTC](https://discuss.flower.ai/t/server-automatically-shutdown-during-implement-flower-on-3-devices/236 "2024-06-06T20:32:38Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![manhbui](https://avatars.discourse-cdn.com/v4/letter/m/ad7895/32.png) [@manhbui](https://discuss.flower.ai/u/manhbui)\ **Post date:** [June 6, 2024, 8:32pm UTC](https://discuss.flower.ai/t/server-automatically-shutdown-during-implement-flower-on-3-devices/236/1 "2024-06-06T20:32:38Z")
Hi guys, I am implementing 1 server and 2 clients on 3 different machines with the FL for the encrypted data. Generally, I tested my code when I tried to simulate locally, and it ran perfectly. However, when I implemented it in three different devices, the Flower framework could only run in 5 epochs, and the client had the following errors: ![image](https://europe1.discourse-cdn.com/flex013/uploads/flower/original/1X/94735082af96e9f48590ec0586fde5568da193ce.png) When I checked the log on the server side, it had no errors. Can someone help me with this issue? ---
**Author:** ![javier](https://dub1.discourse-cdn.com/flex013/user_avatar/discuss.flower.ai/javier/32/116_2.png) [@javier](https://discuss.flower.ai/u/javier)\ **Post date:** [June 11, 2024, 5:50pm UTC](https://discuss.flower.ai/t/server-automatically-shutdown-during-implement-flower-on-3-devices/236/2 "2024-06-11T17:50:24Z")
Hi @manhbui, by _“could only run in 5 epochs”_ do you mean your experiment ran for 5 FL rounds but then this error appeared a the client side? Could you share a bit more details about your setup? Was your code similar to the one in the [examples/quickstart-pytorch](https://github.com/adap/flower/tree/main/examples/quickstart-pytorch) or maybe even [examples/embedded-devices](https://github.com/adap/flower/tree/main/examples/embedded-devices) ? If you could share the log generated at the server side maybe that gives us some hints of what went wrong. ---
**Author:** ![manhbui](https://avatars.discourse-cdn.com/v4/letter/m/ad7895/32.png) [@manhbui](https://discuss.flower.ai/u/manhbui)\ **Post date:** [June 11, 2024, 7:00pm UTC](https://discuss.flower.ai/t/server-automatically-shutdown-during-implement-flower-on-3-devices/236/3 "2024-06-11T19:00:41Z")
Hi @javier , yes I can only run in 5 round of FL training. When the client receive the train message from server, client start training but after about 10 minutes, the server shutdown and client’s grpc channel change to IDLE. When I faced the error in 5 round, I used the default setup related to keepalive ping and the round\_timeout = None. When I changed the round\_timeout to 3600, it can run in 6 rounds and then face the same errors but when I increase the round\_timeout more, it still only can run in 6 rounds. It is noted I am running on the machines on IHPC environment (which run under the industry wifi and network) ---
**Author:** ![manhbui](https://avatars.discourse-cdn.com/v4/letter/m/ad7895/32.png) [@manhbui](https://discuss.flower.ai/u/manhbui)\ **Post date:** [June 11, 2024, 7:03pm UTC](https://discuss.flower.ai/t/server-automatically-shutdown-during-implement-flower-on-3-devices/236/4 "2024-06-11T19:03:12Z")
@javier , it is quite weird that the log on server sides have no errors at all. From my view, it may automatically shutdown the server ![image](https://europe1.discourse-cdn.com/flex013/uploads/flower/original/1X/8a4fc029401aca280c80dcac9937232d37f021c1.png) ---
**Author:** ![javier](https://dub1.discourse-cdn.com/flex013/user_avatar/discuss.flower.ai/javier/32/116_2.png) [@javier](https://discuss.flower.ai/u/javier)\ **Post date:** [June 11, 2024, 7:18pm UTC](https://discuss.flower.ai/t/server-automatically-shutdown-during-implement-flower-on-3-devices/236/5 "2024-06-11T19:18:16Z")
It seems your rounds take quite a long time to complete (just over an hour fro Round 2 to finish for example). Can you try with a faster workload to see your setup is working fine ? Are you using SLURM? Another question is: how are you running your experiment, with `start_server` and 2x `start_client`? ---
**Author:** ![manhbui](https://avatars.discourse-cdn.com/v4/letter/m/ad7895/32.png) [@manhbui](https://discuss.flower.ai/u/manhbui)\ **Post date:** [June 11, 2024, 7:33pm UTC](https://discuss.flower.ai/t/server-automatically-shutdown-during-implement-flower-on-3-devices/236/6 "2024-06-11T19:33:29Z")
Yes, I have tried another simple setup and the code can run perfectly, but because it is simple so it run very fast to finish the learning rounds. That’s why I’m thinking it may related to timeout setup and keepalive ping setup. Yes I am running each machine on ubuntu environment and run via start\_server and start\_client ---
**Author:** ![javier](https://dub1.discourse-cdn.com/flex013/user_avatar/discuss.flower.ai/javier/32/116_2.png) [@javier](https://discuss.flower.ai/u/javier)\ **Post date:** [June 11, 2024, 7:38pm UTC](https://discuss.flower.ai/t/server-automatically-shutdown-during-implement-flower-on-3-devices/236/7 "2024-06-11T19:38:30Z")
Ok. then it should be safe to increase the `timeout` to, let’s say 10000 😆 ? We set it to `3600` since it’s not too big, not too small. But it should be adjusted based on the application. ---
**Author:** ![manhbui](https://avatars.discourse-cdn.com/v4/letter/m/ad7895/32.png) [@manhbui](https://discuss.flower.ai/u/manhbui)\ **Post date:** [June 11, 2024, 7:52pm UTC](https://discuss.flower.ai/t/server-automatically-shutdown-during-implement-flower-on-3-devices/236/8 "2024-06-11T19:52:24Z")
Yeah I tried to increase the round\_timeout for server to 7200 or 10000 but it still seems cannot run reliably. The server still shutdown in round 6. If I set timeout to None, it can only run for 5 rounds ---
**Author:** ![javier](https://dub1.discourse-cdn.com/flex013/user_avatar/discuss.flower.ai/javier/32/116_2.png) [@javier](https://discuss.flower.ai/u/javier)\ **Post date:** [June 11, 2024, 8:11pm UTC](https://discuss.flower.ai/t/server-automatically-shutdown-during-implement-flower-on-3-devices/236/9 "2024-06-11T20:11:24Z")
Is there a maximum time limit in the HPC system you are running these experiments? Some compute clusters have maximum time limits for the experiments if people submit to a queue of jobs…