i am trying to run flower embedded example with two raspberry pie as clients , after i run the code using flwr run . embedded-federation, clients are created but training is not happening, stuck at below error
Fleet.PullMessages] node_id=3527963171899011829
INFO : [Fleet.PullMessages] node_id=12301742394902099172
INFO : Starting Flower ServerApp, config: num_rounds=3, no round_timeout
INFO :
INFO : [INIT]
INFO : Using initial global parameters provided by strategy
INFO : Starting evaluation of initial global parameters
INFO : Evaluation returned no results (None)
INFO :
INFO : [ROUND 1]
INFO : configure_fit: strategy sampled 2 clients (out of 2)
INFO : [Fleet.PullMessages] node_id=3527963171899011829
INFO : [Fleet.PullMessages] node_id=12301742394902099172
INFO : [Fleet.GetRun] Requesting Run for run_id=8148725298008692287
INFO : [Fleet.GetFab] Requesting FAB for fab_hash=56b3fddb2878845c10b30a9717f4c74cc5e67aeeadd57b61659113c071aa2aa2
INFO : [Fleet.GetRun] Requesting Run for run_id=8148725298008692287
INFO : [Fleet.GetFab] Requesting FAB for fab_hash=56b3fddb2878845c10b30a9717f4c74cc5e67aeeadd57b61659113c071aa2aa2
Hallo javier, thanks for the response , i use raspberry pi 4, i installed the packge as per the pyproject toml ,does flower=1.20.0 offer superlink and supernode suport,because i tried to find the error using chatgpt and it says both superlink and supernode are part flower commercial and i think this chatgpt wrong since i was able to launch superlink and supernode using commands mentioned in the repositary- i am trying to launch tinybert using flower into three raspberry pie client for sentiment analysis
Hey @tomyfrancis25 , flower-superlink and flower-supernode are commands that have been around for a while as part of the open source Flower repo.
Could you try something simpler before doing tinybert? one idea would be to simply run the Flower app that comes out from executing:
flwr new
then select the NumPy template, then edit the remote federation section to insert the address of your SuperLink and run it via flwr run
It could be that the execution of your ClientApp in your SuperNodes are crashing for different reasons. So i’d recommend using a Flower App that’s know to work first
HI javier, i tried it using the numpy template it worked perfectly fine,but in embedded federated example using cnn, it is failing and its stuck at requesting RUN and FAB, this is the same error i am getting for tinybert as well i have attached a screenshot of cnn
PS: i have tried to execute the code as it is , i did not change anything
Good to hear the NumPy app worked. The FAB was delivered to your RPis successfully and that’s why you can see that log message saying 🎊 Successfully installed …. However, it seems something breaks when the ClientApp is about to start. Here are some questions:
Do you have all dependencies correctly installed in the SuperNodes in the RPIs?
Is there by any chance some dataset downloading operation taking place when starting the ClientApp . Note that in many of our examples we do just that via Flower Datasets but this is not a requirement. It makes more sense in real world setups like yours to simply load the local dataset from a file. (ofcourse you can download the dataset but that would maybe explain why your ClientApp doesn’t seem to responde?)
Have your RPis frozen? (i.e. RAM is full and therefore the RPI is unresponsive?) Could you check by running, for example, htop?
Could you add some print statements in your ClientApp to see what stages are executed? for example add one print right when client_fn starts. Then add more based on your analysis.
Hi, javier actually numpy app worked within my linux pc, i tried to ping from my pc and from my raspberry pie both worked , the problem is with client.py is not getting executed, i tried debugging with print statements this was the issue that i found , i reinstalled all the required dependencis again in raspberry pie but still the same errror,ram utilization and memory suggest that training was not happening.
Hi @tomyfrancis25 , I just tested running the embedded-devices example on two RaspberryPi 5 I have. It seems to work fine. Did you try running the code as-is without changes? (in the example there is no client.py)
I would recommend testing the simplest setup possible first. For example as I suggested earlier with what comes out of the NumPy template via flwr new. Then submit the run to your SupeLink after creating a new federation in your pyproject.toml. For example like this:
[tool.flwr.federations.embedded-federation]
address = "127.0.0.1:9093" # Address of your superlink
insecure = true
Hallo Javier, thank you for the reply ,i tested like the way you suggested ,it worked like a charm,i need to figure out how i can scale this down to the embedded example
i only changed the following thing in raspberry example
[tool.flwr.federations.embedded-federation]
address = "***.**.**.**:9093" IP Address of my linux pc running as server
insecure = true
flower-supernode --insecure --superlink=“**.**.**.**(ip of superlink):9092” --node-config=“dataset-path=‘home/francis/fashionmnist_part_2’” and same with other supernode rest i did not change anything ,is it because of dependency issue
I’m a bit out of ideas of what else you could try. If i understood you correctly you did try running the app that results from the NumPy template across the SuperNodes running on your RPis, i.e. also using the embedded-federation. Correct?
Could you provide a bit more info how you setup the RaspberryPis ? What version of python are you using? Do they have internet access?
Could you try running your supernodes by first exporting FLWR_LOG_LEVEL=DEBUG, this will enable more detailed logs that might help us uncover what’s failing.
export FLWR_LOG_LEVEL=DEBUG
# then launch supernode as usual
flower-supernode ....
as per email, I am having similar issues. I am away this week so I will post details here next week. In the meantime,
what OS have you on the raspberry pi? What RAM does it have? Mine is Rpi 4 with 4gb ram and raspian lite on one pi and raspian desktop on the other.
What device is the superlink running on?
What wifi network are you using? I have tried both home WiFi (fibre to home) and Vodafone mobile hot-spot. In my case, when I do a netstat. I don’t see a connection to either of the client ports. I have set a parameter suggested by Hans to ignore ssl (can’t remember it’s name)
I’m using the fashionmnist locally. I copied over the folders created with the generate.py in the embedded example.
