No metrics being shown at all in the simulations

Flower Help - Beginners
1

This question was migrated from Github Discussions.

Original questions:
"When i run the tutorial
[Google Colab]

WARNING flower 2022-08-11 20:31:18,657 | fedavg.py:243 | No fit_metrics_aggregation_fn provided
WARNING flower 2022-08-11 20:31:25,887 | fedavg.py:274 | No evaluate_metrics_aggregation_fn provided

And the output of

INFO flower 2022-08-11 20:34:21,317 | app.py:180 | app_fit: losses_distributed [(1, 0.06262678999900817), (2, 0.054536575412750254), (3, 0.05160248675346375), (4, 0.050336861228942875), (5, 0.049149505472183226)]
INFO:flower:app_fit: losses_distributed [(1, 0.06262678999900817), (2, 0.054536575412750254), (3, 0.05160248675346375), (4, 0.050336861228942875), (5, 0.049149505472183226)]
INFO flower 2022-08-11 20:34:21,323 | app.py:181 | app_fit: metrics_distributed {}
INFO:flower:app_fit: metrics_distributed {}
INFO flower 2022-08-11 20:34:21,331 | app.py:182 | app_fit: losses_centralized []
INFO:flower:app_fit: losses_centralized []
INFO flower 2022-08-11 20:34:21,340 | app.py:183 | app_fit: metrics_centralized {}
INFO:flower:app_fit: metrics_centralized {}
History (loss, distributed):
	round 1: 0.06262678999900817
	round 2: 0.054536575412750254
	round 3: 0.05160248675346375
	round 4: 0.050336861228942875
	round 5: 0.049149505472183226

This happens with my own implementation too. It runs with no errors and all successes. But gives these warning and will not give any metrics which I need to see. Is there any reason behind this? or a way to fix it?

EDIT: For my own implementation, updating to 1.10 worked

Thank you"

Answer:
"Thanks for reaching out!
What you’re describing is expected behaviour. The reason for metrics_distributed to be empty (even though you might return something like {"accuracy": 0.8} from your client) is that the framework cannot know how to interpret and aggregate the keys/values in those dictionaries. You therefore have to provide functions that implement the aggregation (fit_metrics_aggregation_fn or evaluate_metrics_aggregation_fn ). If those functions are not provided, Flower will log the warning you’re seeing to point you in the right direction.

Thanks for raising this question though, it motivated us to update the Flower tutorial: https://flower.dev/docs/tutorial/Flower-1-Intro-to-FL-PyTorch.html#Where’s-the-accuracy?

The other two categories, losses_centralized and metrics_centralized, will only contain values if centralized evaluation is configured. This is covered in part two of the Flower tutorial: Flower Framework main"

1 Like
2

Note that this question is closely related to Why don't I see evaluate results in my output? - #2.

2 Likes