How do I plot the metrics from the strategy

[flower]

A frequently asked question we get is:

How do I plot the metrics from the strategy (e.g., the loss/accuracy over rounds)?

[javier]

Let’s assume your strategy does centralized evaluation of the global model (i.e. if you set the evaluate_fn argument) and also specifies a method to aggregate the metrics received by clients in an evaluate round. For example:

strategy = fl.server.strategy.FedAvg(
    ...,
    evaluate_metrics_aggregation_fn=weighted_average,  # aggregates federated metrics
    evaluate_fn=get_evaluate_fn(centralized_testset),  # global evaluation function
)

You’d then pass such strategy to either start_server or start_simulation. They both return a History object.

Printing Centralised evaluation results

These are the results of just 10 rounds on MNIST. I’m plotting the centralized accuracy reported by the function passed to evaluate_fn in my strategy.

import matplotlib.pyplot as plt

print(f"{history.metrics_centralized = }")

global_accuracy_centralised = history.metrics_centralized["accuracy"]
round = [data[0] for data in global_accuracy_centralised]
acc = [100.0 * data[1] for data in global_accuracy_centralised]
plt.plot(round, acc)
plt.grid()
plt.ylabel("Accuracy (%)")
plt.xlabel("Round")
plt.title("MNIST - IID - 100 clients with 10 clients per round")

You can follow a similar logic for the aggregated distributed evaluate metrics. The result of your evaluate_metrics_aggregation_fn can be accessed by:

history.metrics_distributed # as opposed to .metrics_centralized
# then you can plot it 

If interacting with the History object is limiting for your use case, you can write a custom strategy that stores in a different way the result of the strategy’s evaluate(), aggregate_evaluate() methods (and potentially aggregate_fit().