Early Stopping Implementation

Flower Help - Intermediate
1

Hello everyone!

I am doing a custom strategy with FedAVG as base strategy.

I was wondering, if there is any way to implement an Early stopping for the strategy, making use of the Flower framework methods, I haven’t seen in the documentation anything related to this, neither how to stop the federated training process manually.

Thank you in advance!

1 Like
2

Hello

a option could be to implement a new class e.g.

class EarlyStoppingStrategy(fl.server.strategy.FedAvg):                                                          
    def __init__(self, patience, target_metric):
        super().__init__()
        self.patience = patience
        self.target_metric = target_metric   # mainly the loss or someting else e.g. accuracy, f1score...                                                                                
        self.beset_metric = 0  
        self.counter = 0                                                                                                                   

    def aggregate_evaluate(self, server_round, results, failures):
        aggregated_metrics = super().aggregate_evaluate(server_round, results, failures)
        if server_round == 1:
            if self.target_metric in aggregated_metrics:
                self.best_metric = aggregated_metrics[self.target_metric]                                           
        else:
            if self.target_metric in aggregated_metrics:
                current_metric =  aggregated_metrics[self.target_metric]                                                                        
                if current_metric > self.best_metric:
                    self.best_metric = current_metric
                    self.counter = 0
                else:
                    self.counter += 1  
                if self.counter >= self.patience: 
                    return aggregated_metrics
            return aggregated_metrics

The question remains how to stop training. Maybe if you set your server_round to your selected server_round. There is certainly an optimal solution

2 Likes
3 
strategy = EarlyStoppingStrategy(
              fraction_fit= .....
                  patience=3,
                  target_metric = "loss", 
                  evaluate_metrics_aggregation_fn=weighted_average, )

There is definitely a more efficient one. So i am also excited which suggestions comes next :smiley:

2 Likes
4

Exactly! I was trying to do something like this, a FedAvg strategy but with early stopping (with patience, best_metric, counter, min_delta…), but then I couldn’t find any way to stop the federated training! :sweat_smile:

Let’s see if a Flower developer sees this, and recommends us a way to stop the process even if it is in a rudimentary way (closing gRPC communications e.g.) in the meanwhile.

2 Likes
5

Hi,
Unfortunately, I don’t think it’s currently possible to terminate the simulation earlier based on some external condition like early stopping (the server is started as a separate thread, and I don’t see a way to influence its behavior from the strategy level), but note that I haven’t been involved in the development. Maybe we’ll get a different response from @javier but we’ll have to wait a bit for his response (he’s currently not available).

4 Likes
6

Hi Adam,

Thank you for letting me know about that issue, maybe we should think of another kind of solution…

It seems we’ll have to wait for Javier’s opinion on the matter.

Kind regards! :smile:

2 Likes
7

Hi @azulu, @adam-narozniak ,

What some people have done in the past is to modify slightly how the base Server works. Of particular interset is the fit for loop, which runs for the specified number of rounds. What you could do is to add a couple of lines that, given some logic in a custom strategy that you implement, it does the break from the for loop. Does it make sense?

Note you can pass a custom Server object when you write your server_fn that returns an instance of ServerAppComponents.

This is currently the best way to achieve early stopping. It will be easier with upcoming versions of flower.

There is another way: implement a custom strategy and, once your early stopping condition is reached, effectively disable the sampling on clients In this way the remaining rounds will go through but will do no work (which will make your experiment finish very fast).

2 Likes
8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.