# Not clear how loss distributed is calculated

**URL:** <https://discuss.flower.ai/t/not-clear-how-loss-distributed-is-calculated/238>\
**Category:** Flower Help - Beginners\
**Tags:** metrics\
**Created:** [June 7, 2024, 12:33pm UTC](https://discuss.flower.ai/t/not-clear-how-loss-distributed-is-calculated/238 "2024-06-07T12:33:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![vikwal](https://avatars.discourse-cdn.com/v4/letter/v/838e76/32.png) [@vikwal](https://discuss.flower.ai/u/vikwal)\
**Post date:** [June 7, 2024, 12:33pm UTC](https://discuss.flower.ai/t/not-clear-how-loss-distributed-is-calculated/238/1 "2024-06-07T12:33:54Z")

</div>

Hello there,

I run the siumulation-tensorflow example from [flower/examples/simulation-tensorflow at main · adap/flower · GitHub](https://github.com/adap/flower/tree/main/examples/simulation-tensorflow)

I have adapted the script slightly as I am also interested in the loss per client and per epoch.  
I have added the lines in the fit method of the client as follows:

```auto
def fit(self, parameters, config):

        self.model.set_weights(parameters)

        history = self.model.fit(self.trainset, epochs=2, verbose=VERBOSE)
        loss = history.history['loss']
        loss_per_epoch = json.dumps(loss)
        
        return self.model.get_weights(), len(self.trainset), 
{'loss_per_client_and_epoch': f'{self.cid}:{loss_per_epoch}'}

```

So the history of tensorflow training is serialized, because flower metrics only accepts data type Scalar, in which lists are not included, but strings.

Then I formulated a fit\_metrics\_aggregation\_fn and named it loss\_per\_client\_epoch\_fn:

```auto
def loss_per_client_epoch_fn(metrics: List[Tuple[int, Metrics]]) -> Metrics:

    loss_epochs = [m["loss_per_client_and_epoch"] for num_examples, m in metrics]
    
    loss_per_client_and_epoch = {}
    
    for item in loss_epochs:
        key, value = item.split(':')
        value_list = [float(x) for x in value.strip('[]').split(',')]
        new_key = f"Client_{key}"
        loss_per_client_and_epoch[new_key] = value_list
        
    return {"loss_per_client_and_epoch": loss_per_client_and_epoch}

```

Running the code, for 3 rounds and 3 clients, the output of Flower history is this:

```auto
[SUMMARY]
INFO : Run finished 3 rounds in 26.77s
INFO : History (loss, distributed):
INFO : ('\tround 1: 0.7618376016616821\n'
INFO : '\tround 2: 0.46412525574366253\n'
INFO : '\tround 3: 0.3493043581644694\n')History (loss, centralized):
INFO : ('\tround 0: 121.7120361328125\n'
INFO : '\tround 1: 0.7046558260917664\n'
INFO : '\tround 2: 0.4510457515716553\n'
INFO : '\tround 3: 0.3422752916812897\n')History (metrics, distributed, fit):
INFO : {'loss_per_client_and_epoch': [
INFO : (1,
INFO : {'Client_0': [6.12161111831665,
INFO : 0.9191960692405701],
INFO : 'Client_1': [6.517727375030518,
INFO : 0.8578547835350037],
INFO : 'Client_2': [6.666688919067383,
INFO : 0.9658933281898499]}),
INFO : (2,
INFO : {'Client_0': [0.979604184627533,
INFO : 0.6871640682220459],
INFO : 'Client_1': [0.9863572120666504,
INFO : 0.7082828879356384],
INFO : 'Client_2': [0.9567165970802307,
INFO : 0.6330591440200806]}),
INFO : (3,
INFO : {'Client_0': [0.6875190138816833,
INFO : 0.5470158457756042],
INFO : 'Client_1': [0.6803959608078003,
INFO : 0.5474984049797058],
INFO : 'Client_2': [0.6760547757148743,
INFO : 0.5368837714195251]})]}
INFO : History (metrics, distributed, evaluate):
INFO : {'accuracy': [(1, 0.8008333245913187),
INFO : (2, 0.874999980131785),
INFO : (3, 0.9045000076293945)]}History (metrics, centralized):
INFO : {'accuracy': [(0, 0.11739999800920486),
INFO : (1, 0.807699978351593),
INFO : (2, 0.881600022315979),
INFO : (3, 0.9101999998092651)]}

```

So my point is the History (loss, distributed). In my understanding this is the fit loss aggregated from all clients per round. For example for round 1 it is 0.7618… Looking at the loss per client and epoch in round 1, there are no values even smaller than 0.9… so my question is: **How is loss, distributed calculated exactly?** The num\_examples among the clients is equal, so the aggregated loss distributed should be the average of the losses in the dictionary loss\_per\_client\_and\_epoch or am I understanding it entirely wrong?

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![javier](https://dub1.discourse-cdn.com/flex013/user_avatar/discuss.flower.ai/javier/32/116_2.png) [@javier](https://discuss.flower.ai/u/javier)\
**Post date:** [June 11, 2024, 8:09am UTC](https://discuss.flower.ai/t/not-clear-how-loss-distributed-is-calculated/238/2 "2024-06-11T08:09:25Z")

</div>

Hi @vikwal, this is a great first question!

In `fit()` you can return as metrics (i.e. the last return element) anything you want. It could be a loss, accuracy, something else. Therefore it is up to you and your particular setting how to define the _distributed loss_. From my previous experience what I typically do is:

- A client returns the average train loss (so just a single scalar)
- in the aggregate fit metrics function, i take the average of all training losses.

To do something similar to the above, you could try to edit the code in your client’s `fit()` and only return the average loss.

---

<div class="post-metadata">

**Author:** ![system](https://europe1.discourse-cdn.com/flex013/uploads/flower/original/1X/bc7fe36843c391cbd0b4e8b6f48275961d356abc.png) [@system](https://discuss.flower.ai/u/system)\
**Post date:** [June 18, 2024, 8:10am UTC](https://discuss.flower.ai/t/not-clear-how-loss-distributed-is-calculated/238/3 "2024-06-18T08:10:02Z")

</div>

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.