Using MS COCO 2017 dataset for FL simulation

Flower Help - Intermediate
1

Hi
Does someone know if it is possible to load the MSCOCO 2017 dataset with the FederatedDataset class and partition it correctly? I have struggled to do so, since pytorch expect another format than the coco format. Therefore i was wondering if it possible to load a custom pytorch datasetclass created myself in the FederatedDataset, instead of using the string path of it from huggingface?

Appreciate any help!

//Johan

1 Like
2

Hello @johanrubak, I haven’t experienced loading MSCOCO from HuggingFace before. But you can create a FederatedDataset from your own local images or in-memory data. I think there’s potentially one conversion step to convert PyTorch Dataset to HuggingFace’s Dataset. After that, you can create the FederatedDataset from the Dataset.

Here’s the link to the Flower Datasets page: Use with Local Data - Flower Datasets 0.4.0.

2 Likes
3

Thanks a lot, figured a way to load data in a manual way into a huggingface dataset, which is used for partitioning

2 Likes
4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.