Image-to-image models (diffusion-based restyling, background replacement, outpainting, old-photo restoration) seem like a natural fit for federated learning: the most useful training pairs are often private photos that people would never upload to a central dataset. I’m curious how people here would approach this with Flower.
A few questions I keep coming back to:
-
Non-IID visual data. One client may hold mostly portraits, another mostly landscapes or scanned family photos. Would something like FedProx or per-client adapters (LoRA layers kept local, shared backbone aggregated) be the more practical starting point in Flower?
-
Communication cost. Full diffusion UNets are huge. Has anyone tried sending only adapter weights through a custom Strategy, and how did aggregation behave when clients trained on very different styles?
-
Evaluation. For generative tasks, accuracy isn’t meaningful. Would a federated evaluation round computing something like LPIPS or CLIP similarity on each client’s held-out pairs be reasonable, or is it better to evaluate centrally on a public benchmark?
For context on the kind of task I mean: consumer tools such as image to image ai take an uploaded photo plus a short prompt and restyle it, change the background or restore it while keeping pose, composition and lighting. The open question is whether that kind of model could be improved collaboratively without ever centralising users’ photos.
Any pointers to Flower examples (e.g. the diffusion or vision examples) that would be a good base would be appreciated.