Hi @wittenator and @mistersunshine,
I just replied in this post Communicate List[Float] in fit() function from Clients - #2 by javier from @mistersunshine. There I show how you an send back to the ServerApp any data structure containing metrics from the ClientApp’s local training or evaluation.
In short: you can send your arbitrary data strucutres as bytes. Then, you deserialize it on the server-side (e.g. in the strategy) and use it as you normally would.
Does this help solving your problem?