Hybrid frameworks

Flower Help - Intermediate
1

[DISCLAIMER: Not sure if this is a beginner issue or intermediate or more advanced]
I’m setting up a FL example on a compute cluster where the mother node has access to the Internet (and will be the Superlink of the application) and the supernodes are either local nodes on the same cluster (with no access to the Internet) or remote clients.

Prior knowledge:

  • communication between the mother node and the local compute nodes is done over InfiniBand which from what I’ve read does not play well with GRPC.
  • the examples I’ve set up so far do not yet include the remote clients.

My experiments have worked only in insecure mode; I use the InfiniBand addresses and the nodes communicate correctly and everything works fine.
When I try to implement secure mode; the supernodes can’t connect to the superlink WARNING : Connection attempt failed, retrying in XX seconds, but the superlink outputs with ssl_transport_security.cc:2396] No match found for server name: [INFINIBAND ADDRESS HERE] which tells me that the incoming connection does reach somehow. Any idea what might be happening here?

Also, my main problem here is that when I do implement the larger example see attached diagram, I need to have secure enabled to communicate with the remote clients.

framework
framework2154×1667 260 KB

So my question is this: is it possible to implement a hybrid workflow where some clients use insecure and others use secure? If so, is there a tutorial somewhere that I may have missed?

Thanks a lot for your help!

2

Hi @chadib,

I am afraid Flower currently does not support mixing TLS and non-TLS connections to the same SuperLink. If the SuperLink is started with TLS enabled, all connecting SuperNodes must also use TLS.

Regarding the error you’re seeing (No match found for server name), this is likely due to TLS hostname verification: the InfiniBand address used by the SuperNode does not match the SAN / CN in the server certificate.

For a hybrid setup (local insecure + remote secure clients), Flower doesn’t provide native support. One possible workaround is to run the SuperLink without TLS and place a TLS-terminating proxy / relay (e.g. a gRPC-capable proxy) in front of it for remote clients. Local SuperNodes would connect directly, while remote ones connect via the proxy.

There isn’t a Flower tutorial for this setup at the moment, and implementing a custom gRPC relay is possible but fairly advanced.

1 Like