fault tolerance of restarting server

[cpd85]

I'm a little confused, shouldn't a server using local storage that restarts be able to accept/handle downloads if the service can restart in 15 seconds or so, saving its state? it looks like on restart, the server will update its 'running version' to the current time so any existing write clients will fail at this line of code

RemoteShuffleService/src/main/java/com/uber/rss/clients/ServerIdAwareSyncWriteClient.java

Line 149 in 7220c23

throw new RssInvalidServerVersionException(msg);

[cpd85]

@hiboyang just wanted to check if you might have context on the above?

[hiboyang]

Hi @cpd85 , this is due to a special case inside our previous environment, where when each server restarts, the behavior is unpredictable. To be safe, we do not want RSS client to read data from restarted server, to prevent reading bad data. Thus we update the running version.

This is a lacking feature in current RSS to gracefully handle server restart. If you are interested, welcome to contribute this feature.

[cpd85]

@hiboyang definitely planning on contributing this feature. Could you elaborate on 'unpredictable' issue you saw? I think if the localstatestore has good status on what has been committed succesfully, it should be okay to restart right?

Also curious how you deploy code/config changes if a server restart causes such issues

[hiboyang]

The unpredictable issue is mostly related to the internal environment at that time. Kind of hard to explain.

It is better to redesign the server restart/recover feature, look forward to your contribution here.

By the way, are you running in YARN or Kubernetes?

[cpd85]

@hiboyang running on aws ec2 for now, we don't have the infra at the moment to support kubernetes unfortunately. will let you know when i have a diff ready for restart/recover!

[mayurdb]

Hi @cpd85, looking forward to your contribution on gracefully handling server restarts.

In our internal deployments we were facing an issue of the node hosting RSS server going down much more frequently than restarts. We have added a fault tolerance for such cases by triggering a stage retry and picking a new list of RSS servers to execute a shuffle. That patch will help in preventing failures because of server restart cases as well. I'll try to port the feature as soon as possible.

[YutingWang98]

@mayurdb Hi mayurdb! We also have this server down/restart issue quite frequently. Do you mind sharing your progress on the stage retry and new server list picking, or how you implemented it?