Add retries to outbox on confirm failure

[Bruno-DaSilva]

Just FYI, the reason I don't republish the failed message is because it's really hard to know which failed. The way rabbitmq/bunny indicate which message failed is by a message index, which increments from the beginning of your connection every time you publish a message.

We'd have to keep track of the messages indexes for any given loop and what indexes they are, so if a failure happened we would find the right message.

I opted instead to just republish the entire batch of messages for three reasons:

1. It's simple to write and understand whats going on
2. Batch sizes are small enough that there's not a super huge performance impact to republish the batch
3. The failure scenario happens very inoften (I'm pretty sure it hasnt even happened once in prod yet), so who cares if one of the publishes out of 10,000 batches takes twice as long

[bradrobertson]

seems fine. I'm a little worried we don't actually get a notification about the failures... aside from debugging. We also have an MM error handler right? Maybe we could use that? In the hub I think we tap into that to log Honeybadgers

[Bruno-DaSilva]

Yeah, that's a good point. We should figure out how we can bubble up exceptions without entirely killing the producer