Add ability to provide retry options that are passed into urllib3

[Pwnion]

This issue is an extension of this discussion.

The issue is, in fact, worse than we thought. As previously noted, xero-python uses urllib3 under the hood and this results in 3 request retries by default. However, urllib3 also respects the Retry-After rate limiting header, and will time.sleep to wait until it should send another request. This can result in hour or day-long sleeps, which is extremely undesirable.

The proposed solution is the same as in the original discussion:

xero_python.api_client.configuration.Configuration should accept retries as an argument which in turn would be passed down to the urllib3.PoolManager or urllib3.ProxyManager during initialisation.

[github-actions]

PETOSS-808

[github-actions]

Thanks for raising an issue, a ticket has been created to track your request

[timrichardson]

+1 I am overriding ApiClient and AccountingApi to achieve this because monkey patching is not working. Being stuck in sleep_for_retry() for ten hours is not nice.

[uptickmetachu]

urllib3/urllib3#3233

Note this is also an open issue in urllib3. Until this is merged (who knows when); all users of this library suffer from an hours long sleep which can break many background processes.

I would consider this a bug in the API client.

Monkey patch on urllib3 to disable this sleep_for_retry if it is excessively long.

import datetime
from typing import Any

from urllib3.util.retry import Retry

original_sleep_for_retry = Retry.sleep_for_retry


MAX_RETRY_AFTER = datetime.timedelta(minutes=10).seconds


def _patched_sleep_for_retry(self, response: Any) -> bool:
    retry_after = self.get_retry_after(response)
    if retry_after > MAX_RETRY_AFTER:
        return False
    return original_sleep_for_retry(self, response)


Retry.sleep_for_retry = _patched_sleep_for_retry