On a multi-thread context, the use of this method could lead to "HTTPSConnectionPool ... Max retries exceeded with url..."

[fabiofalavinha]

clearbit-python/clearbit/resource.py

Line 46 in 5594e82

response = requests.get(endpoint, **options)

The use of "requests" component library on a multi-thread context with a large number of concurrent threads requests data to Clearbit Servers, as an example, using the Reveal API to find company data by IP. We could got an error from server. But, this is not an issue on server side, because server has a limit and this is fine. The problem is how to handle this issue on client side.

So, I made an improve on my own using the code below:

A little brief on this code: Find company by IP.

To highlight the improvement, the code add a Retry strategy to handle issues when the server returns the status code [429, 500, 502, 503, 504. Just remembering that this is from my case, we should refactor this using a factory + configuration design model to avoid hardocoded configuration on this end.

@staticmethod
def __createSession__():
    session = requests.Session()
    retry = Retry(
        total=3,
        backoff_factor=1,
        status_forcelist=[429, 500, 502, 503, 504],
        method_whitelist=["HEAD", "GET", "OPTIONS"]
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session


 def findCompanyInfoByIp(self, ip: string) -> ClearbitResult:
        if ip is None or ip == "":
            return ErrorClearbitResult(f"Can't find clearbit data with a valid ip")
        url = f"{BASE_URL}/find?ip={urllib.parse.quote(ip)}"
        try:
            response = \
                self.__requestSession.get(
                    url,
                    auth=BearerAuth(self.__authToken))
            return ClearbitResult(response)
        except Exception as e:
            errorMessage = f"Error requesting Clearbit at [{url}]: {e}"
            self.__logger.error(errorMessage)
            return ErrorClearbitResult(errorMessage)

This solution helps me to ensure 100 or maybe more threands to requests Clearbit Reveal API using a simple retry strategy.

Could this be an improvement to help this library?