GH-144179: Use recorded values to make optimizer more robust

GH-144179: Use recorded values to make optimizer more robust #5191