feature: Add $feature_flag_error property to track flag evaluation failures
#390
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
haacked
force-pushed
the
haacked/fix-null-response
branch
2 times, most recently
from
881ff09 to
fe5cb38
Compare
Track errors in feature flag evaluation by adding a `$feature_flag_error` property to the `$feature_flag_called` event.
haacked
force-pushed
the
haacked/fix-null-response
branch
2 times, most recently
from
495b463 to
c8eeffc
Compare
There was a problem hiding this comment.
Pull request overview
This PR adds error tracking to feature flag evaluation by introducing a $feature_flag_error property to the $feature_flag_called event. This enables users to debug flag evaluation failures by identifying the specific error type that occurred.
Key changes:
- Added
$feature_flag_errorproperty to capture different types of flag evaluation failures - Implemented specific error categorization (timeout, connection_error, api_error, quota_limited, etc.)
- Added comprehensive unit tests covering all error scenarios
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| posthog/client.py | Added error tracking logic to capture and categorize flag evaluation failures, including imports for new exception types and error property handling |
| posthog/test/test_feature_flag_result.py | Added unit tests for all error scenarios including timeout, connection errors, API errors, quota limits, and unknown errors |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Export RequestsTimeout and RequestsConnectionError from posthog/request.py to keep all requests library imports in one place and avoid mypy issues.
haacked
force-pushed
the
haacked/fix-null-response
branch
from
c8eeffc to
b4d67bf
Compare
- Fix fallback logic to only trigger on actual exceptions, not when errors_while_computing or flag_missing is set from a successful API response - Change log.exception() to log.warning() for expected operational errors (quota limits, timeouts, connection errors, API errors) to reduce log noise - Keep log.exception() only for truly unexpected errors (unknown_error) - Extract stale cache fallback into _get_stale_flag_fallback() helper method
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Add TestFeatureFlagErrorWithStaleCacheFallback test class with 4 tests: - test_timeout_error_returns_stale_cached_value - test_connection_error_returns_stale_cached_value - test_api_error_returns_stale_cached_value - test_error_without_cache_returns_none - Add negative assertions to verify $feature_flag_error is absent on success: - test_get_feature_flag_result_boolean_local_evaluation - test_get_feature_flag_result_variant_local_evaluation - test_get_feature_flag_result_boolean_decide - test_get_feature_flag_result_variant_decide
When the server returns errorsWhileComputingFlags=true AND the requested flag is not in the response, report both conditions as a comma-separated string: "errors_while_computing_flags,flag_missing" This provides better debugging context when both conditions occur.
- Add FeatureFlagError class to types.py with constants: - ERRORS_WHILE_COMPUTING, FLAG_MISSING, QUOTA_LIMITED - TIMEOUT, CONNECTION_ERROR, UNKNOWN_ERROR - api_error(status) static method for dynamic error strings - Update client.py to use FeatureFlagError constants instead of magic strings - Update all tests to use constants for maintainability This improves maintainability by: - Single source of truth for error values - IDE autocomplete and typo detection - Documentation of analytics-stable values
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Accept Union[int, str] to match APIError.status type.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
dmarticus
approved these changes
Dec 16, 2025
Contributor
dmarticus
left a comment
dmarticus
left a comment
There was a problem hiding this comment.
genuinely no notes; this should work great
haacked
added a commit
to PostHog/posthog-js
that referenced
this pull request
Dec 16, 2025
Add error tracking for feature flag evaluations, mirroring the Python SDK implementation (PostHog/posthog-python#390). Changes: - Add `FeatureFlagError` constants and type to `types.ts` - Update `getFeatureFlag` to track errors and include `$feature_flag_error` property in `$feature_flag_called` events - Add tests for error tracking scenarios Error types tracked: - `errors_while_computing_flags`: Server returned `errorsWhileComputingFlags` - `flag_missing`: Requested flag not in API response - `unknown_error`: Request failed (network error, timeout, etc.) Note: `quota_limited` detection is not supported because the core library filters out the quotaLimited field before returning. A TODO was added to consider updating the core library to preserve this information.
17 tasks
haacked
added a commit
to PostHog/posthog-js
that referenced
this pull request
Dec 17, 2025
Add error tracking for feature flag evaluations, mirroring the Python SDK implementation (PostHog/posthog-python#390). Changes: - Add `FeatureFlagError` constants and type to `types.ts` - Update `getFeatureFlag` to track errors and include `$feature_flag_error` property in `$feature_flag_called` events - Add tests for error tracking scenarios Error types tracked: - `errors_while_computing_flags`: Server returned `errorsWhileComputingFlags` - `flag_missing`: Requested flag not in API response - `unknown_error`: Request failed (network error, timeout, etc.) Note: `quota_limited` detection is not supported because the core library filters out the quotaLimited field before returning. A TODO was added to consider updating the core library to preserve this information.
haacked
added a commit
to PostHog/posthog-js
that referenced
this pull request
Dec 18, 2025
…ion failures (#2782) * Add `$feature_flag_error` property to track flag evaluation failures Add error tracking for feature flag evaluations, mirroring the Python SDK implementation (PostHog/posthog-python#390). Changes: - Add `FeatureFlagError` constants and type to `types.ts` - Update `getFeatureFlag` to track errors and include `$feature_flag_error` property in `$feature_flag_called` events - Add tests for error tracking scenarios Error types tracked: - `errors_while_computing_flags`: Server returned `errorsWhileComputingFlags` - `flag_missing`: Requested flag not in API response - `unknown_error`: Request failed (network error, timeout, etc.) Note: `quota_limited` detection is not supported because the core library filters out the quotaLimited field before returning. A TODO was added to consider updating the core library to preserve this information. * Add quota_limited error detection to feature flag tracking Preserve quotaLimited field in core library response so Node SDK can detect when feature flags are unavailable due to quota limits. * Add changeset * Rename test to specify error type being tested Makes the test name more specific by including the exact error type (unknown_error) being tested, aligning with the pattern used in other tests in the same file. * Remove unused FeatureFlagError constants Remove TIMEOUT, CONNECTION_ERROR, and apiError() which were added for future use but are not currently implemented. These can be added back when the core library is updated to preserve HTTP status information for proper error differentiation.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
$feature_flag_errorproperty to the$feature_flag_calledevent to help debug flag evaluation failures.Error values:
errors_while_computing_flags- Server returnederrorsWhileComputingFlags=trueflag_missing- Requested flag not in API responsequota_limited- Rate/quota limit exceededtimeout- Request timed outconnection_error- Network connectivity issueapi_error_{status}- Server error with HTTP status code (e.g.,api_error_500)unknown_error- Unexpected exceptionsTest plan