Skip to content

feat: implement DynamoDBDataSource + tests#534

Open
beekld wants to merge 9 commits into
mainfrom
beeklimt/SDK-2362
Open

feat: implement DynamoDBDataSource + tests#534
beekld wants to merge 9 commits into
mainfrom
beeklimt/SDK-2362

Conversation

@beekld
Copy link
Copy Markdown
Contributor

@beekld beekld commented May 14, 2026

Summary

Implements DynamoDBDataSource, similar to RedisDataSource, using the Proxy Relay schema as per the schema.

DynamoDBClientOptions options;
options.region = "us-east-1";
options.endpoint = "http://localhost:8000";
auto src = DynamoDBDataSource::Create("my-table", "prefix", options);

Differences from the Redis source

  • All() paginates DynamoDB Query responses via LastEvaluatedKey; Redis hgetall is single-shot.
  • AWS SDK InitAPI/ShutdownAPI are owned by an singleton (AwsSdkGuard); Redis++ needs no equivalent.
  • Config takes a DynamoDBClientOptions struct (region/endpoint/credentials) instead of a single URI string.
  • All reads use ConsistentRead = true to avoid DynamoDB's default eventually-consistent staleness (same as Java)
  • When no prefix is configured, this source uses bare partition-key values (features, $inited), matching what Relay writes per the spec. The Redis source unconditionally does prefix + ":" + name, so an empty Redis prefix would produce leading-colon keys (:features) that no Relay deployment writes. That divergence is never exercised — Redis defaults to a non-empty prefix and the Redis tests always pass one, but wanted to note it.

Test plan

  • 24/24 tests green against amazon/dynamodb-local
  • Unit-only error tests pass without a backend.
  • CI Ubuntu job runs against amazon/dynamodb-local; macOS / Windows build-only (matches Redis workflow).
  • No AWS header leakage in public surface.

Note

Medium Risk
Adds a new DynamoDB-backed data source implementation plus integration tests and CI wiring, and adjusts CMake/AWS SDK build behavior (static vs shared), which could impact build/link outcomes across platforms.

Overview
Implements a real DynamoDBDataSource (replacing the prior scaffold) that reads flags/segments from a Relay-compatible DynamoDB table, including Get, All (with pagination), and Initialized checks, with consistent-read requests and structured error handling for malformed rows.

Introduces DynamoDBClientOptions plus internal helpers (AwsSdkGuard, client factory, namespace/prefix utilities) to manage AWS SDK lifecycle and client configuration without leaking AWS headers in the public API.

Adds a new tests target with DynamoDB Local-backed coverage and updates CI/workflows and release scripts to enable DynamoDB builds/tests when requested; also forces the AWS SDK FetchContent build to static archives with cache save/restore to avoid macOS shared-link issues.

Reviewed by Cursor Bugbot for commit c4f97d6. Bugbot is set up for automated code reviews on this repo. Configure here.

@beekld beekld marked this pull request as ready for review May 14, 2026 23:25
@beekld beekld requested a review from a team as a code owner May 14, 2026 23:25
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default mode and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d936b5d. Configure here.

Comment thread libs/server-sdk-dynamodb-source/src/dynamodb_attributes.hpp Outdated
Error{"DynamoDB row missing expected 'item' attribute"});
}

return SerializedItemDescriptor::Present(0, it->second.GetS());
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should type check or empty check calls to GetS? I have claude working on a test so I can understand scope a little easier.

// first use and torn down during normal program termination via C++ static
// destruction.
//
// Static-destruction ordering caveat: if a caller stashes a raw AWS SDK
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to surface this someplace more customer visible? Readme? Or just when we make docs?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess I need to read further to understand if there is external impact. On some SDKs we allow passing clients, but they would then theoretically be non-owned. So more of an internal problem?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, the AWS Api is a bit icky here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if potentially we should have a way to disable the AWS SDK shutdown and let the customer be responsible for shutting it down?

@kinyoklion
Copy link
Copy Markdown
Member

From Claude.

AttributeValue::GetS() on a non-String item attribute silently returns "". The source builds Present(0, ""), and downstream JsonDeserializer::DeserializeJsonDescriptor calls the throwing boost::json::parse("") overload — uncaught all the way to the SDK's evaluation entry point. DynamoDB enforces type only on key attributes, so a non-Relay writer can trigger this.

Verified with two tests against amazon/dynamodb-local: both fail on current code (result.has_value() == true), pass after the fix.

Proposed fix and tests:

// libs/server-sdk-dynamodb-source/src/dynamodb_source.cpp -- in Get(), after the
// "missing item attribute" check
auto const& serialized = it->second.GetS();
if (serialized.empty()) {
    return tl::make_unexpected(
        Error{"DynamoDB 'item' attribute is empty or not of type S"});
}
return SerializedItemDescriptor::Present(0, serialized);

// Same pattern inside All()'s per-row loop, applied to item_it->second.GetS().
// The third GetS() call (sort key, line 119) is left alone -- DynamoDB enforces
// the key schema at insert time, so it's guaranteed type S.

Test helper added to tests/prefixed_dynamodb_client.hpp:

void PutRowWithNumericItem(std::string const& ns_suffix,
                           std::string const& key) const {
    Aws::DynamoDB::Model::PutItemRequest request;
    request.SetTableName(table_name_);
    request.AddItem("namespace",
                    Aws::DynamoDB::Model::AttributeValue{Prefixed(ns_suffix)});
    request.AddItem("key", Aws::DynamoDB::Model::AttributeValue{key});
    Aws::DynamoDB::Model::AttributeValue numeric_item;
    numeric_item.SetN("12345");
    request.AddItem("item", numeric_item);
    auto outcome = client_.PutItem(request);
    if (!outcome.IsSuccess()) {
        FAIL() << "..." << outcome.GetError().GetMessage();
    }
}

Tests added to tests/dynamodb_source_test.cpp:

TEST_F(DynamoDBTests, GetReturnsErrorWhenItemAttributeIsNotString) {
    WithPrefixedClient(prefix_, [&](auto const& client) {
        client.PutRowWithNumericItem("features", "foo");
    });
    auto const result = source->Get(FlagKind{}, "foo");
    ASSERT_FALSE(result);
}

TEST_F(DynamoDBTests, AllReturnsErrorWhenItemAttributeIsNotString) {
    WithPrefixedClient(prefix_, [&](auto const& client) {
        client.PutFlag(Flag{"foo", 1, true});
        client.PutRowWithNumericItem("features", "bar");
    });
    auto const result = source->All(FlagKind{});
    ASSERT_FALSE(result);
}

Suite is 28/28 green with the fix applied. Reverting the source change makes only the two new tests red, with Actual: true / Expected: false — confirming each test is load-bearing for this specific defect.

Broader hardening worth a separate PR: wrap boost::json::parse in libs/server-sdk/src/data_components/serialization_adapters/json_deserializer.hpp:54 in try/catch (or use the error_code overload) so any serialized data source, including Redis, is one bad row away from a structured error rather than an uncaught exception.

DynamoDB does not enforce a type schema on non-key attributes, so a
non-Relay writer (manual put-item, schema-migration tool, etc.) can
produce a row whose 'item' attribute is stored as Number/Binary/Null/
Bool/List/Map instead of String. AttributeValue::GetS() on such a
value silently returns an empty string. Without a check, the source
returns Present(0, "") and the empty string flows into
JsonDeserializer::DeserializeJsonDescriptor's throwing
boost::json::parse, which propagates an uncaught exception out of
the SDK's evaluation entry point.

Treat an empty GetS() result as a structured error in both Get() and
All(). The sort-key GetS() (All() line 119) is left untouched -- the
DynamoDB service enforces the table's key schema at insert time, so
the sort key is guaranteed type S by table-schema invariant.

Adds PutRowWithNumericItem test helper that stores 'item' as a Number
via AttributeValue::SetN, and two bug-proving tests
(GetReturnsErrorWhenItemAttributeIsNotString,
AllReturnsErrorWhenItemAttributeIsNotString) that fail without the
fix and pass with it. Full dynamodb suite is 28/28 green.

Per #534 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants