Skip to content

[CASSANDRA-21087] Fix non-determinism of repair fuzz tests by passing randomizer to RetryStrategy #4537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nivykani wants to merge 2 commits into
base: trunk
Choose a base branch
from
+38 −1
Open

[CASSANDRA-21087] Fix non-determinism of repair fuzz tests by passing randomizer to RetryStrategy #4537

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a8d2600
CASSANDRA-21087: Fix non-determinism of repair fuzz tests by passing …
Dec 18, 2025
5d1e3e1
Fix style
Jan 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion src/java/org/apache/cassandra/config/RetrySpec.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,15 @@

import javax.annotation.Nullable;

import accord.utils.RandomSource;
import org.apache.cassandra.config.DurationSpec.LongMillisecondsBound;
import org.apache.cassandra.repair.SharedContext;
import org.apache.cassandra.service.RetryStrategy;
import org.apache.cassandra.service.TimeoutStrategy.LatencySourceFactory;
import org.apache.cassandra.service.WaitStrategy;

import static org.apache.cassandra.service.RetryStrategy.randomizers;

public class RetrySpec
{
public static class MaxAttempt
Expand Down Expand Up @@ -161,7 +164,9 @@ public static WaitStrategy toStrategy(SharedContext ctx, RetrySpec spec)
{
if (!spec.isEnabled())
return WaitStrategy.None.INSTANCE;
return RetryStrategy.parse(spec.baseSleepTime.toMilliseconds() + "ms * 2^attempts <= " + spec.maxSleepTime.toMilliseconds() + "ms,retries=" + (spec.maxAttempts.value - 1), LatencySourceFactory.none());
RandomSource randomSource = RandomSource.wrap(ctx.random().get());
RetryStrategy.WaitRandomizer randomizer = randomizers(randomSource).uniform();
return RetryStrategy.parse((int) (0.5 * spec.baseSleepTime.toMilliseconds()) + "ms * 2^attempts ... " + (int) (1.5 * spec.baseSleepTime.toMilliseconds()) + "ms * 2^attempts <= " + spec.maxSleepTime.toMilliseconds() + "ms,retries=" + (spec.maxAttempts.value - 1), LatencySourceFactory.none(), randomizer);
Copy link
Contributor

@dcapwell dcapwell Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for better history: the logic in 5.0 is https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/utils/Backoff.java#L78

to simplify that logic its

baseSleepTime ^ (attempt - 1) * (rnd.double * 0.5)

But this new retry framework doesn't allow that same thing, so we have to do things slightly differently...

* (rnd.double * 0.5) this adds jitter to the sleep, so to map this to the new framework we need to have a min and max value and it will choose a uniform value between those too; this change adds a min/max using the ... syntax and has min * 0.5 and max * 1.5 to keep the old behavior.

}

@Override
Expand Down
32 changes: 32 additions & 0 deletions test/unit/org/apache/cassandra/service/RetryStrategyTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,19 @@

package org.apache.cassandra.service;

import java.util.Random;
import java.util.concurrent.TimeUnit;
import java.util.function.IntFunction;
import java.util.function.Supplier;

import org.junit.Test;

import accord.utils.Gen;
import accord.utils.RandomTestRunner;
import org.apache.cassandra.config.DurationSpec;
import org.apache.cassandra.config.RetrySpec;
import org.apache.cassandra.repair.SharedContext;
import org.assertj.core.api.Assertions;

public class RetryStrategyTest
{
Expand Down Expand Up @@ -64,6 +71,31 @@ public void fuzzParser()
});
}

@Test
public void seededWaitRandomizer()
{
RetrySpec spec = new RetrySpec(new RetrySpec.MaxAttempt(10),
new DurationSpec.LongMillisecondsBound("200ms"),
new DurationSpec.LongMillisecondsBound("1000ms"));
long wait1 = RetrySpec.toStrategy(sharedContext(100), spec).computeWait(1, TimeUnit.MILLISECONDS);
long wait2 = RetrySpec.toStrategy(sharedContext(100), spec).computeWait(1, TimeUnit.MILLISECONDS);
long wait3 = RetrySpec.toStrategy(sharedContext(200), spec).computeWait(1, TimeUnit.MILLISECONDS);
Assertions.assertThat(wait1).isEqualTo(wait2);
Assertions.assertThat(wait1).isNotEqualTo(wait3);
}

private static SharedContext sharedContext(long seed)
{
return new SharedContext.ForwardingSharedContext(SharedContext.Global.instance)
{
@Override
public Supplier<Random> random()
{
return () -> new Random(seed);
}
};
}

private static class TestLatencySourceFactory implements TimeoutStrategy.LatencySourceFactory
{

Expand Down