Configurable index disabling for virtual columns

[nozjkoitop]

Description

In some of our environments we observed a significant query performance regression after upgrading to Druid 29.0. This regression appears to be related to the virtual column bitmap indexing optimization introduced in #15585 and #15633.

While the change improves performance for many cases by enabling bitmap index creation for expression-based virtual columns, we found that in certain workloads the index computation becomes unexpectedly expensive. In our deployments, building these indices accounted for more than 90% of the CPU usage on Historical nodes during query execution, leading to severe degradation in overall query latency.

For one representative workload, the average query runtime increased from 10.01 seconds to 150.18 seconds compared to earlier versions. Once expression-based virtual column bitmap index creation was disabled, the regression disappeared and query performance returned to expected levels.

This PR addresses this issue by preventing the optimization from triggering in scenarios where the cost of index computation outweighs its benefit, avoiding major regressions in affected environments.

Added a new query context parameter, maxVirtualColumnsForBitmapIndexing (default: Integer.MAX_VALUE), which sets the virtual-column count threshold beyond which Druid stops using bitmap indexes for filters on virtual columns.

Release note

Added a safeguard to skip virtual-column bitmap indexing when it is likely to be counterproductive, falling back to non-indexed filtering to preserve expected performance.

---

This PR has:

• been self-reviewed.
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[clintropolis]

In some of our environments we observed a significant query performance regression after upgrading to Druid 29.0. This regression appears to be related to the virtual column bitmap indexing optimization introduced in #15585 and #15633.

have you tried in versions newer than 29? #15838 was in 30, and #17055/#17125 in 31, which should make it so that expression indexes are only used in cases where there would be less work to do than a full scan (which has to compute the expression value and perform the match for every row), which i think should reduce problems like you are seeing.

I would be very interested if there are still cases where expression indexes cause a slowdown after those changes, since it would indicate that perhaps the cost estimate needs adjustment.

[nozjkoitop]

Thanks for the references, unfortunately this still reproduces for us on 31.
To put numbers behind this, i can try adding a benchmark that shows the impact

[clintropolis]

Thanks for the references, unfortunately this still reproduces for us on 31.
To put numbers behind this, i can try adding a benchmark that shows the impact

that would be great, there is SqlExpressionBenchmark which uses a data generator to have a collection of columns with various types and data distributions maybe you can add something that looks approximately like the problem you're running into.

I would like to try to see if this is something we can improve without adding a new manual parameter

[nozjkoitop]

added a case with broad, expression-heavy virtual-column filters
there were likely other improvements in later versions, but I still saw ~12% better performance with indexing disabled

[clintropolis]

hmm, I still show the added benchmark query as faster with indexes, especially for vectorized processing. I didn't pull your branch, just used apache master with the query added and to test no indexes modified this line ExpressionVirtualColumn to return NoIndexesColumnIndexSupplier.getInstance()

current master:

Benchmark                        (complexCompression)  (deferExpressionDimensions)  (jsonObjectStorageEncoding)  (query)  (rowsPerSegment)  (schemaType)  (storageType)  (stringEncoding)  (vectorize)  Mode  Cnt    Score    Error  Units
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       61           1500000      explicit           MMAP              UTF8        false  avgt    5  437.811 ± 57.658  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       61           1500000      explicit           MMAP              UTF8        force  avgt    5  371.432 ±  2.740  ms/op
SqlExpressionBenchmark.querySql                  NONE                   fixedWidth                        SMILE       61           1500000      explicit           MMAP              UTF8        false  avgt    5  417.081 ± 10.159  ms/op
SqlExpressionBenchmark.querySql                  NONE                   fixedWidth                        SMILE       61           1500000      explicit           MMAP              UTF8        force  avgt    5  368.717 ±  8.266  ms/op
SqlExpressionBenchmark.querySql                  NONE         fixedWidthNonNumeric                        SMILE       61           1500000      explicit           MMAP              UTF8        false  avgt    5  417.167 ±  4.085  ms/op
SqlExpressionBenchmark.querySql                  NONE         fixedWidthNonNumeric                        SMILE       61           1500000      explicit           MMAP              UTF8        force  avgt    5  369.220 ± 10.618  ms/op
SqlExpressionBenchmark.querySql                  NONE                       always                        SMILE       61           1500000      explicit           MMAP              UTF8        false  avgt    5  417.864 ± 13.421  ms/op
SqlExpressionBenchmark.querySql                  NONE                       always                        SMILE       61           1500000      explicit           MMAP              UTF8        force  avgt    5  344.602 ±  9.518  ms/op

modified so that no indexes are used:

Benchmark                        (complexCompression)  (deferExpressionDimensions)  (jsonObjectStorageEncoding)  (query)  (rowsPerSegment)  (schemaType)  (storageType)  (stringEncoding)  (vectorize)  Mode  Cnt     Score    Error  Units
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       61           1500000      explicit           MMAP              UTF8        false  avgt    5   505.174 ±  7.250  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       61           1500000      explicit           MMAP              UTF8        force  avgt    5  1165.574 ± 37.120  ms/op
SqlExpressionBenchmark.querySql                  NONE                   fixedWidth                        SMILE       61           1500000      explicit           MMAP              UTF8        false  avgt    5   501.502 ± 33.384  ms/op
SqlExpressionBenchmark.querySql                  NONE                   fixedWidth                        SMILE       61           1500000      explicit           MMAP              UTF8        force  avgt    5  1160.577 ± 43.142  ms/op
SqlExpressionBenchmark.querySql                  NONE         fixedWidthNonNumeric                        SMILE       61           1500000      explicit           MMAP              UTF8        false  avgt    5   501.388 ± 24.289  ms/op
SqlExpressionBenchmark.querySql                  NONE         fixedWidthNonNumeric                        SMILE       61           1500000      explicit           MMAP              UTF8        force  avgt    5  1198.882 ± 57.135  ms/op
SqlExpressionBenchmark.querySql                  NONE                       always                        SMILE       61           1500000      explicit           MMAP              UTF8        false  avgt    5   508.851 ±  4.995  ms/op
SqlExpressionBenchmark.querySql                  NONE                       always                        SMILE       61           1500000      explicit           MMAP              UTF8        force  avgt    5  1206.142 ± 37.526  ms/op

[nozjkoitop]

Interesting, although I was using lz4 compression + FRONT_CODED_16_V1 encoding, ur results surprised me :) lemme give it another try

Could u share the JMH flags used please?

[clintropolis]

DRUID_BENCHMARK_CACHE_DIR=./tmp java --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -server -jar benchmarks/target/benchmarks.jar -p query=61 -p stringEncoding=UTF8 -p complexCompression=NONE -p schemaType=explicit -p storageType=MMAP org.apache.druid.benchmark.query.SqlExpressionBenchmark

I'm using java 21 on a m1 mac for additional context.

Interesting, although I was using lz4 compression + FRONT_CODED_16_V1 encoding

i think by default everything here would be using lz4 since that parameter is only for complex columns to measure #16863 and has no impact if not using complex columns, but I will try with front coding though, since significantly slower perf there would also be quite interesting to look into

[nozjkoitop]

thanks, i'll return it with matching params and let u know about results

[clintropolis]

a neat, I do show front-coded as a bit slower for non-vectorized, which is curious, will look into that a bit more

with index:

Benchmark                        (complexCompression)  (deferExpressionDimensions)  (jsonObjectStorageEncoding)  (query)  (rowsPerSegment)  (schemaType)  (storageType)   (stringEncoding)  (vectorize)  Mode  Cnt    Score    Error  Units
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5  522.387 ± 22.942  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  501.122 ± 17.559  ms/op
SqlExpressionBenchmark.querySql                  NONE                   fixedWidth                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5  547.506 ± 15.055  ms/op
SqlExpressionBenchmark.querySql                  NONE                   fixedWidth                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  446.650 ±  5.308  ms/op
SqlExpressionBenchmark.querySql                  NONE         fixedWidthNonNumeric                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5  572.099 ± 67.823  ms/op
SqlExpressionBenchmark.querySql                  NONE         fixedWidthNonNumeric                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  499.534 ± 19.926  ms/op
SqlExpressionBenchmark.querySql                  NONE                       always                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5  549.607 ± 25.846  ms/op
SqlExpressionBenchmark.querySql                  NONE                       always                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  496.660 ± 16.439  ms/op

without:

Benchmark                        (complexCompression)  (deferExpressionDimensions)  (jsonObjectStorageEncoding)  (query)  (rowsPerSegment)  (schemaType)  (storageType)   (stringEncoding)  (vectorize)  Mode  Cnt     Score    Error  Units
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5   516.153 ± 18.747  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  1154.914 ± 34.838  ms/op
SqlExpressionBenchmark.querySql                  NONE                   fixedWidth                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5   438.316 ± 24.513  ms/op
SqlExpressionBenchmark.querySql                  NONE                   fixedWidth                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  1204.622 ± 57.132  ms/op
SqlExpressionBenchmark.querySql                  NONE         fixedWidthNonNumeric                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5   431.339 ± 23.974  ms/op
SqlExpressionBenchmark.querySql                  NONE         fixedWidthNonNumeric                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  1199.698 ± 34.916  ms/op
SqlExpressionBenchmark.querySql                  NONE                       always                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5   518.188 ± 19.511  ms/op
SqlExpressionBenchmark.querySql                  NONE                       always                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  1235.943 ± 38.730  ms/op

[clintropolis]

re front-coding, i think i see what is going on, the ExpressionPredicateIndexSupplier that is computing the indexes is scanning the dictionary in order to find all of the values that match, but is using random access to get the values which for front coding means seeking in the bucket to the position we actually need which is basically where all of the cost of using it is (in exchange for the smaller sizes). I suspect when it is not using the indexes that perhaps the matcher must be able to rule out a match earlier and so fewer overall calls to FrontCodedIndexed.get are happening.

I think since we are checking every dictionary value, it would be a lot more chill for front-coding if use used the dictionary iterator instead of calling get, it needs to be exposed on DictionaryEncodedValueIndex so that the expression predicate indexes could use it. Will look into this 👍

[clintropolis]

ah yea, it is totally that, using the iterator improves the measurement on using the indexes quite a lot

with indexes using iterator:

Benchmark                        (complexCompression)  (deferExpressionDimensions)  (jsonObjectStorageEncoding)  (query)  (rowsPerSegment)  (schemaType)  (storageType)   (stringEncoding)  (vectorize)  Mode  Cnt    Score     Error  Units
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5  428.333 ±  14.320  ms/op
SqlExpressionBenchmark.querySql                  NONE                 singleString                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  364.073 ±   5.671  ms/op
SqlExpressionBenchmark.querySql                  NONE                   fixedWidth                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5  423.951 ±  12.710  ms/op
SqlExpressionBenchmark.querySql                  NONE                   fixedWidth                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  371.926 ±   5.133  ms/op
SqlExpressionBenchmark.querySql                  NONE         fixedWidthNonNumeric                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5  424.357 ±  10.445  ms/op
SqlExpressionBenchmark.querySql                  NONE         fixedWidthNonNumeric                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  419.708 ±  71.678  ms/op
SqlExpressionBenchmark.querySql                  NONE                       always                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        false  avgt    5  444.724 ± 112.962  ms/op
SqlExpressionBenchmark.querySql                  NONE                       always                        SMILE       61           1500000      explicit           MMAP  FRONT_CODED_16_V1        force  avgt    5  373.843 ±   8.409  ms/op

thanks for bringing this to attention 👍

[clintropolis]

opened #19023 to help perf for front-coding + expression indexes (and maybe a few other things)

[nozjkoitop]

appreciate the input, that's a clean fix, nice