Skip to content

Comments

feat: filter out docs#297

Open
forshev wants to merge 27 commits intomainfrom
290-filter-out-docs
Open

feat: filter out docs#297
forshev wants to merge 27 commits intomainfrom
290-filter-out-docs

Conversation

@forshev
Copy link
Contributor

@forshev forshev commented Dec 12, 2025 •
edited
Loading

Description

This is PoC of "delete imitation"

Fixes #290

  • I have read and followed all requirements in CONTRIBUTING.md;
  • I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

@forshev forshev marked this pull request as draft December 12, 2025 13:33
@codecov-commenter
Copy link

codecov-commenter commented Dec 12, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 54.88810% with 383 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.67%. Comparing base (8aefcc9) to head (3ef55c1).

Files with missing lines Patch % Lines
docsfilter/docs_filter.go 24.55% 159 Missing and 10 partials ⚠️
docsfilter/loader.go 55.17% 20 Missing and 19 partials ⚠️
docsfilter/encoding.go 78.80% 17 Missing and 15 partials ⚠️
util/fs.go 38.77% 17 Missing and 13 partials ⚠️
frac/active_index.go 63.04% 15 Missing and 2 partials ⚠️
docsfilter/filter.go 0.00% 16 Missing ⚠️
cmd/seq-db/seq-db.go 0.00% 14 Missing ⚠️
docsfilter/merged_iterator.go 81.48% 7 Missing and 3 partials ⚠️
frac/sealed_index.go 70.96% 8 Missing and 1 partial ⚠️
docsfilter/iterator_asc.go 75.75% 5 Missing and 3 partials ⚠️
... and 8 more
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #297      +/-   ##
==========================================
- Coverage   71.42%   70.67%   -0.76%     
==========================================
  Files         205      214       +9     
  Lines       14910    15693     +783     
==========================================
+ Hits        10650    11091     +441     
- Misses       3484     3769     +285     
- Partials      776      833      +57

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cheb0
cheb0 reviewed Dec 15, 2025
View reviewed changes
docsfilter/metrics.go Outdated
diskUsage = promauto.NewGaugeVec(prometheus.GaugeOpts{
Namespace: "seq_db_store",
Subsystem: "filters",
Name: "disk_usage_bytes_total",
Copy link
Member

@cheb0 cheb0 Dec 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to company standards, only counters now can (and must) have _total suffix. Gauges can not end with _total

@forshev forshev force-pushed the 290-filter-out-docs branch from 6b1e6c5 to f98406a Compare December 26, 2025 11:24
@forshev forshev force-pushed the 290-filter-out-docs branch from 35c810c to 7baeaef Compare January 21, 2026 09:05
@github-actions
Copy link
Contributor

github-actions bot commented Jan 21, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
FindSequence_Random/medium-4 467eb3 61c22e
9583.66 MB/s 7535.90 MB/s 0.79 🔴
106.80 ns/op 135.90 ns/op 1.27 🔴
FindSequence_Random/small-4 467eb3 61c22e
5128.78 MB/s 4585.96 MB/s 0.89 🔴
49.91 ns/op 55.82 ns/op 1.12 🔴
Indexer-4 467eb3 61c22e
265877562.00 ns/op 296289980.00 ns/op 1.11 🔴

@forshev forshev force-pushed the 290-filter-out-docs branch from 7baeaef to 5e040ab Compare January 21, 2026 09:10
@github-actions
Copy link
Contributor

github-actions bot commented Jan 21, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
FindSequence_Random/small-4 467eb3 1045a0
5128.78 MB/s 3914.73 MB/s 0.76 🔴
49.91 ns/op 65.39 ns/op 1.31 🔴
GenerateDocs-4 467eb3 1045a0
975.70 ns/op 1179.00 ns/op 1.21 🔴

@forshev forshev force-pushed the 290-filter-out-docs branch from 5e040ab to 5fb5af0 Compare January 21, 2026 12:43
@github-actions
Copy link
Contributor

github-actions bot commented Jan 21, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
ESBulk-4 abfd3f 5035be
378.00 B/op 444.00 B/op 1.17 🔴
FindSequence_Random/medium-4 abfd3f 5035be
12756.46 MB/s 9068.94 MB/s 0.71 🔴
80.27 ns/op 112.90 ns/op 1.41 🔴
FindSequence_Random/small-4 abfd3f 5035be
6511.28 MB/s 4151.19 MB/s 0.64 🔴
39.32 ns/op 61.67 ns/op 1.57 🔴

@forshev forshev force-pushed the 290-filter-out-docs branch from 5fb5af0 to 277508f Compare January 23, 2026 09:59
@github-actions
Copy link
Contributor

github-actions bot commented Jan 23, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
Bitmask-4 32489f 90d653
0.00 B/op 3.00 B/op NaN 🔴
FindSequence_Random/medium-4 32489f 90d653
9785.69 MB/s 8062.37 MB/s 0.82 🔴
104.60 ns/op 127.00 ns/op 1.21 🔴
GenerateDocs-4 32489f 90d653
958.10 ns/op 1143.00 ns/op 1.19 🔴

@forshev forshev force-pushed the 290-filter-out-docs branch from 4846a3e to 3deb507 Compare February 3, 2026 11:10
@github-actions
Copy link
Contributor

github-actions bot commented Feb 3, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
FindSequence_Random/small-4 174405 7d5d8f
10280.15 MB/s 4422.80 MB/s 0.43 🔴
24.90 ns/op 57.88 ns/op 2.32 🔴
FindSequence_Random/tiny-4 174405 7d5d8f
2491.86 MB/s 2183.17 MB/s 0.88 🔴
25.68 ns/op 29.32 ns/op 1.14 🔴
GenerateDocs-4 174405 7d5d8f
952.20 ns/op 1126.00 ns/op 1.18 🔴
ProcessDocuments-4 174405 7d5d8f
381.00 B/op 33066.00 B/op 86.79 🔴
4.00 allocs/op 5.00 allocs/op 1.25 🔴

@forshev forshev force-pushed the 290-filter-out-docs branch from 3deb507 to 6b2cfb2 Compare February 3, 2026 14:03
@github-actions
Copy link
Contributor

github-actions bot commented Feb 3, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
FindSequence_Random/small-4 174405 4712c5
10280.15 MB/s 4097.76 MB/s 0.40 🔴
24.90 ns/op 62.47 ns/op 2.51 🔴

@forshev forshev force-pushed the 290-filter-out-docs branch from 6b2cfb2 to 37173f0 Compare February 5, 2026 06:11
@github-actions
Copy link
Contributor

github-actions bot commented Feb 5, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
FindSequence_Random/large-4 77fae2 f8591d
13987.92 MB/s 12148.77 MB/s 0.87 🔴
1171.00 ns/op 1349.00 ns/op 1.15 🔴
FindSequence_Random/small-4 77fae2 f8591d
5061.19 MB/s 4319.71 MB/s 0.85 🔴
50.58 ns/op 59.26 ns/op 1.17 🔴
GenerateDocs-4 77fae2 f8591d
949.50 ns/op 1106.00 ns/op 1.16 🔴

@github-actions
Copy link
Contributor

github-actions bot commented Feb 5, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
FindSequence_Random/tiny-4 77fae2 ef9cdd
2850.02 MB/s 2399.97 MB/s 0.84 🔴
22.46 ns/op 26.67 ns/op 1.19 🔴

@forshev forshev force-pushed the 290-filter-out-docs branch from 1e19ca1 to fc34615 Compare February 5, 2026 14:33
@github-actions
Copy link
Contributor

github-actions bot commented Feb 5, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
FindSequence_Random/medium-4 77fae2 32ec58
11769.73 MB/s 10062.12 MB/s 0.85 🔴
87.00 ns/op 101.80 ns/op 1.17 🔴
FindSequence_Random/tiny-4 77fae2 32ec58
2850.02 MB/s 2247.37 MB/s 0.79 🔴
22.46 ns/op 28.48 ns/op 1.27 🔴
GenerateDocs-4 77fae2 32ec58
949.50 ns/op 1120.00 ns/op 1.18 🔴

@forshev forshev force-pushed the 290-filter-out-docs branch from fc34615 to ef49f29 Compare February 6, 2026 08:47
@github-actions
Copy link
Contributor

github-actions bot commented Feb 6, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
FindSequence_Random/medium-4 6ae568 070d19
13517.73 MB/s 8992.43 MB/s 0.67 🔴
75.75 ns/op 113.90 ns/op 1.50 🔴
FindSequence_Random/small-4 6ae568 070d19
5683.27 MB/s 3800.59 MB/s 0.67 🔴
45.04 ns/op 67.36 ns/op 1.50 🔴
GenerateDocs-4 6ae568 070d19
957.90 ns/op 1118.00 ns/op 1.17 🔴
ProcessDocuments-4 6ae568 070d19
4.00 allocs/op 5.00 allocs/op 1.25 🔴

@github-actions
Copy link
Contributor

github-actions bot commented Feb 6, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
Bitmask-4 6ae568 ce7881
0.00 B/op 3.00 B/op NaN 🔴
FindSequence_Random/medium-4 6ae568 ce7881
13517.73 MB/s 8591.23 MB/s 0.64 🔴
75.75 ns/op 119.20 ns/op 1.57 🔴

@forshev forshev force-pushed the 290-filter-out-docs branch from 2bfa582 to e39db32 Compare February 6, 2026 13:57
@github-actions
Copy link
Contributor

github-actions bot commented Feb 6, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
Bitmask-4 6ae568 0023d4
0.00 B/op 3.00 B/op NaN 🔴
FindSequence_Random/small-4 6ae568 0023d4
5683.27 MB/s 3844.97 MB/s 0.68 🔴
45.04 ns/op 66.58 ns/op 1.48 🔴

@forshev forshev force-pushed the 290-filter-out-docs branch from e39db32 to c9a8505 Compare February 10, 2026 12:49
@github-actions
Copy link
Contributor

github-actions bot commented Feb 10, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
ESBulk-4 772ba3 671de3
345.00 B/op 439.00 B/op 1.27 🔴
FindSequence_Random/small-4 772ba3 671de3
5411.55 MB/s 3841.34 MB/s 0.71 🔴
47.31 ns/op 66.64 ns/op 1.41 🔴
GenerateDocs-4 772ba3 671de3
964.70 ns/op 1156.00 ns/op 1.20 🔴

@dkharms dkharms added the feature New feature or request label Feb 10, 2026
@forshev forshev force-pushed the 290-filter-out-docs branch from c9a8505 to 72d04da Compare February 18, 2026 09:49
forshev added 26 commits February 20, 2026 15:12
@forshev
feat(config): allow multiple filtering requests
231916e
@forshev
feat(config): allow multiple filtering requests
fe3d331
@forshev
feat(config): allow multiple filtering requests
5cbc59e
@forshev
feat: remove imitation PoC - write path
551e995
@forshev
feat(docsfilter): store lids in binary format
b85d410
@forshev
feat(docsfilter): store lids in binary format
1447307
@forshev
feat(docsfilter): store lids in binary format
c8dec96
@forshev
feat(docsfilter): method to read filtered lids
e3bdefa
@forshev
feat(docsfilter): filter ids on search
e6c2807
@forshev
feat(docsfilter): filter ids on fetch for sealed fracs
6d09466
@forshev
feat(docsfilter): support filtering on active fraction
290ab4a
@forshev
feat(docsfilter): refresh frac tobstones after sealing
43ebf5d
@forshev
fix: fix rebase
3da6a91
@forshev
fix: fix tests
79fc418
@forshev
chore(docsfilter): organize tombstone files in blocks
e18b899
@forshev
chore(docsfilter): compress lids blocks in tombstone files
b3d9d59
@forshev
chore(docsfilter): handle errors
6fe5d44
@forshev
chore(docsfilter): add iterator to lazy load tombstones
72599ea
@forshev
chore(docsfilter): merge multiple iterators into one
475f6fd
@forshev
chore(docsfilter): use iterators in search and fetch
dd863ec
@forshev
chore(docsfilter): remove tombstones when frac is evicted
e99b6bf
@forshev
refactor(docsfilter): move common parts from iterators
998acde
@forshev
chore(docsfilter): write metrics
fb00fdc
@forshev
chore(docsfilter): cache tombstones files' headers
785ef7b
@forshev
fix(docs filter): skip missing fracs
4d74ec6
@forshev
fix(docs filter): open tombstone files only when necessary
229fcb8
@forshev forshev force-pushed the 290-filter-out-docs branch from fa17c29 to 229fcb8 Compare February 20, 2026 10:12
@github-actions
Copy link
Contributor

github-actions bot commented Feb 20, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
AggDeep/size=1000-4 8aefcc 4dbe1e
4817.00 ns/op 5860.00 ns/op 1.22 🔴
AggWide/size=1000-4 8aefcc 4dbe1e
4804.00 ns/op 5964.00 ns/op 1.24 🔴
AggWide/size=10000-4 8aefcc 4dbe1e
47761.00 ns/op 59577.00 ns/op 1.25 🔴
And/size=10000-4 8aefcc 4dbe1e
4.78 ns/op 5.31 ns/op 1.11 🔴
AndTree/size=1000-4 8aefcc 4dbe1e
4.70 ns/op 5.29 ns/op 1.13 🔴

@github-actions
Copy link
Contributor

github-actions bot commented Feb 20, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
AggDeep/size=1000-4 8aefcc 5c4d8e
4817.00 ns/op 5755.00 ns/op 1.19 🔴
AggDeep/size=1000000-4 8aefcc 5c4d8e
4837044.00 ns/op 5829087.00 ns/op 1.21 🔴
AggWide/size=1000-4 8aefcc 5c4d8e
4804.00 ns/op 5836.00 ns/op 1.21 🔴
FindSequence_Random/medium-4 8aefcc 5c4d8e
10747.09 MB/s 9448.75 MB/s 0.88 🔴

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cheb0 cheb0 cheb0 left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

feature New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Make better mechanism to hide documents

4 participants

@forshev @codecov-commenter @cheb0 @dkharms