Reuse grpc buffer in querier's store-gateway stream

[eeldaly]

What this PR does:
Reuse grpc buffer in querier's store-gateway stream to reduce memory usage in large queries

Current heap dump during heavy queries:

GRPC Store-Gateway Bench Test NEW:

BenchmarkGrpcStoreGatewayCalls-10    	    1501	    929363 ns/op	   59943 B/op	     455 allocs/op
BenchmarkGrpcStoreGatewayCalls-10    	    1723	    693808 ns/op	   49132 B/op	     456 allocs/op
BenchmarkGrpcStoreGatewayCalls-10    	    1818	    649024 ns/op	   49703 B/op	     454 allocs/op
BenchmarkGrpcStoreGatewayCalls-10    	    1516	    949340 ns/op	   46230 B/op	     453 allocs/op
BenchmarkGrpcStoreGatewayCalls-10    	    1765	    710793 ns/op	   50579 B/op	     453 allocs/op
BenchmarkGrpcStoreGatewayCalls-10    	    1670	    658193 ns/op	   50318 B/op	     458 allocs/op

GRPC Store-Gateway Bench Test OLD:

BenchmarkGrpcStoreGatewayCalls-10    	    1869	    717663 ns/op	  553930 B/op	     482 allocs/op
BenchmarkGrpcStoreGatewayCalls-10    	    1275	    849913 ns/op	  561863 B/op	     477 allocs/op
BenchmarkGrpcStoreGatewayCalls-10    	    1290	    834707 ns/op	  567643 B/op	     478 allocs/op
BenchmarkGrpcStoreGatewayCalls-10    	    1356	   1100831 ns/op	  560738 B/op	     477 allocs/op
BenchmarkGrpcStoreGatewayCalls-10    	    1146	   1179173 ns/op	  573945 B/op	     477 allocs/op
BenchmarkGrpcStoreGatewayCalls-10    	    1335	    875292 ns/op	  556640 B/op	     481 allocs/op

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
• docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

[yeya24]

This seems not the right place for benchmarking store gateways. Can you move to its own folder?

[yeya24]

We shouldn't change Thanos via vendor

[eeldaly]

I'll change upstream if this helps, still in process of testing and wanted to get opinions on draft pr since I'm not very familiar with grpc

[harry671003]

I think we should also apply this fix in distributor_queryable.go when querying from ingesters. I remember seeing a similar kind of heap profile for ingester queries as well.

[harry671003]

This will a problem. Can we avoid having upstream changes in Thanos. We can't have this circular dependency with Thanos importing Cortex.

[harry671003]

Could you scope this PR down to just the detach? That alone fixes the OOM (buffer gets GC'd once nothing references it). The pool-reuse part needs the proto changes which create a circular dep with Thanos.

In parallel we can open an upstream thanos PR to add a generic BufRef to SeriesResponse — once vendored back, we can swap detach for Free() to get the alloc wins.