add QueryBinary, an alloc-free way to read all rows into a buffer#58
add QueryBinary, an alloc-free way to read all rows into a buffer#58
Conversation
| return res | ||
| } | ||
|
|
||
| func (stmt *Stmt) StepAllBinary(dstBuf []byte) (n int, err error) { |
There was a problem hiding this comment.
:ohno:
you could, now you're going this route... make this AppendAllBinary(dst []byte) ([]byte, int, error)
and it could loop over the C function, with it returning every time a row doesn't fit for AppendAllBinary to grow dst, then pass it back in to read more rows.
that way you get a clean Go api, an amortized number of C calls over the growing size of dst, and only a single pass through the query
There was a problem hiding this comment.
Look at the caller of this. That's where clean Go API will go. This is the lower level wrapper.
There was a problem hiding this comment.
I was suggesting that if you move that logic in here, instead of calling ResetAndClear every time you need to grow the array, you just have to start encoding the current row again. That way you only have to step through a query once. (Which is nice if someone throws in a huge complex join that requires sqlite go off and do some huge amount of pre-work.)
There was a problem hiding this comment.
I don't want to attempt to bail out of step iteration in C, go back to Go, and then try to get back into C where I left off. Just not worth the complexity.
All or nothing.
Callers can arrange good enough buffer sizes in pools.
bradfitz
force-pushed
the
bradfitz/binread
branch
3 times, most recently
from
7cd933b to
a6c43b3
Compare
bradfitz
force-pushed
the
bradfitz/binread
branch
3 times, most recently
from
6af984e to
d10e7f3
Compare
WIP; goal is alloc-free reads of a query into a Go-provided buffer. Go code can then parse the simple binary format and alloc if needed (doing its own cache lookups, including alloc-free m[string([]byte)] lookups, and returning existing Views if data is unmodified) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
bradfitz
force-pushed
the
bradfitz/binread
branch
from
d10e7f3 to
1c6bbb2
Compare
2 participants
WIP; goal is alloc-free reads of a query into a Go-provided buffer. Go code can then parse the simple binary format and alloc if needed (doing its own cache lookups, including alloc-free m[string([]byte)] lookups, and returning existing Views if data is unmodified)
(Counterproposal to #43)