wire: Optimize writes to bytes.Buffer

[jrick]

This special cases writes to bytes.Buffer, which is always the writer type written to by WriteMessageN. There are several optimizations that can be implemented by special casing this type:

First, pulling temporary short buffers from binary freelist can be skipped entirely, and instead the binary encoding of integers can be appended directly to its existing capacity. This avoids the synchronization cost to add and remove buffers from the free list, and for applications which only ever write wire messages with WriteMessageN, the allocation and ongoing garbage collection scanning cost to for these buffers can be completely skipped.

Second, special casing the buffer type in WriteVarString saves us from creating a temporary heap copy of a string, as the buffer's WriteString method can be used instead.

Third, special casing the buffer allows WriteMessageN to calculate the serialize size and grow its buffer so all remaining appends for writing block and transactions will not have to reallocate the buffer's backing allocation. This same optimization can be applied to other messages in the future.

[jrick]

Now rebased over #3584. Below is the performance improvement with both PRs, relative to master:

$ benchstat old.txt new.txt                                                   
goos: openbsd
goarch: amd64
pkg: github.com/decred/dcrd/wire
cpu: AMD Ryzen 7 5800X3D 8-Core Processor           
                │   old.txt   │               new.txt               │
                │   sec/op    │   sec/op     vs base                │
WriteMessageN-8   2.784µ ± 2%   1.652µ ± 1%  -40.67% (p=0.000 n=10)

                │  old.txt   │              new.txt               │
                │    B/op    │    B/op     vs base                │
WriteMessageN-8   592.0 ± 0%   328.0 ± 0%  -44.59% (p=0.000 n=10)

                │  old.txt   │              new.txt               │
                │ allocs/op  │ allocs/op   vs base                │
WriteMessageN-8   7.000 ± 0%   3.000 ± 0%  -57.14% (p=0.000 n=10)

[davecgh]

Looks great overall. This help further reduce the number of allocations the GC has to deal with.

I've identified a few nits inline.

[davecgh]

Please name this writeUint16LE to make it clear it encodes it using little endian. This is consistent with other cases in the code such as putUint16LE in txscript and WriteUint16LE of the blake256 hashers.

The only reason the existing exported funcs on the binaryFreeList weren't named that way is because they took the byte order as a parameter.

I really prefer not having to dig further into the write funcs to clearly and ambiguously immediately see which byte order is being used at the call sites.

[davecgh]

Same here, but writeUint32LE.

[davecgh]

writeUint64LE

[davecgh]

I don't really have any issue with changing these, but what's the motivation of not using the arguably slightly more readable version that it's replacing?

Also, if this is done, I believe the *[4]byte: and *[CommandSize]uint8: cases can be removed from writeElement since nothing else uses those sizes.

[jrick]

writeElements causes an additional allocation.

it's obviously handy to have it for the message serialization implementations, but this is quite low hanging fruit as we can knock out an additional allocation on every message written over the wire.

[davecgh]

Ah, right. I was thinking the elems struct would still end up on the heap due to the call to Write, but it won't because it's now calling bytes.Buffer.Write which takes raw bytes.

So, we don't really need the elems struct anymore either then. It was a nice optimization to reduce the individual allocs down to 1 for use with writeElements, but now that none of the individual elements escape, it wouldn't matter.