API for Buffer

[jakobnissen]

I was taking with @nhz2 about how Automa (and presumably other packages) use internals from TranscodingStreams, and how it would be nice to make it stop doing that. However, TS was created by the same author as Automa, and the two packages' internals are entangled. So, for Automa to not rely on TS internals, TS needs to offer these things as API. In particular:

• Access to the .data field of a TranscodingStream, either as a Vector{UInt8} or potentially Memory{UInt8} going forward
• Get and set the .markpos and .bufferpos fields of the read Buffer of a stream
• Get the .marginpos of the read Buffer of a stream

Ideally, the Buffer object can also be obtained so the user can manipulate the Buffer directly.

Another approach is for Automa to stop using TranscodingStreams, and instead use something like BufferedStreams.jl. However, I believe TranscodingStreams might have originally been implemented to support Automa.

[nhz2]

Thank you for making this list as I don't understand the internals of Automa. I think it is fine to keep the Buffer type and .data, .markpos, .marginpos, and .bufferpos fields as they are very well documented in src/buffer.jl

Also, do you know if Automa is using the unread or unsafe_unread functions? I would like to better specify how those functions interact with position and mark.

Lastly, does Automa use any of the write functions?

[jakobnissen]

Automa does not use unread or unsafe_unread. It does use Base.write, but not writing to a TranscodingStream, so it shouldn't be of concern for this package.

From what I can tell, what Automa uses is:

• That you can get a Buffer containing the input data by doing stream.state.buffer1 - this could be changed to a getbuffer function if necessary
• The fields .markpos, .marginpos and .bufferpos, and their precise meaning
• That you can get a Vector{UInt8} by reading the .data field of a Buffer - this could perhaps be changed to a function also, since maybe in the future, this should be a Memory
• That a TranscodingStream behaves like an ordinary IO object, e.g. that eof works and returns a Bool. This is perhaps too trivial to mention.