snappy.uncompress and snappy.compress are not compatible with `python -m snappy -c/d`

[johnrfrank]

It appears that the module-level functions snappy.compress and snappy.uncompress have bitrotted and fallen behind the new framed version produced/consumed by the command-line interface.

Is the framed version adopted enough that those should be made consistent? Regardless, having this inconsistency seems like the wrong thing to do.

$ echo 'hello' | python -m snappy -c > hello.sz
$ python
Python 2.7.6 (default, Mar 22 2014, 22:59:56) [GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import snappy
snappy.uncompress(open('hello.sz').read())
Traceback (most recent call last):
File "", line 1, in
snappy.UncompressError: Error while decompressing: invalid input

open('hello.sz').read()
'\xff\x06\x00\x00sNaPpY\x01\n\x00\x00SU\xffShello\n'
d = open('hello.sz').read()
snappy.uncompress(d)
Traceback (most recent call last):
File "", line 1, in
snappy.UncompressError: Error while decompressing: invalid input

snappy.compress('hello\n')
'\x06\x14hello\n'

from cStringIO import StringIO

fh = StringIO()
snappy.stream_decompress(StringIO(d), fh)
fh.getvalue()
'hello\n'

[andrix]

Hey @johnrfrank - snappy.{compress, uncompress} where both aimed to be similar to - zlib.{compress, uncompress} interface, and internally are wrapped calls to snappy_compress and snappy_uncompress.

Later, @jtolds added support for framed format and as well the CLI, but probably we should make sure, compress & uncompress handle the framed formats as well. @jtolds - thoughts?

[jtolio]

i don't know. the framing format isn't usually what people mean when they talk about snappy compression, so i am reluctant to make it work automatically. i think people should know explicitly what format they're using

[johnrfrank]

Why are there two incompatible formats? Is the framing format not intended to replace the old format?=

[jtolio]

tl;dr: they aren't incompatible really, one is just a wrapper around the other.

Snappy is (or was) intended to compress short pieces of in-memory data. It isn't(/wasn't) intended for compressing large streams of data that don't fit in memory. So, Snappy compression in general is for taking a "short" range of bytes (something like 32K or less) and not doing anything else to it other than compressing it (no large amount of headers and versioning added, etc). Very simple and straightforward. You can get a feel for the simplicity of Snappy's main interface in the README: https://code.google.com/p/snappy/source/browse/trunk/README. Note that the two methods (Compress and Uncompress) only take strings.

Naturally people want to use this technology to compress larger things than just memory strings (like, files, or sockets, or something), so this issue was filed: https://code.google.com/p/snappy/issues/detail?id=34. It's a long discussion, but basically it boils down to this: to run large things through the primary snappy interface, those things need to be broken up into small partitions or frames. In addition, it would be nice to have compatibility across format changes and have a magic number field for mime-type identification, add some headers, etc.

So what the framing format does (final version described here: https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt) is it describes a way to turn a large stream of data (potentially unbounded) into a stream of frames, where each frame is snappy compressed.

So, if you're Google and you're compressing keys and values inside of some large key value store, you have everything in memory and don't want file-type identification headers and so forth, so you just use the standard snappy Compress/Uncompress. You don't want the framing stuff. On the other hand, if you want to use Snappy as a Zip/GZip/BZip2 replacement, then you need a lot more machinery to support large streams, so that's what the frame format is for.

[jtolio]

It's clear from issues like #27 that there's a lot of confusion on this topic. I don't know that the python-snappy wrapper library is the place to fix it as much as trying to get the upstream snappy library to be more clear about the distinction. I'm afraid upstream cares less about the framing format, last I checked they didn't even support it.

[johnrfrank]

three suggestions:

1. offer module-level functions called {un,}compress_framed that take strings and generate strings compatible with the commandline -c/d flags

2. maybe offer a fallback=True kwarg in these module level functions that attempts the other type before raising snappy.UncompressError: Error while decompressing: invalid input

3. when someone uses either set of functions, consider having a python logging.DEBUG level message that says which they are using and specifically what it is compatible/incompatible with

[aeroevan]

To make things worse, hadoop has it's own framing format for snappy compressed files. See #35

[andrix]

@aeroevan Hi Evan, sorry for the delay. I'm afraid to tell you I've been not working in maintain the library anymore because of lack of time.

[aeroevan]

Guessed as much :)

I actually am playing with cython bindings to snappy here: https://github.com/aeroevan/pysnappy (more as a learning exercise than anything), but if I actually get things working I'll probably publish that as a package.

[andrix]

@aeroevan yes, sorry :(. Full time job, father and kind of starting a business myself, leading to no time. If you want, we can migrate the project somewhere else, and change the maintainer. @jtolds has helped as well.