Support column based parsing

[RandomString123]

When parsing large files and using DataStreams to push it into a DataFrame code is very slow parsing one field at a time. It is more efficient when parsing whole columns from the file at a time. This is fairly easy with byte based parsing but character based parsing means lines are not a constant length in bytes making byte indexing across a large file difficult. Figure out a method to do this universally.

[RandomString123]

I spent a lot of time yesterday trying to get the parse column code updated. While I was able to get a working version the method I wrote ended up being slower than just using DataStreams and parsing the file one field at at a time. This is because the looping in DataStreams is highly optimized to remove type ambiguity and any code I would write could not come close to the work done in that package.

I am going to leave this issue open to keep thinking about it over time. But given that the only reliable way to be able to parse one column is to parse the whole file, which is hard to do in a type stable way without optimized looping. It might never be faster to do parsing a whole column at a time. Right now the best performance gain I think we will get is when .7 optimizes the use of missings in code.

[bkamins]

You might want to check out https://github.com/JuliaComputing/TextParse.jl.

[RandomString123]

Interesting...they make a bunch of assumptions that would make it difficult for us to just build on top of their code. However there are a few things in there I think we can learn from.

• util/nonallocating_setindex! is pretty spiffy for breaking up a string. Although since strings are now immutable in Julia I would hope that the compiler/GC does this sort of pooled optimization when allocating a string.
• Thier tryparsenext and tryparsesetindex seem to mimic the inner and outer loops in DataStreams but are more approachable. I think using a similar @generated method wrapped around the parsing methods in parsefilds.jl would introduce type stability for bulk parsing.

Using that package as a base I might take another wack at this in the coming weeks. Thanks for the pointer.