BUG, DOC: Allow custom line terminator with delim_whitespace=True

[gfyoung]

Title is self-explanatory. Closes #12912.

[gfyoung]

The tokenizers in tokenizer.c, and the CParser tests in test_parser.py seem to have a lot of repetition, although reworking the latter seems easier to do than the former. Can / should this PR also include refactoring changes to those parts of the code?

[jreback]

no make smaller / simpler PRs that do 1 thing

[jreback]

they can of course be built upon one another

[jreback]

say this is equivalent to \s+ regex

[gfyoung]

Done.

[jreback]

why do you think adding all of this untested code is a good idea?

you would basically have to duplicate all of the test suite with a custom terminator in order to validate this

you need to use functions that are already tested
iow each of the cases needs to all a function

[gfyoung]

What functions are you talking about? If you look at the other tokenizer functions, they all duplicate the same or have very similar case work. The function I added is in fact a near duplicate of tokenize_whitespace save replacing checks for \n and \r with self->lineterminator. That's why I asked about the refactoring initially.

[jreback]

well then refactor first - we can't add more code like this

[jorisvandenbossche]

One backtick too much before \t

[gfyoung]

Oh, good catch. Will fix.

[gfyoung]

Managed to refactor code so that it only splits based on whether or not whitespace is used as a delimiter, so there are now only two tokenizing functions instead of three (or four after my second commit). Will see if it can be squashed into just one, though it seems less straightforward compared to the custom line terminator split.

[jreback]

@gfyoung ok much better. pls also run asv's for csv to make sure no degredation.

So macros are good because they make the code shorter / more understandable while hopefully not sacrificing perf. more combining code is better (again could be done later), but since you are already working on it....

[jreback]

say (both her and in the docs), that delim_whitespace=True equiv to sep='\s+' (not just that its the same regex)

[gfyoung]

Done.

[gfyoung]

@jreback : there seems to have been a recent update to python-dateutil (just today), causing a test failure in pandas.tslib on OSX. I can reproduce this failure on Linux FYI.

[jreback]

@gfyoung ok! my pip install catches things then, great! (the osx build will pip install python-dateutil to catch things just like this :>), while the 3.5 build uses conda (which is with a stable build). IIRC this has some reverses from previous things, so will have to sort them and see. Maybe we should ban certain versions of dateutil. These can cause weird things to happen.

[gfyoung]

@jreback : is it possible to put in a stopgap measure FTTB (e.g. change the compat condition in that test to being exactly 2.5.2 so that investigation can proceed in isolation from other PR's?)

[jreback]

see #12944 (I skipped the test for now on master), so rebase.

[gfyoung]

Whoot! Squashed it down into one function! Hopefully Travis will give the green light this time around. I will also do some timing analysis for read_csv and post it in the conversation.

[gfyoung]

For PR

io_bench.frame_to_csv.time_frame_to_csv           188.25ms
io_bench.frame_to_csv2.time_frame_to_csv2         312.94ms
...formatting.time_frame_to_csv_date_formatting    13.36ms
...h.frame_to_csv_mixed.time_frame_to_csv_mixed   218.47ms
...m.time_read_csv_infer_datetime_format_custom    15.82ms
....time_read_csv_infer_datetime_format_iso8601     2.64ms
..._ymd.time_read_csv_infer_datetime_format_ymd     2.92ms
...nch.read_csv_skiprows.time_read_csv_skiprows    16.74ms
...nch.read_csv_standard.time_read_csv_standard    14.00ms
..._dates_iso8601.time_read_parse_dates_iso8601     1.93ms
...h.write_csv_standard.time_write_csv_standard    50.49ms

For Master

io_bench.frame_to_csv.time_frame_to_csv           195.29ms
io_bench.frame_to_csv2.time_frame_to_csv2         303.08ms
...formatting.time_frame_to_csv_date_formatting    12.99ms
...h.frame_to_csv_mixed.time_frame_to_csv_mixed   224.39ms
...m.time_read_csv_infer_datetime_format_custom    15.53ms
....time_read_csv_infer_datetime_format_iso8601     2.64ms
..._ymd.time_read_csv_infer_datetime_format_ymd     2.80ms
...nch.read_csv_skiprows.time_read_csv_skiprows    16.04ms
...nch.read_csv_standard.time_read_csv_standard    13.23ms
..._dates_iso8601.time_read_parse_dates_iso8601     1.78ms
...h.write_csv_standard.time_write_csv_standard    51.60ms

[gfyoung]

@jreback : Travis is giving the green light, and I don't see any major timing discrepancies. Ready to merge if there is nothing else.

[jreback]

thanks @gfyoung didn't realize how much duplicative code there was already in tokenizer.c sheesh 👍

[jreback]

as an aside, adding delim_whitespace option to python parser is trivial.