Skip to content

in read_csv, warn_bad_lines and error_bad_lines have misleading names #6478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rcompton opened this issue Feb 25, 2014 · 2 comments
Closed
Labels
Dtype Conversions Unexpected or buggy dtype conversions Enhancement IO CSV read_csv, to_csv

Comments

@rcompton
Copy link

read_csv will still fail if *_bad_lines=False. *_bad_lines only applies when the number of delimiters is wrong. This is misleading, a line with a poorly formatted date is what I would think of as a "bad_line" that should be ignored (cf https://stackoverflow.com/questions/22026181 )

@jreback
Copy link
Contributor

jreback commented Feb 25, 2014

this is not the purpose of error_bad_lines

if you are providing a date parser then you need to handle errors
(in this case you you should either put a try except in your parsing function,parse the dates after with to_datetime, or use infer_datetime_format (in 0.13.1))

infering dtypes is inherently an ambiguous and not foolproof process
the user needs to be in charge of this and be responsible

I'll mark this an enhancement for someday

@jorisvandenbossche
Copy link
Member

jorisvandenbossche commented Jan 12, 2017

Not sure there is an enhancement here (but feel free to formulate a more concrete feature request). Renaming the *_bad_lines is best done in a reworked API to better deal with such "bad lines", see eg #9549 (comment). Therefore closing this issue.

@jorisvandenbossche jorisvandenbossche modified the milestones: No action, Someday Jan 12, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Dtype Conversions Unexpected or buggy dtype conversions Enhancement IO CSV read_csv, to_csv
Projects
None yet
Development

No branches or pull requests

3 participants