COMPAT: Consider Python 2.x tarfiles file-like #16533

gfyoung · 2017-05-29T17:15:56Z

Tarfile.ExFileObject has no next method in Python 2.x, making it an invalid file-like object in read_csv. However, they can be read in just fine, meaning our check is too strict for file-like. This commit relaxes
the check to just look for the __iter__ attribute.

Closes #16530.

codecov · 2017-05-29T21:38:50Z

Codecov Report

Merging #16533 into master will decrease coverage by <.01%.
The diff coverage is 60%.

@@            Coverage Diff             @@
##           master   #16533      +/-   ##
==========================================
- Coverage   90.79%   90.79%   -0.01%     
==========================================
  Files         161      161              
  Lines       51063    51075      +12     
==========================================
+ Hits        46365    46371       +6     
- Misses       4698     4704       +6

Flag	Coverage Δ
#multiple	`88.63% <60%> (-0.01%)`	⬇️
#single	`40.15% <60%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/dtypes/inference.py	`98.33% <100%> (ø)`	⬆️
pandas/io/parsers.py	`95.3% <57.14%> (-0.36%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e60dc4c...a9c9365. Read the comment docs.

codecov · 2017-05-29T21:38:54Z

Codecov Report

Merging #16533 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #16533      +/-   ##
==========================================
+ Coverage   90.75%   90.75%   +<.01%     
==========================================
  Files         161      161              
  Lines       51073    51081       +8     
==========================================
+ Hits        46350    46360      +10     
+ Misses       4723     4721       -2

Flag	Coverage Δ
#multiple	`88.59% <100%> (ø)`	⬆️
#single	`40.16% <81.81%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/dtypes/inference.py	`98.33% <100%> (ø)`	⬆️
pandas/io/parsers.py	`95.68% <100%> (+0.02%)`	⬆️
pandas/core/internals.py	`93.4% <0%> (+0.07%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ee8346d...7c59fc9. Read the comment docs.

jreback · 2017-05-30T11:33:48Z

pandas/io/parsers.py

+        if is_file_like(f):
+            next_attr = "__next__" if PY3 else "next"
+
+            if engine != "c" and not hasattr(f, next_attr):


can you add a comment of why you are doing this

jreback · 2017-05-30T11:34:17Z

setup.py

@@ -702,6 +702,7 @@ def pxd(name):
                                        'parser/data/*.gz',
                                        'parser/data/*.bz2',
                                        'parser/data/*.txt',
+                                        'parser/data/*.tar',


can you add with .tar.gz as well?

I meant test with .tar.gz as well.

Ah, I see. Done.

jreback

lgtm. minor comments.

TomAugspurger · 2017-05-30T21:24:43Z

pandas/io/parsers.py

+            # explicitly calls "next(...)" when iterating through
+            # such an object, meaning it needs to have that attribute.
+            if engine != "c" and not hasattr(f, next_attr):
+                msg = ("The 'python' engine can not iterate "


this block seems untested. Are you able to add some, or is it a hassle to hit it?

Was meaning to do that eventually. Done.

jreback

lgtm. couple of comments.

jreback · 2017-05-30T22:51:03Z

setup.py

@@ -702,6 +702,7 @@ def pxd(name):
                                        'parser/data/*.gz',
                                        'parser/data/*.bz2',
                                        'parser/data/*.txt',
+                                        'parser/data/*.tar',


I meant test with .tar.gz as well.

jreback · 2017-05-31T00:31:22Z

lgtm. ping on green.

gfyoung · 2017-05-31T02:47:45Z

@jreback : Green all around and ready to merge.

bashtage · 2017-05-31T09:26:34Z

pandas/io/parsers.py

+                else:
+                    engine = "c"
+                    msg += " Falling back to the 'c' engine."
+                    warnings.warn(msg, ParserWarning, stacklevel=5)


Appears this isn't being hit by test coverage. Should the warning be checked? Can this actually be hit?

You're right. This actually can't be hit (engine = python means the engine was specified). Removing this block.

jreback · 2017-05-31T22:49:52Z

pandas/io/parsers.py

@@ -801,6 +803,24 @@ def _get_options_with_defaults(self, engine):

        return options

+    @staticmethod


why is this a staticmethod?

Because self. is not used. I pass in the engine and f attributes as parameters.

right but no other methods are like this. you can make it a free function if you want, but prob just make it a method.

Alright. Done.

Tarfile.ExFileObject has no "next" method in Python 2.x, making it an invalid file-like object in read_csv. However, they can be read in just fine, meaning our check is too strict for file-like. This commit relaxes the check to just look for "__iter__". Closes pandas-devgh-16530.

jreback · 2017-06-01T10:38:54Z

thanks!

(cherry picked from commit e0a127a)

gfyoung force-pushed the is-file-like-relax branch 7 times, most recently from f17a20c to a9c9365 Compare May 29, 2017 21:38

jreback reviewed May 30, 2017

View reviewed changes

jreback approved these changes May 30, 2017

View reviewed changes

jreback added Compat pandas objects compatability with Numpy or Python functions IO CSV read_csv, to_csv labels May 30, 2017

jreback added this to the 0.20.2 milestone May 30, 2017

gfyoung force-pushed the is-file-like-relax branch from a9c9365 to 6dd7837 Compare May 30, 2017 13:03

TomAugspurger reviewed May 30, 2017

View reviewed changes

jreback approved these changes May 30, 2017

View reviewed changes

gfyoung force-pushed the is-file-like-relax branch 3 times, most recently from 0df7b2c to e236ba5 Compare May 30, 2017 23:34

gfyoung force-pushed the is-file-like-relax branch from e236ba5 to 8611b79 Compare May 31, 2017 00:40

bashtage reviewed May 31, 2017

View reviewed changes

gfyoung force-pushed the is-file-like-relax branch 2 times, most recently from 0526fd6 to 7a5fcd3 Compare May 31, 2017 16:29

jreback reviewed May 31, 2017

View reviewed changes

gfyoung force-pushed the is-file-like-relax branch from 7a5fcd3 to 08efe9c Compare May 31, 2017 23:59

gfyoung force-pushed the is-file-like-relax branch from 08efe9c to 7c59fc9 Compare June 1, 2017 03:10

jreback added the Needs Backport label Jun 1, 2017

jreback merged commit e0a127a into pandas-dev:master Jun 1, 2017

gfyoung deleted the is-file-like-relax branch June 1, 2017 10:40

TomAugspurger pushed a commit to TomAugspurger/pandas that referenced this pull request Jun 1, 2017

COMPAT: Consider Python 2.x tarfiles file-like (pandas-dev#16533)

81210ee

(cherry picked from commit e0a127a)

TomAugspurger pushed a commit that referenced this pull request Jun 4, 2017

COMPAT: Consider Python 2.x tarfiles file-like (#16533)

4a35ee8

(cherry picked from commit e0a127a)

TomAugspurger removed the Needs Backport label Jun 4, 2017

Kiv pushed a commit to Kiv/pandas that referenced this pull request Jun 11, 2017

COMPAT: Consider Python 2.x tarfiles file-like (pandas-dev#16533)

f92ec38

stangirala pushed a commit to stangirala/pandas that referenced this pull request Jun 11, 2017

COMPAT: Consider Python 2.x tarfiles file-like (pandas-dev#16533)

16ce138

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

COMPAT: Consider Python 2.x tarfiles file-like #16533

COMPAT: Consider Python 2.x tarfiles file-like #16533

gfyoung commented May 29, 2017 •

edited

Loading

codecov bot commented May 29, 2017

codecov bot commented May 29, 2017 •

edited

Loading

jreback May 30, 2017

gfyoung May 30, 2017

jreback May 30, 2017

gfyoung May 30, 2017

jreback May 30, 2017

gfyoung May 30, 2017

jreback left a comment

TomAugspurger May 30, 2017

gfyoung May 30, 2017

jreback left a comment

jreback May 30, 2017

jreback commented May 31, 2017

gfyoung commented May 31, 2017

bashtage May 31, 2017

gfyoung May 31, 2017 •

edited

Loading

jreback May 31, 2017

gfyoung May 31, 2017 •

edited

Loading

jreback May 31, 2017

gfyoung May 31, 2017

jreback commented Jun 1, 2017

		@@ -801,6 +803,24 @@ def _get_options_with_defaults(self, engine):

		return options

		@staticmethod

COMPAT: Consider Python 2.x tarfiles file-like #16533

COMPAT: Consider Python 2.x tarfiles file-like #16533

Conversation

gfyoung commented May 29, 2017 • edited Loading

codecov bot commented May 29, 2017

Codecov Report

codecov bot commented May 29, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented May 31, 2017

gfyoung commented May 31, 2017

Choose a reason for hiding this comment

gfyoung May 31, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gfyoung May 31, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 1, 2017

gfyoung commented May 29, 2017 •

edited

Loading

codecov bot commented May 29, 2017 •

edited

Loading

gfyoung May 31, 2017 •

edited

Loading

gfyoung May 31, 2017 •

edited

Loading