pd.read_pickle, pd.DataFrame.to_pickle do not accept pathlib.Path objects

[thorbjornwolf]

Code Sample

In [1]: import pandas as pd
   ...: import pathlib as pl
   ...: path = pl.Path.home() / 'tmp.pickle' # I have prepared a pickle in this path
   ...: 

In [2]: df = pd.read_pickle(str(path)) # This works, no problem
   ...: 

In [3]: df2 = pd.read_pickle(path) # This breaks
   ...: 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     50         try:
---> 51             with open(path, 'rb') as fh:
     52                 return pkl.load(fh)

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     55             try:
---> 56                 with open(path, 'rb') as fh:
     57                     return pc.load(fh, encoding=encoding, compat=False)

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in read_pickle(path)
     64     try:
---> 65         return try_read(path)
     66     except:

/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     60             except:
---> 61                 with open(path, 'rb') as fh:
     62                     return pc.load(fh, encoding=encoding, compat=True)

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     50         try:
---> 51             with open(path, 'rb') as fh:
     52                 return pkl.load(fh)

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     55             try:
---> 56                 with open(path, 'rb') as fh:
     57                     return pc.load(fh, encoding=encoding, compat=False)

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-3-d33de9a00518> in <module>()
----> 1 df2 = pd.read_pickle(path)

/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in read_pickle(path)
     66     except:
     67         if PY3:
---> 68             return try_read(path, encoding='latin1')
     69         raise
     70 

/home/ntaw/anaconda3/lib/python3.5/site-packages/pandas/io/pickle.py in try_read(path, encoding)
     59             # compat pickle
     60             except:
---> 61                 with open(path, 'rb') as fh:
     62                     return pc.load(fh, encoding=encoding, compat=True)
     63 

TypeError: invalid file: PosixPath('/home/ntaw/tmp.pickle')

Problem description

pandas breaks at df = pd.read_pickle(path). It seems to me that this should have been solved along with issue #11033, but it evidently doesn't work for my setup.

I encountered the same issue with pd.DataFrame.to_pickle.

The usecase appears to be related to open issue #14705 and closed pull request #12930.

Expected Output

I would expect the pandas.read_* functions and pandas.DataFrame.to_* methods to recognise pathlib objects, and handle the conversion str(path) under the hood.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.5.2.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-53-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8

pandas: 0.19.1
nose: 1.3.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.1
numpy: 1.11.2
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: 1.4.8
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.7
blosc: None
bottleneck: 1.1.0
tables: 3.3.0
numexpr: 2.6.1
matplotlib: 1.5.3
openpyxl: 2.4.0
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.3
lxml: 3.6.4
bs4: 4.5.1
html5lib: None
httplib2: 0.9.2
apiclient: 1.5.1
sqlalchemy: 1.1.3
pymysql: None
psycopg2: None
jinja2: 2.8
boto: 2.43.0
pandas_datareader: None

[jreback]

original issue to implement for excel, hdf, csv was here: https://github.com/pandas-dev/pandas/pull/11051/files

pickle should be quite easy if you'd like to do a PR.

[jreback]

xref #11666

this should simply use pandas.io.common._get_handle which will then facilitate both of these requests (pathlib and compression) as that now is our generic handler.

[jreback]

xref #13823

[thorbjornwolf]

Sounds like a fun way to start contributing :) I'll give it a shot!

[thorbjornwolf]

pandas.io.common._get_handle does not support pathlib.Path with python 3.5.2, but it seems that the open call supports it in 3.6. get_filepath_or_buffer, however, seems to handle it well. So, what did you mean with

this should simply use pandas.io.common._get_handle

?