Skip to content

ENH: add nthreads option to feather-format IO #16476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 24, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v0.21.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Other Enhancements
- ``RangeIndex.append`` now returns a ``RangeIndex`` object when possible (:issue:`16212`)
- :func:`to_pickle` has gained a protocol parameter (:issue:`16252`). By default, this parameter is set to `HIGHEST_PROTOCOL <https://docs.python.org/3/library/pickle.html#data-stream-format>`__
- :func:`api.types.infer_dtype` now infers decimals. (:issue: `15690`)
- :func:`read_feather` has gained the ``nthreads`` parameter for multi-threaded operations (:issue:`16359`)

.. _whatsnew_0210.api_breaking:

Expand Down
13 changes: 11 additions & 2 deletions pandas/io/feather_format.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ def to_feather(df, path):
df : DataFrame
path : string
File path

"""
path = _stringify_path(path)
if not isinstance(df, DataFrame):
Expand Down Expand Up @@ -83,7 +84,7 @@ def to_feather(df, path):
feather.write_dataframe(df, path)


def read_feather(path):
def read_feather(path, nthreads=1):
"""
Load a feather-format object from the file path

Expand All @@ -93,6 +94,10 @@ def read_feather(path):
----------
path : string
File path
nthreads : int, default 1
Number of CPU threads to use when reading to pandas.DataFrame

.. versionadded 0.21.0

Returns
-------
Expand All @@ -102,4 +107,8 @@ def read_feather(path):

feather = _try_import()
path = _stringify_path(path)
return feather.read_dataframe(path)

if feather.__version__ < LooseVersion('0.4.0'):
return feather.read_dataframe(path)

return feather.read_dataframe(path, nthreads=nthreads)
10 changes: 8 additions & 2 deletions pandas/tests/io/test_feather.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,11 @@ def check_error_on_write(self, df, exc):
with ensure_clean() as path:
to_feather(df, path)

def check_round_trip(self, df):
def check_round_trip(self, df, **kwargs):

with ensure_clean() as path:
to_feather(df, path)
result = read_feather(path)
result = read_feather(path, **kwargs)
assert_frame_equal(result, df)

def test_error(self):
Expand Down Expand Up @@ -98,6 +98,12 @@ def test_unsupported_other(self):
df = pd.DataFrame({'a': pd.period_range('2013', freq='M', periods=3)})
self.check_error_on_write(df, ValueError)

@pytest.mark.skipif(fv < '0.4.0', reason='new in 0.4.0')
def test_rw_nthreads(self):

df = pd.DataFrame({'A': np.arange(100000)})
self.check_round_trip(df, nthreads=2)

def test_write_with_index(self):

df = pd.DataFrame({'A': [1, 2, 3]})
Expand Down