WIP: add pd.read_ipc and DataFrame.to_ipc to provide efficient serialization to/from memory

[jreback]

No description provided.

[jreback]

cc @mrocklin
@wesm @cpcloud

[chris-b1]

Given that this is a pretty 'advanced' feature (and presumably an unstable format?), maybe would make sense to expose these functions somewhere like pandas.api.lib rather than in the main namespace?

[jreback]

@chris-b1 yeah probably. This is just trying to get working (today) locally. :> (its not actually anywhere atm, but in the local module).

[codecov]

Codecov Report

Merging #15907 into master will decrease coverage by 0.1%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master   #15907      +/-   ##
==========================================
- Coverage      91%    90.9%   -0.11%     
==========================================
  Files         145      146       +1     
  Lines       49576    49631      +55     
==========================================
  Hits        45118    45118              
- Misses       4458     4513      +55

Flag	Coverage Δ
#multiple	88.67% <0%> (-0.1%)	⬇️
#single	40.53% <0%> (-0.05%)	⬇️

Impacted Files	Coverage Δ
pandas/io/ipc.py	0% <0%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f35209e...f727477. Read the comment docs.

[wesm]

I agree putting such things into an experimental namespace would be a good idea

[jorisvandenbossche]

Genuine question: since we are typically trying to limit pandas' scope, why should we include this in the core package? Instead of having this live in the package implementing this / a separate package ('pandas-ipc') providing this API?

[jreback]

Genuine question: since we are typically trying to limit pandas' scope, why should we include this in the core package? Instead of having this live in the package implementing this / a separate package ('pandas-ipc') providing this API?

True, we are trying to limit scope. This is essentialy an ipc version of arrow (which also back feather). So this ATM expands scope, though eventually would limit it somewhat (as the back-end would have less tooling).

Could consider this as pandas-ipc though this is pretty 'simple' and not meant as a full-fledged package, mostly a passthru / convenient interface for others (e.g. dask).

[wesm]

I think it would be useful for pandas to have more robust support for various transient serialization formats, with a choice between pickle (most compatible) vs msgpack vs arrow vs other things. Whether the implementation of this goes into core pandas, or into a "leaf" library that gets imported, I don't have a strong opinion

[jreback]

closing for now. This will be pretty transparent with pyarrow, so this interface would just be a simple wrapper. let's see if its even needed.

[jreback]

so this is now available in a released version in arrow: https://arrow.apache.org/docs/python/ipc.html (IIRC 0.5.0 has full support). appetite for this in main pandas as read_ipc/to_ipc ?

@wesm @cpcloud @jorisvandenbossche

[wesm]

Depending on what memory format these functions create it may affect the name. If it's vanilla arrow stream (schema + sequence of record batches), then it might be better to call it read_arrow_stream / to_arrow_stream (in the latter case, you could select a batch size to use for chunking the DataFrame once someone implements this)