Series shift method breaks for series of pandas Intervals in Pandas 1.0 (works in 0.25.3)

[owenlamont]

Code Sample, a copy-pastable example if possible

# Problem code example 1
import pandas as pd
test = pd.Series(index=[1, 2], data=[pd.Interval(pd.Timestamp("2020-09-04 10:00:00"), pd.Timestamp("2020-11-30 14:00:00")), pd.Interval(pd.Timestamp("2020-08-14 10:00:00"), pd.Timestamp("2020-09-21 14:00:00"))])
test.shift(1)

# Problem code example 2
import pandas as pd
test = pd.Series(index=[1, 2], data=[pd.Interval(1, 2), pd.Interval(3, 4)])
test.shift(1)

Problem description

Calling the shift method on an integer indexed Pandas series of Pandas intervals throws opaque exceptions. The same code works as expected in Pandas 0.25.3. This is definitely a breaking change - I'm unsure if it is intentional. I'm assuming it should still work the same as 0.25.3 for now. I tried searching for any documented changes to the shift method behaviour but didn't find any.

Exception traceback for example 1:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-7a1f088b5a14> in <module>
      4 test = pd.Series(index=[1, 2], data=[pd.Interval(pd.Timestamp("2020-09-04 10:00:00"), pd.Timestamp("2020-11-30 14:00:00")),
      5                                      pd.Interval(pd.Timestamp("2020-08-14 10:00:00"), pd.Timestamp("2020-09-21 14:00:00"))])
----> 6 test.shift(1)

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\series.py in shift(self, periods, freq, axis, fill_value)
   4183     def shift(self, periods=1, freq=None, axis=0, fill_value=None):
   4184         return super().shift(
-> 4185             periods=periods, freq=freq, axis=axis, fill_value=fill_value
   4186         )
   4187 

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\generic.py in shift(self, periods, freq, axis, fill_value)
   9043         if freq is None:
   9044             new_data = self._data.shift(
-> 9045                 periods=periods, axis=block_axis, fill_value=fill_value
   9046             )
   9047         else:

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\internals\managers.py in shift(self, **kwargs)
    571 
    572     def shift(self, **kwargs):
--> 573         return self.apply("shift", **kwargs)
    574 
    575     def fillna(self, **kwargs):

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\internals\managers.py in apply(self, f, filter, **kwargs)
    440                 applied = b.apply(f, **kwargs)
    441             else:
--> 442                 applied = getattr(b, f)(**kwargs)
    443             result_blocks = _extend_blocks(applied, result_blocks)
    444 

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\internals\blocks.py in shift(self, periods, axis, fill_value)
   1908         return [
   1909             self.make_block_same_class(
-> 1910                 self.values.shift(periods=periods, fill_value=fill_value),
   1911                 placement=self.mgr_locs,
   1912                 ndim=self.ndim,

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\arrays\base.py in shift(self, periods, fill_value)
    623 
    624         empty = self._from_sequence(
--> 625             [fill_value] * min(abs(periods), len(self)), dtype=self.dtype
    626         )
    627         if periods > 0:

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\arrays\interval.py in _from_sequence(cls, scalars, dtype, copy)
    243     @classmethod
    244     def _from_sequence(cls, scalars, dtype=None, copy=False):
--> 245         return cls(scalars, dtype=dtype, copy=copy)
    246 
    247     @classmethod

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\arrays\interval.py in __new__(cls, data, closed, dtype, copy, verify_integrity)
    182             copy=copy,
    183             dtype=dtype,
--> 184             verify_integrity=verify_integrity,
    185         )
    186 

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\arrays\interval.py in _simple_new(cls, left, right, closed, copy, dtype, verify_integrity)
    202                 raise TypeError(msg)
    203             elif dtype.subtype is not None:
--> 204                 left = left.astype(dtype.subtype)
    205                 right = right.astype(dtype.subtype)
    206 

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\indexes\numeric.py in astype(self, dtype, copy)
    393         if needs_i8_conversion(dtype):
    394             raise TypeError(
--> 395                 f"Cannot convert Float64Index to dtype {dtype}; integer "
    396                 "values are required for conversion"
    397             )

TypeError: Cannot convert Float64Index to dtype datetime64[ns]; integer values are required for conversion

Exception traceback for example 2

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-6804e066ee18> in <module>
      4 test = pd.Series(index=[1, 2], data=[pd.Interval(1, 2),
      5                                      pd.Interval(3, 4)])
----> 6 test.shift(1)

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\series.py in shift(self, periods, freq, axis, fill_value)
   4183     def shift(self, periods=1, freq=None, axis=0, fill_value=None):
   4184         return super().shift(
-> 4185             periods=periods, freq=freq, axis=axis, fill_value=fill_value
   4186         )
   4187 

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\generic.py in shift(self, periods, freq, axis, fill_value)
   9043         if freq is None:
   9044             new_data = self._data.shift(
-> 9045                 periods=periods, axis=block_axis, fill_value=fill_value
   9046             )
   9047         else:

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\internals\managers.py in shift(self, **kwargs)
    571 
    572     def shift(self, **kwargs):
--> 573         return self.apply("shift", **kwargs)
    574 
    575     def fillna(self, **kwargs):

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\internals\managers.py in apply(self, f, filter, **kwargs)
    440                 applied = b.apply(f, **kwargs)
    441             else:
--> 442                 applied = getattr(b, f)(**kwargs)
    443             result_blocks = _extend_blocks(applied, result_blocks)
    444 

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\internals\blocks.py in shift(self, periods, axis, fill_value)
   1908         return [
   1909             self.make_block_same_class(
-> 1910                 self.values.shift(periods=periods, fill_value=fill_value),
   1911                 placement=self.mgr_locs,
   1912                 ndim=self.ndim,

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\arrays\base.py in shift(self, periods, fill_value)
    623 
    624         empty = self._from_sequence(
--> 625             [fill_value] * min(abs(periods), len(self)), dtype=self.dtype
    626         )
    627         if periods > 0:

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\arrays\interval.py in _from_sequence(cls, scalars, dtype, copy)
    243     @classmethod
    244     def _from_sequence(cls, scalars, dtype=None, copy=False):
--> 245         return cls(scalars, dtype=dtype, copy=copy)
    246 
    247     @classmethod

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\arrays\interval.py in __new__(cls, data, closed, dtype, copy, verify_integrity)
    182             copy=copy,
    183             dtype=dtype,
--> 184             verify_integrity=verify_integrity,
    185         )
    186 

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\arrays\interval.py in _simple_new(cls, left, right, closed, copy, dtype, verify_integrity)
    202                 raise TypeError(msg)
    203             elif dtype.subtype is not None:
--> 204                 left = left.astype(dtype.subtype)
    205                 right = right.astype(dtype.subtype)
    206 

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\indexes\numeric.py in astype(self, dtype, copy)
    399             # TODO(jreback); this can change once we have an EA Index type
    400             # GH 13149
--> 401             arr = astype_nansafe(self.values, dtype=dtype)
    402             return Int64Index(arr)
    403         return super().astype(dtype, copy=copy)

~\Miniconda3\envs\jupyter\lib\site-packages\pandas\core\dtypes\cast.py in astype_nansafe(arr, dtype, copy, skipna)
    866 
    867         if not np.isfinite(arr).all():
--> 868             raise ValueError("Cannot convert non-finite values (NA or inf) to integer")
    869 
    870     elif is_object_dtype(arr):

ValueError: Cannot convert non-finite values (NA or inf) to integer

Expected Output

This is the actual output I got executing with pandas 0.25.3

1 NaN
2 (2020-09-04 10:00:00, 2020-11-30 14:00:00]
dtype: object

1 NaN
2 (1, 2]
dtype: object

Output of pd.show_versions()

commit : None
python : 3.7.3.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 13, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.0.0
numpy : 1.17.5
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.1.0.post20200119
Cython : None
pytest : 5.3.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 1.2.7
lxml.etree : 4.5.0
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.1
IPython : 7.11.1
pandas_datareader: None
bs4 : 4.8.2
bottleneck : 1.3.1
fastparquet : 0.3.2
gcsfs : None
lxml.etree : 4.5.0
matplotlib : 3.1.2
numexpr : None
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
pytest : 5.3.5
pyxlsb : None
s3fs : 0.2.2
scipy : 1.3.1
sqlalchemy : 1.3.13
tables : None
tabulate : 0.8.6
xarray : 0.14.1
xlrd : 1.2.0
xlwt : None
xlsxwriter : 1.2.7
numba : 0.48.0

[TomAugspurger]

I don't agree with the expected output. The dtype was lost (object), and we should be able to mostly preserve the interval dtype.

I think the bug is from the shifted interval sub-dtype not necessarily matching the original sub-dtype.

In IntervalArray.shift, we essentially do

In [21]: pd.arrays.IntervalArray._from_sequence([np.nan], dtype=pd.IntervalDtype('int64'))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-34d8dc2cf5c6> in <module>
----> 1 pd.arrays.IntervalArray._from_sequence([np.nan], dtype=pd.IntervalDtype('int64'))

~/sandbox/pandas/pandas/core/arrays/interval.py in _from_sequence(cls, scalars, dtype, copy)
    243     @classmethod
    244     def _from_sequence(cls, scalars, dtype=None, copy=False):
--> 245         return cls(scalars, dtype=dtype, copy=copy)
    246
    247     @classmethod

~/sandbox/pandas/pandas/core/arrays/interval.py in __new__(cls, data, closed, dtype, copy, verify_integrity)
    182             copy=copy,
    183             dtype=dtype,
--> 184             verify_integrity=verify_integrity,
    185         )
    186

~/sandbox/pandas/pandas/core/arrays/interval.py in _simple_new(cls, left, right, closed, copy, dtype, verify_integrity)
    202                 raise TypeError(msg)
    203             elif dtype.subtype is not None:
--> 204                 left = left.astype(dtype.subtype)
    205                 right = right.astype(dtype.subtype)
    206

~/sandbox/pandas/pandas/core/indexes/numeric.py in astype(self, dtype, copy)
    385             # TODO(jreback); this can change once we have an EA Index type
    386             # GH 13149
--> 387             arr = astype_nansafe(self.values, dtype=dtype)
    388             return Int64Index(arr)
    389         return super().astype(dtype, copy=copy)

~/sandbox/pandas/pandas/core/dtypes/cast.py in astype_nansafe(arr, dtype, copy, skipna)
    866
    867         if not np.isfinite(arr).all():
--> 868             raise ValueError("Cannot convert non-finite values (NA or inf) to integer")
    869
    870     elif is_object_dtype(arr):

ValueError: Cannot convert non-finite values (NA or inf) to integer

@owenlamont are you interested into looking into this?

[jorisvandenbossche]

I don't agree with the expected output. The dtype was lost (object), and we should be able to mostly preserve the interval dtype.

The expected output is just what was in 0.25.3. I agree we can do better (preserve the dtype), but it's still a regression that it errors instead of returning the correct values (just not the correct dtype)

[TomAugspurger]

The expected output is just what was in 0.25.3. I agree we can do better (preserve the dtype)

Agreed that it's a regression, but if we were writing a test we wouldn't want to assert that the result dtype is object right?

[owenlamont]

Thanks for the quick responses. I'd love to contribute to Pandas but I'm not familiar with the code base and probably wouldn't be able to address this issue in a reasonable time. You've got me interested learning so I'll have a shot at compiling from source and see how I go from there.

[TomAugspurger]

Thanks! Since this is a regression I may fix it myself so that we can adrress it quickly. We have lots of "Good first issues" though :)

[jorisvandenbossche]

Ah, this also worked (partly) in 0.25 because there was not yet interval dtype inference in the Series constructor ..