PERF: use .values in index difference

[max-sixty]

The existing .difference method 'unboxed' all the objects, which has a severe performance impact on PeriodIndex in particular.

In [3]: long_index = pd.period_range(start='2000', freq='s', periods=1000)

In [4]: empty_index = pd.PeriodIndex([],freq='s')


In [24]: %timeit long_index.difference(empty_index)

# existing:
1 loops, best of 1: 1.02 s per loop
# updated: 
1000 loops, best of 3: 538 µs per loop

...so around 2000x

I haven't worked with asv or the like - is this a case where a test like that is required?

[jorisvandenbossche]

maybe nice to add a test for this one? (that it keeps the correct class)

[jreback]

there are quite a number of tests in tseries/tests/test_base for this type of behavior FYI

[max-sixty]

OK cheers @jreback. At the moment I'm getting a number of failures similar to the one below - I think it's where this operates on MultiIndexes.
I don't know how well multi_index._shallow_copy(multi_index.values) == multi_index works?
I can branch the logic depending on whether it's a MultiIndex or not - unless you have an alternative?

======================================================================
ERROR: test_stack_partial_multiIndex (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/tests/test_frame.py", line 13998, in test_stack_partial_multiIndex
    _test_stack_with_multiindex(full_multiindex[multiindex_columns])
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/tests/test_frame.py", line 13969, in _test_stack_with_multiindex
    result = df.stack(level=level, dropna=False)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/frame.py", line 3745, in stack
    return stack(self, level, dropna=dropna)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/reshape.py", line 481, in stack
    return _stack_multi_columns(frame, level_num=level_num, dropna=dropna)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/reshape.py", line 648, in _stack_multi_columns
    result = DataFrame(new_data, index=new_index, columns=new_columns)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/frame.py", line 227, in __init__
    mgr = self._init_dict(data, index, columns, dtype=dtype)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/frame.py", line 322, in _init_dict
    data = dict((k, v) for k, v in compat.iteritems(data)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/frame.py", line 323, in <genexpr>
    if k in columns)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/index.py", line 1116, in __contains__
    return key in self._engine
  File "pandas/index.pyx", line 99, in pandas.index.IndexEngine.__contains__ (pandas/index.c:2749)
  File "pandas/index.pyx", line 261, in pandas.index.IndexEngine._ensure_mapping_populated (pandas/index.c:5304)
  File "pandas/index.pyx", line 267, in pandas.index.IndexEngine.initialize (pandas/index.c:5408)
  File "pandas/hashtable.pyx", line 703, in pandas.hashtable.PyObjectHashTable.map_locations (pandas/hashtable.c:12850)
ValueError: Does not understand character buffer dtype format string ('w')

[jreback]

looks like something else is going on
shallow_cooy should work it overridden for MultIndex

[jreback]

any progress?

[max-sixty]

@jreback not yet - will look at it this weekend. Thanks for the ping

[jreback]

@MaximilianR if you'd like to update would be gr8

[max-sixty]

I had a go at debugging this. But I'm struggling, since the errors happen on the Cython side - I need to get up to speed on how to debug those.
If anyone has any guidance, I'm very open to ideas. Otherwise it'll be a few weeks at least, I think.

[jreback]

@MaximilianR can you rebase / update

[max-sixty]

I still get this error below. I'm really not sure how to debug the pyx files - although keen to learn. Any guidance?

======================================================================
ERROR: test_stack_partial_multiIndex (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/tests/test_frame.py", line 14305, in test_stack_partial_multiIndex
    _test_stack_with_multiindex(full_multiindex[multiindex_columns])
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/tests/test_frame.py", line 14276, in _test_stack_with_multiindex
    result = df.stack(level=level, dropna=False)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/frame.py", line 3803, in stack
    return stack(self, level, dropna=dropna)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/reshape.py", line 481, in stack
    return _stack_multi_columns(frame, level_num=level_num, dropna=dropna)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/reshape.py", line 648, in _stack_multi_columns
    result = DataFrame(new_data, index=new_index, columns=new_columns)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/frame.py", line 226, in __init__
    mgr = self._init_dict(data, index, columns, dtype=dtype)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/frame.py", line 323, in _init_dict
    data = dict((k, v) for k, v in compat.iteritems(data)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/frame.py", line 324, in <genexpr>
    if k in columns)
  File "/Users/maximilianroos/Dropbox/workspace/pandas/pandas/core/index.py", line 1161, in __contains__
    return key in self._engine
  File "pandas/index.pyx", line 99, in pandas.index.IndexEngine.__contains__ (pandas/index.c:2749)
  File "pandas/index.pyx", line 261, in pandas.index.IndexEngine._ensure_mapping_populated (pandas/index.c:5304)
  File "pandas/index.pyx", line 267, in pandas.index.IndexEngine.initialize (pandas/index.c:5408)
  File "pandas/hashtable.pyx", line 703, in pandas.hashtable.PyObjectHashTable.map_locations (pandas/hashtable.c:12518)
ValueError: Does not understand character buffer dtype format string ('w')

----------------------------------------------------------------------

[jreback]

go up the stack when debugging. somehow the new_columns is created with a dtype of S1 which is invalid this violates some guarantees there. So you have to trace where this is happening (prob the _shallow_copy may need a hint)

> /Users/jreback/pandas/pandas/core/reshape.py(648)_stack_multi_columns()
-> result = DataFrame(new_data, index=new_index, columns=new_columns)
(Pdb) p new_data
{'A': array([ nan,   2.,  nan,  nan,   5.,  nan,  nan,   8.,  nan]), 'B': array([  0.,  nan,   1.,   3.,  nan,   4.,   6.,  nan,   7.])}
(Pdb) p new_index
MultiIndex(levels=[[0, 1, 2], [u'u', u'x', u'y', u'z']],
           labels=[[0, 0, 0, 1, 1, 1, 2, 2, 2], [1, 2, 3, 1, 2, 3, 1, 2, 3]],
           names=[None, u'Lower'])
(Pdb) p new_columns
Index([u'A', u'B'], dtype='|S1', name=u'Upper')
(Pdb) !new_columns = Index(new_columns.values,name=new_columns.name)
*** NameError: name 'Index' is not defined
(Pdb) from pandas import Index
(Pdb) !new_columns = Index(new_columns.values,name=new_columns.name)
(Pdb) p new_columns
Index([u'A', u'B'], dtype='object', name=u'Upper')
(Pdb) p DataFrame(new_data, index=new_index, columns=new_columns)
Upper     A   B
  Lower        
0 x     NaN   0
  y       2 NaN
  z     NaN   1
1 x     NaN   3
  y       5 NaN
  z     NaN   4
2 x     NaN   6
  y       8 NaN
  z     NaN   7

[max-sixty]

OK thanks, I'll try that angle

[jreback]

@MaximilianR pls reopen if you would like to update

[max-sixty]

OK, I will aim to come back to this one at some point

[jreback]

np. just trying to keep out outstanding PR's to minimum.