Skip to content

Multiindex recurse error fix #29260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Jan 1, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.0.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -946,6 +946,7 @@ Reshaping
- :func:`qcut` and :func:`cut` now handle boolean input (:issue:`20303`)
- Fix to ensure all int dtypes can be used in :func:`merge_asof` when using a tolerance value. Previously every non-int64 type would raise an erroneous ``MergeError`` (:issue:`28870`).
- Better error message in :func:`get_dummies` when `columns` isn't a list-like value (:issue:`28383`)
- Bug in :meth:`Index.join` that caused infinite recursion error for mismatched ``MultiIndex`` name orders. (:issue:`25760`, :issue:`28956`)
- Bug :meth:`Series.pct_change` where supplying an anchored frequency would throw a ValueError (:issue:`28664`)
- Bug where :meth:`DataFrame.equals` returned True incorrectly in some cases when two DataFrames had the same columns in different orders (:issue:`28839`)
- Bug in :meth:`DataFrame.replace` that caused non-numeric replacer's dtype not respected (:issue:`26632`)
Expand Down
9 changes: 7 additions & 2 deletions pandas/core/indexes/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -3368,8 +3368,13 @@ def _join_multi(self, other, how, return_indexers=True):
ldrop_names = list(self_names - overlap)
rdrop_names = list(other_names - overlap)

self_jnlevels = self.droplevel(ldrop_names)
other_jnlevels = other.droplevel(rdrop_names)
# if only the order differs
if not len(ldrop_names + rdrop_names):
self_jnlevels = self
other_jnlevels = other.reorder_levels(self.names)
else:
self_jnlevels = self.droplevel(ldrop_names)
other_jnlevels = other.droplevel(rdrop_names)

# Join left and right
# Join on same leveled multi-index frames is supported
Expand Down
16 changes: 16 additions & 0 deletions pandas/tests/indexes/multi/test_join.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,3 +87,19 @@ def test_join_self_unique(idx, join_type):
if idx.is_unique:
joined = idx.join(idx, how=join_type)
assert (idx == joined).all()


def test_join_multi_wrong_order():
# GH 25760
# GH 28956

midx1 = pd.MultiIndex.from_product([[1, 2], [3, 4]], names=["a", "b"])
midx2 = pd.MultiIndex.from_product([[1, 2], [3, 4]], names=["b", "a"])

join_idx, lidx, ridx = midx1.join(midx2, return_indexers=False)

exp_ridx = np.array([-1, -1, -1, -1], dtype=np.intp)

tm.assert_index_equal(midx1, join_idx)
assert lidx is None
tm.assert_numpy_array_equal(ridx, exp_ridx)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use tm.assert_index_equal here instead of breaking up into individual arrays?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can only do this for midx1 as Index.equals is not sensitive for the order of names, but tm.assert_index_equal is.

Anyway, I expanded both the test and the docs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm not sure I totally follow; can you not set the proper expectation inclusive of the order?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mdx does not equal join_idx, so midx2 just remains the same, basically, which is fine for Index.equals, but there is nothing to compare it with in this example using tm.assert_index_equal. That would require a more complicated example. I didn't add any, as the order of the indices is not agreed upon, I just corrected a bug one way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

anyway, since this I expanded the test a little

19 changes: 19 additions & 0 deletions pandas/tests/reshape/merge/test_multi.py
Original file line number Diff line number Diff line change
Expand Up @@ -828,3 +828,22 @@ def test_single_common_level(self):
).set_index(["key", "X", "Y"])

tm.assert_frame_equal(result, expected)

def test_join_multi_wrong_order(self):
# GH 25760
# GH 28956

midx1 = pd.MultiIndex.from_product([[1, 2], [3, 4]], names=["a", "b"])
midx3 = pd.MultiIndex.from_tuples([(4, 1), (3, 2), (3, 1)], names=["b", "a"])

left = pd.DataFrame(index=midx1, data={"x": [10, 20, 30, 40]})
right = pd.DataFrame(index=midx3, data={"y": ["foo", "bar", "fing"]})

result = left.join(right)

expected = pd.DataFrame(
index=midx1,
data={"x": [10, 20, 30, 40], "y": ["fing", "foo", "bar", np.nan]},
)

tm.assert_frame_equal(result, expected)