ENH: add data hashing routines

[jreback]

xref dask/dask#1807

[jreback]

cc @mikegraham
cc @jcrist
cc @mrocklin

added index=True kw to by default hash the index.

[mikegraham]

With this broader goal in mind, it might be worth considering exactly what the applications are. This current implementation uses the hash builtin, so it will have hash randomization for strings for some Pythons. I don't know what the right move is there, but it's something to keep in mind.

There was a big discussion for the dask application about whether to handle the case where dtype was object but the objects weren't strings, and whether to handle strings in a way that was stable between runs. Part of what shaped that discussion was dask's refraining from introducing C extensions.

[jreback]

@mikegraham yes a stable string hash would be nice. (esp between runs). I don't think performance is an issue ATM (this is pretty fast). But can certainly deal with that if it comes up. Do you have a 'better' way of hashing strings?

[jreback]

should we be using hashlib? https://www.peterbe.com/plog/best-hashing-function-in-python

[jreback]

(pandas) [Thu Nov 24 11:26:03 ~/pandas]$ python -c "import hashlib; print(int(hashlib.md5('foo'.encode('utf8')).hexdigest(), 16) % (10 ** 8))"
985560

I have seen this done. Its stable across runs & compat between python versions.

[jreback]

we could also always stringify object dtype data.

[codecov-io]

Current coverage is 85.28% (diff: 98.14%)

Merging #14729 into master will increase coverage by 0.04%

@@             master     #14729   diff @@
==========================================
  Files           143        144     +1   
  Lines         50849      50903    +54   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43343      43414    +71   
+ Misses         7506       7489    -17   
  Partials          0          0          

Powered by Codecov. Last update 837db72...f5e05a7

[jreback]

In [2]: df = tm.makeMixedDataFrame()

In [3]: df = pd.concat([df]*100000)

In [4]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 500000 entries, 0 to 4
Data columns (total 4 columns):
A    500000 non-null float64
B    500000 non-null float64
C    500000 non-null object
D    500000 non-null datetime64[ns]
dtypes: datetime64[ns](1), float64(2), object(1)
memory usage: 19.1+ MB

In [5]: df.hash()
Out[5]: 
array([ 7765496462383768478,  9003743364327424712, 10642375279821279843,
       ..., 10642375279821279843, 15490030136262591728,
        1988273298325137642], dtype=uint64)

In [6]: %timeit df.hash()
1 loop, best of 3: 995 ms per loop

note great about 1/2 time is spent in hashlib. but can work on that later.

[mikegraham]

should we be using hashlib?

It will be slow has heck. I have a Cython siphash implementation I can contribute if you want. This should also be able to be GIL-releasing.

we could also always stringify object dtype data.

You mean str(x)? I don't think that's super valid -- it will often have unstable data (like the id) in it and will often not have real information about the object in it.

[mikegraham]

https://gist.github.com/mikegraham/feab49dd9a0ac98cc1fb26a3366e84e4

[jreback]

@mikegraham ok, using your cython code we now get about a 6x speedup. so 120ms for 500k rows.
couple of issues.

• have to handle things like tuples (e.g. from a multiindex) / non-strings
• the hash does not appear consistent. maybe I am doing something wrong. can you have a look.

[jreback]

ok, fixed the consistency issue. key has to be 16 long! hahha

[jreback]

and latest push makes this nogil!

[jreback]

note: have to regen these (from old way of hashing)

[jreback]

ok, added tests for mixed dtypes, empty objects.
support for encoding and different hash_keys.

Timing is not bad

In [33]: len(s2)
Out[33]: 500000

In [34]: %timeit s2.hash()
10 loops, best of 3: 126 ms per loop

We can do even better by factorizing first & then hashing (so we wouldn't even need the object hashing code).

In [35]: %timeit Series(s2.factorize()[0]).hash()
10 loops, best of 3: 28 ms per loop

Is this a valid thing to do?

Acutally its not very hard to:

• categorize
• hash the categories
• broadcast back to the original.

[jreback]

are repeated hashes likely?

In [21]: N=1000000; s = pd.Series(tm.makeStringIndex(10).take(np.random.randint(0, 10, size=N)))

In [22]: c = s.astype('category')

In [23]: c.cat.categories
Out[23]: Index(['56bjQF4AsK', '8ZAYAT8GPT', 'AR3qPSViwT', 'I7RtoX1MVN', 'IlKrENATh1', 'QWnGtQeaPG', 'YhkZCbpQHJ', 'a7N5Sb78rE', 'fE08yrQbJ7', 'jVm54MmpWD'], dtype='object')

In [24]: c.cat.categories.hash()
Out[24]: 
array([16874492830844911956, 17129770899522711342, 16874492830844911956,
       17129770899522711342, 16874492830844911956, 17129770899522711342,
       16874492830844911956, 17129770899522711342, 16874492830844911956,
       17129770899522711342], dtype=uint64)

hmm, something funny going on........

In [27]: Index(['foo', 'bar', 'baz']).hash()
Out[27]: array([ 477881037637427054, 1374399572096150070,  477881037637427054], dtype=uint64)

[jreback]

nvm, was incorretly assigning the pointer.

In [1]: Index(['foo', 'bar', 'baz']).hash()
Out[1]: array([3600424527151052760, 1374399572096150070,  477881037637427054], dtype=uint64)

[mikegraham]

We can do even better by factorizing first & then hashing (so we wouldn't even need the object hashing code).

That will make the hash only valid for one given series/df, I think, not between different dfs. I don't know what all the uses are for hashing, but for the dask uses, I think it wouldn't since the hash values for similar values wouldn't agree between partitions.

[jreback]

now for dask I don't actually think this is an issue. as you always know the meta-data anyhow.

In [10]: pd.DataFrame([[1, 'foo'], [2, 'bar'], [3, 'baz']]).hash()
Out[10]: array([11603696091789712533,  5345384428795788641,    46691607209239364], dtype=uint64)

In [11]: pd.DataFrame([[1, 'foo'], [2, 'bar'], [3, 'baz']], columns=list('AB')).hash()
Out[11]: array([11603696091789712533,  5345384428795788641,    46691607209239364], dtype=uint64)

[mikegraham]

I meant to ask what the specific use cases for a whole-dataframe hash were.

[mrocklin]

We use hashing for other things in dask as well. For example we use whole-dataframe hashing to determine keys if they are not given.

In [1]: from dask.base import tokenize

In [2]: import pandas as pd

In [3]: tokenize(pd.DataFrame({'x': [1, 2, 3]}))
Out[3]: '873595b6236c19b206f2e28992546e10'

https://github.com/dask/dask/blob/master/dask/base.py#L292-L315

[chris-b1]

Since this is a row-based hash, it may make sense to wrap the results back in a Series with the original row index?

I'm not 100% clear on the use-case - I would have guessed the DataFrame one is column based, so I also think the docstring could use some stronger language to make it clear it is row-based.

[jreback]

@chris-b1 addressed your comments.

[jorisvandenbossche]

No comments directly about the implementation, but regarding the API: is it needed to have this as a method on the DataFrame/Series/Index object itself?
I think this is not a functionality that will be used a lot directly by users, where interactive tab completion and method chaining is important, but rather used by library developers (eg dask)? Is that correct? In that case, I don't think it is that much an inconvenience for dask to eg have to use pd.tools.hash(df) instead of df.hash()?

The reason I raise this points is because we already have so many methods on dataframe, we should be weigh the added value of having it as a method.

Also not that fond of having it in 0.19.2, but I understand you want to have this quickly :-)

[mrocklin]

I'm quite happy to call an internal method within Dask. It's all the same to me.

…

On Fri, Nov 25, 2016 at 4:58 PM, Joris Van den Bossche < ***@***.***> wrote: No comments directly about the implementation, but regarding the API: is it needed to have this as a method on the DataFrame/Series/Index object itself? I think this is not a functionality that will be used a lot directly by users, where interactive tab completion and method chaining is important, but rather used by library developers (eg dask)? Is that correct? In that case, I don't think it is that much an inconvenience for dask to eg have to use pd.tools.hash(df) instead of df.hash()? The reason I raise this points is because we already have so many methods on dataframe, we should be weigh the added value of having it as a method. Also not that fond of having it in 0.19.2, but I understand you want to have this quickly :-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14729 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszCekgrXO2rWA94uE7P1qqe91HXVgks5rB1oGgaJpZM4K7wlL> .

[jreback]

ok, removed the public API. should be just along for the ride now. We can think about when (or if) to expose this publicly.

[max-sixty]

Thought about using this to override __hash__(), and so making the pandas objects Hashable?

[mikegraham]

@MaximilianR I don't think that's appropriate -- the __eq__ of a pandas object is subject to change.

[jreback]

@MaximilianR in theory this could provide __hash__, but its not cheap to produce, so it couldn't be cached (unless we then track whether data actually is changed, IOW, to mark it dirty). Its mostly useful when a frame is serialized / deserialized anyhow, not the typical scenario for __hash__.

[mikegraham]

Even if you could detected stale cached values, I'm not sure it would be appropriate to define __hash__ for pandas objects. According to the Python Language Reference

If a class defines mutable objects and implements a __cmp__() or __eq__() method, it should not implement __hash__()

[jreback]

@mikegraham we r leaving 'hash' out for now
but this IS a valid use case to provide a data hash that does change when the data is changed . pandas objects are mutable but occasionally you do want to compare versions of them (there are explicit ways of doing this now - namely compare the elements)

For Python structures this is non trivial and requires knowing objects IN a container change. this could be quite non performant for say a python list (where u may have to arbitrarily descend the object tree); a dataframe is more limited in how it can change and generally doesn't hold arbitrary other objects), so this is feasible. whether it is in practice reallly useful is another matter.

[jreback]

any further comments?

[mikegraham]

@jreback Thanks for relating, I agree 100%, I was just saying that it seems like __hash__ isn't the right name for the class of use cases you seem to be describing.