Add file based locks

[mrocklin]

I've also reverted #98 . It looks like libhdfs3 already does retries.

[mrocklin]

cc @martindurant

[martindurant]

So, lock around everything...
If you are unconcerned with the the extra overhead, this is fine with me, although I'm not 100% convinced that this will solve unpickling errors in every case.

We should have a requirements.txt now.

[mrocklin]

Are there other situations that concern you?

I believe that all access to libhdfs3 is now controlled under a lock.

On Thu, Nov 10, 2016 at 9:13 AM, Martin Durant [email protected]
wrote:

So, lock around everything...
If you are unconcerned with the the extra overhead, this is fine with me,
although I'm not 100% convinced that this will solve unpickling errors in
every case.

We should have a requirements.txt now.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#99 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszFVenm94mD_Sl1ZSvxN9NxXmAxcXks5q8yaVgaJpZM4KuqAo
.

[martindurant]

The methods under HDFile have not been done. Whilst acting on a file may be immune to this because it's a separate data-node connection (I don't know), at least open must be protected. Probably best to protect everything.

I don't have concrete situations in mind, just a vague feeling that we might have to revisit this.

[mrocklin]

OK, I've extended locks to almost all methods. We don't lock around close or flush. These will cause deadlocks, I suspect because they are called within __del__. I'm going to let these lie for now.

I've benchmarked this against toy data and verified that the way the locks are used now they do not significantly impact write or read performance.

[mrocklin]

@martindurant this could use your review and, if you think it's ok, merge.

[pitrou]

After diagnosing some additional woes on Travis-CI, it seems this PR introduced some deadlocks (which I can reproduce at home on the distributed test suite). See dask/distributed#687 (comment)