"spawn" method doesn't work on 2.7 (under Linux) #201

pitrou · 2016-11-21T18:12:27Z

The following simple script:

from billiard import get_context

ctx = get_context('spawn')


def f(x):
    return x*x

if __name__ == '__main__':
    with ctx.Pool(2) as p:
        print(p.map(f, list(range(10))))

fails with:

Traceback (most recent call last):
  File "d.py", line 11, in <module>
    with ctx.Pool(2) as p:
  File "/home/antoine/billiard/billiard/context.py", line 165, in Pool
    context=self.get_context())
  File "/home/antoine/billiard/billiard/pool.py", line 1008, in __init__
    self._create_worker_process(i)
  File "/home/antoine/billiard/billiard/pool.py", line 1117, in _create_worker_process
    w.start()
  File "/home/antoine/billiard/billiard/process.py", line 122, in start
    self._popen = self._Popen(self)
  File "/home/antoine/billiard/billiard/context.py", line 341, in _Popen
    return Popen(process_obj)
  File "/home/antoine/billiard/billiard/popen_spawn_posix.py", line 39, in __init__
    super(Popen, self).__init__(process_obj)
  File "/home/antoine/billiard/billiard/popen_fork.py", line 24, in __init__
    self._launch(process_obj)
  File "/home/antoine/billiard/billiard/popen_spawn_posix.py", line 56, in _launch
    reduction.dump(process_obj, fp)
  File "/home/antoine/billiard/billiard/reduction.py", line 99, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 401, in save_reduce
    save(args)
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 568, in save_tuple
    save(element)
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/antoine/miniconda3/envs/dask27/lib/python2.7/pickle.py", line 754, in save_global
    (obj, module, name))
pickle.PicklingError: Can't pickle <class 'billiard.pool.Process'>: it's not found as billiard.pool.Process

The text was updated successfully, but these errors were encountered:

auvipy · 2017-07-17T13:41:26Z

can you verify it against new release?

pitrou · 2017-07-17T13:51:09Z

It still doesn't work. Now it doesn't raise anymore, but it hangs:

No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
No handlers could be found for logger "multiprocessing"
[ hangs here ]

pitrou · 2017-07-17T13:52:30Z

If I use faulthandler to set a process timeout, I see the following traces:

Timeout (0:00:04)!
Thread 0x00007f7a6274b700 <python> (most recent call first):
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/connection.py", line 976 in _poll
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/connection.py", line 998 in wait
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/connection.py", line 465 in _poll
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/connection.py", line 287 in poll
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 1326 in _poll_result
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 826 in _process_result
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 861 in body
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 478 in run
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x00007f7a62f4c700 <python> (most recent call first):
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 340 in wait
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/Queue.py", line 168 in get
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 561 in body
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 478 in run
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x00007f7a6374d700 <python> (most recent call first):
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 538 in body
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 478 in run
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 801 in __bootstrap_inner
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 774 in __bootstrap

Current thread 0x00007f7a69124700 <python> (most recent call first):
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 340 in wait
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/threading.py", line 614 in wait
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 1733 in wait
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 1736 in get
  File "/home/antoine/miniconda3/envs/bbb/lib/python2.7/site-packages/billiard/pool.py", line 1372 in map
  File "b.py", line 17 in <module>

auvipy · 2017-07-17T14:51:35Z

do you have any suggested fix?

pitrou · 2017-07-17T16:49:13Z

None. I don't use billiard (I evaluated it for Dask/distributed and decided against it) and I don't have time to get acquainted with it.

That said, if the functionality doesn't work and doesn't stand a chance of being fixed, it would be reasonable to remove it so that people don't get lured :-)

auvipy · 2018-01-15T17:56:35Z

we have a plan to rewrite celeray billiard part with asyncio

tonal · 2018-06-05T08:49:54Z

see issue #241 and MR #245

jakirkham · 2018-06-23T00:56:57Z

Still failing it seems. Used the OPs script.

Traceback (most recent call last):
  File "test_fork_exec.py", line 10, in <module>
    with ctx.Pool(2) as p:
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/context.py", line 165, in Pool
    context=self.get_context())
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/pool.py", line 959, in __init__
    self._setup_queues()
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/pool.py", line 1328, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/context.py", line 150, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/queues.py", line 377, in __init__
    self._rlock = ctx.Lock()
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/context.py", line 105, in Lock
    return Lock(ctx=self.get_context())
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/synchronize.py", line 182, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/synchronize.py", line 101, in __init__
    register(self._semlock.name)
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/semaphore_tracker.py", line 72, in register
    self._send('REGISTER', name)
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/semaphore_tracker.py", line 79, in _send
    self.ensure_running()
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/semaphore_tracker.py", line 61, in ensure_running
    spawnv_passfds(exe, args, fds_to_pass)
  File "/Users/kirkhamj/Developer/PyCharmCE/billiard/billiard/compat.py", line 217, in spawnv_passfds
    False, False, None)
TypeError: fork_exec() argument 4 must be tuple, not list

tonal · 2018-07-06T04:20:42Z

On https://github.com/celery/billiard/blob/master/billiard/compat.py#L217 only return pid
@jakirkham need use last source.

tonal · 2018-07-06T06:17:49Z

For latest master bf04adf all tests in 2.7 - Ok.

And start code sample work ok:

from __future__ import absolute_import
from billiard import get_context

ctx = get_context('spawn')

def f(x):
    return x*x

def test_spawn():
    with ctx.Pool(2) as p:
        print(p.map(f, list(range(10))))

if __name__ == '__main__':
    test_spawn()

jakirkham · 2018-07-16T23:09:50Z

Have tried with the Python 2.7 version and Billiard 3.5.0.4 on the OP's script and it fails as described in this comment. Had to use kill -s 9 <PID> to get rid of the processes. Nothing short of that worked.

jakirkham · 2018-07-16T23:14:48Z

On Python 3.6.5 and Billiard 3.5.0.4, I get a different error using the OPs script.

Pool process <billiard.pool.Worker object at 0x10e948320> error: AttributeError("Can't get attribute 'f' on <module '__main__' (built-in)>",)
Traceback (most recent call last):
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 289, in __call__
    sys.exit(self.workloop(pid=pid))
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 347, in workloop
    req = wait_for_job()
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 447, in receive
    ready, req = _receive(1.0)
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 419, in _recv
    return True, loads(get_payload())
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/common.py", line 107, in pickle_loads
    return load(BytesIO(s))
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Pool process <billiard.pool.Worker object at 0x100f2a390> error: AttributeError("Can't get attribute 'f' on <module '__main__' (built-in)>",)
Traceback (most recent call last):
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 289, in __call__
    sys.exit(self.workloop(pid=pid))
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 347, in workloop
    req = wait_for_job()
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 447, in receive
    ready, req = _receive(1.0)
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/pool.py", line 419, in _recv
    return True, loads(get_payload())
  File "/zopt/conda3/envs/billiard3/lib/python3.6/site-packages/billiard/common.py", line 107, in pickle_loads
    return load(BytesIO(s))
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Process 'SpawnPoolWorker-2' pid:48250 exited with 'exitcode 1'
Process 'SpawnPoolWorker-1' pid:48249 exited with 'exitcode 1'

It seems to create new spawn pool workers that fail the same way. This continues until the 5th worker is launched and hangs. Though it is at least responsive to ^c.

pitrou mentioned this issue Nov 21, 2016

Use forkserver on Unix and Python 3 dask/distributed#687

Merged

auvipy added the Issue Type: Bug label Jul 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

"spawn" method doesn't work on 2.7 (under Linux) #201

"spawn" method doesn't work on 2.7 (under Linux) #201

pitrou commented Nov 21, 2016

auvipy commented Jul 17, 2017

Uh oh!

pitrou commented Jul 17, 2017

Uh oh!

pitrou commented Jul 17, 2017

Uh oh!

auvipy commented Jul 17, 2017

Uh oh!

pitrou commented Jul 17, 2017

Uh oh!

auvipy commented Jan 15, 2018

Uh oh!

tonal commented Jun 5, 2018

Uh oh!

jakirkham commented Jun 23, 2018

Uh oh!

tonal commented Jul 6, 2018

Uh oh!

tonal commented Jul 6, 2018

Uh oh!

jakirkham commented Jul 16, 2018 •

edited

Loading

Uh oh!

jakirkham commented Jul 16, 2018

Uh oh!

"spawn" method doesn't work on 2.7 (under Linux) #201

"spawn" method doesn't work on 2.7 (under Linux) #201

Comments

pitrou commented Nov 21, 2016

auvipy commented Jul 17, 2017

Uh oh!

pitrou commented Jul 17, 2017

Uh oh!

pitrou commented Jul 17, 2017

Uh oh!

auvipy commented Jul 17, 2017

Uh oh!

pitrou commented Jul 17, 2017

Uh oh!

auvipy commented Jan 15, 2018

Uh oh!

tonal commented Jun 5, 2018

Uh oh!

jakirkham commented Jun 23, 2018

Uh oh!

tonal commented Jul 6, 2018

Uh oh!

tonal commented Jul 6, 2018

Uh oh!

jakirkham commented Jul 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jakirkham commented Jul 16, 2018

Uh oh!

jakirkham commented Jul 16, 2018 •

edited

Loading