Skip to content

gh-74028: concurrent.futures.Executor.map: avoid reference cycles when an exception is raised #131701

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 49 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
7dc850b
`test_concurrent_futures.test_map_exception`: assert no reference cyc…
ebonnal Mar 24, 2025
19edced
test there is no future in referrers
ebonnal Mar 25, 2025
1c3d01a
switch to assertTrue(all(...))
ebonnal Mar 25, 2025
fa790f1
add test msg
ebonnal Mar 25, 2025
e48d0e0
format msgs
ebonnal Mar 25, 2025
99850b8
switch to assert list empty for a better assertion error message
ebonnal Mar 25, 2025
6e78014
edit comment
ebonnal Mar 26, 2025
48ed15d
Merge remote-tracking branch 'cpython/main' into test/executor-map-re…
ebonnal Mar 26, 2025
f59c749
Merge branch 'main' into test/executor-map-ref-cycles
graingert Mar 30, 2025
3dc2461
assert there are no reference cycles at all
graingert Mar 30, 2025
4dce280
avoid a reference cycle in subinterpreter pools
graingert Mar 30, 2025
9bc69b9
test that traceback frames should not contain any variables referring…
ebonnal Mar 30, 2025
9c781c1
interpreter.WorkerContext.run: finally unassign exc_wrapper
ebonnal Mar 30, 2025
a438301
fix msg grammar
ebonnal Mar 30, 2025
ada6140
fix access to exception
ebonnal Mar 30, 2025
d5e8c7a
narrow test: traceback frames should not contain any exception
ebonnal Mar 30, 2025
c4b771a
clean up exc_wrapper with minimal indentation
ebonnal Mar 31, 2025
03f8ab4
test_map_exception: only search for exception that captures itself in…
ebonnal Mar 31, 2025
ef30122
concurrent.futures.process: avoid ref cycle in _sendback_result
ebonnal Mar 31, 2025
09b0819
concurrent.futures.process: avoid ref cycle in _process_worker
ebonnal Mar 31, 2025
4fab860
format test msg
ebonnal Mar 31, 2025
0714013
Merge remote-tracking branch 'cpython/main' into test/executor-map-re…
ebonnal Mar 31, 2025
f5135ab
refactor ZeroDivisionError assert
ebonnal Mar 31, 2025
855c427
Revert "concurrent.futures.process: avoid ref cycle in _process_worker"
ebonnal Mar 31, 2025
4504516
Revert "concurrent.futures.process: avoid ref cycle in _sendback_result"
ebonnal Mar 31, 2025
fc7a569
skip referrers test for free-threading build on win/linux
ebonnal Mar 31, 2025
cae6474
call gc.collect for free-threading on linux/windows
ebonnal Mar 31, 2025
4abb239
Revert "call gc.collect for free-threading on linux/windows"
ebonnal Apr 1, 2025
4bc07c0
Revert "skip referrers test for free-threading build on win/linux"
ebonnal Apr 1, 2025
7b7a5e6
Merge branch 'main' into test/executor-map-ref-cycles
graingert Apr 3, 2025
3b00c5f
more gc referrers
graingert Apr 3, 2025
1948eb3
even more gc referrers
graingert Apr 3, 2025
37b7cb3
Merge branch 'main' into test/executor-map-ref-cycles
graingert Apr 4, 2025
9184964
delete a ref to work_item
graingert Apr 4, 2025
7e5300c
delete reference to result_item before joining process
graingert Apr 4, 2025
c5e39ee
unpack ResultItem and WorkItem
graingert Apr 4, 2025
548d517
fix typo
graingert Apr 5, 2025
fac7e89
Merge branch 'main' into test/executor-map-ref-cycles
graingert Apr 5, 2025
fd9a89e
bit of a hack?
graingert Apr 5, 2025
dbaa044
Revert investigations
ebonnal Apr 7, 2025
869e4b7
pause before checking referrers
ebonnal Apr 7, 2025
a40b9b8
Merge remote-tracking branch 'cpython/main' into test/executor-map-re…
ebonnal Apr 7, 2025
1ee9bf4
add comment about the avoided ref cycle with exc_wrapper
ebonnal Apr 7, 2025
95765f6
format test
ebonnal Apr 7, 2025
9020971
Merge remote-tracking branch 'cpython/main' into test/executor-map-re…
ebonnal Apr 7, 2025
a5c6b17
make the self-captured exceptions check not rely on var name
ebonnal Apr 7, 2025
0f8748c
use traceback.walk_tb
ebonnal Apr 7, 2025
85a3c36
comment the skipping of the current frame
ebonnal Apr 7, 2025
82542f3
test_map_exception: improve readability
ebonnal Apr 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions Lib/concurrent/futures/interpreter.py
Original file line number Diff line number Diff line change
Expand Up @@ -205,8 +205,11 @@ def run(self, task):
assert res is None, res
assert pickled
assert exc_wrapper is not None
exc = pickle.loads(excdata)
raise exc from exc_wrapper
try:
raise pickle.loads(excdata) from exc_wrapper
finally:
# avoid a ref cycle where exc_wrapper is captured in its traceback
exc_wrapper = None
return pickle.loads(res) if pickled else res


Expand Down
39 changes: 37 additions & 2 deletions Lib/test/test_concurrent_futures/executor.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import gc
import itertools
import threading
import time
import traceback
import weakref
from concurrent import futures
from operator import add
Expand Down Expand Up @@ -55,8 +57,41 @@ def test_map_exception(self):
i = self.executor.map(divmod, [1, 1, 1, 1], [2, 3, 0, 5])
self.assertEqual(i.__next__(), (0, 1))
self.assertEqual(i.__next__(), (0, 1))
with self.assertRaises(ZeroDivisionError):

exception = None
try:
i.__next__()
except Exception as e:
exception = e
self.assertTrue(
isinstance(exception, ZeroDivisionError),
msg="should raise a ZeroDivisionError",
)

# pause needed for free-threading builds on Ubuntu (ARM) and Windows
time.sleep(1)

self.assertFalse(
gc.get_referrers(exception),
msg="the exception should not have any referrers",
)

self.assertFalse(
[
(var, val)
# go through the frames of the exception's traceback
for frame, _ in traceback.walk_tb(exception.__traceback__)
# skipping the current frame
if frame is not exception.__traceback__.tb_frame
# go through the locals captured in that frame
for var, val in frame.f_locals.items()
# check if one of them is an exception
if isinstance(val, Exception)
# check if it is captured in its own traceback
and frame is val.__traceback__.tb_frame
],
msg=f"the exception's traceback should not contain an exception captured in its own traceback",
)

@support.requires_resource('walltime')
def test_map_timeout(self):
Expand Down Expand Up @@ -140,7 +175,7 @@ def test_map_buffersize_when_buffer_is_full(self):
self.assertEqual(
next(ints),
buffersize,
msg="should have fetched only `buffersize` elements from `ints`.",
msg="should have fetched only `buffersize` elements from `ints`",
)

def test_shutdown_race_issue12456(self):
Expand Down
Loading