Crash when inspecting frame of generator

[kumaraditya303]

Crash report

This was discovered in an asyncio program when interrupted with CTRL - C.
Minimal Reproducer:

import gc
import inspect

gc.set_threshold(1, 0, 0)
f = []

def cb(*args):
    f.append(inspect.stack())

gc.callbacks.append(cb)

def gen():
    yield 1

g = gen()
g.__next__()

Error messages

python: Python/ceval.c:5436: _PyEval_EvalFrameDefault: Assertion `frame->frame_obj == NULL' failed.
Aborted (core dumped)

Your environment

• CPython versions tested on: Python 3.12.0a0 51fd4de, Python 3.11.0b3+ 41e4b42
• Operating system and architecture: Linux 5.4.0-1074-azure

cc @markshannon @pablogsal

[markshannon]

This is tricky.

The fundamental problem, IMO, is that almost any memory allocation can run arbitrary code. This makes it really difficult to reason about the state of the VM.

In the short term, we should handle the case where frame->frame_obj != NULL when creating a generator, but there are bound other issues, as the generator and frame object are may not be an introspectable state.

In the longer term, I'd like to only run the GC, and/or finalizers, when the VM is a known state; as we do with interrupts.

I'm concerned that this shows up from an interrupt. It shouldn't.

[kumaraditya303]

The fundamental problem, IMO, is that almost any memory allocation can run arbitrary code. This makes it really difficult to reason about the state of the VM.

I agree, not only the state of VM but also any other object state can be mutated which in some can lead to memory leak or worse crash.

I'm concerned that this shows up from an interrupt. It shouldn't.

It was really hard to reproduce the error, my current theory is that if an interrupt happens while executing __del__ of an object and the interpreter creates a traceback to print, it can create a frame object for a generator and it causes a hard crash while the exception is propagated to the generator.

[kumaraditya303]

I was correct about my theory, the following code sample demonstrates how KeyboardInterrupt can lead to crash:

import gc
gc.set_threshold(1, 0, 0)

class Sneaky:
    def __del__(self):
        raise KeyboardInterrupt


def gen():
    yield 1

sneaky = Sneaky()
sneaky._s = Sneaky()
sneaky._s._s = sneaky
del sneaky

g = gen()

Crash:

Exception ignored in: <function Sneaky.__del__ at 0x7fbfa49220a0>
Traceback (most recent call last):
  File "/workspaces/cpython/main.py", line 6, in __del__
    raise KeyboardInterrupt
    ^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt: 
Exception ignored in: <function Sneaky.__del__ at 0x7fbfa49220a0>
Traceback (most recent call last):
  File "/workspaces/cpython/main.py", line 6, in __del__
    raise KeyboardInterrupt
    ^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt: 
python: Python/ceval.c:5477: _PyEval_EvalFrameDefault: Assertion `frame->frame_obj == NULL' failed.

[markshannon]

Thanks @kumaraditya303 that's really helpful.

[markshannon]

Turns out that the fix isn't that hard.

The problem isn't really the GC behavior* but that partially constructed frames are visible to Python and C extensions.
We fix this by not creating frame objects for partially constructed frames.

*The behavior of the GC and finalization is a problem, but not the problem here.

[pablogsal]

Thanks, @markshannon and @kumaraditya303 for the work! 🎉