Subshells on 6.x branch

[ianthomas23]

We've been planning the next release of ipykernel to be version 7 with the headline backward-incompatible changes being anyio and the addition of experimental subshell support. The anyio implementation is almost complete but it is proving difficult to finish it without breaking too much downstream, and some concerns have been expressed about the choice of anyio and if we really want to support its use going forwards.

I would like to have subshells available in an ipykernel release and although the current implementation in the main branch builds on top of anyio it does not have to do so, and hence I am proposing to try implementing on the 6.x branch so on top of tornado/asyncio to if it is possible. There was a discussion about this at yesterday's Jupyter Server/Kernel meeting which met with some support.

My plan is to submit PRs to the 6.x branch which I will clearly label with 6.x in each PR title. The initial PRs will be to fix the currently failing CI on that branch, hopefully using backported PRs from main but I may to write different fixes. When CI is passing I will attempt to implement subshells on top of that. If this works and looks reasonable, only then will there need to be a discussion of whether we actually do an ipykernel 6.30.0 release with subshells or not.

[davidbrochart]

So that we're clear, the breaking changes are not due to AnyIO per say, but to the fact that we dropped Tornado and the callback-based ZMQ streams. It would have been the same using asyncio.
AnyIO brings more guaranties than asyncio because of structured concurrency and many more design choices, while providing Trio support for free, thus I think we should keep on using it.