Connection._cleanup not being called when the connection is dropped

[ioistired]

This issue is continued from comments on #421 and #283.

Problem steps

Run the following code:

#!/usr/bin/env python3

import asyncpg
import asyncio

class MyConnection(asyncpg.Connection):
	def _cleanup(self):
		print(1)
		return super()._cleanup()

async def amain():
	conn = await asyncpg.connect(connection_class=MyConnection)
	await asyncio.sleep(float('inf'))

if __name__ == '__main__':
	asyncio.run(amain())

Then restart postgres.

Expected results

"1" is printed.

Actual results

Nothing is printed.

System info

• asyncpg version: commit 851d586
• PostgreSQL version: 12.1
• Python version: CPython 3.8.1
• Platform: Arch Linux
• Do you use pgbouncer?: No
• Did you install asyncpg with pip?: No
• If you built asyncpg locally, which version of Cython did you use?: 0.29.14
• Can the issue be reproduced under both asyncio and
  uvloop?: Yes

[elprans]

Well, the code is clearly there:

asyncpg/asyncpg/protocol/protocol.pyx

Lines 863 to 883 in 851d586

	cdef _on_connection_lost(self, exc):
	if self.closing:
	# The connection was lost because
	# Protocol.close() was called
	if self.waiter is not None and not self.waiter.done():
	if exc is None:
	self.waiter.set_result(None)
	else:
	self.waiter.set_exception(exc)
	self.waiter = None
	else:
	# The connection was lost because it was
	# terminated or due to another error;
	# Throw an error in any awaiting waiter.
	self.closing = True
	# Cleanup the connection resources, including, possibly,
	# releasing the pool holder.
	con = self.get_connection()
	if con is not None:
	con._cleanup()
	self._handle_waiter_on_connection_lost(exc)

So there are two options: 1) Either asyncio is not calling connection_lost, or 2) You don't keep a live reference to the connection so it gets garbage collected before _cleanup() is called.

[ioistired]

How is (2) possible? amain() runs forever and does not del conn.

[elprans]

Well, your particular example is broken, since the amain task is never woken up.

[ioistired]

Uhh… hm. This code:

#!/usr/bin/env python3

import asyncpg
import asyncio

cleanup_ran = False

class MyConnection(asyncpg.Connection):
	def _cleanup(self):
		global cleanup_ran
		cleanup_ran = True
		return super()._cleanup()

async def amain():
	conn = await asyncpg.connect(connection_class=MyConnection)
	while True:
		await asyncio.sleep(1)
		if conn.is_closed():
			print(cleanup_ran)
			break

if __name__ == '__main__':
	asyncio.run(amain())

prints True after restarting postgres. I see.

[ioistired]

@elprans this code:

#!/usr/bin/env python3

import asyncpg
import asyncio

async def amain():
	conn = await asyncpg.connect()
	cleanup_ran = False
	def close_listener(_):
		nonlocal cleanup_ran
		cleanup_ran = True
	conn.add_close_listener(close_listener)
	while await asyncio.sleep(1, True):
		if conn.is_closed():
			print(cleanup_ran)

if __name__ == '__main__':
	asyncio.run(amain())

when run against my fork prints False every second after restarting postgres. Any clues as to why?

[elprans]

Put prints into def connection_lost() and def _on_connection_lost() right before it is calling _cleanup() to see where the call chain gets broken.

[ioistired]

Oh hm… I was using pip install -e . which wasn't rebuilding the cython code. Running make causes the above code to print True. Thanks for the help!