Plugins are set up too late for custom exchange types

[michaelklishin]

Custom exchange types may need to re-populate their state during binding recovery (consistent hashing exchange is one example) and starting with 3.8.4, plugin activation happens too late
so said exchanges are skipped:

2020-06-12 08:45:30.687 [debug] <0.490.0> Supervisor {<0.490.0>,rabbit_vhost_sup} started rabbit_amqqueue_sup_sup:start_link() at pid <0.506.0>
2020-06-12 08:45:30.687 [debug] <0.508.0> Supervisor {<0.508.0>,rabbit_amqqueue_sup} started rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"q1">>},true,false,none,[],<0.509.0>,[],[],[],undefined,undefined,...}, recovery, <0.507.0>) at pid <0.509.0>
2020-06-12 08:45:30.687 [debug] <0.511.0> Supervisor {<0.511.0>,rabbit_amqqueue_sup} started rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"q4">>},true,false,none,[],<0.512.0>,[],[],[],undefined,undefined,...}, recovery, <0.510.0>) at pid <0.512.0>
2020-06-12 08:45:30.687 [debug] <0.514.0> Supervisor {<0.514.0>,rabbit_amqqueue_sup} started rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"q3">>},true,false,none,[],<0.515.0>,[],[],[],undefined,undefined,...}, recovery, <0.513.0>) at pid <0.515.0>
2020-06-12 08:45:30.688 [debug] <0.517.0> Supervisor {<0.517.0>,rabbit_amqqueue_sup} started rabbit_prequeue:start_link({amqqueue,{resource,<<"/">>,queue,<<"q2">>},true,false,none,[],<0.518.0>,[],[],[],undefined,undefined,...}, recovery, <0.516.0>) at pid <0.518.0>
2020-06-12 08:45:30.694 [warning] <0.531.0> Could not find exchange type x-consistent-hash.
2020-06-12 08:45:30.694 [warning] <0.491.0> Could not find exchange type x-consistent-hash.
2020-06-12 08:45:30.694 [warning] <0.431.0> Could not find exchange type x-consistent-hash.
2020-06-12 08:45:30.694 [warning] <0.432.0> Could not find exchange type x-consistent-hash.
2020-06-12 08:45:30.694 [warning] <0.433.0> Could not find exchange type x-consistent-hash.
2020-06-12 08:45:30.694 [warning] <0.434.0> Could not find exchange type x-consistent-hash.

...

2020-06-12 08:45:30.700 [debug] <0.566.0> == Postlaunch phase ==
2020-06-12 08:45:30.700 [debug] <0.566.0>
2020-06-12 08:45:30.700 [debug] <0.566.0> == Plugins ==
2020-06-12 08:45:30.700 [debug] <0.566.0> Setting plugins up

This means that consistent hashing ring is not repopulated for durable consistent hashing exchanges, and such exchanges do not route any published messages as a result.

[michaelklishin]

So the sequence of events starting with 3.8.4 is this:

• During a "launch phase", all boot steps are executed and the step named recovery will recover all virtual hosts, exchanges, queues, and bindings
• Exchanges of unknown type are filtered out
• Then in a "post-launch phase" plugins are activated

[michaelklishin]

In this particular case it was easy enough to make the plugin itself recover its state => closing this for now.

This is a fundamental chicken-and-egg problem that will affect plugins that depend on the state of the topology in the future, so perhaps investigating a more fundamental solution would be worth our time.

[lukebakken]

@michaelklishin does this affect the delayed message exchange?

[michaelklishin]

As far as I can tell, it does not: that plugin stores its state in a local Mnesia table and does not reconstruct it using add_binding/2 and similar callbacks. Such exchanges usually don't have any bindings so as long as the exchange itself is recovered, it should be functional.

[tbnguyen1407]

Hello this seems does not fix loading x-consistent-hash exchange type from load_definitions or management.load_definitions (error invalid exchange type).

The plugin boot step happens after definitions are loaded. Anyway to workaround this?

[michaelklishin]

@tbnguyen1407 the chicken-and-egg problem of definition load, default data seeding and plugin activation is not a new problem and it affects more than just custom exchange types. Definitions can be imported at any time after the node is fully booted.

[michaelklishin]

@dumbbell agrees with the solution proposed in #2384. We'll try to squeeze it into 3.8.6.

[michaelklishin]

Addressed in #2384 which will ship in 3.8.6. Those on earlier versions can always import definitions after node startup.