Multiple Start/Stop do not work anymore

[scrocquesel]

From quarkiverse/quarkus-operator-sdk#388

I guessed it was a regression introduced with #1358.
I tried to fix it by adding started = false at the end of Operator::stop but then :

With ConfigurationServiceProvider.overrideCurrent(o -> o.withCloseClientOnStop(false));

(main) Couldn't start informer for exposedapps.halkyon.io/v1alpha1 resources: java.lang.IllegalStateException: Cannot restart a stopped informer

With ConfigurationServiceProvider.overrideCurrent(o -> o.withCloseClientOnStop(true));

(main) Error starting operator: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [CustomResourceDefinition]  with name: [exposedapps.halkyon.io]  in namespace: [test]  failed.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159)
KubernetesClientException.java:159
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:177)
BaseOperation.java:177
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:139)
BaseOperation.java:139
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:88)
BaseOperation.java:88
	at io.javaoperatorsdk.operator.processing.Controller.validateCRDWithLocalModelIfRequired(Controller.java:313)
Controller.java:313
	at io.javaoperatorsdk.operator.processing.Controller.start(Controller.java:290)
Controller.java:290
	at io.javaoperatorsdk.operator.ControllerManager.lambda$start$0(ControllerManager.java:35)
ControllerManager.java:35
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
ForEachOps.java:183
	at java.base/java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1675)
HashMap.java:1675
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
AbstractPipeline.java:484
	at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
ForEachOps.java:290
	at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
CountedCompleter.java:746
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
ForkJoinTask.java:290
	at java.base/java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:408)
ForkJoinTask.java:408
...
Caused by: java.io.IOException: executor rejected
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:514)
OperationSupport.java:514
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:551)
OperationSupport.java:551
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:463)
OperationSupport.java:463
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:713)
BaseOperation.java:713
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.getMandatory(BaseOperation.java:172)
BaseOperation.java:172
	... 97 more
Caused by: java.io.InterruptedIOException: executor rejected
	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:157)
RealCall.java:157
	at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:204)
Dispatcher.java:204
	at okhttp3.Dispatcher.enqueue(Dispatcher.java:144)
Dispatcher.java:144
	at okhttp3.RealCall.enqueue(RealCall.java:93)
RealCall.java:93
	at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl.sendAsync(OkHttpClientImpl.java:273)
OkHttpClientImpl.java:273
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:593)
OperationSupport.java:593
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:570)
OperationSupport.java:570
	... 101 more
Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@d3cce46 rejected from java.util.concurrent.ThreadPoolExecutor@30eedaa4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 10]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
ThreadPoolExecutor.java:2055
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825)
ThreadPoolExecutor.java:825
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355)
	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:154)
RealCall.java:154
	... 107 more

[csviri]

What is the use case of multiple start / stop?

[scrocquesel]

What is the use case of multiple start / stop?

That's what the JUnit extension does before/after each tests. This allow to prepare resources without triggering informer.

[csviri]

That would mean our tests would fail too, that is not the case, but maybe I'm overlooking something. It not meant to be restarted. But if that helps with some practicalities and does not brings complexity we should fix it, sure.

[scrocquesel]

That would mean our tests would fail too, that is not the case, but maybe I'm overlooking something. It not meant to be restarted. But if that helps with some practicalities and does not brings complexity we should fix it, sure.

I guess JOSDK samples only have one test per project, so the junit extension do not need to stop the operator.

[metacosm]

I think the solution here would be to destroy the informers on stop and re-create them if there's a restart.

[csviri]

Yep, since informers cannot be restarted atm, this is the only solution