-
Notifications
You must be signed in to change notification settings - Fork 218
WIP: feat: depends on wait condition design #994
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
ef4bded
feat: depends on wait condition design
csviri a57bca1
fix: format
csviri eb979c9
fix: wip
csviri ac27c25
fix: wait impl
csviri d948537
fix: wip
csviri a203d25
fix: impl, integration test
csviri d0aacfd
fix: test
csviri fb2c3da
fix: test
csviri a115232
fix: test
csviri ac4e862
fix: test
csviri 37a2c28
fix: format
csviri 8d3f01c
fix: test
csviri 193dfa7
test
csviri c746c9b
test
csviri 49e14fa
test
csviri 38a8f10
fix: compilation?
csviri 39d9d3e
fix?
csviri c150a37
Merge branch 'next' into wait-conidtion-design
csviri 32cad58
fix merge
csviri a293dda
fix: build
csviri b68ec17
fix: unify nginx
csviri 9c56414
fix: naming
csviri 522f494
fix: naming
csviri 385edee
fix: add unit tests
csviri File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
8 changes: 8 additions & 0 deletions
8
...ore/src/main/java/io/javaoperatorsdk/operator/processing/dependent/waitfor/Condition.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
package io.javaoperatorsdk.operator.processing.dependent.waitfor; | ||
|
||
@FunctionalInterface | ||
public interface Condition<R> { | ||
|
||
boolean isFulfilled(R resource); | ||
|
||
} |
111 changes: 111 additions & 0 deletions
111
.../main/java/io/javaoperatorsdk/operator/processing/dependent/waitfor/ConditionChecker.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,111 @@ | ||
package io.javaoperatorsdk.operator.processing.dependent.waitfor; | ||
|
||
import java.time.Duration; | ||
import java.time.Instant; | ||
import java.time.temporal.ChronoUnit; | ||
import java.util.Objects; | ||
import java.util.Optional; | ||
import java.util.function.Supplier; | ||
|
||
import io.fabric8.kubernetes.api.model.HasMetadata; | ||
import io.javaoperatorsdk.operator.api.reconciler.UpdateControl; | ||
import io.javaoperatorsdk.operator.api.reconciler.dependent.DependentResource; | ||
|
||
import static java.lang.Thread.sleep; | ||
|
||
public class ConditionChecker<R> { | ||
|
||
public static final Duration DEFAULT_POLLING_INTERVAL = Duration.ofSeconds(1); | ||
public static final Duration DEFAULT_TIMEOUT = Duration.ZERO; | ||
private static final int MIN_THREAD_SLEEP = 25; | ||
|
||
private Duration pollingInterval; | ||
private Duration timeout; | ||
private UnfulfillmentHandler<?> unfulfillmentHandler; | ||
private Condition<R> condition; | ||
|
||
public static <T> ConditionChecker<T> checker() { | ||
return new ConditionChecker<>(); | ||
} | ||
|
||
public ConditionChecker() { | ||
this(DEFAULT_POLLING_INTERVAL, DEFAULT_TIMEOUT, UpdateControl::noUpdate); | ||
} | ||
|
||
public ConditionChecker(Duration pollingInterval, Duration timeout, | ||
UnfulfillmentHandler unfulfillmentHandler) { | ||
this.pollingInterval = pollingInterval; | ||
this.timeout = timeout; | ||
this.unfulfillmentHandler = unfulfillmentHandler; | ||
} | ||
|
||
public <P extends HasMetadata> void check(DependentResource<R, P> resource, P primary) { | ||
check(() -> resource.getResource(primary)); | ||
} | ||
|
||
public void check(Supplier<Optional<R>> supplier) { | ||
checkSetup(); | ||
Optional<R> resource = supplier.get(); | ||
if (timeout.isNegative() || timeout.isZero()) { | ||
if (resource.isPresent() && condition.isFulfilled(resource.get())) { | ||
return; | ||
} else { | ||
handleConditionNotMet(); | ||
} | ||
} | ||
var deadline = Instant.now().plus(timeout.toMillis(), ChronoUnit.MILLIS); | ||
while (Instant.now().isBefore(deadline)) { | ||
resource = supplier.get(); | ||
if (resource.isPresent() && condition.isFulfilled(resource.get())) { | ||
return; | ||
} else { | ||
var timeLeft = Duration.between(Instant.now(), deadline); | ||
if (timeLeft.isZero() || timeLeft.isNegative()) { | ||
handleConditionNotMet(); | ||
} else { | ||
sleepUntilNextPoll(timeLeft); | ||
} | ||
} | ||
} | ||
handleConditionNotMet(); | ||
} | ||
|
||
private void checkSetup() { | ||
Objects.requireNonNull(unfulfillmentHandler, "ConditionNotFulfilledHandler is not set"); | ||
Objects.requireNonNull(condition, "Condition is not set"); | ||
} | ||
|
||
private void sleepUntilNextPoll(Duration timeLeft) { | ||
try { | ||
sleep(Math.max(MIN_THREAD_SLEEP, Math.min(pollingInterval.toMillis(), timeLeft.toMillis()))); | ||
} catch (InterruptedException e) { | ||
Thread.currentThread().interrupt(); | ||
throw new IllegalStateException("Thread interrupted.", e); | ||
} | ||
} | ||
|
||
private void handleConditionNotMet() { | ||
throw new ConditionUnfulfilledException(unfulfillmentHandler); | ||
} | ||
|
||
public ConditionChecker<R> withPollingInterval(Duration pollingInterval) { | ||
this.pollingInterval = pollingInterval; | ||
return this; | ||
} | ||
|
||
public ConditionChecker<R> withTimeout(Duration timeout) { | ||
this.timeout = timeout; | ||
return this; | ||
} | ||
|
||
public ConditionChecker<R> withUnfulfilledHandler( | ||
UnfulfillmentHandler unFulfillmentHandler) { | ||
this.unfulfillmentHandler = unFulfillmentHandler; | ||
return this; | ||
} | ||
|
||
public ConditionChecker<R> withCondition(Condition<R> condition) { | ||
this.condition = condition; | ||
return this; | ||
} | ||
} |
22 changes: 22 additions & 0 deletions
22
.../javaoperatorsdk/operator/processing/dependent/waitfor/ConditionUnfulfilledException.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
package io.javaoperatorsdk.operator.processing.dependent.waitfor; | ||
|
||
import io.javaoperatorsdk.operator.OperatorException; | ||
|
||
public class ConditionUnfulfilledException extends OperatorException { | ||
|
||
private final UnfulfillmentHandler unfulfillmentHandler; | ||
|
||
public ConditionUnfulfilledException(UnfulfillmentHandler unfulfillmentHandler) { | ||
this.unfulfillmentHandler = unfulfillmentHandler; | ||
} | ||
|
||
public ConditionUnfulfilledException(String message, | ||
UnfulfillmentHandler unfulfillmentHandler) { | ||
super(message); | ||
this.unfulfillmentHandler = unfulfillmentHandler; | ||
} | ||
|
||
public UnfulfillmentHandler getUnfulfillmentHandler() { | ||
return unfulfillmentHandler; | ||
} | ||
} |
9 changes: 9 additions & 0 deletions
9
...n/java/io/javaoperatorsdk/operator/processing/dependent/waitfor/UnfulfillmentHandler.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,9 @@ | ||
package io.javaoperatorsdk.operator.processing.dependent.waitfor; | ||
|
||
import io.javaoperatorsdk.operator.api.reconciler.BaseControl; | ||
|
||
public interface UnfulfillmentHandler<P extends BaseControl<P>> { | ||
|
||
BaseControl<P> control(); | ||
|
||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
77 changes: 77 additions & 0 deletions
77
...t/java/io/javaoperatorsdk/operator/processing/dependent/waitfor/ConditionCheckerTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
package io.javaoperatorsdk.operator.processing.dependent.waitfor; | ||
|
||
import java.time.Duration; | ||
import java.util.Optional; | ||
import java.util.concurrent.TimeUnit; | ||
|
||
import org.junit.jupiter.api.Assertions; | ||
import org.junit.jupiter.api.Test; | ||
import org.junit.jupiter.api.Timeout; | ||
|
||
import io.javaoperatorsdk.operator.sample.simple.TestCustomResource; | ||
|
||
import static io.javaoperatorsdk.operator.processing.dependent.waitfor.ConditionChecker.checker; | ||
|
||
class ConditionCheckerTest { | ||
|
||
@Test | ||
void returnsIfDurationIsZeroAndConditionMet() { | ||
checker() | ||
.withTimeout(Duration.ZERO) | ||
.withCondition(r -> true) | ||
.check(() -> Optional.of(new TestCustomResource())); | ||
} | ||
|
||
@Test | ||
void throwsExceptionIfDurationZeroConditionNotFulfilled() { | ||
Assertions.assertThrows( | ||
ConditionUnfulfilledException.class, | ||
() -> new ConditionChecker<TestCustomResource>() | ||
.withTimeout(Duration.ZERO) | ||
.withCondition(r -> false) | ||
.check(() -> Optional.of(new TestCustomResource()))); | ||
} | ||
|
||
@Test | ||
void throwsExceptionNoDurationIfResourceNotPresentWithinTimeout() { | ||
Assertions.assertThrows( | ||
ConditionUnfulfilledException.class, | ||
() -> new ConditionChecker<TestCustomResource>() | ||
.withTimeout(Duration.ZERO) | ||
.withCondition(r -> true) | ||
.check(Optional::empty)); | ||
} | ||
|
||
@Test | ||
@Timeout(value = 100, unit = TimeUnit.MILLISECONDS) | ||
void waitsForTheConditionToFulfill() { | ||
new ConditionChecker<TestCustomResource>() | ||
.withTimeout(Duration.ofMillis(200)) | ||
.withPollingInterval(Duration.ofMillis(50)) | ||
.withCondition(r -> true) | ||
.check(() -> Optional.of(new TestCustomResource())); | ||
} | ||
|
||
@Test | ||
@Timeout(value = 230, unit = TimeUnit.MILLISECONDS) | ||
void throwsExceptionIfConditionNotFulfilledWithinTimeout() { | ||
Assertions.assertThrows( | ||
ConditionUnfulfilledException.class, () -> new ConditionChecker<TestCustomResource>() | ||
.withTimeout(Duration.ofMillis(200)) | ||
.withPollingInterval(Duration.ofMillis(50)) | ||
.withCondition(r -> false) | ||
.check(() -> Optional.of(new TestCustomResource()))); | ||
|
||
} | ||
|
||
@Test | ||
@Timeout(value = 230, unit = TimeUnit.MILLISECONDS) | ||
void throwsExceptionIfResourceNotPresentWithinTimeout() { | ||
Assertions.assertThrows( | ||
ConditionUnfulfilledException.class, () -> new ConditionChecker<TestCustomResource>() | ||
.withTimeout(Duration.ofMillis(200)) | ||
.withPollingInterval(Duration.ofMillis(50)) | ||
.withCondition(r -> true) | ||
.check(() -> Optional.empty())); | ||
} | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this is supposed to be called in a reconcile loop, then it is a synchronuous wait. IMHO,
while-sleep-until timeout
is not a good pattern especially because timeout can be very long and it will lock the loop. It can be start in a parrallel thread but then, I think the same waiting loop can be achieve with event source. No need to poll, the dependent should trigger an event when it change and the condition will eventually be met in a latter reconcile loop. We only miss to setup to trigger an event at most when deadline occurs.Also, the sleep loop is not resilient to an operator restart. I expect the timeout to last from the creation. If the operator stop just after the creation and restart after the timeout, then it should reconcile and immediatly report a timeout and not wait for the timeout.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @scrocquesel , pls take a look here, that explains the use case for this:
#995 (comment)
TBH, I'm also kinda thinking also to put it there or not (I mean the sync wating), to explicitly support that use cases mentioned there. Since it's just for some small subset of operators. But as described, and alos shown in the integration tests, most condition check is async or that is perferred.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would require a state management, what we deliberatily want to avoid if possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that it's still a goal that on a reconciliation we reconcile every resource we manage. So if we want to have an analogy here the goal is not to have a state machine rather a workflow engine that is executed then on every reconciliation from beginning to end.
Well at least that is the idea :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@csviri In the test below, what would happen if nginxNginxDeploymentDependentResource trigger an event before the checker has poll for the condition to be met ? Does a new reconcile is started on another thread or is it discarded because there is already a reconcile ongoing ? A deployment can take quite some time and the more the reconcile loop is running, the more the update is exposed to concurrency edition from the operator itself or from outside. What if an error occurs with the deployment and I want to delete the primary ? Does the check with the default infinite will be interrupted ?
Also, this is may works well for creation, but what should happen if the condition is unmet latter on. The reconcile will again wait in a loop and the status of the primary status will not reflect the actual state of the unmet condition.