Skip to content

Test generation for Optional<T> in the plugin (#226) #305

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 28, 2022

Conversation

dtim
Copy link
Collaborator

@dtim dtim commented Jun 27, 2022

Description

Added a check for overridden classes in shouldMock to avoid access to engine classes that are not available in the plugin.

Implemented more accurate speculative marking of final fields as not null to avoid losing paths involving Optional.empty(),
to enable NPE checks for final fields in user code, and to avoid generating non-informative NPE tests for final fields in system
classes.

UtSettings.checkNpeForFinalFields is now set default (false) in AbstractTestCaseGEneratorTest and SummaryTestCaseGeneratorTest.

Fixes #226

Details

There were two issues with test generation for Optional<T>.

  1. The shouldMock function tried to load overridden classes (in particular, UtOptional) and failed as the entire
    org.utbot.engine.overrides package is not included in the UtContext class loader created by the plugin.

    This commit fixes the problem an explicit check for overridden classes in shouldMock.

  2. Execution paths involving Optional.empty() were lost. This was caused by the assumption that all final fields
    can be considered non-nullable to reduce the number of irrelevant NPE execution paths, especially in system classes,
    that can usually be reproduced only by reflection.

    The Optional class is different from most other system classes as it uses null as the marker of the empty value in its final field. The "non-nullable final" optimization leads to discarding execution paths with Optional.empty() as UNSAT, as the final
    value field should be both null (explicit assignment) and not null (assumption for final fields).

    This commit introduces a new memory array to track final fields. For each NPE branch, the "value is null" condition is augmented by the "field is not final" condition. As a result, NPE branches for final fields are discarded with UNSAT, while non-final fields are not affected. The finality check is enabled for system classes only, and may be turned off by setting UtSettings.checkNpeForFinalFields to true.

Type of Change

  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

This commit changes the default behavior of the engine with respect to final fields nullability assumptions.

Previous behavior:

  • null values for final fields are not allowed.
  • Branches where the final field gets null value are discarded even if they are legitimate. No NPE tests are generated.

New behavior:

  • null values for final fields are allowed.
  • NPE branches for final fields of system classes are discarded (the only expected change for system classes is fixing the Optional.empty() bug and any similar bugs if they exist), but can be allowed by setting UtSettings.checkNpeForFinalFields to true.
  • NPE tests for application classes are now generated (they may be legit or redundant, but the best way to avoid redundant tests here and not to lose legit branches is to create objects using the public API instead of reflection).

How Has This Been Tested?

Automated Testing

The UtSettings.checkNpeForFinalFields = true setting was previously used to allow NPE branch generation for final fields in utbot-framework unit tests. Setting it to false would break some tests.

This commit turns off this special mode, so utbot-framework use the same settings w.r.t. final fields nullability as the plugin. No new test failures have been detected in this mode.

A new test case was added: org.utbot.examples.collections.OptionalsTest#testOptionalOfPositive.

Manual Scenario

Generate tests for the following examples using the plugin.

Optional.empty()

public class OptionalExamples {
    public Optional<Integer> nonEmptyIfPositive(int n) {
        if (n > 0) {
            return Optional.of(n);
        } else {
            return Optional.empty();
        }
    }

    public Optional<Boolean> nonEmptyIfTrue(boolean condition) {
        return condition ? Optional.of(true) : Optional.empty();
    }
}

For both methods, two tests should be generated, one expecting a non-empty value, and one expecting the Optional.empty() result. No NPE tests should be generated.

NPE checks for application classes

public class FinalNulls {
    private final Integer shift;

    public FinalNulls() {
        this.shift = null;
    }

    public FinalNulls(int shift) {
        this.shift = shift;
    }

    public int plus(int a){
        return a + shift;
    }
}

Two tests should be generated: a test for shift != null and a NPE test (shift == null).

Checklist (remove irrelevant options):

  • The change followed the style guidelines of the UTBot project
  • Self-review of the code is passed
  • The change contains enough commentaries, particularly in hard-to-understand areas
  • No new warnings
  • Tests that prove my change is effective
  • All tests pass locally with my changes

@dtim dtim requested review from CaelmBleidd and Damtev June 27, 2022 07:40
@dtim dtim force-pushed the dtim/226-test-generation-for-optional branch from 333d563 to 2840303 Compare June 27, 2022 12:11
Copy link
Member

@CaelmBleidd CaelmBleidd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, LGTM

@dtim dtim force-pushed the dtim/226-test-generation-for-optional branch from 90444e7 to 22cb771 Compare June 27, 2022 16:24
@dtim dtim force-pushed the dtim/226-test-generation-for-optional branch 3 times, most recently from 4a46744 to 646c12c Compare June 28, 2022 09:55
dtim added 2 commits June 28, 2022 12:57
Added a check for overridden classes in `shouldMock` to avoid
access to engine classes that are not available in the plugin.

Implemented more accurate speculative marking of final fields
as not null to avoid losing paths involving `Optional.empty()`,
to enable NPE checks for final fields in user code, and to avoid
generating non-informative NPE tests for final fields in system
classes.

UtSettings.checkNpeForFinalFields is now set default (false)
in `AbstractTestCaseGEneratorTest` and `SummaryTestCaseGeneratorTest`.
@dtim dtim force-pushed the dtim/226-test-generation-for-optional branch from 646c12c to fa9e67b Compare June 28, 2022 09:57
@dtim dtim enabled auto-merge (rebase) June 28, 2022 10:00
@dtim dtim merged commit fa3ef8d into main Jun 28, 2022
@dtim dtim deleted the dtim/226-test-generation-for-optional branch June 28, 2022 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Test generation fails for Optional<T> class in the plugin
3 participants