Rework type inference #4080

odersky · 2018-03-06T20:41:54Z

This is an attempt to put type inference on a more solid basis. The main two commits are

Drop Ephemeral `90a661b`

Changing the interpolation scheme uncovered several cache invalidation problems with

the asSeenFrom cache in Denotations
the superType cache in AppliedType
the lastDenotation cache in NamedType

The new interpolation scheme performed essentially the same operations as the old one, but sometimes in a different order. I am still not quite sure how the differences made the cache invalidations fail. On the other hand, it is quite plausible (obvious even, in retrospect) that the previous invalidation schemes are incomplete. So this commit replaces them with a common algorithm that does not rely on the previous global state represented by ephemeral.

New interpolation scheme `417f0c8`

Variable interpolation "improves" types by instantiating type variables that do not appear in the result of a type derivation, or that appear only co- or only contra-variantly. This is convenient because it keeps the constraints small. It is also necessary since some operations have more or better solutions once type variables are instantiated. For instance, the members of a type
variable

X >: T <: U

are the members of U. But assuming X is covariant once it is instantiated to T we get more members. Similarly for implicit searches.

The major changes from the previous interpolation scheme are the following:

We explicitly keep track in typer of which variables should and
which should not be interpolated. This replaces searching trees
for embedded variable definitions, which is fragile e.g. in the
presence of eta expansion.
We compute variances starting with all variables found in the type,
not just the qualifying ones. The previous scheme caused some
variance information to be missed, which caused some variables
to be mis-classified as non-occurring. i4032.scala is a test case.
Unfortunately, fixing this caused several other tricky inference
failures which were previously hidden because some variables
were already instantiated prematurely. Examples were hmap.scala,
hmap-covariant.scala, and i2300.scala. In all these cases there was another
problem which was masked by the fact that some type variables had
already been instantiated where they should not have been.
We interpolate at the end of typedUnadapted instead of at the
beginning of adapt. Tracking instantiatable variables turned out
to be easier this way.

smarter · 2018-03-06T20:56:52Z

compiler/src/dotty/tools/dotc/core/Types.scala

+          x || t.mightBeProvisional && {
+            t.mightBeProvisional = t match {
+              case t: TypeVar =>
+                !t.inst.exists


An instantiated TypeVar could have its underlying type refer to another uninstantiated TypeVar and therefore still be provisional I think.

odersky · 2018-03-06T21:32:34Z

test performance please

dottybot · 2018-03-06T21:33:17Z

performance test scheduled: 1 job(s) in queue, 0 running.

dottybot · 2018-03-07T00:01:21Z

performance test failed:

Error line number: 24

[check /data/workspace/bench/logs/pull-4080-03-07-00.16.out for more information]

odersky · 2018-03-07T07:53:58Z

test performance please

dottybot · 2018-03-07T07:54:51Z

performance test scheduled: 1 job(s) in queue, 0 running.

dottybot · 2018-03-07T09:40:00Z

Performance test finished successfully:

Visit http://dotty-bench.epfl.ch/4080/ to see the changes.

Benchmarks is based on merging with master (4be2e76)

odersky · 2018-03-07T13:11:52Z

test performance please

dottybot · 2018-03-07T13:12:01Z

performance test scheduled: 1 job(s) in queue, 0 running.

Changing the interpolation scheme uncovered several cache invalidation problems with - the asSeenFrom cache in Denotations - the superType cache in AppliedType - the lastDenotation cache in NamedType The new denotation scheme performed essentially same operations as the old one, but sometimes in a different order. I am still not quite sure how the differences made the cache invalidations fail. On the other hand, it is quite plausible (obvious even, in retrospect) that the previous invalidation schemes are incomplete. So this commit replaces them with a common algorithm that does not rely on the previous global state represented by `ephemeral`.

Allow to define what gets shown as a result on a backtrace

The previous scheme created a `val showOp = <some closure>` value for each trace operation. It was unused if tracing was disabled. Still might be better to avoid its creation in the first place.

-Yshow-no-inline suppresses "inlined from" parts when printing trees. This is useful when one has deeply inlined structures, as is the case when looking at `trace`ed code.

Used for small, linked sets. Normal immutable sets are about as fast for 0 - 4 elements, but are not linked for larger sizes.

Identify them by number. Helps in the same way other fixed numbering schemes help understand debug output.

- Fix isProvisional condition for TypeVars - Force recomputation via memberDenot in NamedType if previous prefix was provisional

Need to follow up later on what caused it to fail.

Avoids needlessly complicated inferred types such as C[_ >: 1.type <: Singleton] by detecting that that this is equivalent to `C[1.type]`.

Hashes ruin diffability; replace them with the serial `id` numbers.

We missed some cases before.

Major changes from previous one: - We explicitly keep track in typer of which variables should and which should not be interpolated. This replaces searching trees for embedded variable definitions, which is fragile e.g. in the presence of eta expansion. - We compute variances starting with all variables found in the type, not just teh qualifying ones. The previous scheme caused some variance information to be missed, which caused some variables to be mis-classified as non-occurring. i4032.scala is a test case. Unfortunately, fixing this caused several other tricky inference failures because which were previously hidden because some variables were already instantiated prematurely. Examples were hamp.scala, hmap-covariant.scala, and i2300.scala. - We interpolate at the end of typedUnadapted instead of at the beginning of `adapt`. Managing instantiatable variables turned out easier this way.

odersky · 2018-03-07T14:38:08Z

Rebased

dottybot · 2018-03-07T14:57:16Z

Performance test finished successfully:

Visit http://dotty-bench.epfl.ch/4080/ to see the changes.

Benchmarks is based on merging with master (bdfe740)

Call simplify and interpolateTypeVars in the same situations.

odersky · 2018-03-07T15:56:04Z

test performance please

dottybot · 2018-03-07T15:56:14Z

performance test scheduled: 1 job(s) in queue, 1 running.

smarter

I like it!

smarter · 2018-03-07T17:25:44Z

compiler/src/dotty/tools/dotc/core/TyperState.scala

@@ -76,6 +76,11 @@ class TyperState(previous: TyperState /* | Null */) {
  /** The uninstantiated variables */
  def uninstVars = constraint.uninstVars

+  /** The set of uninstantiated type varibles which have this state as their owning state */


typo: varibles -> variables

smarter · 2018-03-07T17:27:10Z

compiler/src/dotty/tools/dotc/transform/Erasure.scala

      trace(i"adapting ${tree.showSummary}: ${tree.tpe} to $pt", show = true) {
-        assert(ctx.phase == ctx.erasurePhase.next, ctx.phase)
+        assert(ctx.phase == ctx.erasurePhase || ctx.phase == ctx.erasurePhase.next, ctx.phase)


Why did this change?

Not sure why. Some transform ran at phase erasure and used the erasure typer to do it. Seems legit. Not sure why it did not run before.

smarter · 2018-03-07T17:41:46Z

compiler/src/dotty/tools/dotc/typer/Inferencing.scala

+   *      Y <: X
+   *
+   *  Then `Y` also occurs co-variantly in `T` because it needs to be minimized in order to constrain
+   *  `T` teh least. See `variances` for more detail.


typo: teh -> the

dottybot · 2018-03-07T19:11:32Z

Performance test finished successfully:

Visit http://dotty-bench.epfl.ch/4080/ to see the changes.

Benchmarks is based on merging with master (bdfe740)

Type variable instantiation should only occur in Typer, TreeChecker and during exhaustiveness checking. This was not enforced until now, and wa not always true before scala#4080.

Type variable instantiation should only occur in Typer, TreeChecker and during exhaustiveness checking, this means that no uninstantiated type variable should exist outside of these phases. This was not enforced until now, and was not always true before scala#4080.

smarter reviewed Mar 6, 2018

View reviewed changes

odersky added 15 commits March 7, 2018 15:31

More flexible tracing

860d102

Allow to define what gets shown as a result on a backtrace

Avoid creation of an unused closure value in each trace op

0ff0189

The previous scheme created a `val showOp = <some closure>` value for each trace operation. It was unused if tracing was disabled. Still might be better to avoid its creation in the first place.

Allow to suppress "inlined from" parts when printing trees

086f33d

-Yshow-no-inline suppresses "inlined from" parts when printing trees. This is useful when one has deeply inlined structures, as is the case when looking at `trace`ed code.

Add SimpleIdentitySet data structure

8a19054

Used for small, linked sets. Normal immutable sets are about as fast for 0 - 4 elements, but are not linked for larger sizes.

Specialize SimpleIdentitySet with 3 elements

06fd9aa

Better printing of TyperStates

2b6ce9c

Identify them by number. Helps in the same way other fixed numbering schemes help understand debug output.

Show resulting type when tracing "adapt" calls

e3d5a7f

Two fixes to invalidation scheme

93a0bca

- Fix isProvisional condition for TypeVars - Force recomputation via memberDenot in NamedType if previous prefix was provisional

Workaroound for testOptimized failure.

1215dd2

Need to follow up later on what caused it to fail.

More precise characterization of singleton bounds

c460627

Avoids needlessly complicated inferred types such as C[_ >: 1.type <: Singleton] by detecting that that this is equivalent to `C[1.type]`.

Micro-optimization for denotAt

59bb75b

Get rid of TypeState.hashesStr

a800a10

Hashes ruin diffability; replace them with the serial `id` numbers.

Strengthen isMultiSingleton

d17cab4

We missed some cases before.

odersky force-pushed the change-interpolation-3 branch from 1060d12 to 04d1588 Compare March 7, 2018 14:37

odersky changed the title ~~Drop ephemeral~~ Rework type inference Mar 7, 2018

Reduce # calls to simplify

a6e1d19

Call simplify and interpolateTypeVars in the same situations.

odersky force-pushed the change-interpolation-3 branch from 04d1588 to a6e1d19 Compare March 7, 2018 14:59

Get rid of bindingTree and ownerSym in TypeVars

95f8a84

odersky mentioned this pull request Mar 7, 2018

[WIP] Changes to Interpolation #4065

Closed

Fix typo in extends

c810d5a

smarter approved these changes Mar 7, 2018

View reviewed changes

Fix typos

855e0e9

odersky merged commit 14fb071 into scala:master Mar 7, 2018

allanrenucci deleted the change-interpolation-3 branch March 8, 2018 08:23

smarter mentioned this pull request Mar 8, 2018

Check that we don't leak uninstantiated type variables #4084

Merged

adriaanm mentioned this pull request Mar 8, 2018

implicit resolution of F[A] for introduced F[_] and A scala/bug#10753

Closed

Blaisorblade mentioned this pull request Apr 23, 2018

Unexpected literal widening without a Singleton upperbound (post-SIP23) scala/bug#10838

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework type inference #4080

Rework type inference #4080

odersky commented Mar 6, 2018 •

edited

Loading

smarter Mar 6, 2018

odersky commented Mar 6, 2018

dottybot commented Mar 6, 2018

dottybot commented Mar 7, 2018

odersky commented Mar 7, 2018

dottybot commented Mar 7, 2018

dottybot commented Mar 7, 2018

odersky commented Mar 7, 2018

dottybot commented Mar 7, 2018

odersky commented Mar 7, 2018

dottybot commented Mar 7, 2018

odersky commented Mar 7, 2018

dottybot commented Mar 7, 2018

smarter left a comment

smarter Mar 7, 2018

smarter Mar 7, 2018

odersky Mar 7, 2018

smarter Mar 7, 2018

dottybot commented Mar 7, 2018

Rework type inference #4080

Rework type inference #4080

Conversation

odersky commented Mar 6, 2018 • edited Loading

Drop Ephemeral 90a661b

New interpolation scheme 417f0c8

smarter Mar 6, 2018

Choose a reason for hiding this comment

odersky commented Mar 6, 2018

dottybot commented Mar 6, 2018

dottybot commented Mar 7, 2018

odersky commented Mar 7, 2018

dottybot commented Mar 7, 2018

dottybot commented Mar 7, 2018

odersky commented Mar 7, 2018

dottybot commented Mar 7, 2018

odersky commented Mar 7, 2018

dottybot commented Mar 7, 2018

odersky commented Mar 7, 2018

dottybot commented Mar 7, 2018

smarter left a comment

Choose a reason for hiding this comment

smarter Mar 7, 2018

Choose a reason for hiding this comment

smarter Mar 7, 2018

Choose a reason for hiding this comment

odersky Mar 7, 2018

Choose a reason for hiding this comment

smarter Mar 7, 2018

Choose a reason for hiding this comment

dottybot commented Mar 7, 2018

odersky commented Mar 6, 2018 •

edited

Loading

Drop Ephemeral `90a661b`

New interpolation scheme `417f0c8`