Skip to content

[SwiftDriver] PathRemapping and Cache Replay Support #1453

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Nov 14, 2023
Merged

[SwiftDriver] PathRemapping and Cache Replay Support #1453

merged 4 commits into from
Nov 14, 2023

Conversation

cachemeifyoucan
Copy link
Contributor

Teach swift-driver to use newly added functions in swift-frontend and libSwiftScan.

@cachemeifyoucan
Copy link
Contributor Author

Depends on: swiftlang/swift#68660 and swiftlang/swift#68684

artemcm
artemcm reviewed Sep 26, 2023
View reviewed changes
Sources/SwiftDriver/Jobs/CompileJob.swift Outdated
}
}

// Add cached diagnostics entry for primary files, or the first input for multithreaded compilation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing the order of inputs here will invalidate the corresponding cache entries for this whole compilation, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the order of the file matters for cache key. Not sure if that really matters for the compilation but the order is unlikely to change in most cases.

Currently, file passed through filelist vs. argument will get different key, and there is a FIXME in swift somewhere to address that sometime.

@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch 2 times, most recently from d9e75f7 to 03054e2 Compare September 26, 2023 19:54
@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch from 03054e2 to 33bd1b1 Compare October 27, 2023 18:45
@cachemeifyoucan
Copy link
Contributor Author

Updated to match new APIs.

@cachemeifyoucan cachemeifyoucan mentioned this pull request Oct 27, 2023
Merged
@cachemeifyoucan
Copy link
Contributor Author

@swift-ci please test

@cachemeifyoucan
Copy link
Contributor Author

ping

@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch from 33bd1b1 to c3cc47d Compare October 30, 2023 22:36
@cachemeifyoucan
Copy link
Contributor Author

@swift-ci please test

@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch from c3cc47d to be14c2d Compare October 30, 2023 22:56
@cachemeifyoucan
Copy link
Contributor Author

@swift-ci please test

@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch from be14c2d to 56516af Compare November 2, 2023 16:42
@cachemeifyoucan
Copy link
Contributor Author

@swift-ci please test

@cachemeifyoucan
Copy link
Contributor Author

@swift-ci please smoke test macOS

@cachemeifyoucan
Copy link
Contributor Author

@swift-ci please test macOS platform

@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch from 56516af to eefc658 Compare November 2, 2023 20:25
@cachemeifyoucan
Copy link
Contributor Author

@swift-ci please test

@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch from eefc658 to 36503c8 Compare November 3, 2023 17:20
@cachemeifyoucan
Copy link
Contributor Author

@swift-ci please test

@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch from 36503c8 to 474fc4e Compare November 3, 2023 18:28
@cachemeifyoucan
Copy link
Contributor Author

@swift-ci please test

@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch from 474fc4e to 808cc4d Compare November 3, 2023 22:15
akyrtzi
akyrtzi approved these changes Nov 3, 2023
View reviewed changes
Copy link
Contributor

@akyrtzi akyrtzi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nitpicks, otherwise LGTM!

Sources/SwiftDriver/SwiftScan/SwiftScanCAS.swift Outdated
return callback
}

func getContext() -> UnsafeMutableRawPointer {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be more clearly named as takeContext() since it returns a +1 object.

Copy link
Contributor Author

@cachemeifyoucan cachemeifyoucan Nov 3, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is take indicating +1 or -1? Should this be copyContext() or retainContext()?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I change to retain().

@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch 2 times, most recently from 347ff17 to bc87263 Compare November 4, 2023 00:15
@cachemeifyoucan
Copy link
Contributor Author

@swift-ci please test

@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch from bc87263 to 95345ae Compare November 6, 2023 17:41
@cachemeifyoucan
[Caching] Path remapping support
f2248d5 
Teach swift-driver to remap path when needed. This involves:
* Infer SDK/toolchain related path that needs to remapped
* Send path remap related arguments to swift-frontend for depscanning
* Send reverse map to swift-frontend for diagnostics replay
* Constructing the compilation command for main module using remapped
  path
@cachemeifyoucan
[Caching] Direct cache replay support
8b3926b 
Teach swift-driver to replay cached compilation result using
libSwiftScan APIs directly. Jobs constructed from swift-driver is going
to have a new field for all the cache keys associated with the job. The
Executor can choose to query and replay using those cache keys if
possible, instead of executing the build command.

rdar://117947710
@cachemeifyoucan
[Caching] Update swift-driver to reflect the frontend caching configu…
2e11874 
…ration

Re-unify caching availability check behind swift feature query and
adjust the default caching build configuration changes.
@cachemeifyoucan
Fix README for building makeOptions
2de866d 
Remove extra `-Xcc` in the command-line.`
@cachemeifyoucan cachemeifyoucan force-pushed the eng/PR-swift-driver-path-remap-and-replay branch from 95345ae to 2de866d Compare November 13, 2023 17:59
@cachemeifyoucan
Copy link
Contributor Author

@artemcm Some changes after I change the configuration in swift-frontend and swift most of the checks for if swift-frontend/libSwiftScan supports caching behind swift feature json flag.

@cachemeifyoucan
Copy link
Contributor Author

@swift-ci please test

@cachemeifyoucan cachemeifyoucan merged commit be09eb4 into swiftlang:main Nov 14, 2023
@artemcm artemcm mentioned this pull request Nov 15, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@akyrtzi akyrtzi akyrtzi approved these changes

@benlangmuir benlangmuir Awaiting requested review from benlangmuir

@artemcm artemcm Awaiting requested review from artemcm

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@cachemeifyoucan @akyrtzi @artemcm