Migrate away from u32 as an offset/length type

[saethlin]

Proposal

Rustc stores a lot of lengths in u32, which reduces the size of data structures and on-disk formats that contain these lengths. But this strategy imposes an artificial limit on the size of various things, which users can and do hit. This MCP is a general proposal to stop doing this, as opposed to a single strategy to adopt instead of using u32. The right alternative strategy will be determined on a case-by-case basis by experimentation.

Here are the examples I know of where users have reported running into the u32 limitation:

• rmeta offsets: 'rustc' panicked at 'called Result::unwrap() on an Err value: TryFromIntError(()) rust#112934
• incr comp disk cache offsets: Disk cache ICE with big array rust#76037 ICE trying to compile very large file contents to wasm32-wasi and maybe x86_64-unknown-linux-gnu rust#95780 Compiler error encountered when attempting to get the size of a string created from [21u8; u32::MAX as usize] rust#111613

I expect that it is also possible to exceed the u32-based limit on a rustc_index::newtype_index! type, but we do not yet have any reports of those. I wouldn't be surprised if ICEs due to newtype_index! exhaustion are hiding behind the two cases we do have reports for.

Mentors or Reviewers

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

cc @rust-lang/compiler @rust-lang/compiler-contributors

[saethlin]

I'm splitting this into two different MCPs. The first half is #666