Stack overflow when tls_callback called on x86_64-pc-windows-gnu

[Berrysoft]

Recently the CI of libloading could not pass. I tested it and found this issue. It only occurs on *-gnu target, not *-msvc or *-gnullvm target. I have noticed that *-msvc and *-gnullvm has native thread local support, but *-gnu does not. Is it the cause?

Meta

rustc --version --verbose:

rustc 1.88.0-nightly (10fa3c449 2025-04-26)
binary: rustc
commit-hash: 10fa3c449f6b1613b352a6cbf78d3d91fd9a1d81
commit-date: 2025-04-26
host: x86_64-pc-windows-msvc
release: 1.88.0-nightly
LLVM version: 20.1.2

Backtrace

    frame #19175: 0x00007ffd8d392efe libtest_helpers.module`__rustc::rust_begin_unwind at panicking.rs:697:5
    frame #19176: 0x00007ffd8d3d6461 libtest_helpers.module`core::panicking::panic_fmt::hf3da1af663b61a8d at panicking.rs:75:14
    frame #19177: 0x00007ffd8d3a6fb0 libtest_helpers.module`std::sys::thread_local::key::windows::LazyKey::init::h27769e3038b4ecac at windows.rs:92:21
    frame #19178: 0x00007ffd8d392b0b libtest_helpers.module`std::panicking::panic_count::increase::hfd0da68402c75e3a at windows.rs:64:27
    frame #19179: 0x00007ffd8d393489 libtest_helpers.module`std::panicking::rust_panic_with_hook::hbc6c43d9ad374d65 at panicking.rs:801:22
    frame #19180: 0x00007ffd8d39324b libtest_helpers.module`std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h5b6ff7969aa6723e at panicking.rs:699:13
    frame #19181: 0x00007ffd8d38e24f libtest_helpers.module`std::sys::backtrace::__rust_end_short_backtrace::hb7b70c0f3a6d4258 at backtrace.rs:168:18
    frame #19182: 0x00007ffd8d392efe libtest_helpers.module`__rustc::rust_begin_unwind at panicking.rs:697:5
    frame #19183: 0x00007ffd8d3d6461 libtest_helpers.module`core::panicking::panic_fmt::hf3da1af663b61a8d at panicking.rs:75:14
    frame #19184: 0x00007ffd8d3a6fb0 libtest_helpers.module`std::sys::thread_local::key::windows::LazyKey::init::h27769e3038b4ecac at windows.rs:92:21
    frame #19185: 0x00007ffd8d392b0b libtest_helpers.module`std::panicking::panic_count::increase::hfd0da68402c75e3a at windows.rs:64:27
    frame #19186: 0x00007ffd8d393489 libtest_helpers.module`std::panicking::rust_panic_with_hook::hbc6c43d9ad374d65 at panicking.rs:801:22
    frame #19187: 0x00007ffd8d39324b libtest_helpers.module`std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h5b6ff7969aa6723e at panicking.rs:699:13
    frame #19188: 0x00007ffd8d38e24f libtest_helpers.module`std::sys::backtrace::__rust_end_short_backtrace::hb7b70c0f3a6d4258 at backtrace.rs:168:18
    frame #19189: 0x00007ffd8d392efe libtest_helpers.module`__rustc::rust_begin_unwind at panicking.rs:697:5
    frame #19190: 0x00007ffd8d3d6461 libtest_helpers.module`core::panicking::panic_fmt::hf3da1af663b61a8d at panicking.rs:75:14
    frame #19191: 0x00007ffd8d3a6fb0 libtest_helpers.module`std::sys::thread_local::key::windows::LazyKey::init::h27769e3038b4ecac at windows.rs:92:21
    frame #19192: 0x00007ffd8d392b0b libtest_helpers.module`std::panicking::panic_count::increase::hfd0da68402c75e3a at windows.rs:64:27
    frame #19193: 0x00007ffd8d393489 libtest_helpers.module`std::panicking::rust_panic_with_hook::hbc6c43d9ad374d65 at panicking.rs:801:22
    frame #19194: 0x00007ffd8d39324b libtest_helpers.module`std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h5b6ff7969aa6723e at panicking.rs:699:13
    frame #19195: 0x00007ffd8d38e24f libtest_helpers.module`std::sys::backtrace::__rust_end_short_backtrace::hb7b70c0f3a6d4258 at backtrace.rs:168:18
    frame #19196: 0x00007ffd8d392efe libtest_helpers.module`__rustc::rust_begin_unwind at panicking.rs:697:5
    frame #19197: 0x00007ffd8d3d6461 libtest_helpers.module`core::panicking::panic_fmt::hf3da1af663b61a8d at panicking.rs:75:14
    frame #19198: 0x00007ffd8d3a6fb0 libtest_helpers.module`std::sys::thread_local::key::windows::LazyKey::init::h27769e3038b4ecac at windows.rs:92:21
    frame #19199: 0x00007ffd8d392b0b libtest_helpers.module`std::panicking::panic_count::increase::hfd0da68402c75e3a at windows.rs:64:27
    frame #19200: 0x00007ffd8d393489 libtest_helpers.module`std::panicking::rust_panic_with_hook::hbc6c43d9ad374d65 at panicking.rs:801:22
    frame #19201: 0x00007ffd8d39324b libtest_helpers.module`std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h5b6ff7969aa6723e at panicking.rs:699:13
    frame #19202: 0x00007ffd8d38e24f libtest_helpers.module`std::sys::backtrace::__rust_end_short_backtrace::hb7b70c0f3a6d4258 at backtrace.rs:168:18
    frame #19203: 0x00007ffd8d392efe libtest_helpers.module`__rustc::rust_begin_unwind at panicking.rs:697:5
    frame #19204: 0x00007ffd8d3d6461 libtest_helpers.module`core::panicking::panic_fmt::hf3da1af663b61a8d at panicking.rs:75:14
    frame #19205: 0x00007ffd8d3a6fb0 libtest_helpers.module`std::sys::thread_local::key::windows::LazyKey::init::h27769e3038b4ecac at windows.rs:92:21
    frame #19206: 0x00007ffd8d392b0b libtest_helpers.module`std::panicking::panic_count::increase::hfd0da68402c75e3a at windows.rs:64:27
    frame #19207: 0x00007ffd8d393489 libtest_helpers.module`std::panicking::rust_panic_with_hook::hbc6c43d9ad374d65 at panicking.rs:801:22
    frame #19208: 0x00007ffd8d393282 libtest_helpers.module`std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h5b6ff7969aa6723e at panicking.rs:706:13
    frame #19209: 0x00007ffd8d38e24f libtest_helpers.module`std::sys::backtrace::__rust_end_short_backtrace::hb7b70c0f3a6d4258 at backtrace.rs:168:18
    frame #19210: 0x00007ffd8d392efe libtest_helpers.module`__rustc::rust_begin_unwind at panicking.rs:697:5
    frame #19211: 0x00007ffd8d3d6461 libtest_helpers.module`core::panicking::panic_fmt::hf3da1af663b61a8d at panicking.rs:75:14
    frame #19212: 0x00007ffd8d3d699a libtest_helpers.module`core::panicking::assert_failed_inner::h42d3026cd4391138 at panicking.rs:0
    frame #19213: 0x00007ffd8d35de4a libtest_helpers.module`core::panicking::assert_failed::h3c4a6a625213a78c at panicking.rs:380:5
    frame #19214: 0x00007ffd8d3a6ff4 libtest_helpers.module`std::sys::thread_local::key::windows::LazyKey::init::h27769e3038b4ecac at windows.rs:115:13
    frame #19215: 0x00007ffd8d3a6ddf libtest_helpers.module`std::sys::thread_local::guard::windows::tls_callback::h295e5d63f45f6d01 at windows.rs:64:27
    frame #19216: 0x00007ffde207296f ntdll.dll`RtlDeactivateActivationContext + 3519
    frame #19217: 0x00007ffde207265e ntdll.dll`RtlDeactivateActivationContext + 2734
    frame #19218: 0x00007ffde20ea209 ntdll.dll`ShipAssert + 2089
    frame #19219: 0x00007ffde200484e ntdll.dll`TpAllocTimer + 1022
    frame #19220: 0x00007ffde2004694 ntdll.dll`TpAllocTimer + 580
    frame #19221: 0x00007ffde2005b30 ntdll.dll`LdrUnloadDll + 496
    frame #19222: 0x00007ffddf802979 KernelBase.dll`FreeLibrary + 25
    frame #19223: 0x00007ff79b670d76 functions-bc6b1e7bfc14d1a7.exe`libloading::os::windows::Library::close::_$u7b$$u7b$closure$u7d$$u7d$::h6569003bd33c340f at mod.rs:248:25
    frame #19224: 0x00007ff79b670dfa functions-bc6b1e7bfc14d1a7.exe`libloading::os::windows::with_get_last_error::h05288a6ff5b11dcc(wrap=(functions-bc6b1e7bfc14d1a7.exe`core::ops::function::FnOnce::call_once::h3e274d6b7eeffe13 at function.rs:250), closure={closure_env#1} @ 0x000000e3643ff600) at mod.rs:384:5
    frame #19225: 0x00007ff79b6704cc functions-bc6b1e7bfc14d1a7.exe`libloading::os::windows::Library::close::hdbc65720515e50c7(self=(__0 = 140726972514304)) at mod.rs:247:22
    frame #19226: 0x00007ff79b670fa6 functions-bc6b1e7bfc14d1a7.exe`libloading::safe::Library::close::h60ce41f0bf5a34d0(self=Library @ 0x000000e3643ff6e0) at safe.rs:163:9
    frame #19227: 0x00007ff79b624894 functions-bc6b1e7bfc14d1a7.exe`functions::manual_close_many_times::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h61f718895dea9fc6((null)={closure_env#0} @ 0x000000e3643ff7b7) at functions.rs:205:21
    frame #19228: 0x00007ff79b614b39 functions-bc6b1e7bfc14d1a7.exe`std::sys::backtrace::__rust_begin_short_backtrace::h2b7c386dc932afd5(f={closure_env#0} @ 0x000000e3643ff7f6) at backtrace.rs:152:18
    frame #19229: 0x00007ff79b61f9e9 functions-bc6b1e7bfc14d1a7.exe`std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h34b33b572d25be08 at mod.rs:559:17
    frame #19230: 0x00007ff79b6214e1 functions-bc6b1e7bfc14d1a7.exe`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hda80a62a80afd918(self=<unavailable>, (null)=<unavailable>) at unwind_safe.rs:272:9
    frame #19231: 0x00007ff79b61fd90 functions-bc6b1e7bfc14d1a7.exe`std::panicking::try::do_call::h7f5123721e4e2c12(data="\U00000004") at panicking.rs:589:40
    frame #19232: 0x00007ff79b61fcf8 functions-bc6b1e7bfc14d1a7.exe`__rust_try + 24
    frame #19233: 0x00007ff79b61f621 functions-bc6b1e7bfc14d1a7.exe`std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::h06b588143189d0f0 [inlined] std::panicking::try::h5c8446479547f809(f=AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<functions::manual_close_many_times::{closure#0}::{closure_env#0}, ()>> @ 0x000000e3643ffa38) at panicking.rs:552:19
    frame #19234: 0x00007ff79b61f576 functions-bc6b1e7bfc14d1a7.exe`std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::h06b588143189d0f0 [inlined] std::panic::catch_unwind::h0f3613ed9cdfbdf9(f=AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked_::{closure#1}::{closure_env#0}<functions::manual_close_many_times::{closure#0}::{closure_env#0}, ()>> @ 0x000000e3643ffa38) at panic.rs:359:14
    frame #19235: 0x00007ff79b61f576 functions-bc6b1e7bfc14d1a7.exe`std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::h06b588143189d0f0 at mod.rs:557:30
    frame #19236: 0x00007ff79b61649e functions-bc6b1e7bfc14d1a7.exe`core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h063d282d3b2d4b0a((null)=0x000001f2722d3cf0, (null)=<unavailable>) at function.rs:250:5
    frame #19237: 0x00007ff79b6ba82d functions-bc6b1e7bfc14d1a7.exe`std::sys::pal::windows::thread::Thread::new::thread_start::h008373d82c2d2513 at boxed.rs:1966:9
    frame #19238: 0x00007ffde121e8d7 kernel32.dll`BaseThreadInitThunk + 23
    frame #19239: 0x00007ffde203c34c ntdll.dll`RtlUserThreadStart + 44

[Berrysoft]

The code for the dll is this one:

https://github.com/nagisa/rust_libloading/blob/master/src/test_helpers.rs

Compile it with

rustc test_helpers.rs -o libtest_helpers.module --target x86_64-pc-windows-gnu

Then write some code to load and unload it:

fn main() {
    let join_handles: Vec<_> = (0..16)
        .map(|_| {
            std::thread::spawn(|| unsafe {
                for _ in 0..10000 {
                    // Use libloading, or write LoadLibrary & FreeLibrary yourself.
                    let lib = Library::new("libtest_helpers.module").expect("open library");
                    let _: Symbol<unsafe extern "C" fn(u32) -> u32> =
                        lib.get(b"test_identity_u32").expect("get fn");
                    lib.close().expect("close is successful");
                }
            })
        })
        .collect();
    for handle in join_handles {
        handle.join().expect("thread should succeed");
    }
}

This issue doesn't always trigger. I have found that, debug it with lldb could improve the possibility.

[Berrysoft]

searched toolchains nightly-2022-06-24 through nightly-2025-05-08


********************************************************************************
Regression in nightly-2024-10-04
********************************************************************************

fetching https://static.rust-lang.org/dist/2024-10-03/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2024-10-03: 40 B / 40 B [==================================================================================================================================================] 100.00 % 76.96 KB/s converted 2024-10-03 to 18b1161ec9eeab8927f91405bca0ddf59a4a26c9
fetching https://static.rust-lang.org/dist/2024-10-04/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2024-10-04: 40 B / 40 B [==================================================================================================================================================] 100.00 % 40.48 KB/s converted 2024-10-04 to 9ff5fc4ffbbe1e911527aa054e789b05ae55ffcc
looking for regression commit between 2024-10-03 and 2024-10-04
fetching (via remote github) commits from max(18b1161ec9eeab8927f91405bca0ddf59a4a26c9, 2024-10-01) to 9ff5fc4ffbbe1e911527aa054e789b05ae55ffcc
ending github query because we found starting sha: 18b1161ec9eeab8927f91405bca0ddf59a4a26c9
get_commits_between returning commits, len: 10
  commit[0] 2024-10-02: Auto merge of #130821 - lcnr:nalgebra-hang-2, r=compiler-errors
  commit[1] 2024-10-02: Auto merge of #131006 - RalfJung:immediate-sanity, r=saethlin
  commit[2] 2024-10-03: Auto merge of #131148 - Urgau:hashbrown-0.15, r=Amanieu
  commit[3] 2024-10-03: Auto merge of #127912 - joboet:tls_dtor_thread_current, r=cuviper
  commit[4] 2024-10-03: Auto merge of #131185 - workingjubilee:rollup-wkcqe2j, r=workingjubilee
  commit[5] 2024-10-03: Auto merge of #128711 - clarfonthey:default-iters-hash, r=dtolnay
  commit[6] 2024-10-03: Auto merge of #13492 - flip1995:rustup, r=flip1995
  commit[7] 2024-10-03: Auto merge of #131196 - matthiaskrgr:rollup-3it3zqp, r=matthiaskrgr
  commit[8] 2024-10-03: Auto merge of #131205 - flip1995:clippy-subtree-update, r=Manishearth
  commit[9] 2024-10-03: Auto merge of #131145 - ismailarilik:handle_potential_query_instability_lint_for_rustc_metadata, r=compiler-errors
ERROR: no CI builds available between 18b1161ec9eeab8927f91405bca0ddf59a4a26c9 and 9ff5fc4ffbbe1e911527aa054e789b05ae55ffcc within last 167 days

Maybe #127912 , I guess.

[moxian]

Looking at the stack trace, we panic here

rust/library/std/src/sys/thread_local/key/windows.rs

Lines 114 to 115 in e964cca

	let key = unsafe { c::TlsAlloc() };
	assert_ne!(key, c::TLS_OUT_OF_INDEXES, "out of TLS indexes");

and then when trying to check whether we're panicking while inside another panic and we need to abort, we hit the same code path

rust/library/std/src/panicking.rs

Line 433 in e964cca

LOCAL_PANIC_COUNT.with(|c| {

It seems that the issue can occur in more than just windows-gnu, but windows-gnu simply more likely to exhaust tls indices than others (somehow.. I'm not entirely clear on this part)

cc @joboet

@rustbot label: -E-needs-mcve -E-needs-bisection +S-has-bisection -needs-triage +A-thread-locals +A-panic

[Berrysoft]

You're right. I asked the question about target_thread_local, because I didn't find a proper calling chain from std::sys::thread_local::guard::windows::tls_callback to std::sys::thread_local::key::windows::LazyKey::init. Could you explain about that?

[mati865]

Just FYI, there are two similar issues, albeit the trace is a bit different: #139519 and #139070