runtime: unexpected fault address 0

[karalabe]

Go version

go1.22.3 linux-amd64

Output of go env in your module/workspace:

Hard to say exactly, I use `golang:1.22-alpine` (latest) docker container to build a Go app and add it to an `alpine:latest` docker container to run.

What did you do?

I can reproduce this consistently (i.e. happens 100% of the time), but it is extremely deep in a live system, so I have no idea how to make a proper repro out of it.

I have the following Go code crashing:

		values := someFunc(...)

		values = append(make([][]byte, first), values...)
		values = append(values, make([][]byte, len(keys)-last)...)

The crash is in the runtime memory management. With Go 1.22.2 I got the following dump:

	unexpected fault address 0x0
	fatal error: fault
	[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x41b0f7]
	goroutine 226018 gp=0xc000e1f500 m=32 mp=0xc0079b1008 [running]:
	runtime.throw({0x1a826de?, 0xc047f3fcb8?})
		runtime/panic.go:1023 +0x5c fp=0xc047f3fc58 sp=0xc047f3fc28 pc=0x43fc3c
	runtime.sigpanic()
	    runtime/signal_unix.go:895 +0x285 fp=0xc047f3fcb8 sp=0xc047f3fc58 pc=0x4588e5
	runtime.(*mspan).typePointersOfUnchecked(0x0?, 0x0?)
	    runtime/mbitmap_allocheaders.go:202 +0x37 fp=0xc047f3fcd8 sp=0xc047f3fcb8 pc=0x41b0f7
	runtime.(*mspan).typePointersOf(0x7f20aca3af30, 0xc047ceca00, 0x0)
	    runtime/mbitmap_allocheaders.go:162 +0x45 fp=0xc047f3fd20 sp=0xc047f3fcd8 pc=0x41b065
	runtime.bulkBarrierPreWrite(0xc047ceca00, 0x0, 0x0, 0x35?)
	    runtime/mbitmap_allocheaders.go:465 +0x12d fp=0xc047f3fd90 sp=0xc047f3fd20 pc=0x41b52d
	runtime.memclrHasPointers(0xc047ceca00, 0x0)
	    runtime/mbarrier.go:371 +0x25 fp=0xc047f3fdc0 sp=0xc047f3fd90 pc=0x41a1e5
        [...]
        at this point the line is the last line from my snippet above

With Go 1.22.3 I still see the exact crash but it didn't spit out this much crash dump, only a shorter one (which 1.22.2 also spits out sometimes instead of the long one)

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x41b0f7]
goroutine 115123 [running]:
github.com/ethereum/go-ethereum/trie.(*Trie).getBatch(0xc03ad422d0, {0x2009770?, 0xc0489ab450}, {0xc0488daa88, 0x25, 0x25}, 0x7)
 	github.com/ethereum/go-ethereum/trie/trie.go:286 +0xf65

The line referenced is the same exact last line the previous crash references; a simple slice append.

What did you see happen?

Boom

What did you expect to see?

No boom

[karalabe]

For what it's worth, the boom seems to happen with first == 0; len(keys) == len(values) == N and last == n, so something akin to:

	values := f(keys)

	values = append(make([][]byte, 0), values...)
	values = append(values, make([][]byte, 0)...)

Obviously I can't make Go crash on such a simple repro :)) Something needs to bork within the garbage collector first I assume.

[karalabe]

Managed to blow it up verbose with Go 1.22.3 too:

unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x41b0fb]
goroutine 876768 gp=0xc0009e8c40 m=23 mp=0xc000981808 [running]:
runtime.throw({0x1a826de?, 0xc06e640148?})
	runtime/panic.go:1023 +0x5c fp=0xc06e6400e8 sp=0xc06e6400b8 pc=0x43fc3c
runtime.sigpanic()
	runtime/signal_unix.go:895 +0x285 fp=0xc06e640148 sp=0xc06e6400e8 pc=0x4588e5
runtime.readUintptr(...)
	runtime/mbitmap.go:466
runtime.(*mspan).typePointersOfUnchecked(0x0?, 0x0?)
	runtime/mbitmap_allocheaders.go:203 +0x3b fp=0xc06e640168 sp=0xc06e640148 pc=0x41b0fb
runtime.(*mspan).typePointersOf(0x7f0d81965270, 0xc07403a2c0, 0x0)
	runtime/mbitmap_allocheaders.go:162 +0x45 fp=0xc06e6401b0 sp=0xc06e640168 pc=0x41b065
runtime.bulkBarrierPreWrite(0xc07403a2c0, 0x0, 0x0, 0x1d?)
	runtime/mbitmap_allocheaders.go:465 +0x12d fp=0xc06e640220 sp=0xc06e6401b0 pc=0x41b52d
runtime.memclrHasPointers(0xc07403a2c0, 0x0)
	runtime/mbarrier.go:371 +0x25 fp=0xc06e640250 sp=0xc06e640220 pc=0x41a1e5
github.com/ethereum/go-ethereum/trie.(*Trie).getBatch(0xc06fb279e0, {0x2009730?, 0xc073ba11d0}, {0xc07402e588, 0x1d, 0x1d}, 0x7)
	github.com/ethereum/go-ethereum/trie/trie.go:285 +0xe93 fp=0xc06e640410 sp=0xc06e640250 pc=0xd0c693

Mostly the same crash, just one line difference in mbitmap_allocheaders

[mauri870]

Without a repro these issues are hard to investigate. Are you sure that running with -race does not point any issues? Thanks.

[randall77]

I think I see the bug here. Simplified:

func g(x []*byte, z int) []*byte {
	return append(x, make([]*byte, z)...)
}

When we call this with z==0 and x is a slice that points to an object that completely fills its size class, then we call memclrHasPointers with x.ptr + 8*x.len, 0, which may point to the next object in the span. That is a no-no, as the next object may be junk.

We need some sort of defense against past-the-end pointers in this generated code.

Possibly introduced in CL 418554

[karalabe]

Ha, reproed it.

https://go.dev/play/p/zfi-jTPIIp_K

[karalabe]

Further simplified: https://go.dev/play/p/pLh04Z_lzSd

[karalabe]

Interestingly, with 100 items in the slice, it doesn't crash, with 53, it does. I'm assuming the former expands the array "nicely" whereas the latter hits some cornercase.

[karalabe]

PS: The reason I pasted the images and not just stack traces is because I'm unsure how consistent the crashes are, so figured I'd better make some "proof".

[karalabe]

Got a scarier one now: https://go.dev/play/p/AbxNigAFhCk

[randall77]

@gopherbot please open a backport issue for 1.22.

[gopherbot]

Backport issue(s) opened: #67258 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[randall77]

Seems to be ok in 1.21 and earlier.

[gopherbot]

Change https://go.dev/cl/584116 mentions this issue: cmd/compile: avoid past-the-end pointer when zeroing