roachtest: network/tpcc/nodes=4 failed #37259

cockroach-teamcity · 2019-05-02T02:00:49Z

SHA: https://github.com/cockroachdb/cockroach/commits/24feca7a4106f08c73534e16ebb79d949a479f35

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1268176&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1552,cluster.go:1655,cluster.go:1626,network.go:151,network.go:194,cluster.go:1814,errgroup.go:57: context canceled
	cluster.go:1835,network.go:231,network.go:248,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1268176-network-tpcc-nodes-4:4 -- ./workload run tpcc --warehouses=1 --wait=false --histograms=logs/stats.json --duration=1h0m0s {pgurl:2-3} returned:
		stderr:
		Error: error in stockLevel: ERROR: rpc error: code = Unavailable desc = transport is closing (SQLSTATE XXUUU)
		Error:  exit status 1
		
		stdout:
		l
		     16s        0            0.0            1.4      0.0      0.0      0.0      0.0 delivery
		     16s        0            0.0           14.2      0.0      0.0      0.0      0.0 newOrder
		     16s        0            0.0            1.4      0.0      0.0      0.0      0.0 orderStatus
		     16s        0            0.0           14.1      0.0      0.0      0.0      0.0 payment
		     16s        0            0.0            1.4      0.0      0.0      0.0      0.0 stockLevel
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     17s        0            0.0            1.3      0.0      0.0      0.0      0.0 delivery
		     17s        0            0.0           13.4      0.0      0.0      0.0      0.0 newOrder
		     17s        0            0.0            1.3      0.0      0.0      0.0      0.0 orderStatus
		     17s        0            0.0           13.3      0.0      0.0      0.0      0.0 payment
		     17s        0            0.0            1.3      0.0      0.0      0.0      0.0 stockLevel
		: exit status 1
	test.go:1237: test timed out (11h49m4.723910957s)

The text was updated successfully, but these errors were encountered:

cockroach-teamcity · 2019-05-08T04:42:40Z

SHA: https://github.com/cockroachdb/cockroach/commits/d554884a4e474cc06213230d5ba7d757a88e9e46

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1279548&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
	cluster.go:1550,cluster.go:1653,cluster.go:1624,network.go:157,network.go:200,cluster.go:1812,errgroup.go:57: context canceled
	cluster.go:1833,network.go:237,network.go:254,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1279548-network-tpcc-nodes-4:4 -- ./workload run tpcc --warehouses=1 --wait=false --histograms=logs/stats.json --duration=1h0m0s {pgurl:2-3} returned:
		stderr:
		Error: error in delivery: ERROR: rpc error: code = Unavailable desc = transport is closing (SQLSTATE XX000)
		Error:  exit status 1
		
		stdout:
		       0            0.0            1.4      0.0      0.0      0.0      0.0 stockLevel
		     15s        0            0.0            1.3      0.0      0.0      0.0      0.0 delivery
		     15s        0            0.0           13.3      0.0      0.0      0.0      0.0 newOrder
		     15s        0            0.0            1.3      0.0      0.0      0.0      0.0 orderStatus
		     15s        0            0.0           13.3      0.0      0.0      0.0      0.0 payment
		     15s        0            0.0            1.3      0.0      0.0      0.0      0.0 stockLevel
		     16s        0            0.0            1.2      0.0      0.0      0.0      0.0 delivery
		     16s        0            0.0           12.4      0.0      0.0      0.0      0.0 newOrder
		     16s        0            0.0            1.2      0.0      0.0      0.0      0.0 orderStatus
		     16s        0            0.0           12.4      0.0      0.0      0.0      0.0 payment
		     16s        0            0.0            1.2      0.0      0.0      0.0      0.0 stockLevel
		: exit status 1

cockroach-teamcity · 2019-05-09T01:48:34Z

SHA: https://github.com/cockroachdb/cockroach/commits/8abb47a1c9795c1463183bc44e776b054bece682

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1279683&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1550,cluster.go:1653,cluster.go:1624,network.go:157,network.go:200,cluster.go:1812,errgroup.go:57: context canceled
	cluster.go:1833,network.go:237,network.go:254,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1279683-network-tpcc-nodes-4:4 -- ./workload run tpcc --warehouses=1 --wait=false --histograms=logs/stats.json --duration=1h0m0s {pgurl:2-3} returned:
		stderr:
		Error: error in delivery: ERROR: rpc error: code = Unavailable desc = transport is closing (SQLSTATE XXUUU)
		Error:  exit status 1
		
		stdout:
		l
		     16s        0            0.0            1.5      0.0      0.0      0.0      0.0 delivery
		     16s        0            0.0           15.6      0.0      0.0      0.0      0.0 newOrder
		     16s        0            0.0            1.5      0.0      0.0      0.0      0.0 orderStatus
		     16s        0            0.0           15.3      0.0      0.0      0.0      0.0 payment
		     16s        0            0.0            1.6      0.0      0.0      0.0      0.0 stockLevel
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     17s        0            0.0            1.4      0.0      0.0      0.0      0.0 delivery
		     17s        0            0.0           14.6      0.0      0.0      0.0      0.0 newOrder
		     17s        0            0.0            1.4      0.0      0.0      0.0      0.0 orderStatus
		     17s        0            0.0           14.4      0.0      0.0      0.0      0.0 payment
		     17s        0            0.0            1.5      0.0      0.0      0.0      0.0 stockLevel
		: exit status 1
	test.go:1237: test timed out (11h49m4.871769742s)

andreimatei · 2019-05-09T20:29:00Z

Two of the 3 previous failures are test timeouts (11h50m)...
@ajwerner any clue what that's about?

cockroach-teamcity · 2019-05-17T12:54:33Z

SHA: https://github.com/cockroachdb/cockroach/commits/7b2651400b2003d0a381cba9dbfc0b7bc0dfee00

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1293898&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,network.go:116,network.go:254,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1293898-network-tpcc-nodes-4:4 -- ./workload fixtures load tpcc --warehouses=1 {pgurl:1} returned:
		stderr:
		
		stdout:
		-status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4572 bytes, in 8.6 seconds
		Bytes per second: sent 368.5, received 532.5
		debug1: Exit status 1
		: exit status 1
		: exit status 1

cockroach-teamcity · 2019-05-18T12:43:39Z

SHA: https://github.com/cockroachdb/cockroach/commits/923a3b2a6f4a6492883141092280d1041de1381a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295056&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,network.go:116,network.go:254,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1295056-network-tpcc-nodes-4:4 -- ./workload fixtures load tpcc --warehouses=1 {pgurl:1} returned:
		stderr:
		
		stdout:
		-status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4528 bytes, in 8.2 seconds
		Bytes per second: sent 383.6, received 549.0
		debug1: Exit status 1
		: exit status 1
		: exit status 1

cockroach-teamcity · 2019-05-19T12:52:59Z

SHA: https://github.com/cockroachdb/cockroach/commits/cab299a0ef983f8b4ffe5d724e44587d9665d3a3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1295811&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,network.go:116,network.go:254,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1295811-network-tpcc-nodes-4:4 -- ./workload fixtures load tpcc --warehouses=1 {pgurl:1} returned:
		stderr:
		
		stdout:
		-status reply 0
		debug3: receive packet: type 96
		debug2: channel 0: rcvd eof
		debug2: channel 0: output open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4572 bytes, in 7.9 seconds
		Bytes per second: sent 398.8, received 576.2
		debug1: Exit status 1
		: exit status 1
		: exit status 1

cockroach-teamcity · 2019-05-20T12:12:19Z

SHA: https://github.com/cockroachdb/cockroach/commits/58c567a325056033b326cb9c4ed9ba490e8956da

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1296592&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,network.go:116,network.go:254,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1296592-network-tpcc-nodes-4:4 -- ./workload fixtures load tpcc --warehouses=1 {pgurl:1} returned:
		stderr:
		
		stdout:
		 open -> drain
		debug2: channel 0: obuf empty
		debug2: channel 0: close_write
		debug2: channel 0: output drain -> closed
		debug3: receive packet: type 98
		debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
		debug3: receive packet: type 97
		debug2: channel 0: rcvd close
		debug3: channel 0: will not send data after close
		debug2: channel 0: almost dead
		debug2: channel 0: gc: notify user
		debug2: channel 0: gc: user detached
		debug2: channel 0: send close
		debug3: send packet: type 97
		debug2: channel 0: is dead
		debug2: channel 0: garbage collecting
		debug1: channel 0: free: client-session, nchannels 1
		debug3: channel 0: status: The following connections are open:
		  #0 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
		
		debug3: send packet: type 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug3: fd 2 is not O_NONBLOCK
		Transferred: sent 3164, received 4592 bytes, in 7.5 seconds
		Bytes per second: sent 419.9, received 609.4
		debug1: Exit status 1
		: exit status 1
		: exit status 1

nvanbenschoten · 2019-05-21T20:14:00Z

Previous two issues addressed by #37701.

cockroach-teamcity · 2019-05-31T01:48:37Z

SHA: https://github.com/cockroachdb/cockroach/commits/8f42e0d9948256af8b3e1994d514314ba1718c48

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1315162&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1592,cluster.go:1695,cluster.go:1666,network.go:157,network.go:200,cluster.go:1854,errgroup.go:57: context canceled
	cluster.go:1875,network.go:237,network.go:254,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1315162-network-tpcc-nodes-4:4 -- ./workload run tpcc --warehouses=1 --wait=false --histograms=logs/stats.json --duration=1h0m0s {pgurl:2-3} returned:
		stderr:
		Error: error in newOrder: ERROR: result is ambiguous (error=unable to dial n1: breaker open [exhausted]) (SQLSTATE 40003)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.237.56.114_2019-05-30T13:43:42Z: exit status 1
		
		stdout:
		l
		     20s        0            0.0            1.3      0.0      0.0      0.0      0.0 delivery
		     20s        0            0.0           11.9      0.0      0.0      0.0      0.0 newOrder
		     20s        0            0.0            1.2      0.0      0.0      0.0      0.0 orderStatus
		     20s        0            0.0           12.2      0.0      0.0      0.0      0.0 payment
		     20s        0            0.0            1.2      0.0      0.0      0.0      0.0 stockLevel
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     21s        0            0.0            1.2      0.0      0.0      0.0      0.0 delivery
		     21s        0            0.0           11.4      0.0      0.0      0.0      0.0 newOrder
		     21s        0            0.0            1.1      0.0      0.0      0.0      0.0 orderStatus
		     21s        0            0.0           11.7      0.0      0.0      0.0      0.0 payment
		     21s        0            0.0            1.2      0.0      0.0      0.0      0.0 stockLevel
		: exit status 1
	test.go:1237: test timed out (11h47m24.408494012s)

cockroach-teamcity · 2019-05-31T14:42:33Z

SHA: https://github.com/cockroachdb/cockroach/commits/a059c2ed6edf59894885af9371c584b155d5e8f1

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1317018&tab=buildLog

The test failed on branch=master, cloud=gce:
	network.go:111,network.go:254,test.go:1251: toxify: PutString: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1317018-network-tpcc-nodes-4:1-3 /home/agent/temp/buildTmp/cockroach929017713 ./cockroach returned:
		stderr:
		
		stdout:
		ceived
		debug1: Authentications that can continue: publickey
		debug1: Next authentication method: publickey
		debug1: Offering RSA public key: /root/.ssh/id_rsa
		debug1: Authentications that can continue: publickey
		debug1: Offering RSA public key: /root/.ssh/google_compute_engine
		debug1: Authentication succeeded (publickey).
		Authenticated to 34.73.224.171 ([34.73.224.171]:22).
		debug1: channel 0: new [client-session]
		debug1: Requesting [email protected]
		debug1: Entering interactive session.
		debug1: pledge: network
		debug1: channel 0: free: client-session, nchannels 1
		debug1: fd 0 clearing O_NONBLOCK
		debug1: fd 1 clearing O_NONBLOCK
		debug1: fd 2 clearing O_NONBLOCK
		Connection to 34.73.224.171 closed by remote host.
		Transferred: sent 2308, received 1408 bytes, in 0.0 seconds
		Bytes per second: sent 23048699.1, received 14060904.8
		debug1: Exit status -1
		lost connection
		: exit status 1
		I190531 14:41:03.674782 1 cluster_synced.go:1009  put /home/agent/temp/buildTmp/cockroach929017713 failed
		: exit status 1

cockroach-teamcity · 2019-06-01T11:29:53Z

SHA: https://github.com/cockroachdb/cockroach/commits/24b7b441f56ce8d559380bd6a42bb8104c5cd68a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1318721&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
	cluster.go:1592,cluster.go:1695,cluster.go:1666,network.go:157,network.go:200,cluster.go:1854,errgroup.go:57: context canceled
	cluster.go:1875,network.go:237,network.go:254,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1318721-network-tpcc-nodes-4:4 -- ./workload run tpcc --warehouses=1 --wait=false --histograms=logs/stats.json --duration=1h0m0s {pgurl:2-3} returned:
		stderr:
		Error: error in delivery: ERROR: rpc error: code = Unavailable desc = transport is closing (SQLSTATE XX000)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_34.74.150.216_2019-06-01T11:26:15Z: exit status 1
		
		stdout:
		       0            0.0            1.8      0.0      0.0      0.0      0.0 stockLevel
		     14s        0            0.0            1.8      0.0      0.0      0.0      0.0 delivery
		     14s        0            0.0           16.9      0.0      0.0      0.0      0.0 newOrder
		     14s        0            0.0            1.6      0.0      0.0      0.0      0.0 orderStatus
		     14s        0            0.0           16.6      0.0      0.0      0.0      0.0 payment
		     14s        0            0.0            1.6      0.0      0.0      0.0      0.0 stockLevel
		     15s        0            0.0            1.7      0.0      0.0      0.0      0.0 delivery
		     15s        0            0.0           15.8      0.0      0.0      0.0      0.0 newOrder
		     15s        0            0.0            1.5      0.0      0.0      0.0      0.0 orderStatus
		     15s        0            0.0           15.5      0.0      0.0      0.0      0.0 payment
		     15s        0            0.0            1.5      0.0      0.0      0.0      0.0 stockLevel
		: exit status 1

tbg · 2019-06-04T13:24:56Z

^- last failure is a fluke here

cockroach/pkg/cmd/roachtest/cluster.go

Line 1592 in 40781f1

c.t.Fatal(err)

tbg · 2019-06-04T13:26:52Z

This happens only a few minutes into the run, so... not sure.

cockroach-teamcity · 2019-06-06T11:32:57Z

SHA: https://github.com/cockroachdb/cockroach/commits/962d4a8c3de09a8d5a389efd437889bf078f9b1a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1325811&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
	cluster.go:1589,cluster.go:1692,cluster.go:1663,network.go:154,network.go:197,cluster.go:1851,errgroup.go:57: context canceled
	cluster.go:1872,network.go:234,network.go:251,test.go:1248: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1325811-network-tpcc-nodes-4:4 -- ./workload run tpcc --warehouses=1 --wait=false --histograms=logs/stats.json --duration=1h0m0s {pgurl:2-3} returned:
		stderr:
		Error: error in delivery: ERROR: rpc error: code = Unavailable desc = transport is closing (SQLSTATE XX000)
		Error:  ssh verbose log retained in /root/.roachprod/debug/ssh_35.196.93.213_2019-06-06T11:29:30Z: exit status 1
		
		stdout:
		       0            0.0            1.8      0.0      0.0      0.0      0.0 stockLevel
		     14s        0            0.0            1.6      0.0      0.0      0.0      0.0 delivery
		     14s        0            0.0           16.7      0.0      0.0      0.0      0.0 newOrder
		     14s        0            0.0            1.6      0.0      0.0      0.0      0.0 orderStatus
		     14s        0            0.0           16.7      0.0      0.0      0.0      0.0 payment
		     14s        0            0.0            1.7      0.0      0.0      0.0      0.0 stockLevel
		     15s        0            0.0            1.5      0.0      0.0      0.0      0.0 delivery
		     15s        0            0.0           15.6      0.0      0.0      0.0      0.0 newOrder
		     15s        0            0.0            1.5      0.0      0.0      0.0      0.0 orderStatus
		     15s        0            0.0           15.6      0.0      0.0      0.0      0.0 payment
		     15s        0            0.0            1.6      0.0      0.0      0.0      0.0 stockLevel
		: exit status 1

cockroach-teamcity · 2019-09-07T05:06:36Z

SHA: https://github.com/cockroachdb/cockroach/commits/4784fe3c51545db5fb5d411937ec1db2ef2b9761

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1472753&tab=buildLog

The test failed on branch=provisional_201909060000_v19.2.0-beta.20190910, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190906-1472753/network/tpcc/nodes=4/run_1
	cluster.go:1797,cluster.go:1900,cluster.go:1871,network.go:155,network.go:198,cluster.go:2091,errgroup.go:57: context canceled

cockroach-teamcity · 2019-09-18T21:30:09Z

SHA: https://github.com/cockroachdb/cockroach/commits/93c030e0677283cdea2c9b97a5c91dfe78dc63c1

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1495889&tab=artifacts#/network/tpcc/nodes=4

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190918-1495889/network/tpcc/nodes=4/run_1
	network.go:161,network.go:223,cluster.go:2091,errgroup.go:57: Get http://35.184.169.176:26258/debug/pprof/goroutine?debug=2: dial tcp 35.184.169.176:26258: connect: connection refused

cockroach-teamcity · 2019-09-18T22:32:40Z

SHA: https://github.com/cockroachdb/cockroach/commits/7c8323ad186dc4c4aef43882a08c5a75c0648695

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1496043&tab=artifacts#/network/tpcc/nodes=4

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190918-1496043/network/tpcc/nodes=4/run_1
	network.go:161,network.go:223,cluster.go:2091,errgroup.go:57: Get http://35.193.188.237:26258/debug/pprof/goroutine?debug=2: dial tcp 35.193.188.237:26258: connect: connection refused

cockroach-teamcity · 2019-09-19T06:11:20Z

SHA: https://github.com/cockroachdb/cockroach/commits/c6342c90a7fa4ceb1b674faa47a95e1726d05e79

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1496387&tab=artifacts#/network/tpcc/nodes=4

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190919-1496387/network/tpcc/nodes=4/run_1
	network.go:161,network.go:223,cluster.go:2091,errgroup.go:57: Get http://34.70.179.106:26258/debug/pprof/goroutine?debug=2: dial tcp 34.70.179.106:26258: connect: connection refused

cockroach-teamcity · 2019-10-01T06:06:38Z

SHA: https://github.com/cockroachdb/cockroach/commits/250f4c36de2b88eff443cf9be9cd5d2759312c88

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1514702&tab=artifacts#/network/tpcc/nodes=4

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190930-1514702/network/tpcc/nodes=4/run_1
	cluster.go:1826,cluster.go:1929,cluster.go:1900,network.go:155,network.go:198,cluster.go:2120,errgroup.go:57: context canceled

cockroach-teamcity · 2019-10-05T07:15:27Z

SHA: https://github.com/cockroachdb/cockroach/commits/1c99165c39c3714f1ce9986bff75ce517f977630

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1522970&tab=artifacts#/network/tpcc/nodes=4

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191005-1522970/network/tpcc/nodes=4/run_1
	cluster.go:1524,network.go:103,network.go:252,test_runner.go:689: cluster.Put: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod put teamcity-1570252270-14-n4cpu4:1-4 /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/workload ./workload returned:
		stderr:
		
		stdout:
		......................................................................................................................................
		   1: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine [email protected]:./workload [email protected]:./workload
		packet_write_wait: Connection to 34.69.184.172 port 22: Broken pipe
		: exit status 1
		   2: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine [email protected]:./workload [email protected]:./workload
		packet_write_wait: Connection to 34.69.184.172 port 22: Broken pipe
		: exit status 1
		   3: done
		   4: ~ scp -r -C -o StrictHostKeyChecking=no -i /root/.ssh/id_rsa -i /root/.ssh/google_compute_engine [email protected]:./workload [email protected]:./workload
		packet_write_wait: Connection to 34.69.184.172 port 22: Broken pipe
		: exit status 1
		I191005 07:15:13.308331 1 cluster_synced.go:1088  put /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/workload failed
		: exit status 1

cockroach-teamcity · 2019-10-11T18:06:30Z

SHA: https://github.com/cockroachdb/cockroach/commits/41bac2fd99a91d92cb3ff6426789f7e64dd6b14a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1534498&tab=artifacts#/network/tpcc/nodes=4

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191011-1534498/network/tpcc/nodes=4/run_1
	cluster.go:1831,cluster.go:1934,cluster.go:1905,network.go:155,network.go:198,cluster.go:2125,errgroup.go:57: context canceled

cockroach-teamcity · 2019-10-15T18:06:40Z

SHA: https://github.com/cockroachdb/cockroach/commits/a40cefbb5bd5e9d34f5db930334a385d0934d2d0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1538863&tab=artifacts#/network/tpcc/nodes=4

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191015-1538863/network/tpcc/nodes=4/run_1
	cluster.go:1831,cluster.go:1934,cluster.go:1905,network.go:155,network.go:198,cluster.go:2125,errgroup.go:57: context canceled

cockroach-teamcity · 2019-11-06T21:27:30Z

SHA: https://github.com/cockroachdb/cockroach/commits/2535f0363b5d9b1e466ca3144fe5bcf7ff17e63c

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1575491&tab=artifacts#/network/tpcc/nodes=4

The test failed on branch=provisional_201911052027_v19.2.0-rc.4, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191106-1575491/network/tpcc/nodes=4/run_1
	cluster.go:1848,cluster.go:1951,cluster.go:1922,network.go:155,network.go:198,cluster.go:2142,errgroup.go:57: context canceled

cockroach-teamcity · 2019-12-05T03:06:46Z

SHA: https://github.com/cockroachdb/cockroach/commits/20376489afba84e533d8511f4c9a59b15eec1371

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=network/tpcc/nodes=4 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1624121&tab=artifacts#/network/tpcc/nodes=4

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191204-1624121/network/tpcc/nodes=4/run_1
	cluster.go:1848,cluster.go:1951,cluster.go:1922,network.go:155,network.go:201,cluster.go:2142,errgroup.go:57: exit status 1

cockroach-teamcity · 2019-12-18T02:58:16Z

(roachtest).network/tpcc/nodes=4 failed on release-19.2@dd838bcaa68d3dbb2af8d162d5ba1a7a23fe5098:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191217-1647459/network/tpcc/nodes=4/run_1
	cluster.go:1848,cluster.go:1951,cluster.go:1922,network.go:155,network.go:201,cluster.go:2142,errgroup.go:57: context canceled

Repro

Artifacts: /network/tpcc/nodes=4

make stressrace TESTS=network/tpcc/nodes=4 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS=-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-22T03:15:27Z

(roachtest).network/tpcc/nodes=4 failed on master@dd9b1c1f40dda59ee9d446416106d311ae5ce1e6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191221-1658785/network/tpcc/nodes=4/run_1
	cluster.go:1851,cluster.go:1954,cluster.go:1925,network.go:155,network.go:201,cluster.go:2145,errgroup.go:57: context canceled

Repro

Artifacts: /network/tpcc/nodes=4

make stressrace TESTS=network/tpcc/nodes=4 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-28T00:58:33Z

(roachtest).network/tpcc/nodes=4 failed on release-19.2@599c988773fa6fb4544369a50fe8d06b90c64171:

The test failed on branch=release-19.2, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191227-1664336/network/tpcc/nodes=4/run_1
	network.go:169,network.go:201,cluster.go:2145,errgroup.go:57: context canceled

Repro

Artifacts: /network/tpcc/nodes=4

make stressrace TESTS=network/tpcc/nodes=4 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2019-12-31T07:05:17Z

(roachtest).network/tpcc/nodes=4 failed on release-2.1@c96086a9f10c5dc3a79cf1e8e80e4a0dc878b120:

The test failed on branch=release-2.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191231-1668225/network/tpcc/nodes=4/run_1
	network.go:169,network.go:201,cluster.go:2168,errgroup.go:57: net/http: request canceled (Client.Timeout exceeded while reading body)

Repro

Artifacts: /network/tpcc/nodes=4

make stressrace TESTS=network/tpcc/nodes=4 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-04T03:25:03Z

(roachtest).network/tpcc/nodes=4 failed on master@8facb469d66426a3f372a1aed4ea6f34e287b501:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200103-1671733/network/tpcc/nodes=4/run_1
	cluster.go:1874,cluster.go:1977,cluster.go:1948,network.go:155,network.go:201,cluster.go:2168,errgroup.go:57: context canceled

Repro

Artifacts: /network/tpcc/nodes=4

make stressrace TESTS=network/tpcc/nodes=4 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-06T20:05:21Z

(roachtest).network/tpcc/nodes=4 failed on master@f180501b7c2ccb946d7f7c8944d5509b87cb0394:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200106-1674751/network/tpcc/nodes=4/run_1
	cluster.go:1874,cluster.go:1977,cluster.go:1948,network.go:155,network.go:201,cluster.go:2168,errgroup.go:57: context canceled

Repro

Artifacts: /network/tpcc/nodes=4

make stressrace TESTS=network/tpcc/nodes=4 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-09T09:02:21Z

(roachtest).network/tpcc/nodes=4 failed on master@8d05b3279385d6c21bccb8610039c2678a7e2391:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200109-1680645/network/tpcc/nodes=4/run_1
	network.go:169,network.go:226,cluster.go:2168,errgroup.go:57: context canceled

Repro

Artifacts: /network/tpcc/nodes=4

make stressrace TESTS=network/tpcc/nodes=4 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-09T18:06:41Z

(roachtest).network/tpcc/nodes=4 failed on provisional_202001090312_v20.1.0-alpha.20200113@617cb39e3ced263cb4c253aa2e3edd172368f7cf:

The test failed on branch=provisional_202001090312_v20.1.0-alpha.20200113, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200109-1680508/network/tpcc/nodes=4/run_1
	network.go:164,network.go:201,cluster.go:2168,errgroup.go:57: Get http://35.230.83.113:26258/debug/pprof/goroutine?debug=2: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Repro

Artifacts: /network/tpcc/nodes=4

make stressrace TESTS=network/tpcc/nodes=4 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-09T20:08:53Z

(roachtest).network/tpcc/nodes=4 failed on release-19.1@3d6bb038f3afcf805fb48dddcffa342195d3d7e0:

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200109-1680663/network/tpcc/nodes=4/run_1
	network.go:164,network.go:201,cluster.go:2168,errgroup.go:57: Get http://34.70.20.2:26258/debug/pprof/goroutine?debug=2: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Repro

Artifacts: /network/tpcc/nodes=4

make stressrace TESTS=network/tpcc/nodes=4 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-16T20:11:12Z

(roachtest).network/tpcc/nodes=4 failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200116-1693614/network/tpcc/nodes=4/run_1
	cluster.go:1886,cluster.go:1989,cluster.go:1960,network.go:155,network.go:201,cluster.go:2180,errgroup.go:57: context canceled

Repro

Artifacts: /network/tpcc/nodes=4

make stressrace TESTS=network/tpcc/nodes=4 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-17T20:08:52Z

(roachtest).network/tpcc/nodes=4 failed on release-19.1@265263f40589020c2c6505e102a3c5d398054377:

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200117-1695931/network/tpcc/nodes=4/run_1
	network.go:164,network.go:201,cluster.go:2180,errgroup.go:57: Get http://35.222.157.33:26258/debug/pprof/goroutine?debug=2: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Repro

Artifacts: /network/tpcc/nodes=4

make stressrace TESTS=network/tpcc/nodes=4 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-01-25T21:08:34Z

(roachtest).network/tpcc/nodes=4 failed on master@72ce65798f55af352c6a38c306542f63dfba465f:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200125-1710860/network/tpcc/nodes=4/run_1
	cluster.go:2000,cluster.go:2103,cluster.go:2074,network.go:155,network.go:201,cluster.go:2294,errgroup.go:57: context canceled

More

Artifacts: /network/tpcc/nodes=4
Related:

roachtest: network/tpcc/nodes=4 failed #44369 roachtest: network/tpcc/nodes=4 failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-04T06:51:32Z

(roachtest).network/tpcc/nodes=4 failed on master@4ec5929a08687b82b05ef80231fddd0d53421973:

		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1725352-1580762820-07-n4cpu4:4 -- ./workload run tpcc --warehouses=1 --wait=false --histograms=perf/stats.json --duration=1h0m0s {pgurl:2-3} returned:
		    stderr:
		    
		    stdout:
		         0            0.0            0.4      0.0      0.0      0.0      0.0 payment
		      694.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		      695.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		      695.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 newOrder
		      695.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		      695.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 payment
		      695.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		      696.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		      696.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 newOrder
		      696.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		      696.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 payment
		      696.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      697.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		      697.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 newOrder
		      697.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		      697.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 payment
		      697.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		      698.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		      698.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 newOrder
		      698.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		      698.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 payment
		      698.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		      699.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		      699.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 newOrder
		      699.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		      699.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 payment
		      699.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		      700.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		      700.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 newOrder
		      700.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		      700.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 payment
		      700.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		    _elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		      701.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		      701.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 newOrder
		      701.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		      701.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 payment
		      701.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		      702.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		      702.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 newOrder
		      702.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		      702.0s        0            0.0            0.4      0.0      0.0      0.0      0.0 payment
		      702.0s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel:
		  - context canceled

More

Artifacts: /network/tpcc/nodes=4
Related:

roachtest: network/tpcc/nodes=4 failed #44369 roachtest: network/tpcc/nodes=4 failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

cockroach-teamcity · 2020-02-13T08:09:16Z

(roachtest).network/tpcc/nodes=4 failed on master@2739821b911d777fa2a927295d699b559360a802:

test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/network/tpcc/nodes=4/run_1
	cluster.go:1935,network.go:112,network.go:252,test_runner.go:741: error with attached stack trace:
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runNetworkTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/network.go:112
		    main.registerNetwork.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/network.go:252
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: output in %s
		    -- arg 1: <string>
		  - output in run_080849.335_n4_workload_fixtures_load_tpcc:
		  - error with attached stack trace:
		    main.execCmd
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
		    main.(*cluster).RunL
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
		    main.(*cluster).RunE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
		    main.(*cluster).Run
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:1933
		    main.runNetworkTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/network.go:112
		    main.registerNetwork.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/network.go:252
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - error with embedded safe details: %s returned:
		    stderr:
		    %s
		    stdout:
		    %s
		    -- arg 1: <string>
		    -- arg 2: <string>
		    -- arg 3: <string>
		  - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1747374-1581581035-01-n4cpu4:4 -- ./workload fixtures load tpcc --warehouses=1 {pgurl:1} returned:
		    stderr:
		    I200213 08:08:51.343868 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
		    Error: restoring fixture: sql: expected 6 destination arguments in Scan, not 7
		    Error:  exit status 1
		    
		    stdout::
		  - exit status 1

More

Artifacts: /network/tpcc/nodes=4
Related:

roachtest: network/tpcc/nodes=4 failed #44960 roachtest: network/tpcc/nodes=4 failed C-test-failure O-roachtest O-robot branch-provisional_202002062136_v19.1.8 release-blocker
roachtest: network/tpcc/nodes=4 failed #44875 roachtest: network/tpcc/nodes=4 failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker
roachtest: network/tpcc/nodes=4 failed #44369 roachtest: network/tpcc/nodes=4 failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

tbg · 2020-02-13T09:33:18Z

cc @lucy-zhang:

      Error: restoring fixture: sql: expected 6 destination arguments in Scan, not 7

Any idea who changed something around here? I think the error bubbles up to here:

cockroach/pkg/ccl/workloadccl/cliccl/fixtures.go

Lines 281 to 284 in 84b22c2

    
           bytes, err := workloadccl.RestoreFixture(ctx, db, l.fixture, l.database, true /* injectStats */) 
        
           if err != nil { 
        
           	return 0, errors.Wrap(err, `restoring fixture`) 
        
           }

cockroach-teamcity · 2020-02-14T09:15:09Z

(roachtest).network/tpcc/nodes=4 failed on master@e671a4ef97cbc6cf5d22f8f322fd45733d302094:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/network/tpcc/nodes=4/run_1
	network.go:164,network.go:221,cluster.go:2344,errgroup.go:57: context canceled

	cluster.go:2368,network.go:233,network.go:252,test_runner.go:741: error with attached stack trace:
		    main.(*monitor).WaitE
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2356
		    main.(*monitor).Wait
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2364
		    main.runNetworkTPCC
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/network.go:233
		    main.registerNetwork.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/network.go:252
		    main.(*testRunner).runTest.func2
		    	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
		    runtime.goexit
		    	/usr/local/go/src/runtime/asm_amd64.s:1357
		  - monitor failure:
		  - unexpected node event: 3: dead

	cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1749385-1581667748-09-n4cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: 5326
		2: 4706
		3: dead
		4: skipped
		Error:  3: dead

More

Artifacts: /network/tpcc/nodes=4
Related:

roachtest: network/tpcc/nodes=4 failed #45077 roachtest: network/tpcc/nodes=4 failed C-test-failure O-roachtest O-robot branch-44941 release-blocker
roachtest: network/tpcc/nodes=4 failed #44960 roachtest: network/tpcc/nodes=4 failed C-test-failure O-roachtest O-robot branch-provisional_202002062136_v19.1.8 release-blocker
roachtest: network/tpcc/nodes=4 failed #44875 roachtest: network/tpcc/nodes=4 failed C-test-failure O-roachtest O-robot branch-release-19.1 release-blocker
roachtest: network/tpcc/nodes=4 failed #44369 roachtest: network/tpcc/nodes=4 failed C-test-failure O-roachtest O-robot branch-release-19.2 release-blocker

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

irfansharif · 2020-03-04T19:23:20Z

restoring fixture: sql: expected 6 destination arguments in Scan, not 7

This was fixed in #45078.

cockroach-teamcity added this to the 19.2 milestone May 2, 2019

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels May 2, 2019

ajwerner self-assigned this May 9, 2019

This comment has been minimized.

Sign in to view

knz mentioned this issue Oct 1, 2019

release: v19.2.0-beta.20190930 #41128

Closed

18 tasks

tbg added the branch-master Failures and bugs on the master branch. label Jan 22, 2020

cockroach-teamcity mentioned this issue Jan 24, 2020

roachtest: network/tpcc/nodes=4 failed #44369

Closed

This was referenced Feb 7, 2020

roachtest: network/tpcc/nodes=4 failed #44875

Closed

roachtest: network/tpcc/nodes=4 failed #44960

Closed

cockroach-teamcity mentioned this issue Feb 13, 2020

roachtest: network/tpcc/nodes=4 failed #45077

Closed

tbg closed this as completed Mar 11, 2020

roachtest: network/tpcc/nodes=4 failed #37259

roachtest: network/tpcc/nodes=4 failed #37259

Comments

cockroach-teamcity commented May 2, 2019

cockroach-teamcity commented May 8, 2019

Uh oh!

cockroach-teamcity commented May 9, 2019

Uh oh!

andreimatei commented May 9, 2019

Uh oh!

cockroach-teamcity commented May 17, 2019

Uh oh!

cockroach-teamcity commented May 18, 2019

Uh oh!

cockroach-teamcity commented May 19, 2019

Uh oh!

cockroach-teamcity commented May 20, 2019

Uh oh!

This comment has been minimized.

This comment has been minimized.

nvanbenschoten commented May 21, 2019

Uh oh!

cockroach-teamcity commented May 31, 2019

Uh oh!

cockroach-teamcity commented May 31, 2019

Uh oh!

cockroach-teamcity commented Jun 1, 2019

Uh oh!

tbg commented Jun 4, 2019

Uh oh!

tbg commented Jun 4, 2019

Uh oh!

cockroach-teamcity commented Jun 6, 2019

Uh oh!

cockroach-teamcity commented Sep 7, 2019

Uh oh!

cockroach-teamcity commented Sep 18, 2019

Uh oh!

cockroach-teamcity commented Sep 18, 2019

Uh oh!

cockroach-teamcity commented Sep 19, 2019

Uh oh!

cockroach-teamcity commented Oct 1, 2019

Uh oh!

cockroach-teamcity commented Oct 5, 2019

Uh oh!

cockroach-teamcity commented Oct 11, 2019

Uh oh!

cockroach-teamcity commented Oct 15, 2019

Uh oh!

cockroach-teamcity commented Nov 6, 2019

Uh oh!

cockroach-teamcity commented Dec 5, 2019

Uh oh!

cockroach-teamcity commented Dec 18, 2019

Uh oh!

cockroach-teamcity commented Dec 22, 2019

Uh oh!

cockroach-teamcity commented Dec 28, 2019

Uh oh!

cockroach-teamcity commented Dec 31, 2019

Uh oh!

cockroach-teamcity commented Jan 4, 2020

Uh oh!

cockroach-teamcity commented Jan 6, 2020

Uh oh!

cockroach-teamcity commented Jan 9, 2020

Uh oh!

cockroach-teamcity commented Jan 9, 2020

Uh oh!

cockroach-teamcity commented Jan 9, 2020

Uh oh!

cockroach-teamcity commented Jan 16, 2020

Uh oh!

cockroach-teamcity commented Jan 17, 2020

Uh oh!

cockroach-teamcity commented Jan 25, 2020

Uh oh!

cockroach-teamcity commented Feb 4, 2020

Uh oh!