Skip to content

roachtest: schemachange/indexrollback/tpcc/w=1000 failed #36321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
cockroach-teamcity opened this issue Mar 29, 2019 · 28 comments
Closed

roachtest: schemachange/indexrollback/tpcc/w=1000 failed #36321

cockroach-teamcity opened this issue Mar 29, 2019 · 28 comments
Assignees
@vivekmenezes
Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Milestone
19.1

Comments

@cockroach-teamcity
Copy link
Member

SHA: https://github.com/cockroachdb/cockroach/commits/3aadd20bbf0940ef65f8b2cdcda498401ba5d9c6

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1206925&tab=buildLog

The test failed on release-19.1:
	cluster.go:1293,tpcc.go:130,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1206925-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		    2.1      0.0      0.0      0.0      0.0 delivery
		  50m50s    68081            0.0           20.5      0.0      0.0      0.0      0.0 newOrder
		  50m50s    68081            0.0            2.1      0.0      0.0      0.0      0.0 orderStatus
		  50m50s    68081            0.0           18.9      0.0      0.0      0.0      0.0 payment
		  50m50s    68081            0.0            2.0      0.0      0.0      0.0      0.0 stockLevel
		E190329 03:45:02.793850 1 workload/cli/run.go:420  error in newOrder: dial tcp 10.128.0.30:26257: connect: connection refused
		  50m51s    93052            0.0            2.1      0.0      0.0      0.0      0.0 delivery
		  50m51s    93052            0.0           20.5      0.0      0.0      0.0      0.0 newOrder
		  50m51s    93052            0.0            2.1      0.0      0.0      0.0      0.0 orderStatus
		  50m51s    93052            1.0           18.9  42949.7  42949.7  42949.7  42949.7 payment
		  50m51s    93052            0.0            2.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1652,tpcc.go:140,schemachange.go:375,test.go:1223: unexpected node event: 1: dead
	cluster.go:953,context.go:90,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:774,test.go:760,cluster.go:1652,tpcc.go:140,schemachange.go:375,test.go:1223: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1206925-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1
@cockroach-teamcity cockroach-teamcity added this to the 19.1 milestone Mar 29, 2019
@cockroach-teamcity cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Mar 29, 2019
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/23f9707873abbd2de91a42055535529d7ff296ce

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1209900&tab=buildLog

The test failed on release-19.1:
	cluster.go:1293,tpcc.go:130,cluster.go:1631,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1209900-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		       0            0.0            3.1      0.0      0.0      0.0      0.0 stockLevel
		   9m43s        0            5.0            3.2   2281.7   2952.8   2952.8   2952.8 delivery
		   9m43s        0           35.9           28.3   3355.4  85899.3 103079.2 103079.2 newOrder
		   9m43s        0            3.0            3.1   1275.1   1476.4   1476.4   1476.4 orderStatus
		   9m43s        0           32.9           23.8   3087.0 103079.2 103079.2 103079.2 payment
		   9m43s        0            2.0            3.1   1543.5   1879.0   1879.0   1879.0 stockLevel
		   9m44s        0            0.0            3.2      0.0      0.0      0.0      0.0 delivery
		   9m44s        0           21.0           28.3  11274.3 103079.2 103079.2 103079.2 newOrder
		   9m44s        0            2.0            3.1    805.3    973.1    973.1    973.1 orderStatus
		   9m44s        0           20.0           23.7   2080.4 103079.2 103079.2 103079.2 payment
		   9m44s        0            3.0            3.1    906.0    906.0    906.0    906.0 stockLevel
		: signal: killed
	cluster.go:1652,tpcc.go:140,schemachange.go:375,test.go:1223: sql: no rows in result set

@vivekmenezes
Copy link
Contributor

the latest failure seems to be a situation where the query : SELECT status FROM [SHOW JOBS] WHERE job_id = $1 given the rollback job id failed. It's a harmless failure but we should look into finding out why.

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/877ebd1ece299b9ee621aa0d091657621593d844

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1215372&tab=buildLog

The test failed on release-19.1:
	cluster.go:1329,tpcc.go:130,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1215372-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		.0      0.0 newOrder
		   7m37s    63516            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   7m37s    63516            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m37s    63516            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190402 14:16:51.182444 1 workload/cli/run.go:420  error in newOrder: dial tcp 10.142.0.67:26257: connect: connection refused
		   7m38s   103769            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   7m38s   103769            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   7m38s   103769            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   7m38s   103769            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m38s   103769            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190402 14:16:52.181646 1 workload/cli/run.go:420  error in payment: dial tcp 10.142.0.67:26257: connect: connection refused
		: signal: killed
	test.go:1214: test timed out (3h0m0s)
	cluster.go:1688,tpcc.go:140,schemachange.go:375,test.go:1226: unexpected node event: 3: dead
	cluster.go:1405,cluster.go:1424,cluster.go:1528,cluster.go:968,asm_amd64.s:522,panic.go:397,test.go:774,test.go:760,cluster.go:1688,tpcc.go:140,schemachange.go:375,test.go:1226: context canceled

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/b88a6ce86bfe507e14e1e80fdaefd219b5f0b046

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1222890&tab=buildLog

The test failed on release-19.1:
	cluster.go:1329,tpcc.go:130,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1222890-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		Error: read tcp 10.142.0.77:54644->10.142.0.68:26257: read: connection reset by peer
		Error:  exit status 1
		: exit status 1
	cluster.go:1688,tpcc.go:140,schemachange.go:375,test.go:1228: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/c03c307b1a4100a8a1edd2804bdfaf3903097756

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1229036&tab=buildLog

The test failed on release-19.1:
	test.go:1225: test timed out (3h0m0s)
	cluster.go:1329,tpcc.go:169,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1229036-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		eating load: dial tcp :26257: connect: connection refused
		I190409 15:46:27.943959 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:46:39.205529 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:46:51.616948 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:47:04.566491 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:47:16.429664 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:47:27.551983 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		I190409 15:47:39.100901 1 workload/cli/run.go:337  retrying after error while creating load: dial tcp :26257: connect: connection refused
		: signal: killed
	cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: Goexit() was called

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/bf399d2677783dc1eea7f5ede6d4561f95c0ea10

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1234662&tab=buildLog

The test failed on master:
	cluster.go:1329,tpcc.go:169,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1234662-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		Initializing 10000 connections...
		Initializing 10000 workers and preparing statements...
		: signal: killed
	cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: pq: crdb-internal-jobs-table: no inbound stream connection

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/134478e4dde16919eb86efb81fb22d8cce8a9701

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1236263&tab=buildLog

The test failed on release-19.1:
	cluster.go:1329,tpcc.go:169,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1236263-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		n.go:428  error in newOrder: dial tcp 10.128.0.109:26257: connect: connection refused
		   4m47s     6380            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   4m47s     6380            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   4m47s     6380            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   4m47s     6380            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   4m47s     6380            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		   4m48s     6380            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   4m48s     6380            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   4m48s     6380            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   4m48s     6380            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   4m48s     6380            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: unexpected node event: 4: dead
	cluster.go:953,context.go:90,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1236263-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		4: dead
		1: 3140
		3: 3306
		2: 3185
		Error:  4: dead

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/509c5b130fb1ad0042beb74e083817aa68e4fc92

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1237068&tab=buildLog

The test failed on release-19.1:
	cluster.go:1329,tpcc.go:169,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1237068-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		un.go:428  error in payment: dial tcp 10.128.0.134:26257: connect: connection refused
		   3m51s    11346            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   3m51s    11346            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   3m51s    11346            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   3m51s    11346            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   3m51s    11346            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		   3m52s    11346            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   3m52s    11346            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   3m52s    11346            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   3m52s    11346            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   3m52s    11346            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: unexpected node event: 2: dead
	cluster.go:953,context.go:90,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,tpcc.go:179,schemachange.go:375,test.go:1237: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1237068-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		2: dead
		1: 3011
		4: 3033
		3: 3251
		Error:  2: dead

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/83de585d331b05a4aa02a65b353bed6bf829b696

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1247383&tab=buildLog

The test failed on master:
	cluster.go:1329,tpcc.go:168,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1247383-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		   0.0 delivery
		   6m29s    16869            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   6m29s    16869            0.0            0.4      0.0      0.0      0.0      0.0 orderStatus
		   6m29s    16869            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   6m29s    16869            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190418 14:31:53.585047 1 workload/cli/run.go:428  error in newOrder: ERROR: operation "intent_resolver_ir_batcher.sendBatch" timed out after 30s (SQLSTATE XXUUU)
		   6m30s    16870            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   6m30s    16870            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   6m30s    16870            0.0            0.4      0.0      0.0      0.0      0.0 orderStatus
		   6m30s    16870            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   6m30s    16870            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: dial tcp 35.196.209.187:26257: connect: connection refused
	cluster.go:953,context.go:89,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1247383-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		1: dead
		2: 3947
		3: 4174
		4: 3955
		Error:  1: dead

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/837e946efc272bd8a9e0e08484733f8755ff5ab1

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1247401&tab=buildLog

The test failed on release-19.1:
	cluster.go:1329,tpcc.go:168,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1247401-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		un.go:428  error in newOrder: dial tcp 10.142.0.68:26257: connect: connection refused
		   7m42s    11446            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   7m42s    11446            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   7m42s    11446            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   7m42s    11446            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m42s    11446            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		   7m43s    11446            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   7m43s    11446            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   7m43s    11446            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   7m43s    11446            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m43s    11446            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	test.go:1225: test timed out (3h0m0s)
	cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: unexpected node event: 4: dead
	cluster.go:1405,cluster.go:1424,cluster.go:1528,cluster.go:968,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: context canceled

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/46f8608c4fe2d94b771beb37bcee19136040fd74

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1253450&tab=buildLog

The test failed on master:
	cluster.go:1329,tpcc.go:168,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1253450-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		operation "intent_resolver_ir_batcher.sendBatch" timed out after 30s (SQLSTATE XXUUU)
		  12m10s    20979            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  12m10s    20979            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  12m10s    20979            0.0            0.2      0.0      0.0      0.0      0.0 orderStatus
		  12m10s    20979            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  12m10s    20979            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  12m11s    20979            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  12m11s    20979            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  12m11s    20979            0.0            0.2      0.0      0.0      0.0      0.0 orderStatus
		  12m11s    20979            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  12m11s    20979            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: unexpected node event: 1: dead

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/ec4728ae986b46d4f57009233b86971198b275ed

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1255121&tab=buildLog

The test failed on master:
	cluster.go:1329,tpcc.go:168,cluster.go:1667,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1255121-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		.0      0.0      0.0      0.0 orderStatus
		   7m14s    17981            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m14s    17981            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190423 14:45:20.129701 1 workload/cli/run.go:428  error in newOrder: ERROR: operation "intent_resolver_ir_batcher.sendBatch" timed out after 30s (SQLSTATE XXUUU)
		   7m15s    18008            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   7m15s    18008            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   7m15s    18008            0.0            0.2      0.0      0.0      0.0      0.0 orderStatus
		   7m15s    18008            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   7m15s    18008            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190423 14:45:21.157249 1 workload/cli/run.go:428  error in newOrder: ERROR: operation "intent_resolver_ir_batcher.sendBatch" timed out after 30s (SQLSTATE XXUUU)
		: signal: killed
	cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: unexpected node event: 1: dead
	cluster.go:953,context.go:89,cluster.go:942,asm_amd64.s:522,panic.go:397,test.go:785,test.go:771,cluster.go:1688,tpcc.go:178,schemachange.go:375,test.go:1237: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1255121-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		1: dead
		2: 4198
		4: 4089
		3: 4086
		Error:  1: dead

@vivekmenezes
Copy link
Contributor

I'm seeing node liveness problems on this cluster. Will run one by hand and check

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/a22a3780ebb5482be04b6f31344c764b2fb5f765

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1257361&tab=buildLog

The test failed on release-19.1:
	cluster.go:1423,tpcc.go:168,cluster.go:1761,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1257361-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   4m57s        0            4.0            3.0   6174.0  22548.6  22548.6  22548.6 delivery
		   4m57s        0           70.2           25.9  38654.7 103079.2 103079.2 103079.2 newOrder
		   4m57s        0            3.0            3.1    125.8    369.1    369.1    369.1 orderStatus
		   4m57s        0           17.0           19.2  98784.2 103079.2 103079.2 103079.2 payment
		   4m57s        0            5.0            2.9   2013.3  30064.8  30064.8  30064.8 stockLevel
		   4m58s        0            3.0            3.0  11274.3  23622.3  23622.3  23622.3 delivery
		   4m58s        0           27.9           25.9  64424.5 103079.2 103079.2 103079.2 newOrder
		   4m58s        0            0.0            3.1      0.0      0.0      0.0      0.0 orderStatus
		   4m58s        0           24.0           19.2  57982.1 103079.2 103079.2 103079.2 payment
		   4m58s        0            1.0            2.9    318.8    318.8    318.8    318.8 stockLevel
		: signal: killed
	cluster.go:1782,tpcc.go:178,schemachange.go:375,test.go:1245: sql: no rows in result set

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/99306ec3e9fcbba01c05431cbf496e8b5b8954b4

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1260033&tab=buildLog

The test failed on master:
	cluster.go:1423,tpcc.go:168,cluster.go:1761,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1260033-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		d
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		  14m25s    18711            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  14m25s    18711            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  14m25s    18711            0.0            0.1      0.0      0.0      0.0      0.0 orderStatus
		  14m25s    18711            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  14m25s    18711            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		  14m26s    18711            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		  14m26s    18711            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		  14m26s    18711            0.0            0.1      0.0      0.0      0.0      0.0 orderStatus
		  14m26s    18711            0.0            0.0      0.0      0.0      0.0      0.0 payment
		  14m26s    18711            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	test.go:1231: test timed out (3h0m0s)
	cluster.go:1782,tpcc.go:178,schemachange.go:375,test.go:1245: unexpected node event: 2: dead

@vivekmenezes vivekmenezes mentioned this issue Apr 29, 2019
Merged
craig bot pushed a commit that referenced this issue Apr 29, 2019
@nvanbenschoten @lucy-zhang
@RaduBerinde @vivekmenezes
Merge #36744 #36999 #37169 #37182
f7018cf 
36744: storage/rangefeed: use fine-grained locking around Processor, add metrics r=nvanbenschoten a=nvanbenschoten

It is critical for correctness that operations like providing the processor
with logical ops or informing the processor of closed timestamp updates
be properly synchronized via the raftMu. This ensures that events published
on the processors eventC accurately reflect the order of events in the Raft
log.

However, it is not critical that lifecycle events around starting a
Rangefeed processor, stopping a Rangefeed processor, and registering
with a Rangefeed processor be synchronized with Raft application. This
change exploits this to break up the locking around rangefeed.Processor.

Using more fine-grained locking opens up the opportunity to interact
with the rangefeed Processor without needing to synchronize with the
Range's raft application, which can be very expensive. The next commit
uses this improvement to add a new `rangefeed_registrations` metric to
`RangeInfo`.

36999: roachtest: add schemachange/bulkingest test r=lucy-zhang a=lucy-zhang

Add a test to index the random `payload` column for the `bulkingest` workload,
to test the index backfiller for an index on randomly ordered values relative
to the primary key.

Release note: None

37169: opt: don't use delegateQuery for ShowZoneConfig r=RaduBerinde a=RaduBerinde

Part of a larger work item to remove `delegateQuery`, as it doesn't
work with the optimizer.

I could not use the new delegate infrastructure because it doesn't support hiding columns returned by the delegated query.

Release note: None

37182: roachtest: provision more cpu for schemachange tpcc tests r=vivekmenezes a=vivekmenezes

related to 
#36094 
#36024 
#36321 

Release note: None

Co-authored-by: Nathan VanBenschoten <[email protected]>
Co-authored-by: Lucy Zhang <[email protected]>
Co-authored-by: Radu Berinde <[email protected]>
Co-authored-by: Vivek Menezes <[email protected]>
@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/73765b6d168fb999466756b112fd590747a3a8c4

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1266059&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1476,tpcc.go:168,cluster.go:1814,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1266059-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		   4m20s        0           54.9           28.1   1040.2   5637.1  12348.0  18253.6 delivery
		   4m20s        0          365.3          275.0   2818.6  26843.5  47244.6  85899.3 newOrder
		   4m20s        0           34.9           28.1     48.2   1140.9   3489.7   3489.7 orderStatus
		   4m20s        0          332.4          267.7   2550.1  62277.0 103079.2 103079.2 payment
		   4m20s        0           31.9           27.9    201.3   6710.9  19327.4  19327.4 stockLevel
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   4m21s        0           26.0           28.1   1073.7   4160.7   5100.3   5100.3 delivery
		   4m21s        0          186.2          274.7   1879.0  26843.5  55834.6 103079.2 newOrder
		   4m21s        0           16.0           28.1    151.0   1476.4   8589.9   8589.9 orderStatus
		   4m21s        0          194.2          267.4   2415.9  90194.3 103079.2 103079.2 payment
		   4m21s        0           17.0           27.8    318.8   1543.5   2147.5   2147.5 stockLevel
		: signal: killed
	cluster.go:1835,tpcc.go:178,schemachange.go:375,test.go:1253: sql: no rows in result set

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/efb45869b242137e5c178b10c646c3ed025fff36

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1266041&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1476,tpcc.go:168,cluster.go:1814,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1266041-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		.0      0.0      0.0 orderStatus
		   2m40s    11822            0.0           16.2      0.0      0.0      0.0      0.0 payment
		   2m40s    11822            0.0            0.2      0.0      0.0      0.0      0.0 stockLevel
		E190430 14:22:12.588196 1 workload/cli/run.go:428  error in payment: dial tcp 10.142.0.74:26257: connect: connection refused
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   2m41s    11915            0.0            2.5      0.0      0.0      0.0      0.0 delivery
		   2m41s    11915            2.0           20.4 103079.2 103079.2 103079.2 103079.2 newOrder
		   2m41s    11915            0.0            3.0      0.0      0.0      0.0      0.0 orderStatus
		   2m41s    11915            0.0           16.1      0.0      0.0      0.0      0.0 payment
		   2m41s    11915            0.0            0.2      0.0      0.0      0.0      0.0 stockLevel
		E190430 14:22:13.588267 1 workload/cli/run.go:428  error in newOrder: dial tcp 10.142.0.74:26257: connect: connection refused
		: signal: killed
	cluster.go:1835,tpcc.go:178,schemachange.go:375,test.go:1253: unexpected node event: 4: dead
	test.go:1239: test timed out (3h0m0s)

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/24feca7a4106f08c73534e16ebb79d949a479f35

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1268176&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1476,tpcc.go:168,cluster.go:1814,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1268176-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		       0           15.0           22.3    503.3  16643.0  22548.6  22548.6 stockLevel
		    1m2s        0           12.0           22.7   1275.1   2147.5  28991.0  28991.0 delivery
		    1m2s        0          113.8          211.2   1610.6  31138.5  51539.6  60129.5 newOrder
		    1m2s        0           22.0           23.0    285.2  14495.5  17179.9  17179.9 orderStatus
		    1m2s        0          209.7          207.5   1140.9  28991.0  60129.5  62277.0 payment
		    1m2s        0           24.0           22.3    419.4   1543.5  20401.1  20401.1 stockLevel
		    1m3s        0           18.0           22.6   1208.0   3221.2   9663.7   9663.7 delivery
		    1m3s        0          313.3          212.8   1677.7  34359.7  47244.6  55834.6 newOrder
		    1m3s        0           19.0           23.0    285.2  16106.1  28991.0  28991.0 orderStatus
		    1m3s        0          119.1          206.1   1476.4  40802.2  55834.6  60129.5 payment
		    1m3s        0           17.0           22.2    335.5    805.3  21474.8  21474.8 stockLevel
		: signal: killed
	cluster.go:1835,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/9f039a1237bb7571d910b3a3b0e4f2ffc5a84357

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1270756&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1476,tpcc.go:168,cluster.go:1814,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1270756-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   3m17s        0           33.0           28.1   1208.0   1677.7   1811.9   1811.9 delivery
		   3m17s        0          199.7          270.8   1342.2  15569.3  66572.0  94489.3 newOrder
		   3m17s        0           19.0           27.7    352.3    704.6    872.4    872.4 orderStatus
		   3m17s        0          268.7          266.5   1476.4  73014.4 103079.2 103079.2 payment
		   3m17s        0           13.0           27.6    234.9    453.0    486.5    486.5 stockLevel
		   3m18s        0           23.0           28.0   1275.1   3623.9   3892.3   3892.3 delivery
		   3m18s        0          211.3          270.5   1744.8  22548.6  53687.1 103079.2 newOrder
		   3m18s        0           35.0           27.8    234.9    906.0   1208.0   1208.0 orderStatus
		   3m18s        0          261.4          266.5   1409.3   5368.7  24696.1  81604.4 payment
		   3m18s        0           26.0           27.6    335.5  10200.5  22548.6  22548.6 stockLevel
		: signal: killed
	cluster.go:1835,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/ff969dd6cb0e087327f0b210f728f588b2f5aeb0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1270738&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1476,tpcc.go:168,cluster.go:1814,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1270738-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		       0           29.0           21.1    151.0  23622.3  28991.0  28991.0 stockLevel
		   1m38s        0           15.0           20.6   1946.2  26843.5  27917.3  27917.3 delivery
		   1m38s        0          254.0          188.9   2684.4  47244.6  77309.4  85899.3 newOrder
		   1m38s        0           14.0           21.5    184.5    604.0  22548.6  22548.6 orderStatus
		   1m38s        0          106.0          180.5   2281.7  47244.6  94489.3  94489.3 payment
		   1m38s        0           17.0           21.0    704.6  42949.7  68719.5  68719.5 stockLevel
		   1m39s        0            9.0           20.4   2013.3  24696.1  24696.1  24696.1 delivery
		   1m39s        0          224.0          189.2   2415.9  49392.1  81604.4  85899.3 newOrder
		   1m39s        0           25.0           21.5    151.0  10737.4  12884.9  12884.9 orderStatus
		   1m39s        0          288.0          181.6   1811.9  57982.1  81604.4  94489.3 payment
		   1m39s        0           25.0           21.1    570.4  23622.3  30064.8  30064.8 stockLevel
		: signal: killed
	cluster.go:1835,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/bb50fd396f6ce79258b744e1f8efa2d1bc9dfbd2

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1272551&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1272551-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		       0           35.1           32.4    184.5   6979.3  15032.4  15032.4 stockLevel
		   1m46s        0           44.9           32.4    872.4   1543.5   3087.0   3087.0 delivery
		   1m46s        0          390.4          321.1   1476.4  15032.4  30064.8  40802.2 newOrder
		   1m46s        0           30.9           32.4     33.6    130.0    184.5    184.5 orderStatus
		   1m46s        0          393.4          314.8    771.8  28991.0  49392.1  68719.5 payment
		   1m46s        0           33.9           32.4    142.6   1073.7   4563.4   4563.4 stockLevel
		   1m47s        0           41.0           32.5    805.3   1140.9   1275.1   1275.1 delivery
		   1m47s        0          270.2          320.6    771.8  22548.6  27917.3  32212.3 newOrder
		   1m47s        0           33.0           32.4     58.7    352.3    385.9    385.9 orderStatus
		   1m47s        0          276.2          314.4    805.3  15569.3  38654.7  47244.6 payment
		   1m47s        0           46.0           32.5    125.8   6979.3  15569.3  15569.3 stockLevel
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/29bf8a1eb0b5b943127206f49333337f7f3079b4

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1272533&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1272533-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   3m41s        0           22.0           20.5    436.2  16643.0  25769.8  25769.8 delivery
		   3m41s        0          245.9          193.0    906.0  62277.0 103079.2 103079.2 newOrder
		   3m41s        0           23.0           20.4     39.8  13958.6  19327.4  19327.4 orderStatus
		   3m41s        0          225.9          187.7    637.5  57982.1 103079.2 103079.2 payment
		   3m41s        0           34.0           20.1    130.0  42949.7 103079.2 103079.2 stockLevel
		   3m42s        0           14.0           20.5    469.8  15032.4  51539.6  51539.6 delivery
		   3m42s        0          143.8          192.8    805.3  64424.5 103079.2 103079.2 newOrder
		   3m42s        0           18.0           20.4    234.9  51539.6  57982.1  57982.1 orderStatus
		   3m42s        0          178.8          187.7    771.8  42949.7 103079.2 103079.2 payment
		   3m42s        0           21.0           20.1    260.0  24696.1  25769.8  25769.8 stockLevel
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/b0c0444481f40a772798c0f6eef3af2fd6a8ada0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1274193&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1274193-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		   3m33s        0           30.0           30.8    503.3   1208.0   1275.1   1275.1 delivery
		   3m33s        0          262.0          301.1    838.9  18253.6  31138.5  45097.2 newOrder
		   3m33s        0           35.0           30.6     62.9    260.0    260.0    260.0 orderStatus
		   3m33s        0          394.9          293.2   1040.2  53687.1 103079.2 103079.2 payment
		   3m33s        0           36.0           30.8    151.0   4563.4  40802.2  40802.2 stockLevel
		   3m34s        0           29.0           30.8   1208.0   3892.3   3892.3   3892.3 delivery
		   3m34s        0          195.9          300.6   1040.2  21474.8  53687.1  68719.5 newOrder
		   3m34s        0           21.0           30.6     65.0    369.1    503.3    503.3 orderStatus
		   3m34s        0          315.9          293.3   1342.2  45097.2 103079.2 103079.2 payment
		   3m34s        0           21.0           30.7    142.6    453.0  32212.3  32212.3 stockLevel
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/856ba9108f112f85d406bbe88d2208651859336e

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1274175&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1274175-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		    0.0      0.0      0.0      0.0      0.0 delivery
		   3m31s    10043            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   3m31s    10043            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   3m31s    10043            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   3m31s    10043            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		   3m32s    10043            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   3m32s    10043            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   3m32s    10043            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   3m32s    10043            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   3m32s    10043            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		E190504 14:43:39.885336 1 workload/cli/run.go:428  error in payment: dial tcp 10.142.0.161:26257: connect: connection refused
		: signal: killed
	test.go:1237: test timed out (3h0m0s)
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: unexpected node event: 3: dead
	cluster.go:1550,cluster.go:1569,cluster.go:1673,cluster.go:1053,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: context canceled

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/23155799e92e54915ae66259d06a630e981afbeb

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1277061&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1277061-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		4  15032.4 newOrder
		   1m13s    94392            3.0            5.5   1140.9   7516.2   7516.2   7516.2 orderStatus
		   1m13s    94392           11.0           47.1   3892.3  24696.1  27917.3  27917.3 payment
		   1m13s    94392            0.0            0.5      0.0      0.0      0.0      0.0 stockLevel
		E190507 14:32:07.988144 1 workload/cli/run.go:428  error in payment: dial tcp 10.142.0.165:26257: connect: connection refused
		   1m14s   212773            2.0            4.6  17179.9  22548.6  22548.6  22548.6 delivery
		   1m14s   212773           13.0           40.1  13958.6  19327.4  36507.2  36507.2 newOrder
		   1m14s   212773            3.0            5.4   1946.2   5100.3   5100.3   5100.3 orderStatus
		   1m14s   212773           26.0           46.8   6174.0  22548.6  51539.6  51539.6 payment
		   1m14s   212773            0.0            0.5      0.0      0.0      0.0      0.0 stockLevel
		E190507 14:32:08.988142 1 workload/cli/run.go:428  error in payment: dial tcp 10.142.0.165:26257: connect: connection refused
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: unexpected node event: 4: dead
	cluster.go:1038,context.go:89,cluster.go:1027,asm_amd64.s:522,panic.go:397,test.go:788,test.go:774,cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1277061-schemachange-indexrollback-tpcc-w-1000 --oneshot --ignore-empty-nodes: exit status 1 5: skipped
		4: dead
		2: 4446
		1: 4481
		3: 4463
		Error:  4: dead

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/72d089cfc1a0b67c299439d328587345f1125b74

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1277079&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1277079-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		l
		_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
		     41s        0           38.9           29.8   2281.7   3355.4   3489.7   3489.7 delivery
		     41s        0          524.1          305.6   1342.2   5100.3   6442.5   9126.8 newOrder
		     41s        0           38.9           31.7     30.4    113.2    142.6    142.6 orderStatus
		     41s        0          281.5          296.9    402.7   5100.3   8589.9  11811.2 payment
		     41s        0           38.9           30.8    100.7   1409.3   4563.4   4563.4 stockLevel
		     42s        0           54.1           30.4   1006.6   1946.2   2952.8   3221.2 delivery
		     42s        0          433.7          308.6    704.6   3221.2   8053.1  10737.4 newOrder
		     42s        0           33.1           31.7     18.9     37.7     50.3     50.3 orderStatus
		     42s        0          300.5          297.0    419.4   5905.6  10737.4  16106.1 payment
		     42s        0           31.0           30.8     96.5    385.9   1275.1   1275.1 stockLevel
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: sql: no rows in result set

@cockroach-teamcity
Copy link
Member Author

SHA: https://github.com/cockroachdb/cockroach/commits/8abb47a1c9795c1463183bc44e776b054bece682

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=schemachange/indexrollback/tpcc/w=1000 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1279683&tab=buildLog

The test failed on branch=master, cloud=gce:
	cluster.go:1474,tpcc.go:168,cluster.go:1812,errgroup.go:57: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1279683-schemachange-indexrollback-tpcc-w-1000:5 -- ./workload run tpcc --warehouses=1000 --histograms=logs/stats.json --wait=false --tolerate-errors --ramp=5m0s --duration=1h0m0s {pgurl:1-4} returned:
		stderr:
		
		stdout:
		run.go:428  error in newOrder: dial tcp 10.142.0.5:26257: connect: connection refused
		   6m10s     9455            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   6m10s     9455            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   6m10s     9455            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   6m10s     9455            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   6m10s     9455            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		   6m11s     9455            0.0            0.0      0.0      0.0      0.0      0.0 delivery
		   6m11s     9455            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
		   6m11s     9455            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
		   6m11s     9455            0.0            0.0      0.0      0.0      0.0      0.0 payment
		   6m11s     9455            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
		: signal: killed
	cluster.go:1833,tpcc.go:178,schemachange.go:375,test.go:1251: read tcp 172.17.0.2:43292->34.74.137.171:26257: read: connection reset by peer

@vivekmenezes
Copy link
Contributor

this test has been deleted,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vivekmenezes vivekmenezes

Labels
C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot.
Projects
None yet
Milestone
19.1
Development

No branches or pull requests
2 participants
@vivekmenezes @cockroach-teamcity