Mac + docker-machine failed

[kirinse]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   $ git clone https://github.com/pingcap/tidb-docker-compose.git
   $ cd tidb-docker-compose && docker-compose up -d

2. What did you expect to see?

3. What did you see instead?
   docker logs -f tidbdockercompose_pd0_1
   2018/05/27 23:50:01.700 log.go:78: [fatal] mvcc/backend: [cannot open database at /data/pd0/member/snap/db (invalid argument)]

4. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?
   2.0

[tennix]

@kirinse What's your docker and docker-compose version?

[kirinse]

@tennix thanks for your replying,
docker version
Client:
Version: 17.12.0-ce
API version: 1.35
Go version: go1.9.2
Git commit: c97c6d6
Built: Wed Dec 27 20:03:51 2017
OS/Arch: darwin/amd64

Server:
Engine:
Version: 17.12.1-ce
API version: 1.35 (minimum version 1.12)
Go version: go1.9.4
Git commit: 7390fc6
Built: Tue Feb 27 22:20:43 2018
OS/Arch: linux/amd64
Experimental: false

docker-compose version
docker-compose version 1.18.0, build 8dd22a9
docker-py version: 2.6.1
CPython version: 2.7.12
OpenSSL version: OpenSSL 1.0.2j 26 Sep 2016

[tennix]

Your Docker and Docker Compose version is OK. Is it the first time to run tidb-docker-compose, I mean is the data directory clean from scratch. Try with a clean data directory:

docker-compose down
rm -rf data
docker-compose up -d

[kirinse]

Hi @tennix, thanks for your replying, i tried your suggestion, but no luck.
2018/05/30 04:45:34.642 log.go:78: [fatal] mvcc/backend: [cannot open database at /data/pd0/member/snap/db (invalid argument)]
still get this error.

[tennix]

I've no clue about your problem. I suggest you debug it yourself

1. Reset docker-compose by docker-compose down && rm -rf data
2. Replace pd command in docker-compose.yml

command:
  - --name=pd0
  - --client-urls=http://0.0.0.0:2379
  ...

with entrypoint

entrypoint: ["tail", "-f", "/dev/null"]

2. Bring up docker-compose by docker-compose up -d
3. Exec into pd container by docker-compose exec pd0 sh
4. Check /data directory and its permission in pd container
5. Try to start pd-server manually inside pd container by concatenaing command in previous docker-compose.yml. That is /pd-server --name=pd0 --client-urls=http://0.0.0.0:2379 ...

See if you can find something from above steps

[kirinse]

I found what's the problem.
docker-library/mysql#99

but i don't have any clue how to fix it.

in pd container, i can't modify uid/gid
usermod: not found/groupmod: not found

[kirinse]

Comment out --data-dir, Finally, it's running.

[tennix]

@kirinse comment that will result non-persistent data, if container is removed due to some reasons, all data will be wiped out.

[tennix]

I'll try your version to see if the latest docker introduces permission problem

[kirinse]

@tennix, FYI, i am using docker-machine on Mac

[kirinse]

Hi @tennix, any updates?

[walijoy]

☁ tidb-docker-compose [master] docker-compose ps
Name Command State Ports
tidb-docker-compose_dashboard-installer_1 /tidb-dashboard-installer. ... Exit 0
tidb-docker-compose_grafana_1 /run.sh Up 0.0.0.0:3000->3000/tcp
tidb-docker-compose_pd0_1 /pd-server --name=pd0 --cl ... Restarting
tidb-docker-compose_pd1_1 /pd-server --name=pd1 --cl ... Restarting
tidb-docker-compose_pd2_1 /pd-server --name=pd2 --cl ... Restarting
tidb-docker-compose_prometheus_1 /bin/prometheus --storage. ... Restarting
tidb-docker-compose_pushgateway_1 /bin/pushgateway Up 9091/tcp
tidb-docker-compose_tidb-vision_1 /bin/sh -c sed -i -e "s/PD ... Up 2015/tcp, 443/tcp, 80/tcp, 0.0.0.0:8010->8010/tcp
tidb-docker-compose_tidb_1 /tidb-server --store=tikv ... Up 0.0.0.0:10080->10080/tcp, 0.0.0.0:4000->4000/tcp
tidb-docker-compose_tikv0_1 /tikv-server --addr=0.0.0. ... Restarting
tidb-docker-compose_tikv1_1 /tikv-server --addr=0.0.0. ... Restarting
tidb-docker-compose_tikv2_1 /tikv-server --addr=0.0.0. ... Restarting
tidb-docker-compose_tispark-master_1 /opt/spark/sbin/start-mast ... Up 0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp
tidb-docker-compose_tispark-slave0_1 /opt/spark/sbin/start-slav ... Up 0.0.0.0:38081->38081/tcp
☁ tidb-docker-compose [master] docker-compose exec pd0 sh
Error response from daemon: Container 39dbab4588a60c1b3803ef53bce8836e0e44432ab43b7c0e902cb37fd62d29b3 is restarting, wait until the container is running

☁ tidb-docker-compose [master] docker-compose logs pd0
Attaching to tidb-docker-compose_pd0_1
pd0_1 | 2018/07/19 10:11:37.156 log.go:78: [fatal] mvcc/backend: [cannot open database at /data/pd0/member/snap/db (invalid argument)]
pd0_1 | 2018/07/19 10:11:39.776 log.go:78: [fatal] mvcc/backend: [cannot open database at /data/pd0/member/snap/db (invalid argument)]
pd0_1 | 2018/07/19 10:11:41.916 log.go:78: [fatal] mvcc/backend: [cannot open database at /data/pd0/member/snap/db (invalid argument)]
pd0_1 | 2018/07/19 10:11:44.154 log.go:78: [fatal] mvcc/backend: [cannot open database at /data/pd0/member/snap/db (invalid argument)]
pd0_1 | 2018/07/19 10:11:46.294 log.go:78: [fatal] mvcc/backend: [cannot open database at /data/pd0/member/snap/db (invalid argument)]

[tennix]

I don't have MacOS atm, so don't know if it's a bug introduced by latest docker.

[walijoy]

@tennix please help me. can you tell me how to try?

[walijoy]

我在这里找到看起来与之相似的错误，不知道能否对你们修复此bug有帮助？
etcd-io/etcd#5923
boltdb/bolt#459

[tennix]

@hookover What's your Docker File Sharing settings

[tennix]

Can you write to a directory that is mounted in a container?

docker run -it --rm -v $HOME/Documents:/documents --entrypoint=sh pingcap/pd:latest
touch /documents/something

[tennix]

@kirinse I only have a MacOS running 18.03.1-ce Docker, and I can't find older Docker App download link.

[liufuyang]

@kirinse are you using docker-machine? (The one that runs in a virtual machine in the virtual box?)
If so, I would recommend you download the newest docker from the official site and it doesn't require virtual box anymore and the mapped port on host will be directly available on localhost as well.

I've tried this repo with the newest docker on Mac, it works out of the box.

So in short - currently on Mac it is not recommend to use docker-machine anymore. Just use newer version of docker for Mac (which run on hyperkit internally but you get similar experience as if you are using docker on Linux)

[walijoy]

@tennix
☁ tidb-docker-compose [master] brew search docker
==> Formulae
boot2docker docker-clean docker-compose ✔ docker-gen docker-machine-completion docker-machine-nfs docker-swarm
boot2docker-completion docker-cloud docker-compose-completion docker-ls docker-machine-driver-vultr docker-machine-parallels docker2aci
docker ✔ docker-completion docker-credential-helper docker-machine ✔ docker-machine-driver-xhyve docker-squash

==> Casks
docker docker-toolbox homebrew/cask-versions/docker-edge
☁ tidb-docker-compose [master] docker -v
Docker version 18.05.0-ce, build f150324
☁ tidb-docker-compose [master] docker-compose -v
docker-compose version 1.22.0, build unknown
☁ tidb-docker-compose [master] docker-machine -v
docker-machine version 0.15.0, build b48dc28

docker run -it --rm -v $HOME/Documents:/documents --entrypoint=sh pingcap/pd:latest
touch /documents/something

i can write any files

[walijoy]

@tennix @liufuyang 我从官网重新下载了Docker-ce for mac 后，成功正确运行了测试集群，谢谢你们

[tennix]

Hi, @kirinse could you download Docker for Mac App from Docker official website https://store.docker.com/editions/community/docker-ce-desktop-mac and try to run docker-compose up again to see if the problem still exists?
As @liufuyang said, docker-machine is not recommended for Mac any more, users should uninstall docker-toolbox/docker-machine and use Docker for Mac instead.