cmd/go: go mod init fails to retrieve company.github.com ssl only repositories

[MelleKoning]

What version of Go are you using (go version)?

go version go1.12 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GOARCH=amd64
set GOBIN=C:\git\go\bin
set GOCACHE=C:\Users\xxx\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=c:\git\go
set GOPROXY=
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\git\notgo\engineering\veritasapi\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\xxx\AppData\Local\Temp\go-build238261902=/tmp/go-build -gno-record-gcc-switches

What did you do?

go mod init mymodule

this starts reading the current used glide.yaml to process project dependencies.

There are some imports in this go project that refer to a company located git-host which is accessible via git using private ssl certificates

for exampe in code:

import "oucompany.github.com/engineering/myrepo/package"

whereby in the glide.yaml we might have:

- package: github.ourcompany.local/engineering/go-utils
  repo: [email protected]:engineering/go-utils.git
  vcs: git
  subpackages:
  - env

What did you expect to see?

A properly initialized go.mod

What did you see instead?

errors/ exception thrown like:

$ go mod init veritasapi
go: creating new go.mod: module veritasapi
go: copying requirements from glide.lock
go: converting glide.lock: stat github.mycompany.local/engineering/protos@7462cb9dbaba285e3c931bc1edc0b62ebe8d005e: git fetch -f https://github.mycompany.local/engineering/protos.git refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in c:\git\go\pkg\mod\cache\vcs\5626dfafc13719b661e5e62c892d3563fa39030a15a5ca1fdcad3e06591ea342: exit status 128:
        fatal: unable to access 'https://github.mycompany.local/engineering/protos.git/': schannel: next InitializeSecurityContext failed: Unknown error (0x80092013) - The revocation function was unable to check revocation because the revocation server was offline.

Also tried to change the .gitconfig setting:

[url "ssh://[email protected]:engineering/"]
        insteadOf = https://github.ourcompany.local/engineering/

but this does not show any noticable difference.

Also ensured the repository is available/readable on console:

>git ls-remote [email protected]:engineering/mygitrepo

outputs latest commit hashes

Question: how come

go mod init

is trying to access the git repository using https instead of via ssl / git@etc..?

[agnivade]

Question: how come go mod init is trying to access the git repository using https instead of via ssl / git@etc..?

See https://golang.org/doc/faq#git_https.

/cc @bcmills

[MelleKoning]

Question: how come go mod init is trying to access the git repository using https instead of via ssl / git@etc..?

See https://golang.org/doc/faq#git_https.

/cc @bcmills

Hi @agnivade thanks for that reference. As from the docs:

Git can also be configured to use SSH in place of HTTPS for URLs matching a given prefix. For example, to use SSH for all GitHub access, add these lines to your ~/.gitconfig:

[url "ssh://[email protected]/"]
insteadOf = https://github.com/

This has been tried but it seems the 'insteadOf' workaround to explicitly use SSH is not functioning as mentioned in the original issue remark.

[ update ]

turns out that only providing the domain name actually makes a difference. so:

[url "ssh://[email protected]"]
insteadOf = https://github.ourcompany.local

(without trailing slash or /engineering)

This still shows the following error though:

$ go mod init modulename
go: creating new go.mod: module modulename
go: copying requirements from glide.lock
go: converting glide.lock: stat github.ourcompany.local/engineering/protos@7462cb9dbaba285e3c931bc1edc0b62ebe8d005e: git fetch -f https://github.ourcompany.local/engineering/protos.git refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in c:\git\go\pkg\mod\cache\vcs\5626dfafc13719b661e5e62c892d3563fa39030a15a5ca1fdcad3e06591ea342: **exit status 128:
        fatal: unable to access 'https://github.ourcompany.local/engineering/protos.git/': SSL certificate problem: unable to get local issuer certificate**

Even when providing this part in the gitconfig, to give a hint on the used private SSL certificate:

[http "https://github.ourcompany.local"]
sslCaInfo = /c/Users/me/.ssh/github-ourcompany.pem
schannelUseSSLCAInfo = true

what would be the proper settings to point to the private ssl certificate or gitconfig?

[bcmills]

accessible via git using private ssl certificates

Do you mean that the go command needs to trust SSL certificates from a private CA, or that the go command needs to present a client certificate to the server? (If the latter, is this the same issue as #30119?)

[bcmills]

Question: how come

go mod init

is trying to access the git repository using https instead of via ssl / git@etc..?

Presumably that's what the go-import tags served by https://github.ourcompany.local/engineering/protos?go-get=1 told it to use. (See also #30304.)

You should be able to verify that by running

curl https://github.ourcompany.local/engineering/protos?go-get=1

and checking for a <meta name="go-import" […]> tag.

[bcmills]

Did the go mod init veritasapi command actually fail, or did it emit some errors and write a go.mod file anyway? (It should have done the latter; if not, please file a separate issue and mention #26603 in it.)

[MelleKoning]

Did the go mod init veritasapi command actually fail, or did it emit some errors and write a go.mod file anyway? (It should have done the latter; if not, please file a separate issue and mention #26603 in it.)

It wrote a go.mod file anyway, because of other dependencies that it could find from public repositories like from public github.com.

on checking with curl, the https version returns:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This is fine, because of the gitconfig 'insteadOf' setting moves to the SSH git@ way of connecting.

Thus, using the SSH method returns no error:

curl [email protected]/engineering/protos?go-get=1
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0

Still, using go mod init does not seem to be able to make the SSH connection. Not entirely sure how to setup the certificate configured in github.ourcompany.local within gitconfig, or if more certificate-access would be needed.

[bcmills]

on checking with curl, the https version returns:

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This is fine, because of the gitconfig 'insteadOf' setting moves to the SSH git@ way of connecting.

No, that's not fine. There are two steps to fetching a module with a give path: first we resolve that path to a repo or module server path (via HTTPS), then we fetch the module from that server. If we can't resolve the path, then we have no way of knowing that we need to use git in the first place, let alone that we should use it with the ssh protocol.

That means that your local machine needs to be configured to recognize the issuer of the private certificate. On Windows, I believe that you can use the MMC Certificates snap-in or the Certmgr tool for that, but that configuration is independent of the Go toolchain itself.

If you can configure your certificates so that curl […]?go-get=1 works, then cmd/go should also work. Please get curl working and then let us know where you land.

[MelleKoning]

@bcmills how come

glide update

does work and retrieves the company.github.com code from remote while

go mod init myname

does not?
I did make sure I first executed a glide cc (to clean glide cache).
Is there a certain difference in retrieving code that exposes via that curl command? To clarify - I just do not know the underlying git commands the different go tools are using to get dependencies and what 'go mod init' would do differently than 'glide update', if anything.

[bcmills]

how come

glide update

does work and retrieves the company.github.com code

I don't know; you'll have to ask the glide folks that. Perhaps they use heuristics (or data from the glide.lock file) rather than go-import meta-tags.

[MelleKoning]

@bcmills

Ok, so I got curl working only by providing the server side certificates separately. That took me some work, let me explain.

So at first I saw this:

$ curl https://github.ourcompany.local/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

After some digging through several options I bumped on this website http://javamemento.blogspot.com/2015/10/using-curl-with-ssl-cert-chain.html, which steps I've followed through to get a cacert file containing all certificates of the entire trust-chain necessary for server-access.

Remark: Not sure why this would be needed for 'go mod' command though; because a command like 'go get github.ourcompany.local/somerepo' simply works, but ok.

Now using that certificate (certRepo) in the curl command this results in:

$ curl --cacert certRepo https://github.ourcompany.local
<html><body>You are being <a href="https://github.ourcompany.local/login">redirected</a>.</body></html>

So, the SSL access error is gone and we have access via curl command.

However:
Now how do I tell go mod init mycoderepo to use that certificate-file for the https communication it wants to do? The command still fails.

I do not see a command line param for go mod to give a certificate to help.

[MelleKoning]

Addition:
Note: Everything is running outside of $GOPATH.

When in the gitconfig I set the following, changing sslBackend option from 'schannel' to 'openssl' and point to the generated .pem file with the server certificates:

[http "https://github.ourcompany.local"]
schannelUseSSLCAInfo = true
sslBackend=openssl
sslCAInfo=C:\\git\\notgo\\engineering\\veritasapi\\certRepo.pem

then go mod init gives this failure:

fatal: unable to access 'https://github.ourcompany.local/somedependency/whatever.git/': schannel: next InitializeSecurityContext failed: Unknown error (0x80092013) - The revocation function was unable to check revocation because the revocation server was offline.

However, when I take the git fetch command of one of the dependencies that go mod wants to execute, and try to execute that 'git fetch' on the command line myself I get the following message:

> git fetch -f https://github.ourcompany.local/somedependency/whatever.git refs/heads/*:refs/heads/* refs/tags/*:refs/tags/*

remote: Password authentication is not available for Git operations.
remote: You must use a personal access token or SSH key.
remote: See https://github.ourcompany.local/settings/tokens or https://github.ourcompany.local/settings/ssh
fatal: unable to access 'https://github.ourcompany.local/somedependency/whatever.git/': The requested URL returned error: 403

I'm not all too familiar with the connection details, but does 'git mod' not accept SSH keys/connections yet?

[MelleKoning]

@agnivade @bcmills does the above provide enough information? Let me know if I can try anything else!

[bcmills]

@MelleKoning, thanks for following up, and my apologies for the delayed response.

The go command does not have an equivalent to curl's --cacert or --capath flags. It uses only the certificates provided by the operating system itself, so the private CA for github.ourcompany.local needs to be installed at the system level (that's what you need mmc or Certmgr for, as mentioned previously).

When that is done, I would be surprised if you needed any of the explicit ssl configuration in .gitconfig, since I would expect that Git also uses the system-level certificates by default.

The choice between SSH and HTTPS is currently whatever is indicated in the response to https://[…]?go-get=1. (The proposal to allow the server to leave that choice up to the client is #30304, and the workaround in the meantime is to use insteadOf in your .gitconfig as described in #26134 (comment).)

[MelleKoning]

Thanks @bcmills; I have installed the certificates from github.ourcompany.local in the windows root certificate store (computer trusted), but now getting a CERT_TRUST_REVOCATION_STATUS_UNKNOWN error.

> certutil -verify -urlfetch <certificatename.cer>
provides much more information and ends with:

ERROR: Verifying leaf certificate revocation status returned The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613 CRYPT_E_REVOCATION_OFFLINE)
CertUtil: -verify command completed successfully.

I will ask our cloudops department in the company more information about this. Thank you for your feedback so far.

[MelleKoning]

As of date: Company sticks with glide for go development as 'go mod' commands are not functioning for the ssh accessible private github.company.com endpoints.

[agnivade]

@MelleKoning - Were you able to figure out the certificate error that you were facing ?

[MelleKoning]

Hi @agnivade unfortunately not. a 'go mod init myname' keeps coming back with multiple of the following errors.

fatal: unable to access 'https://github.ourcompany.local/engineering/protos.git/': schannel: next InitializeSecurityContext failed: Unknown error (0x80092013) - The revocation function was unable to check revocation because the revocation server was offline.

so we simply keep developing within the go path.

[mvdan]

Closing old issues that still have the WaitingForInfo label where enough details to investigate weren't provided. Feel free to leave a comment with more details and we can reopen.