net/http: ProxyFromEnvironment does not support socks5 #18508

bruceauyeung · 2017-01-04T01:38:32Z

What version of Go are you using (`go version`)?

go version go1.7.4 linux/amd64

What operating system and processor architecture are you using (`go env`)?

actually it's Manjaro

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/bruce/go_ext_path"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build134643851=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

git config --global http.proxy 'socks5://127.0.0.1:1080'
git config --global https.proxy 'socks5://127.0.0.1:1080'
export http_proxy=socks5://127.0.0.1:1080
export https_proxy=socks5://127.0.0.1:1080
go get -u -v github.com/zmb3/gogetdoc

What did you expect to see?

Git can use socks5 proxy to clone a repo, so i expect go get can use socks5 proxy to download/update package github.com/zmb3/gogetdoc

What did you see instead?

github.com/zmb3/gogetdoc (download)
Fetching https://golang.org/x/tools/go/buildutil?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/buildutil: unrecognized import path "golang.org/x/tools/go/buildutil" (https fetch: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)
Fetching https://golang.org/x/tools/go/loader?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/loader: unrecognized import path "golang.org/x/tools/go/loader" (https fetch: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)
[bruce@bruce-manjaro ~]$ set https_proxy=socks5://127.0.0.1:1080
[bruce@bruce-manjaro ~]$ go get -u -v github.com/zmb3/gogetdoc
github.com/zmb3/gogetdoc (download)
Fetching https://golang.org/x/tools/go/buildutil?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/buildutil: unrecognized import path "golang.org/x/tools/go/buildutil" (https fetch: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)
Fetching https://golang.org/x/tools/go/loader?go-get=1
https fetch failed: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout
package golang.org/x/tools/go/loader: unrecognized import path "golang.org/x/tools/go/loader" (https fetch: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout)

The text was updated successfully, but these errors were encountered:

davecheney · 2017-01-04T01:39:46Z

Is there a specification for http_proxy ?

…

On Wed, Jan 4, 2017 at 12:38 PM, Bruce Auyeung ***@***.***> wrote: What version of Go are you using (go version)? go version go1.7.4 linux/amd64 What operating system and processor architecture are you using (go env)? actually it's Manjaro GOARCH="amd64" GOBIN="" GOEXE="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GOPATH="/home/bruce/go_ext_path" GORACE="" GOROOT="/usr/lib/go" GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64" CC="gcc" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build134643851=/tmp/go-build -gno-record-gcc-switches" CXX="g++" CGO_ENABLED="1" What did you do? git config --global https.proxy 'socks5://127.0.0.1:1080' git config --global https.proxy 'socks5://127.0.0.1:1080' export http_proxy=socks5://127.0.0.1:1080 export https_proxy=socks5://127.0.0.1:1080 go get -u -v github.com/zmb3/gogetdoc What did you expect to see? Git can use socks5 proxy to clone a repo, so i expect go get can use socks5 proxy to download/update package github.com/zmb3/gogetdoc What did you see instead? github.com/zmb3/gogetdoc (download) Fetching https://golang.org/x/tools/go/buildutil?go-get=1 https fetch failed: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout package golang.org/x/tools/go/buildutil: unrecognized import path " golang.org/x/tools/go/buildutil" (https fetch: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout) Fetching https://golang.org/x/tools/go/loader?go-get=1 https fetch failed: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout package golang.org/x/tools/go/loader: unrecognized import path " golang.org/x/tools/go/loader" (https fetch: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout) ***@***.*** ~]$ set https_proxy=socks5://127.0.0.1:1080 ***@***.*** ~]$ go get -u -v github.com/zmb3/gogetdoc github.com/zmb3/gogetdoc (download) Fetching https://golang.org/x/tools/go/buildutil?go-get=1 https fetch failed: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout package golang.org/x/tools/go/buildutil: unrecognized import path " golang.org/x/tools/go/buildutil" (https fetch: Get https://golang.org/x/tools/go/buildutil?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout) Fetching https://golang.org/x/tools/go/loader?go-get=1 https fetch failed: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout package golang.org/x/tools/go/loader: unrecognized import path " golang.org/x/tools/go/loader" (https fetch: Get https://golang.org/x/tools/go/loader?go-get=1: http: error connecting to proxy http://socks5://127.0.0.1:1080: dial tcp 59.50.43.82:0: i/o timeout) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#18508>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA0krlQiUzE1sCC7YLFI_VhbLOyhRks5rOvgZgaJpZM4LaNf9> .

josharian · 2017-01-04T02:04:01Z

From a quick glance, looks like maybe cmd/go needs to use http.ProxyFromEnvironment.

davecheney · 2017-01-04T02:05:58Z

I'm pretty confident there is no socks handling code in the std lib, so it wouldn't really help.

…

On Wed, 4 Jan 2017, 13:04 Josh Bleecher Snyder ***@***.***> wrote: From a quick glance, looks like maybe cmd/go needs to use http.ProxyFromEnvironment. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18508 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA9dxWPfJsACfbawXpwN_z7EdCQvhks5rOv4SgaJpZM4LaNf9> .

davecheney · 2017-01-04T02:07:04Z

If there is a standard for a socks URL, we can at least spot it and give a better error rather than blindly using the provided URL as a http proxy.

…

On Wed, 4 Jan 2017, 13:05 Dave Cheney ***@***.***> wrote: I'm pretty confident there is no socks handling code in the std lib, so it wouldn't really help. On Wed, 4 Jan 2017, 13:04 Josh Bleecher Snyder ***@***.***> wrote: From a quick glance, looks like maybe cmd/go needs to use http.ProxyFromEnvironment. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18508 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA9dxWPfJsACfbawXpwN_z7EdCQvhks5rOv4SgaJpZM4LaNf9> .

bruceauyeung · 2017-01-04T02:08:56Z

i tried to find if there is a spec for http.proxy, but found nothing.

bradfitz · 2017-01-04T02:10:17Z

Yes, we don't support SOCK5 proxies in the standard library and thus we don't support it in the cmd/go tool.

We have SOCK5 support elsewhere, so we could vendor it into std at least for use by cmd/go, but it's not clear how much it matters. Is SOCK5 still widely used?

bruceauyeung · 2017-01-04T02:14:21Z

@davecheney useful informations that i can found about http.proxy is http://unix.stackexchange.com/a/212972

@bradfitz In China, shadowsocks is used widely

davecheney · 2017-01-04T02:14:40Z

If you use tor, the answer is a strong yes.

…

On Wed, 4 Jan 2017, 13:10 Brad Fitzpatrick ***@***.***> wrote: Yes, we don't support SOCK5 proxies in the standard library and thus we don't support it in the cmd/go tool. We have SOCK5 support elsewhere, so we could vendor it into std at least for use by cmd/go, but it's not clear how much it matters. Is SOCK5 still widely used? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#18508 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcAzw8FVQU52L3HaRFAueumkP2QYP9ks5rOv-LgaJpZM4LaNf9> .

bradfitz · 2017-01-04T02:31:32Z

That still wouldn't get you SOCKS5 support.

…

On Jan 3, 2017 6:04 PM, "Josh Bleecher Snyder" ***@***.***> wrote: From a quick glance, looks like maybe cmd/go needs to use http.ProxyFromEnvironment. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#18508 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAKPblcSHbNwESPYf2DMDbooilOvvlrks5rOv4SgaJpZM4LaNf9> .

rsc · 2017-01-04T15:50:26Z

cmd/go does use http.ProxyFromEnvironment, because http.DefaultTransport does.

http.ProxyFromEnvironment does not support socks5. @bradfitz, where is the socks5 code we have elsewhere? It seems like it should be a tiny amount of code (under 100 lines plus tests) to proxy TCP connections into socks5. Assuming it really is small, I would not object to putting that into the standard library, on by default. It's very useful combined with ssh -D. I used to use that and tsocks, but Go doesn't play well with tsocks if I remember correctly, and tsocks is a bit of a hack.

As @davecheney says, at the least we should give a better error. But it would be nice to add support.

bradfitz · 2017-01-04T20:37:35Z

@rsc, SOCK5 support is at https://godoc.org/golang.org/x/net/proxy#SOCKS5

iamzhout · 2017-01-05T02:54:30Z

@bruceauyeung Just suggest to use privoxy / squid alike tool to setup a local http proxy server over socks5 one to get workaround of this issue.

@bradfitz it would be useful if socks proxy could be supported. As to me it would also be ok if not as for the workaround I mentioned above.

bruceauyeung · 2017-01-05T03:13:16Z

@bruceauyeung Just suggest to use privoxy / squid alike tool to setup a local http proxy server over socks5 one to get workaround of this issue.

@iamzhout thanks for your tip. now i go get packages through a http proxy (XX-Net), although it's slow.

iamzhout · 2017-01-05T03:27:37Z

@bradfitz actually there is another issue #18519 which really bothers me a lot. Hope there would be a fix or workaround also.

davecheney · 2017-01-05T03:32:50Z

@iamzhout it looks like your employer is MITM'ing your SSL communications.

Go get is just a wrapper over git, I recommend git cloning that repository directly.

iamzhout · 2017-01-05T03:47:04Z

@bruceauyeung, I do suggest you to give privoxy/squid a try, as the speed should be the same as to your socks5 one. and the configuration is really simple, take privoxy as an example, you just add below line to your /etc/privoxy/config file, and then do service privoxy restart, all done.

listen-address  0.0.0.0:12345
forward-socks5  /  your-socks5-ip:port  .

icexin · 2017-01-12T02:28:48Z

@bruceauyeung You may want this github.com/icexin/sockhttp. It's a small tool written in go that makes a HTTP proxy over SOCK5.

bruceauyeung · 2017-01-12T03:34:16Z

@icexin i'll give it a shot, thanks for your work!

mattn · 2017-01-13T07:23:53Z

Maybe, implementaion will be like below.

package main

import (
	"context"
	"io"
	"log"
	"net"
	"net/http"
	"os"

	"golang.org/x/net/proxy"
)

type dialer struct {
	addr   string
	socks5 proxy.Dialer
}

func (d *dialer) DialContext(ctx context.Context, network, addr string) (net.Conn, error) {
	// TODO: golang.org/x/net/proxy need to add DialContext
	return d.Dial(network, addr)
}

func (d *dialer) Dial(network, addr string) (net.Conn, error) {
	var err error
	if d.socks5 == nil {
		d.socks5, err = proxy.SOCKS5("tcp", d.addr, nil, proxy.Direct)
		if err != nil {
			return nil, err
		}
	}
	return d.socks5.Dial(network, addr)
}

func Socks5Proxy(addr string) *http.Transport {
	d := &dialer{addr: addr}
	return &http.Transport{
		DialContext: d.DialContext,
		Dial:        d.Dial,
	}
}

func main() {
	http.DefaultTransport = Socks5Proxy("127.0.0.1:1080")

	resp, err := http.Get("http://www.google.com/")
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()
	io.Copy(os.Stdout, resp.Body)
}

tested with server written in go https://github.com/armon/go-socks5

mattn · 2017-01-13T07:35:00Z

So, to add common interfaces between http_proxy and socks5, I guess we should not modify ProxyFromEnvironment. it's better to add new API for set-up proxy.

walken-google · 2017-01-18T05:26:39Z

I could use this feature too, and I've been looking into implementing it.

This has probably been answered before, but how do I go about importing a golang.org package (such as golang_org/x/net/proxy) so that the net/http package can use it ? I know this would require creating a src/vendor/golang_org/x/net/proxy directory but I'm not sure if it needs any tracking metadata, if the import needs to be a separate commit, etc...

walken-google · 2017-01-21T00:59:52Z

Prototype in https://go-review.googlesource.com/35488

gopherbot · 2017-01-23T18:18:59Z

CL https://golang.org/cl/35488 mentions this issue.

See #18508 This commit adds http Client support for socks5 proxies. Change-Id: Ib015f3819801da13781d5acdd780149ae1f5857b Reviewed-on: https://go-review.googlesource.com/35488 Reviewed-by: Brad Fitzpatrick <[email protected]> Run-TryBot: Brad Fitzpatrick <[email protected]> TryBot-Result: Gobot Gobot <[email protected]>

gopherbot · 2017-08-09T08:30:52Z

Change https://golang.org/cl/41031 mentions this issue: net/http: replace SOCKS client implementation

This change replaces the vendored socks client implementation with the bundle of golang.org/x/net/internal/socks package which contains fixes for 19354 and 11682. golang.org/x/net/internal/socks becomes socks_bundle.go. At git rev 61147c4. (golang.org/cl/38278) Updates #11682. Updates #18508. Updates #19354. Fixes #19688. Updates #21333. Change-Id: I8cf6c3f5eb87c24685a7592be015729f84fbed77 Reviewed-on: https://go-review.googlesource.com/41031 Run-TryBot: Mikio Hara <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Brad Fitzpatrick <[email protected]>

josharian changed the title ~~go does not recognize socks5 proxy~~ cmd/go: get does not use socks5 proxy Jan 4, 2017

rsc added this to the Go1.9Early milestone Jan 4, 2017

rsc changed the title ~~cmd/go: get does not use socks5 proxy~~ net/http: ProxyFromEnvironment does not support socks5 Jan 4, 2017

bradfitz added the NeedsFix The path to resolution is known, but the work has not been done. label Jan 4, 2017

iamzhout mentioned this issue Jan 5, 2017

cmd/go: "go get gopkg.in/yaml.v2" will run into "x509: certificate signed by unknown authority" behind corporate proxy #18519

Closed

walken-google self-assigned this Jan 21, 2017

walken-google closed this as completed Mar 2, 2017

sorsasampo mentioned this issue Feb 26, 2018

net/http: ProxyFromEnvironment does not support socks5h #24135

Closed

mholt mentioned this issue May 28, 2018

Allow specifying a proxy for certificate generation caddyserver/caddy#2186

Closed

golang locked and limited conversation to collaborators Aug 9, 2018

gopherbot unassigned walken-google Aug 9, 2018

gopherbot added the FrozenDueToAge label Aug 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: ProxyFromEnvironment does not support socks5 #18508

net/http: ProxyFromEnvironment does not support socks5 #18508

bruceauyeung commented Jan 4, 2017 •

edited

Loading

davecheney commented Jan 4, 2017 via email

josharian commented Jan 4, 2017

davecheney commented Jan 4, 2017 via email

davecheney commented Jan 4, 2017 via email

bruceauyeung commented Jan 4, 2017

bradfitz commented Jan 4, 2017

bruceauyeung commented Jan 4, 2017

davecheney commented Jan 4, 2017 via email

bradfitz commented Jan 4, 2017 via email

rsc commented Jan 4, 2017

bradfitz commented Jan 4, 2017

iamzhout commented Jan 5, 2017

bruceauyeung commented Jan 5, 2017

iamzhout commented Jan 5, 2017

davecheney commented Jan 5, 2017 •

edited

Loading

iamzhout commented Jan 5, 2017

icexin commented Jan 12, 2017 •

edited

Loading

bruceauyeung commented Jan 12, 2017

mattn commented Jan 13, 2017 •

edited

Loading

mattn commented Jan 13, 2017

walken-google commented Jan 18, 2017

walken-google commented Jan 21, 2017

gopherbot commented Jan 23, 2017

gopherbot commented Aug 9, 2017

net/http: ProxyFromEnvironment does not support socks5 #18508

net/http: ProxyFromEnvironment does not support socks5 #18508

Comments

bruceauyeung commented Jan 4, 2017 • edited Loading

What version of Go are you using (go version)?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

davecheney commented Jan 4, 2017 via email

josharian commented Jan 4, 2017

davecheney commented Jan 4, 2017 via email

davecheney commented Jan 4, 2017 via email

bruceauyeung commented Jan 4, 2017

bradfitz commented Jan 4, 2017

bruceauyeung commented Jan 4, 2017

davecheney commented Jan 4, 2017 via email

bradfitz commented Jan 4, 2017 via email

rsc commented Jan 4, 2017

bradfitz commented Jan 4, 2017

iamzhout commented Jan 5, 2017

bruceauyeung commented Jan 5, 2017

iamzhout commented Jan 5, 2017

davecheney commented Jan 5, 2017 • edited Loading

iamzhout commented Jan 5, 2017

icexin commented Jan 12, 2017 • edited Loading

bruceauyeung commented Jan 12, 2017

mattn commented Jan 13, 2017 • edited Loading

mattn commented Jan 13, 2017

walken-google commented Jan 18, 2017

walken-google commented Jan 21, 2017

gopherbot commented Jan 23, 2017

gopherbot commented Aug 9, 2017

bruceauyeung commented Jan 4, 2017 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

davecheney commented Jan 5, 2017 •

edited

Loading

icexin commented Jan 12, 2017 •

edited

Loading

mattn commented Jan 13, 2017 •

edited

Loading