Skip to content

Commit f9acad8

Browse files
lunny6543zeripath
authored
Add proxy settings and support for migration and webhook (#16704)
* Add proxy settings and support for migration and webhook * Fix default value * Add newline for example ini * Add lfs proxy support * Fix lint * Follow @zeripath's review * Fix git clone * Fix test * missgin http requests for proxy * use empty Co-authored-by: zeripath <[email protected]> Co-authored-by: 6543 <[email protected]> Co-authored-by: zeripath <[email protected]>
1 parent 422c30d commit f9acad8

20 files changed

+302
-41
lines changed

‎custom/conf/app.example.ini

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2127,3 +2127,11 @@ PATH =
21272127
;;
21282128
;; Minio enabled ssl only available when STORAGE_TYPE is `minio`
21292129
;MINIO_USE_SSL = false
2130+
2131+
;[proxy]
2132+
;; Enable the proxy, all requests to external via HTTP will be affected
2133+
;PROXY_ENABLED = false
2134+
;; Proxy server URL, support http://, https//, socks://, blank will follow environment http_proxy/https_proxy/no_proxy
2135+
;PROXY_URL =
2136+
;; Comma separated list of host names requiring proxy. Glob patterns (*) are accepted; use ** to match all hosts.
2137+
;PROXY_HOSTS =

‎docs/content/doc/advanced/config-cheat-sheet.en-us.md

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -549,8 +549,8 @@ Define allowed algorithms and their minimum key length (use -1 to disable a type
549549
- `DELIVER_TIMEOUT`: **5**: Delivery timeout (sec) for shooting webhooks.
550550
- `SKIP_TLS_VERIFY`: **false**: Allow insecure certification.
551551
- `PAGING_NUM`: **10**: Number of webhook history events that are shown in one page.
552-
- `PROXY_URL`: ****: Proxy server URL, support http://, https//, socks://, blank will follow environment http_proxy/https_proxy
553-
- `PROXY_HOSTS`: ****: Comma separated list of host names requiring proxy. Glob patterns (*) are accepted; use ** to match all hosts.
552+
- `PROXY_URL`: **\<empty\>**: Proxy server URL, support http://, https//, socks://, blank will follow environment http_proxy/https_proxy. If not given, will use global proxy setting.
553+
- `PROXY_HOSTS`: **\<empty\>`**: Comma separated list of host names requiring proxy. Glob patterns (*) are accepted; use ** to match all hosts. If not given, will use global proxy setting.
554554

555555
## Mailer (`mailer`)
556556

@@ -950,6 +950,7 @@ Task queue configuration has been moved to `queue.task`. However, the below conf
950950
- `ALLOWED_DOMAINS`: **\<empty\>**: Domains allowlist for migrating repositories, default is blank. It means everything will be allowed. Multiple domains could be separated by commas.
951951
- `BLOCKED_DOMAINS`: **\<empty\>**: Domains blocklist for migrating repositories, default is blank. Multiple domains could be separated by commas. When `ALLOWED_DOMAINS` is not blank, this option will be ignored.
952952
- `ALLOW_LOCALNETWORKS`: **false**: Allow private addresses defined by RFC 1918, RFC 1122, RFC 4632 and RFC 4291
953+
- `SKIP_TLS_VERIFY`: **false**: Allow skip tls verify
953954

954955
## Mirror (`mirror`)
955956

@@ -1023,6 +1024,19 @@ is `data/repo-archive` and the default of `MINIO_BASE_PATH` is `repo-archive/`.
10231024
- `MINIO_BASE_PATH`: **repo-archive/**: Minio base path on the bucket only available when `STORAGE_TYPE` is `minio`
10241025
- `MINIO_USE_SSL`: **false**: Minio enabled ssl only available when `STORAGE_TYPE` is `minio`
10251026

1027+
## Proxy (`proxy`)
1028+
1029+
- `PROXY_ENABLED`: **false**: Enable the proxy if true, all requests to external via HTTP will be affected, if false, no proxy will be used even environment http_proxy/https_proxy
1030+
- `PROXY_URL`: **\<empty\>**: Proxy server URL, support http://, https//, socks://, blank will follow environment http_proxy/https_proxy
1031+
- `PROXY_HOSTS`: **\<empty\>**: Comma separated list of host names requiring proxy. Glob patterns (*) are accepted; use ** to match all hosts.
1032+
1033+
i.e.
1034+
```ini
1035+
PROXY_ENABLED = true
1036+
PROXY_URL = socks://127.0.0.1:1080
1037+
PROXY_HOSTS = *.github.com
1038+
```
1039+
10261040
## Other (`other`)
10271041

10281042
- `SHOW_FOOTER_BRANDING`: **false**: Show Gitea branding in the footer.

‎docs/content/doc/advanced/config-cheat-sheet.zh-cn.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -332,6 +332,7 @@ IS_INPUT_FILE = false
332332
- `ALLOWED_DOMAINS`: **\<empty\>**: 迁移仓库的域名白名单，默认为空，表示允许从任意域名迁移仓库，多个域名用逗号分隔。
333333
- `BLOCKED_DOMAINS`: **\<empty\>**: 迁移仓库的域名黑名单，默认为空，多个域名用逗号分隔。如果 `ALLOWED_DOMAINS` 不为空，此选项将会被忽略。
334334
- `ALLOW_LOCALNETWORKS`: **false**: Allow private addresses defined by RFC 1918
335+
- `SKIP_TLS_VERIFY`: **false**: 允许忽略 TLS 认证
335336

336337
## LFS (`lfs`)
337338

@@ -397,6 +398,19 @@ Repository archive 的存储配置。 如果 `STORAGE_TYPE` 为空，则此配
397398
- `MINIO_BASE_PATH`: **repo-archive/**: Minio base path ，仅当 `STORAGE_TYPE` 为 `minio` 时有效。
398399
- `MINIO_USE_SSL`: **false**: Minio 是否启用 ssl ，仅当 `STORAGE_TYPE` 为 `minio` 时有效。
399400

401+
## Proxy (`proxy`)
402+
403+
- `PROXY_ENABLED`: **false**: 是否启用全局代理。如果为否，则不使用代理，环境变量中的代理也不使用
404+
- `PROXY_URL`: **\<empty\>**: 代理服务器地址，支持 http://, https//, socks://，为空则不启用代理而使用环境变量中的 http_proxy/https_proxy
405+
- `PROXY_HOSTS`: **\<empty\>**: 逗号分隔的多个需要代理的网址，支持 * 号匹配符号， ** 表示匹配所有网站
406+
407+
i.e.
408+
```ini
409+
PROXY_ENABLED = true
410+
PROXY_URL = socks://127.0.0.1:1080
411+
PROXY_HOSTS = *.github.com
412+
```
413+
400414
## Other (`other`)
401415

402416
- `SHOW_FOOTER_BRANDING`: 为真则在页面底部显示Gitea的字样。

‎modules/git/command.go

Lines changed: 37 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -110,24 +110,47 @@ func (c *Command) RunInDirTimeoutEnvFullPipeline(env []string, timeout time.Dura
110110
// RunInDirTimeoutEnvFullPipelineFunc executes the command in given directory with given timeout,
111111
// it pipes stdout and stderr to given io.Writer and passes in an io.Reader as stdin. Between cmd.Start and cmd.Wait the passed in function is run.
112112
func (c *Command) RunInDirTimeoutEnvFullPipelineFunc(env []string, timeout time.Duration, dir string, stdout, stderr io.Writer, stdin io.Reader, fn func(context.Context, context.CancelFunc) error) error {
113-
if timeout == -1 {
114-
timeout = defaultCommandExecutionTimeout
113+
return c.RunWithContext(&RunContext{
114+
Env: env,
115+
Timeout: timeout,
116+
Dir: dir,
117+
Stdout: stdout,
118+
Stderr: stderr,
119+
Stdin: stdin,
120+
PipelineFunc: fn,
121+
})
122+
}
123+
124+
// RunContext represents parameters to run the command
125+
type RunContext struct {
126+
Env []string
127+
Timeout time.Duration
128+
Dir string
129+
Stdout, Stderr io.Writer
130+
Stdin io.Reader
131+
PipelineFunc func(context.Context, context.CancelFunc) error
132+
}
133+
134+
// RunWithContext run the command with context
135+
func (c *Command) RunWithContext(rc *RunContext) error {
136+
if rc.Timeout == -1 {
137+
rc.Timeout = defaultCommandExecutionTimeout
115138
}
116139

117-
if len(dir) == 0 {
140+
if len(rc.Dir) == 0 {
118141
log.Debug("%s", c)
119142
} else {
120-
log.Debug("%s: %v", dir, c)
143+
log.Debug("%s: %v", rc.Dir, c)
121144
}
122145

123-
ctx, cancel := context.WithTimeout(c.parentContext, timeout)
146+
ctx, cancel := context.WithTimeout(c.parentContext, rc.Timeout)
124147
defer cancel()
125148

126149
cmd := exec.CommandContext(ctx, c.name, c.args...)
127-
if env == nil {
150+
if rc.Env == nil {
128151
cmd.Env = os.Environ()
129152
} else {
130-
cmd.Env = env
153+
cmd.Env = rc.Env
131154
}
132155

133156
cmd.Env = append(
@@ -141,23 +164,23 @@ func (c *Command) RunInDirTimeoutEnvFullPipelineFunc(env []string, timeout time.
141164
if goVersionLessThan115 {
142165
cmd.Env = append(cmd.Env, "GODEBUG=asyncpreemptoff=1")
143166
}
144-
cmd.Dir = dir
145-
cmd.Stdout = stdout
146-
cmd.Stderr = stderr
147-
cmd.Stdin = stdin
167+
cmd.Dir = rc.Dir
168+
cmd.Stdout = rc.Stdout
169+
cmd.Stderr = rc.Stderr
170+
cmd.Stdin = rc.Stdin
148171
if err := cmd.Start(); err != nil {
149172
return err
150173
}
151174

152175
desc := c.desc
153176
if desc == "" {
154-
desc = fmt.Sprintf("%s %s %s [repo_path: %s]", GitExecutable, c.name, strings.Join(c.args, " "), dir)
177+
desc = fmt.Sprintf("%s %s %s [repo_path: %s]", GitExecutable, c.name, strings.Join(c.args, " "), rc.Dir)
155178
}
156179
pid := process.GetManager().Add(desc, cancel)
157180
defer process.GetManager().Remove(pid)
158181

159-
if fn != nil {
160-
err := fn(ctx, cancel)
182+
if rc.PipelineFunc != nil {
183+
err := rc.PipelineFunc(ctx, cancel)
161184
if err != nil {
162185
cancel()
163186
_ = cmd.Wait()

‎modules/git/repo.go

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,15 @@ import (
99
"bytes"
1010
"context"
1111
"fmt"
12+
"io"
13+
"net/url"
1214
"os"
1315
"path"
1416
"strconv"
1517
"strings"
1618
"time"
19+
20+
"code.gitea.io/gitea/modules/proxy"
1721
)
1822

1923
// GPGSettings represents the default GPG settings for this repository
@@ -99,12 +103,12 @@ type CloneRepoOptions struct {
99103
}
100104

101105
// Clone clones original repository to target path.
102-
func Clone(from, to string, opts CloneRepoOptions) (err error) {
106+
func Clone(from, to string, opts CloneRepoOptions) error {
103107
return CloneWithContext(DefaultContext, from, to, opts)
104108
}
105109

106110
// CloneWithContext clones original repository to target path.
107-
func CloneWithContext(ctx context.Context, from, to string, opts CloneRepoOptions) (err error) {
111+
func CloneWithContext(ctx context.Context, from, to string, opts CloneRepoOptions) error {
108112
cargs := make([]string, len(GlobalCommandArgs))
109113
copy(cargs, GlobalCommandArgs)
110114
return CloneWithArgs(ctx, from, to, cargs, opts)
@@ -146,8 +150,24 @@ func CloneWithArgs(ctx context.Context, from, to string, args []string, opts Clo
146150
opts.Timeout = -1
147151
}
148152

149-
_, err = cmd.RunTimeout(opts.Timeout)
150-
return err
153+
var envs = os.Environ()
154+
u, err := url.Parse(from)
155+
if err == nil && (strings.EqualFold(u.Scheme, "http") || strings.EqualFold(u.Scheme, "https")) {
156+
if proxy.Match(u.Host) {
157+
envs = append(envs, fmt.Sprintf("https_proxy=%s", proxy.GetProxyURL()))
158+
}
159+
}
160+
161+
var stderr = new(bytes.Buffer)
162+
if err = cmd.RunWithContext(&RunContext{
163+
Timeout: opts.Timeout,
164+
Env: envs,
165+
Stdout: io.Discard,
166+
Stderr: stderr,
167+
}); err != nil {
168+
return ConcatenateError(err, stderr.String())
169+
}
170+
return nil
151171
}
152172

153173
// PullRemoteOptions options when pull from remote

‎modules/lfs/client.go

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,9 @@ type Client interface {
2424
}
2525

2626
// NewClient creates a LFS client
27-
func NewClient(endpoint *url.URL) Client {
27+
func NewClient(endpoint *url.URL, skipTLSVerify bool) Client {
2828
if endpoint.Scheme == "file" {
2929
return newFilesystemClient(endpoint)
3030
}
31-
return newHTTPClient(endpoint)
31+
return newHTTPClient(endpoint, skipTLSVerify)
3232
}

‎modules/lfs/client_test.go

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,10 @@ import (
1313

1414
func TestNewClient(t *testing.T) {
1515
u, _ := url.Parse("file:///test")
16-
c := NewClient(u)
16+
c := NewClient(u, true)
1717
assert.IsType(t, &FilesystemClient{}, c)
1818

1919
u, _ = url.Parse("https://test.com/lfs")
20-
c = NewClient(u)
20+
c = NewClient(u, true)
2121
assert.IsType(t, &HTTPClient{}, c)
2222
}

‎modules/lfs/http_client.go

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ package lfs
77
import (
88
"bytes"
99
"context"
10+
"crypto/tls"
1011
"errors"
1112
"fmt"
1213
"net/http"
@@ -15,6 +16,7 @@ import (
1516

1617
"code.gitea.io/gitea/modules/json"
1718
"code.gitea.io/gitea/modules/log"
19+
"code.gitea.io/gitea/modules/proxy"
1820
)
1921

2022
const batchSize = 20
@@ -32,8 +34,13 @@ func (c *HTTPClient) BatchSize() int {
3234
return batchSize
3335
}
3436

35-
func newHTTPClient(endpoint *url.URL) *HTTPClient {
36-
hc := &http.Client{}
37+
func newHTTPClient(endpoint *url.URL, skipTLSVerify bool) *HTTPClient {
38+
hc := &http.Client{
39+
Transport: &http.Transport{
40+
TLSClientConfig: &tls.Config{InsecureSkipVerify: skipTLSVerify},
41+
Proxy: proxy.Proxy(),
42+
},
43+
}
3744

3845
client := &HTTPClient{
3946
client: hc,

‎modules/migrations/gitea_downloader.go

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ package migrations
66

77
import (
88
"context"
9+
"crypto/tls"
910
"errors"
1011
"fmt"
1112
"io"
@@ -17,6 +18,8 @@ import (
1718
"code.gitea.io/gitea/models"
1819
"code.gitea.io/gitea/modules/log"
1920
"code.gitea.io/gitea/modules/migrations/base"
21+
"code.gitea.io/gitea/modules/proxy"
22+
"code.gitea.io/gitea/modules/setting"
2023
"code.gitea.io/gitea/modules/structs"
2124

2225
gitea_sdk "code.gitea.io/sdk/gitea"
@@ -87,6 +90,12 @@ func NewGiteaDownloader(ctx context.Context, baseURL, repoPath, username, passwo
8790
gitea_sdk.SetToken(token),
8891
gitea_sdk.SetBasicAuth(username, password),
8992
gitea_sdk.SetContext(ctx),
93+
gitea_sdk.SetHTTPClient(&http.Client{
94+
Transport: &http.Transport{
95+
TLSClientConfig: &tls.Config{InsecureSkipVerify: setting.Migrations.SkipTLSVerify},
96+
Proxy: proxy.Proxy(),
97+
},
98+
}),
9099
)
91100
if err != nil {
92101
log.Error(fmt.Sprintf("Failed to create NewGiteaDownloader for: %s. Error: %v", baseURL, err))
@@ -266,6 +275,13 @@ func (g *GiteaDownloader) convertGiteaRelease(rel *gitea_sdk.Release) *base.Rele
266275
Created: rel.CreatedAt,
267276
}
268277

278+
httpClient := &http.Client{
279+
Transport: &http.Transport{
280+
TLSClientConfig: &tls.Config{InsecureSkipVerify: setting.Migrations.SkipTLSVerify},
281+
Proxy: proxy.Proxy(),
282+
},
283+
}
284+
269285
for _, asset := range rel.Attachments {
270286
size := int(asset.Size)
271287
dlCount := int(asset.DownloadCount)
@@ -282,7 +298,11 @@ func (g *GiteaDownloader) convertGiteaRelease(rel *gitea_sdk.Release) *base.Rele
282298
return nil, err
283299
}
284300
// FIXME: for a private download?
285-
resp, err := http.Get(asset.DownloadURL)
301+
req, err := http.NewRequest("GET", asset.DownloadURL, nil)
302+
if err != nil {
303+
return nil, err
304+
}
305+
resp, err := httpClient.Do(req)
286306
if err != nil {
287307
return nil, err
288308
}

‎modules/migrations/github.go

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ package migrations
77

88
import (
99
"context"
10+
"crypto/tls"
1011
"fmt"
1112
"io"
1213
"net/http"
@@ -17,6 +18,8 @@ import (
1718

1819
"code.gitea.io/gitea/modules/log"
1920
"code.gitea.io/gitea/modules/migrations/base"
21+
"code.gitea.io/gitea/modules/proxy"
22+
"code.gitea.io/gitea/modules/setting"
2023
"code.gitea.io/gitea/modules/structs"
2124
"code.gitea.io/gitea/modules/util"
2225

@@ -90,7 +93,7 @@ func NewGithubDownloaderV3(ctx context.Context, baseURL, userName, password, tok
9093
Transport: &http.Transport{
9194
Proxy: func(req *http.Request) (*url.URL, error) {
9295
req.SetBasicAuth(userName, password)
93-
return nil, nil
96+
return proxy.Proxy()(req)
9497
},
9598
},
9699
}
@@ -269,6 +272,13 @@ func (g *GithubDownloaderV3) convertGithubRelease(rel *github.RepositoryRelease)
269272
r.Published = rel.PublishedAt.Time
270273
}
271274

275+
httpClient := &http.Client{
276+
Transport: &http.Transport{
277+
TLSClientConfig: &tls.Config{InsecureSkipVerify: setting.Migrations.SkipTLSVerify},
278+
Proxy: proxy.Proxy(),
279+
},
280+
}
281+
272282
for _, asset := range rel.Assets {
273283
var assetID = *asset.ID // Don't optimize this, for closure we need a local variable
274284
r.Assets = append(r.Assets, &base.ReleaseAsset{
@@ -295,7 +305,7 @@ func (g *GithubDownloaderV3) convertGithubRelease(rel *github.RepositoryRelease)
295305
if err != nil {
296306
return nil, err
297307
}
298-
resp, err := http.DefaultClient.Do(req)
308+
resp, err := httpClient.Do(req)
299309
err1 := g.RefreshRate()
300310
if err1 != nil {
301311
log.Error("g.client.RateLimits: %s", err1)

0 commit comments

Comments
 (0)