enhance: automatically set up credential helpers (#468)

Signed-off-by: Grant Linville <[email protected]>

Author: g-linville

docs/docs/02-credentials.md

@@ -0,0 +1,71 @@
+# Credentials
+
+Some GPTScript tools will use [credential tools](03-tools/04-credential-tools.md) to get sensitive information like API keys from the user.
+These credentials will be stored in a credential store and are used to set environment variables before executing a tool.
+
+GPTScript itself will also prompt you for your OpenAI API key and save it in the credential store if the
+`OPENAI_API_KEY` environment variable is not set. The environment variable always overrides the value stored in the
+credential store.
+
+## Credential Store
+
+There are different options available for credential stores, depending on your operating system.
+When you first run GPTScript, the default credential store for your operating system will be selected.
+
+You can change the credential store by modifying the `credsStore` field in your GPTScript configuration file.
+The configuration file is located in the following location based on your operating system:
+- Windows: `%APPDATA%\Local\gptscript\config.json`
+- macOS: `$HOME/Library/Application Support/gptscript/config.json`
+- Linux: `$XDG_CONFIG_HOME/gptscript/config.json`
+
+(Note: if you set the `XDG_CONFIG_HOME` environment variable on macOS, then the same path as Linux will be used.)
+
+The configured credential store will be automatically downloaded and compiled from the [gptscript-ai/gptscript-credential-helpers](https://github.com/gptscript-ai/gptscript-credential-helpers)
+repository, other than the `file` store, which is built-in to GPTScript itself.
+The `wincred` and `osxkeychain` stores do not require any external dependencies in order to compile correctly.
+The `secretservice` store on Linux may require some extra packages to be installed, depending on your distribution.
+
+## Credential Store Options
+
+### Wincred (Windows)
+
+Wincred, or the Windows Credential Manager, is the default credential store for Windows.
+This is Windows' built-in credential manager that securely stores credentials for Windows applications.
+This credential store is called `wincred` in GPTScript's configuration.
+
+### macOS Keychain (macOS)
+
+The macOS Keychain is the default credential store for macOS.
+This is macOS' built-in password manager that securely stores credentials for macOS applications.
+This credential store is called `osxkeychain` in GPTScript's configuration.
+
+### File (all operating systems)
+
+"File" is the default credential store for every other operating system besides Windows and macOS, but it
+can also be configured on Windows and macOS. This will store credentials **unencrypted** inside GPTScript's
+configuration file.
+This credential store is called `file` in GPTScript's configuration.
+
+### D-Bus Secret Service (Linux)
+
+The D-Bus Secret Service can be used as the credential store for Linux systems with a desktop environment that supports it.
+This credential store is called `secretservice` in GPTScript's configuration.
+
+### Pass (Linux)
+
+Pass can be used as the credential store for Linux systems. This requires the `pass` package to be installed
+and configured. See [this guide](https://www.howtogeek.com/devops/how-to-use-pass-a-command-line-password-manager-for-linux-systems/)
+for information about how to set it up.
+This credential store is called `pass` in GPTScript's configuration.
+
+## GPTScript `credential` Command
+
+The `gptscript credential` command can be used to interact with your stored credentials.
+`gptscript credential` without any arguments will list all stored credentials.
+`gptscript credential delete <credential name>` will delete the specified credential, and you will be
+prompted to enter it again the next time a tool that requires it is run.
+
+## See Also
+
+For more advanced credential usage, including credential contexts, writing credential tools, and using
+credential tools, see [the credential tools documentation](03-tools/04-credential-tools.md).

docs/docs/03-tools/04-credentials.md renamed to docs/docs/03-tools/04-credential-tools.md

@@ -1,4 +1,4 @@
-# Credentials
+# Credential Tools
 
 GPTScript supports credential provider tools. These tools can be used to fetch credentials from a secure location (or
 directly from user input) and conveniently set them in the environment before running a script.
@@ -92,19 +92,8 @@ In this example, the tool's output would be `{"env":{"MY_ENV_VAR":"my value"}}`
 
 ## Storing Credentials
 
-By default, credentials are automatically stored in a config file at `$XDG_CONFIG_HOME/gptscript/config.json`.
-This config file also has another parameter, `credsStore`, which indicates where the credentials are being stored.
-
-- `file` (default): The credentials are stored directly in the config file.
-- `osxkeychain`: The credentials are stored in the macOS Keychain.
-- `wincred`: The credentials are stored in the Windows Credential Manager.
-
-In order to use `osxkeychain` or `wincred` as the credsStore, you must have the `gptscript-credential-*` executable
-available in your PATH. There will probably be better packaging for this in the future, but for now, you can build them
-from the [repo](https://github.com/gptscript-ai/gptscript-credential-helpers). (For wincred, make sure the executable
-is called `gptscript-credential-wincred.exe`.)
-
-There will likely be support added for other credential stores in the future.
+By default, credentials are automatically stored in the credential store. Read the [main credentials page](../02-credentials.md)
+for more information about the credential store.
 
 :::note
 Credentials received from credential provider tools that are not on GitHub (such as a local file) and do not have an alias
@@ -128,7 +117,7 @@ or when you want to store credentials that were provided by a tool that is not o
 ## Credential Contexts
 
 Each stored credential is uniquely identified by the name of its provider tool (or alias, if one was specified) and the name of its context.
-A credential  context is basically a namespace for credentials. If you have multiple credentials from the same provider tool,
+A credential context is basically a namespace for credentials. If you have multiple credentials from the same provider tool,
 you can switch between them by defining them in different credential contexts. The default context is called `default`,
 and this is used if none is specified.
 

pkg/cli/credential.go

@@ -8,6 +8,7 @@ import (
 	"text/tabwriter"
 
 	cmd2 "github.com/acorn-io/cmd"
+	"github.com/gptscript-ai/gptscript/pkg/cache"
 	"github.com/gptscript-ai/gptscript/pkg/config"
 	"github.com/gptscript-ai/gptscript/pkg/credentials"
 	"github.com/spf13/cobra"
@@ -38,7 +39,13 @@ func (c *Credential) Run(_ *cobra.Command, _ []string) error {
 		ctx = "*"
 	}
 
-	store, err := credentials.NewStore(cfg, ctx)
+	opts, err := c.root.NewGPTScriptOpts()
+	if err != nil {
+		return err
+	}
+	opts.Cache = cache.Complete(opts.Cache)
+
+	store, err := credentials.NewStore(cfg, ctx, opts.Cache.CacheDir)
 	if err != nil {
 		return fmt.Errorf("failed to get credentials store: %w", err)
 	}

pkg/cli/credential_delete.go

@@ -3,6 +3,7 @@ package cli
 import (
 	"fmt"
 
+	"github.com/gptscript-ai/gptscript/pkg/cache"
 	"github.com/gptscript-ai/gptscript/pkg/config"
 	"github.com/gptscript-ai/gptscript/pkg/credentials"
 	"github.com/spf13/cobra"
@@ -21,12 +22,18 @@ func (c *Delete) Customize(cmd *cobra.Command) {
 }
 
 func (c *Delete) Run(_ *cobra.Command, args []string) error {
+	opts, err := c.root.NewGPTScriptOpts()
+	if err != nil {
+		return err
+	}
+	opts.Cache = cache.Complete(opts.Cache)
+
 	cfg, err := config.ReadCLIConfig(c.root.ConfigFile)
 	if err != nil {
 		return fmt.Errorf("failed to read CLI config: %w", err)
 	}
 
-	store, err := credentials.NewStore(cfg, c.root.CredentialContext)
+	store, err := credentials.NewStore(cfg, c.root.CredentialContext, opts.Cache.CacheDir)
 	if err != nil {
 		return fmt.Errorf("failed to get credentials store: %w", err)
 	}

pkg/config/cliconfig.go

@@ -3,17 +3,24 @@ package config
 import (
 	"encoding/base64"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"os"
-	"os/exec"
 	"runtime"
+	"slices"
 	"strings"
 	"sync"
 
 	"github.com/adrg/xdg"
 	"github.com/docker/cli/cli/config/types"
 )
 
+var (
+	darwinHelpers  = []string{"osxkeychain", "file"}
+	windowsHelpers = []string{"wincred", "file"}
+	linuxHelpers   = []string{"secretservice", "pass", "file"}
+)
+
 const GPTScriptHelperPrefix = "gptscript-credential-"
 
 type AuthConfig types.AuthConfig
@@ -132,21 +139,54 @@ func ReadCLIConfig(gptscriptConfigFile string) (*CLIConfig, error) {
 	}
 
 	if result.CredentialsStore == "" {
-		result.setDefaultCredentialsStore()
+		if err := result.setDefaultCredentialsStore(); err != nil {
+			return nil, err
+		}
+	}
+
+	if !isValidCredentialHelper(result.CredentialsStore) {
+		errMsg := fmt.Sprintf("invalid credential store '%s'", result.CredentialsStore)
+		switch runtime.GOOS {
+		case "darwin":
+			errMsg += " (use 'osxkeychain' or 'file')"
+		case "windows":
+			errMsg += " (use 'wincred' or 'file')"
+		case "linux":
+			errMsg += " (use 'secretservice', 'pass', or 'file')"
+		default:
+			errMsg += " (use 'file')"
+		}
+		errMsg += fmt.Sprintf("\nPlease edit your config file at %s to fix this.", result.GPTScriptConfigFile)
+
+		return nil, errors.New(errMsg)
 	}
 
 	return result, nil
 }
 
-func (c *CLIConfig) setDefaultCredentialsStore() {
-	if runtime.GOOS == "darwin" {
-		// Check for the existence of the helper program
-		fullPath, err := exec.LookPath(GPTScriptHelperPrefix + "osxkeychain")
-		if err == nil && fullPath != "" {
-			c.CredentialsStore = "osxkeychain"
-		}
+func (c *CLIConfig) setDefaultCredentialsStore() error {
+	switch runtime.GOOS {
+	case "darwin":
+		c.CredentialsStore = "osxkeychain"
+	case "windows":
+		c.CredentialsStore = "wincred"
+	default:
+		c.CredentialsStore = "file"
+	}
+	return c.Save()
+}
+
+func isValidCredentialHelper(helper string) bool {
+	switch runtime.GOOS {
+	case "darwin":
+		return slices.Contains(darwinHelpers, helper)
+	case "windows":
+		return slices.Contains(windowsHelpers, helper)
+	case "linux":
+		return slices.Contains(linuxHelpers, helper)
+	default:
+		return helper == "file"
 	}
-	c.CredentialsStore = "file"
 }
 
 func readFile(path string) ([]byte, error) {

pkg/credentials/noop.go

@@ -0,0 +1,19 @@
+package credentials
+
+type NoopStore struct{}
+
+func (s NoopStore) Get(_ string) (*Credential, bool, error) {
+	return nil, false, nil
+}
+
+func (s NoopStore) Add(_ Credential) error {
+	return nil
+}
+
+func (s NoopStore) Remove(_ string) error {
+	return nil
+}
+
+func (s NoopStore) List() ([]Credential, error) {
+	return nil, nil
+}

pkg/credentials/store.go

@@ -2,28 +2,39 @@ package credentials
 
 import (
 	"fmt"
+	"path/filepath"
 	"regexp"
+	"strings"
 
 	"github.com/docker/cli/cli/config/credentials"
 	"github.com/gptscript-ai/gptscript/pkg/config"
 )
 
+type CredentialStore interface {
+	Get(toolName string) (*Credential, bool, error)
+	Add(cred Credential) error
+	Remove(toolName string) error
+	List() ([]Credential, error)
+}
+
 type Store struct {
-	credCtx string
-	cfg     *config.CLIConfig
+	credCtx        string
+	credHelperDirs CredentialHelperDirs
+	cfg            *config.CLIConfig
 }
 
-func NewStore(cfg *config.CLIConfig, credCtx string) (*Store, error) {
+func NewStore(cfg *config.CLIConfig, credCtx, cacheDir string) (CredentialStore, error) {
 	if err := validateCredentialCtx(credCtx); err != nil {
 		return nil, err
 	}
-	return &Store{
-		credCtx: credCtx,
-		cfg:     cfg,
+	return Store{
+		credCtx:        credCtx,
+		credHelperDirs: GetCredentialHelperDirs(cacheDir),
+		cfg:            cfg,
 	}, nil
 }
 
-func (s *Store) Get(toolName string) (*Credential, bool, error) {
+func (s Store) Get(toolName string) (*Credential, bool, error) {
 	store, err := s.getStore()
 	if err != nil {
 		return nil, false, err
@@ -46,7 +57,7 @@ func (s *Store) Get(toolName string) (*Credential, bool, error) {
 	return &cred, true, nil
 }
 
-func (s *Store) Add(cred Credential) error {
+func (s Store) Add(cred Credential) error {
 	cred.Context = s.credCtx
 	store, err := s.getStore()
 	if err != nil {
@@ -59,15 +70,15 @@ func (s *Store) Add(cred Credential) error {
 	return store.Store(auth)
 }
 
-func (s *Store) Remove(toolName string) error {
+func (s Store) Remove(toolName string) error {
 	store, err := s.getStore()
 	if err != nil {
 		return err
 	}
 	return store.Erase(toolNameWithCtx(toolName, s.credCtx))
 }
 
-func (s *Store) List() ([]Credential, error) {
+func (s Store) List() ([]Credential, error) {
 	store, err := s.getStore()
 	if err != nil {
 		return nil, err
@@ -103,6 +114,12 @@ func (s *Store) getStoreByHelper(helper string) (credentials.Store, error) {
 	if helper == "" || helper == config.GPTScriptHelperPrefix+"file" {
 		return credentials.NewFileStore(s.cfg), nil
 	}
+
+	// If the helper is referencing one of the credential helper programs, then reference the full path.
+	if strings.HasPrefix(helper, "gptscript-credential-") {
+		helper = filepath.Join(s.credHelperDirs.BinDir, helper)
+	}
+
 	return NewHelper(s.cfg, helper)
 }
 

pkg/credentials/util.go

@@ -0,0 +1,17 @@
+package credentials
+
+import (
+	"path/filepath"
+)
+
+type CredentialHelperDirs struct {
+	RevisionFile, BinDir, RepoDir string
+}
+
+func GetCredentialHelperDirs(cacheDir string) CredentialHelperDirs {
+	return CredentialHelperDirs{
+		RevisionFile: filepath.Join(cacheDir, "repos", "gptscript-credential-helpers", "revision"),
+		BinDir:       filepath.Join(cacheDir, "repos", "gptscript-credential-helpers", "bin"),
+		RepoDir:      filepath.Join(cacheDir, "repos", "gptscript-credential-helpers", "repo"),
+	}
+}

pkg/engine/engine.go

@@ -7,6 +7,7 @@ import (
 	"strings"
 	"sync"
 
+	"github.com/gptscript-ai/gptscript/pkg/config"
 	"github.com/gptscript-ai/gptscript/pkg/counter"
 	"github.com/gptscript-ai/gptscript/pkg/system"
 	"github.com/gptscript-ai/gptscript/pkg/types"
@@ -19,6 +20,7 @@ type Model interface {
 
 type RuntimeManager interface {
 	GetContext(ctx context.Context, tool types.Tool, cmd, env []string) (string, []string, error)
+	SetUpCredentialHelpers(ctx context.Context, cliCfg *config.CLIConfig, env []string) error
 }
 
 type Engine struct {