Add find and exec

Author: ibuildthecloud

README.md

@@ -1,81 +1,55 @@
-# README for GPT File Syntax
+# README.md for GPTscript Project
 
-## Overview
-GPT files found in this project use a custom syntax to describe various shell tools and their respective functionalities. Each file contains definitions for tools including their names, descriptions, arguments, and executable code. Below is an explanation of the syntax based on the contents of the GPT files reviewed (`./describe.gpt`, `./summarize-syntax.gpt`, `./tables.gpt`).
+## Project Description
 
-## Syntax Description
+This project utilizes `GPTscript`, a powerful scripting tool that leverages OpenAI's GPT plugins to execute tasks specified in `.gpt` files. These scripts can combine traditional scripting with GPT's AI capabilities to automate complex workflows, parse information, and more.
 
-### General Format
-GPT files follow a structured format that includes a preamble (indicating available tools), followed by metadata (descriptions of each tool), and Bash script examples for the tool's operation. 
+## Usage
 
-Here's the general layout of a GPT file definition:
-```
-Tools: <tool_list>
-<empty_line>
---- (3 hyphens to separate each tool definition)
-name: <tool_name>
-description: <description_of_tool>
-arg: <argument_description> (Optional and re-occurring)
-<empty_line>
-<executable_code_block>
-```
+The primary command used to run GPT scripts is `gptscript`, which can be invoked with various flags to control the script execution. A typical usage pattern looks like this:
 
-### Tool Definition Header
-A tool definition always starts with three hyphens (`---`). This is a separator used to distinguish between different tool descriptions within the file.
+```bash
+gptscript [flags] PROGRAM_FILE [INPUT...]
+```
 
-### Name
-The `name` field defines the name of the tool. It is used to invoke a specific functionality.
+Where `PROGRAM_FILE` is the script to be executed, optionally followed by `INPUT` arguments to pass into the script.
 
-### Description
-The `description` field provides a concise explanation of what the tool does.
+## GPT File Syntax
 
-### Arguments
-The `arg` fields describe the arguments that the tool accepts. Each argument is paired with a brief description. Argument definitions are optional and can be repeated if a tool accepts multiple arguments.
+GPT script files (`*.gpt`) follow a syntax which allows for defining tools, their descriptions, arguments, and embedded traditional scripting codes. Here's a breakdown of the syntax:
 
-### Executable Code Block
-Following the metadata, an executable code block is provided. It is written in Bash and contains the script to execute the tool's functionality.
+- `tools:` Followed by the tool names to be used.
+- `name:` Defines the tool's name.
+- `description:` Describes what the tool does.
+- `args:` Lists arguments the tool accepts, describing the expected input.
+- `tools:` Below the main tool definition, you can specify other tools used by this tool.
+- A line containing `---` signifies the separation between tool metadata and the script code itself.
 
-### Examples
-Here are examples extracted from the GPT files which follow the syntax described:
+## Example `.gpt` Files and Their Syntax
 
-#### From `./describe.gpt`
-```
----
-name: ls
-description: Recursively lists all go files
+The repository contains several example `.gpt` files, such as:
 
-#!/bin/bash
-
-find . -name '*.go'
-```
-#### From `./summarize-syntax.gpt`
-```
----
-name: cat
-description: Reads the contents of a file
-arg: file: The filename to read
+- `describe.gpt`: Provides functionalities for working with Go files, with tools like `ls`, `count`, `summarize`, and `compare`.
+- `echo.gpt`: A simple script that echoes the provided input.
+- `fib.gpt`: An example demonstrating a recursive function `myfunction`.
+- `git-commit.gpt`: Automates the creation of a well-formed git commit message.
+- `helloworld.gpt`: A basic script outputs the traditional "hello world" message.
+- `summarize-syntax.gpt`: Meta-script for generating documentation based on `.gpt` files in a directory.
+- `tables.gpt`: Using SQLite, identifies the table in a database with the most rows.
 
-#!/bin/bash
-
-cat ${FILE} </dev/null
-```
-#### From `./tables.gpt`
-```
----
-Name: sqlite
-Description: The sqlite command line program ... to run sqlite command or SQL statements against a database.
-Arg: databaseFile: The filename of the database to open
-Arg: cmd: The sqlite command or sql statement to run.
-
-#!/bin/bash
-
-sqlite3 ${DATABASEFILE} -cmd "${CMD}" </dev/null
-```
+## gptscript Command Help
 
-## Additional Observations
-- The fields and code block are separated by a new line.
-- The arguments are indicated as `arg: argument_name: argument_description`.
-- The executable code is introduced with a `#!/bin/bash` shebang line and is expected to replace the argument placeholders (`${NAME}`) with actual values during execution.
+The help output for `gptscript` provides vital information on how to use the command along with the available flags. Here's an abbreviated listing of the flags:
 
-Please note that this description of the syntax is based on the provided examples and may not cover variations which are not included in those files. Additional GPT files may include diverse structures and should be reviewed to check for consistency with this syntax.
+- `--assemble`: Assemble tool to a single artifact, with output specified by `--output`.
+- `--cache`, `--cache-dir`: Controls caching behavior and cache directory location.
+- `--debug`: Enable debug logging.
+- `--help`: Displays help information for `gptscript`.
+- `--input`: Specify an input file or stdin.
+- `--list-models`, `--list-tools`: Lists available models or built-in tools.
+- `--openai-api-key`: Specifies the OpenAI API key to use.
+- `--output`: Saves output to a specified file.
+- `--quiet`: Suppresses output logging.
+- `--sub-tool`: Selects a specific sub-tool to use from the `.gpt` file.
 
+For a detailed description and more options, refer to the `gptscript --help` command.

examples/summarize-syntax.gpt

@@ -1,19 +1,9 @@
-tools: ls, sys.read, help
+tools: sys.find, sys.read, sys.exec
 
 Write a README.md for this github repo that describes this project, it's usage, and a description of the
-GPT file syntax. Example GPT files can be found by listing and read them. Also document the help of the
-gptscript command.
+GPT file syntax. 
 
----
-name: ls
-description: list all gpt files
+Example GPT files can be found by looking for all *.gpt files in the current directory. Analyze
+the contents of those files to discover the syntax and then describe how the syntax works.
 
-#!/bin/bash
-
-find . -name '*.gpt'
-
----
-name: help
-description: Get the help output of gptscript
-
-#!gptscript --help
+Also document the help of the gptscript command which can be obtained from running "gptscript --help".

pkg/builtin/builtin.go

@@ -5,8 +5,11 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
+	"io/fs"
 	"net/http"
 	"os"
+	"os/exec"
+	"path/filepath"
 	"sort"
 	"strings"
 
@@ -48,6 +51,22 @@ var Tools = map[string]types.Tool{
 			"contentType", "The \"content type\" of the content such as application/json or text/plain"),
 		BuiltinFunc: SysHTTPPost,
 	},
+	"sys.find": {
+		Description: "Traverse a directory looking for files that match a pattern in the style of the unix find command",
+		Arguments: types.ObjectSchema(
+			"pattern", "The file pattern to look for. The pattern is a traditional unix glob format with * matching any character and ? matching a single character",
+			"directory", "The directory to search in. The current directory \".\" will be used as the default if no argument is passed",
+		),
+		BuiltinFunc: SysFind,
+	},
+	"sys.exec": {
+		Description: "Execute a command and get the output of the command",
+		Arguments: types.ObjectSchema(
+			"command", "The command to run including all applicable arguments",
+			"directory", "The directory to use as the current working directory of the command. The current directory \".\" will be used if no argument is passed",
+		),
+		BuiltinFunc: SysExec,
+	},
 }
 
 func ListTools() (result []types.Tool) {
@@ -73,6 +92,64 @@ func Builtin(name string) (types.Tool, bool) {
 	return t, ok
 }
 
+func SysFind(ctx context.Context, env []string, input string) (string, error) {
+	var result []string
+	var params struct {
+		Pattern   string `json:"pattern,omitempty"`
+		Directory string `json:"directory,omitempty"`
+	}
+	if err := json.Unmarshal([]byte(input), &params); err != nil {
+		return "", err
+	}
+
+	if params.Directory == "" {
+		params.Directory = "."
+	}
+
+	log.Debugf("Finding files %s in %s", params.Pattern, params.Directory)
+	err := fs.WalkDir(os.DirFS(params.Directory), params.Directory, func(pathname string, d fs.DirEntry, err error) error {
+		if err != nil {
+			return err
+		}
+		if d.IsDir() {
+			return nil
+		}
+		if ok, err := filepath.Match(params.Pattern, d.Name()); err != nil {
+			return err
+		} else if ok {
+			result = append(result, filepath.Join(pathname))
+		}
+		return nil
+	})
+	if err != nil {
+		return "", nil
+	}
+	sort.Strings(result)
+	return strings.Join(result, "\n"), nil
+}
+
+func SysExec(ctx context.Context, env []string, input string) (string, error) {
+	var params struct {
+		Command   string `json:"command,omitempty"`
+		Directory string `json:"directory,omitempty"`
+	}
+	if err := json.Unmarshal([]byte(input), &params); err != nil {
+		return "", err
+	}
+
+	if params.Directory == "" {
+		params.Directory = "."
+	}
+
+	log.Debugf("Running %s in %s", params.Command, params.Directory)
+
+	cmd := exec.Command("/bin/sh", "-c", params.Command)
+	cmd.Env = env
+	cmd.Dir = params.Directory
+	out, err := cmd.CombinedOutput()
+	return string(out), err
+}
+
 func SysRead(ctx context.Context, env []string, input string) (string, error) {
 	var params struct {
 		Filename string `json:"filename,omitempty"`