diff --git a/docs/guide/executables.md b/docs/guide/executables.md index 4f6ffa81..2a9ca089 100644 --- a/docs/guide/executables.md +++ b/docs/guide/executables.md @@ -387,8 +387,7 @@ You can use special comments to override executable metadata: # f:name=production f:verb=deploy # f:description="Deploy to production environment" -# f:tags=production,critical -# f:aliases=prod-deploy +# f:tag=production f:tag=critical # f:visibility=internal # f:timeout=10m @@ -396,17 +395,7 @@ echo "Deploying to production..." kubectl apply -f k8s/ ``` -**Supported comment keys:** -- `name`, `verb`, `description`, `tags`, `aliases`, `visibility`, `timeout` - -**Multi-line descriptions:** -```bash -# f:name=backup f:verb=run -# -# Creates a backup of the database -# and uploads it to S3 storage -# -``` +See the [generated configuration reference](generated-config.md) for more details. #### **Makefiles** @@ -428,7 +417,7 @@ clean: rm -rf bin/ ``` -The same comment parsing syntax works in Makefiles - use `# f:key=value` to override executable configuration. +See the [generated configuration reference](generated-config.md) for more details on overriding executable configuration. #### **Package.json Scripts** diff --git a/docs/guide/generated-config.md b/docs/guide/generated-config.md new file mode 100644 index 00000000..25dd4964 --- /dev/null +++ b/docs/guide/generated-config.md @@ -0,0 +1,98 @@ +# Imported Executables Config Reference + +flow can automatically generate executables from shell scripts and Makefiles using special comments. +flow parses these comments during workspace synchronization and creates executable definitions that can be run +like any other flow executable. See [Importing Executables](../guide/executables.md#importing-executables) for more details. + +> [!NOTE] The configuration comments must be at the top of the shell script or right above the Makefile target definition. + +## Supported Fields + +| Field | Description | Example | +|--------------------|-------------|---------| +| `name` | Executable name | `f:name=deploy-app` | +| `verb` | Action verb | `f:verb=deploy` | +| `description` | Executable description | `f:description=Deploy to production` | +| `tag` or `tags` | Pipe-separated tags | `f:tags=deployment\|production` | +| `alias`or `aliases` | Pipe-separated aliases | `f:aliases=prod-deploy\|deploy-prod` | +| `timeout` | Execution timeout | `f:timeout=10m` | +| `visibility` | Executable visibility | `f:visibility=private` | +| `dir` | Working directory | `f:dir=//` | +| `logMode` | Log output format | `f:logMode=json` | + +### Environment Parameters + +Define environment variables that will be available to your script with `f:params` or `f:param`: + +```bash +#!/bin/bash +# f:name=deploy-with-secrets f:verb=deploy +# f:params=secretRef:api-key:API_TOKEN|prompt:Environment?:ENV_NAME|text:production:DEFAULT_ENV + +echo "Deploying to $ENV_NAME with token: ${API_TOKEN:0:8}..." +``` + +**Parameter Types:** +- `secretRef:secret-name:ENV_VAR` - Reference a vault secret +- `prompt:Question text:ENV_VAR` - Prompt user for input +- `text:static-value:ENV_VAR` - Set static value + +### Command Line Arguments + +Define command line arguments that users can pass when running the executable with `f:args` or `f:arg`: + +```bash +#!/bin/bash +# f:name=build-app f:verb=build +# f:args=flag:dry-run:DRY_RUN|pos:1:VERSION|flag:verbose:VERBOSE + +if [ "$DRY_RUN" = "true" ]; then + echo "DRY RUN: Would build version $VERSION" +else + echo "Building version $VERSION" +fi +``` + +**Argument Types:** +- `flag:flag-name:ENV_VAR` - Named flag (`--flag-name`) +- `pos:1:ENV_VAR` - Positional argument (position 1, 2, etc.) + +## Configuration Syntax + +**Single Line Format** + +Multiple configurations can be defined on a single line: + +```bash +# f:name=my-task f:verb=run f:timeout=5m f:visibility=private +``` + +**Multi-Line Format** + +Configurations can be split across multiple lines for readability: + +```bash +# f:name=complex-task +# f:verb=deploy +# f:description="Complex deployment with multiple parameters" +# f:params=secretRef:api-key:API_TOKEN +# f:params=prompt:Target environment?:ENV_NAME +# f:args=flag:dry-run:DRY_RUN +# f:args=pos:1:VERSION +``` + +**Multi-Line Descriptions** + +For longer descriptions, use the multi-line description syntax: + +```bash +# f:name=complex-deploy f:verb=deploy +# +# Deploy application to production environment +# +# This executable handles the complete deployment process including: +# - Database migrations +# - Service deployment +# - Health checks +# +``` diff --git a/internal/fileparser/config.go b/internal/fileparser/config.go new file mode 100644 index 00000000..8c22b69b --- /dev/null +++ b/internal/fileparser/config.go @@ -0,0 +1,471 @@ +package fileparser + +import ( + "fmt" + "regexp" + "strconv" + "strings" + "time" + + "github.com/flowexec/tuikit/io" + "github.com/pkg/errors" + + "github.com/flowexec/flow/types/executable" +) + +const ( + TimeoutConfigurationKey = "timeout" + VerbConfigurationKey = "verb" + NameConfigurationKey = "name" + AliasConfigurationKey = "alias" + DescriptionConfigurationKey = "description" + VisibilityConfigurationKey = "visibility" + TagConfigurationKey = "tag" + ParamConfigurationKey = "param" + ArgConfigurationKey = "arg" + DirConfigurationKey = "dir" + LogModeConfigurationKey = "logmode" + + InternalListSeparator = "|" + InternalValueSeparator = ":" + + commentPrefix = "# " + multiLineKeyPrefix = "f|" + descriptionSeparator = "\n" + descriptionAlias = "desc" +) + +var multiLineDescriptionTag = fmt.Sprintf("<%s%s>", multiLineKeyPrefix, DescriptionConfigurationKey) + +// Regex to extract flow configuration fields from comment lines. +var flowConfigStartRegex = regexp.MustCompile(`f:(\w+)=`) + +type ParseResult struct { + SimpleFields map[string]string + Params executable.ParameterList + Args executable.ArgumentList +} + +func ExtractExecConfig(data, prefix string) (*ParseResult, error) { + result := &ParseResult{ + SimpleFields: make(map[string]string), + Params: make(executable.ParameterList, 0), + Args: make(executable.ArgumentList, 0), + } + processingMultiLineDescription := false + for _, line := range strings.Split(data, "\n") { + isComment := strings.HasPrefix(line, strings.TrimSpace(prefix)) + if trimmedLine := strings.TrimSpace(line); !isComment && trimmedLine != "" { + // If the line is not a comment or empty, break out of the loop. + // All flow executable configuration should be at the top of the file. + break + } + + line = strings.TrimPrefix(line, commentPrefix) + if processingMultiLineDescription = parseMultiLineDescription( + line, result.SimpleFields, processingMultiLineDescription, + ); processingMultiLineDescription { + continue + } + + lineResult, err := parseConfigurations(line) + if err != nil { + return nil, fmt.Errorf("unable to extract executable configurations: %w", err) + } + + result.Params = append(result.Params, lineResult.Params...) + result.Args = append(result.Args, lineResult.Args...) + for key, value := range lineResult.SimpleFields { + switch key { + case DescriptionConfigurationKey: + if existingValue, ok := result.SimpleFields[DescriptionConfigurationKey]; ok { + result.SimpleFields[DescriptionConfigurationKey] = + fmt.Sprintf("%s%s%s", existingValue, descriptionSeparator, value) + } else { + result.SimpleFields[DescriptionConfigurationKey] = value + } + case AliasConfigurationKey, TagConfigurationKey: + value = strings.TrimSpace(value) + if existingValue, ok := result.SimpleFields[key]; ok { + result.SimpleFields[key] = fmt.Sprintf("%s%s%s", existingValue, InternalListSeparator, value) + } else { + result.SimpleFields[key] = value + } + default: + result.SimpleFields[key] = strings.TrimSpace(value) + } + } + } + return result, nil +} + +func ApplyExecConfig(exec *executable.Executable, result *ParseResult) error { + if exec.Exec == nil { + exec.Exec = &executable.ExecExecutableType{} + } + + exec.Exec.Params = result.Params + exec.Exec.Args = result.Args + + for key, value := range result.SimpleFields { + switch key { + case TimeoutConfigurationKey: + dur, err := time.ParseDuration(value) + if err != nil { + return errors.Wrapf(err, "unable to parse timeout duration %s", value) + } + exec.Timeout = &dur + case VerbConfigurationKey: + exec.Verb = executable.Verb(value) + case NameConfigurationKey: + exec.Name = value + case VisibilityConfigurationKey: + v := executable.ExecutableVisibility(value) + exec.Visibility = &v + case DescriptionConfigurationKey: + exec.Description = value + case AliasConfigurationKey: + if value != "" { + exec.Aliases = strings.Split(value, InternalListSeparator) + } + case TagConfigurationKey: + if value != "" { + exec.Tags = strings.Split(value, InternalListSeparator) + } + case DirConfigurationKey: + exec.Exec.Dir = executable.Directory(value) + case LogModeConfigurationKey: + exec.Exec.LogMode = io.LogMode(value) + } + } + + return nil +} + +func parseConfigurations(line string) (*ParseResult, error) { + result := &ParseResult{ + SimpleFields: make(map[string]string), + Params: make(executable.ParameterList, 0), + Args: make(executable.ArgumentList, 0), + } + + matches := flowConfigStartRegex.FindAllStringSubmatchIndex(line, -1) + if len(matches) == 0 { + return result, nil + } + + for i, match := range matches { + key := strings.TrimSpace(strings.ToLower(line[match[2]:match[3]])) + + valueStart := match[1] + var valueEnd int + if i+1 < len(matches) { + valueEnd = matches[i+1][0] // Start of next f:key= + } else { + valueEnd = len(line) + } + + rawValue := strings.TrimSpace(line[valueStart:valueEnd]) + if rawValue == "" { + continue + } + + value := cleanValue(rawValue) + normalizedKey := normalizeKey(key) + + if !validateKey(normalizedKey) { + return nil, fmt.Errorf("invalid key (%s)", key) + } + + switch normalizedKey { + case ParamConfigurationKey: + params, err := parseParams(value) + if err != nil { + return nil, fmt.Errorf("error parsing params in line '%s': %w", line, err) + } + result.Params = append(result.Params, params...) + + case ArgConfigurationKey: + args, err := parseArgs(value) + if err != nil { + return nil, fmt.Errorf("error parsing args in line '%s': %w", line, err) + } + result.Args = append(result.Args, args...) + + default: + processSimpleField(result.SimpleFields, normalizedKey, value) + } + } + + return result, nil +} + +func parseMultiLineDescription(line string, configMap map[string]string, processing bool) bool { + multiLineDescPrefixed := strings.HasPrefix(line, multiLineDescriptionTag) + multiLineDescSuffixed := strings.HasSuffix(line, multiLineDescriptionTag) + existingValue := configMap[DescriptionConfigurationKey] + switch { + case processing && multiLineDescPrefixed: + processing = false + case processing && multiLineDescSuffixed: + processing = false + line = strings.TrimSuffix(line, multiLineDescriptionTag) + if existingValue != "" { + configMap[DescriptionConfigurationKey] = fmt.Sprintf( + "%s%s%s", + existingValue, + descriptionSeparator, + line, + ) + } else { + configMap[DescriptionConfigurationKey] = line + } + case processing: + if existingValue != "" { + configMap[DescriptionConfigurationKey] = fmt.Sprintf( + "%s%s%s", + existingValue, + descriptionSeparator, + line, + ) + } else { + configMap[DescriptionConfigurationKey] = line + } + case multiLineDescPrefixed: + processing = true + line = strings.TrimPrefix(line, multiLineDescriptionTag) + if trimmedLine := strings.TrimSpace(line); trimmedLine == "" { + return processing + } else if existingValue != "" { + configMap[DescriptionConfigurationKey] = fmt.Sprintf( + "%s%s%s", + existingValue, + descriptionSeparator, + line, + ) + } else { + configMap[DescriptionConfigurationKey] = line + } + } + return processing +} + +func parseParams(value string) (executable.ParameterList, error) { + if strings.TrimSpace(value) == "" { + return executable.ParameterList{}, nil + } + + var params executable.ParameterList + items := splitValue(value, InternalListSeparator) + + for i, item := range items { + item = strings.TrimSpace(item) + if item == "" { + continue + } + + fields := splitValue(item, InternalValueSeparator) + if len(fields) < 3 { + return nil, fmt.Errorf("param %d requires at least 3 fields (type:value:envKey), got %d: %s", + i+1, len(fields), item) + } + + paramType := strings.TrimSpace(fields[0]) + paramValue := cleanValue(strings.TrimSpace(fields[1])) + envKey := strings.TrimSpace(fields[2]) + + param := executable.Parameter{ + EnvKey: envKey, + } + + switch paramType { + case "secretRef": + param.SecretRef = paramValue + case "prompt": + param.Prompt = paramValue + case "text": + param.Text = paramValue + default: + return nil, fmt.Errorf("invalid parameter type: %s (expected secretRef, prompt, or text)", paramType) + } + + if err := param.Validate(); err != nil { + return nil, fmt.Errorf("error validating parameter %d: %w", i+1, err) + } + + params = append(params, param) + } + + return params, nil +} + +func parseArgs(value string) (executable.ArgumentList, error) { + if strings.TrimSpace(value) == "" { + return executable.ArgumentList{}, nil + } + + var args executable.ArgumentList + items := splitValue(value, InternalListSeparator) + + for i, item := range items { + item = strings.TrimSpace(item) + if item == "" { + continue + } + + fields := splitValue(item, InternalValueSeparator) + if len(fields) != 3 { + return nil, fmt.Errorf("arg %d requires exactly 3 fields (type:name:envKey), got %d: %s", + i+1, len(fields), item) + } + + argType := strings.TrimSpace(fields[0]) + typeVal := cleanValue(strings.TrimSpace(fields[1])) + envKey := strings.TrimSpace(fields[2]) + + arg := executable.Argument{ + EnvKey: envKey, + Type: executable.ArgumentTypeString, + } + + switch argType { + case "flag": + arg.Flag = typeVal + case "pos": + pos, err := strconv.Atoi(typeVal) + if err != nil { + return nil, fmt.Errorf("invalid position number: %s", typeVal) + } + arg.Pos = pos + default: + return nil, fmt.Errorf("invalid argument type: %s (expected flag or pos)", argType) + } + + if err := arg.Validate(); err != nil { + return nil, fmt.Errorf("error validating argument %d: %w", i+1, err) + } + + args = append(args, arg) + } + + return args, nil +} + +func splitValue(s, delimiter string) []string { + if s == "" { + return []string{} + } + + var result []string + var current strings.Builder + escaped := false + + runes := []rune(s) + delimiterRunes := []rune(delimiter) + + for i := 0; i < len(runes); i++ { + if escaped { + current.WriteRune(runes[i]) + escaped = false + continue + } + + if runes[i] == '\\' { + escaped = true + current.WriteRune(runes[i]) // Keep the backslash for later processing + continue + } + + // Check if we're at the start of a delimiter + if i+len(delimiterRunes) <= len(runes) { + match := true + for j, delimRune := range delimiterRunes { + if runes[i+j] != delimRune { + match = false + break + } + } + + if match { + result = append(result, current.String()) + current.Reset() + i += len(delimiterRunes) - 1 // Skip the delimiter + continue + } + } + + current.WriteRune(runes[i]) + } + + // Add the last part + if current.Len() > 0 || len(result) == 0 { + result = append(result, current.String()) + } + + return result +} + +func normalizeKey(key string) string { + switch key { + case "tags": + return TagConfigurationKey + case "aliases": + return AliasConfigurationKey + case "params": + return ParamConfigurationKey + case "args": + return ArgConfigurationKey + case descriptionAlias: + return DescriptionConfigurationKey + default: + return key + } +} + +func cleanValue(value string) string { + if len(value) >= 2 && + ((value[0] == '"' && value[len(value)-1] == '"') || + (value[0] == '\'' && value[len(value)-1] == '\'')) { + value = value[1 : len(value)-1] + } + + // Unescape characters in the correct order + value = strings.ReplaceAll(value, `\|`, `|`) + value = strings.ReplaceAll(value, `\:`, `:`) + value = strings.ReplaceAll(value, `\"`, `"`) + value = strings.ReplaceAll(value, `\'`, `'`) + value = strings.ReplaceAll(value, `\\`, `\`) + + return strings.TrimSpace(value) +} + +func processSimpleField(fields map[string]string, key, value string) { + switch key { + case AliasConfigurationKey, TagConfigurationKey: + if existing, ok := fields[key]; ok { + fields[key] = fmt.Sprintf("%s|%s", existing, value) + } else { + fields[key] = value + } + case DescriptionConfigurationKey, descriptionAlias: + if existing, ok := fields[DescriptionConfigurationKey]; ok { + fields[DescriptionConfigurationKey] = fmt.Sprintf("%s%s%s", existing, descriptionSeparator, value) + } else { + fields[DescriptionConfigurationKey] = value + } + default: + fields[key] = value + } +} + +func validateKey(key string) bool { + switch key { + case VerbConfigurationKey, NameConfigurationKey, DescriptionConfigurationKey, descriptionAlias, + AliasConfigurationKey, VisibilityConfigurationKey, TagConfigurationKey, + TimeoutConfigurationKey, ParamConfigurationKey, ArgConfigurationKey, + DirConfigurationKey, LogModeConfigurationKey: + return true + default: + return false + } +} diff --git a/internal/fileparser/config_test.go b/internal/fileparser/config_test.go new file mode 100644 index 00000000..6fcd85ae --- /dev/null +++ b/internal/fileparser/config_test.go @@ -0,0 +1,210 @@ +package fileparser_test + +import ( + "os" + "path/filepath" + "slices" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + + "github.com/flowexec/flow/internal/fileparser" + "github.com/flowexec/flow/types/executable" +) + +var _ = Describe("ExtractExecConfig", func() { + DescribeTable("should correctly parse simple configurations", + func(file string, expectedFields map[string]string) { + filePath := filepath.Join("testdata", file) + fileBytes, err := os.ReadFile(filepath.Clean(filePath)) + Expect(err).ToNot(HaveOccurred()) + + result, err := fileparser.ExtractExecConfig(string(fileBytes), "# ") + Expect(err).ToNot(HaveOccurred()) + Expect(result.SimpleFields).To(Equal(expectedFields)) + Expect(result.Params).To(BeEmpty()) + Expect(result.Args).To(BeEmpty()) + }, + Entry( + "simple key-value pairs", + "simple.sh", + map[string]string{ + fileparser.NameConfigurationKey: "hello", + fileparser.VerbConfigurationKey: "show", + }), + Entry( + "values with spaces in quotes", + "quoted.sh", + map[string]string{ + fileparser.NameConfigurationKey: "value 1", + fileparser.VerbConfigurationKey: "value2", + fileparser.DescriptionConfigurationKey: "value 3", + }), + Entry( + "values with escaped characters", + "escaped.sh", + map[string]string{ + fileparser.NameConfigurationKey: "value 1' one", + fileparser.DescriptionConfigurationKey: "'value two'", + fileparser.TagConfigurationKey: "tag1|tag2", + }), + Entry( + "repeated key configurations", + "repeated.sh", + map[string]string{ + fileparser.TagConfigurationKey: "tag1|tag2|tag3|tag4|tag5", + fileparser.AliasConfigurationKey: "alias", + fileparser.DescriptionConfigurationKey: "first line\nsecond line\nthird line", + }), + Entry( + "complex configuration parsing", + "complex.sh", + map[string]string{ + fileparser.NameConfigurationKey: "name", + fileparser.VerbConfigurationKey: "verb", + fileparser.DescriptionConfigurationKey: "first line\nsecond line\nthird line with 'quotes', and commas\nclosin'", + }), + Entry( + "multi-line description", + "multiline.sh", + map[string]string{ + fileparser.DescriptionConfigurationKey: expectedMultiLineDescription, + }), + Entry( + "directory and log mode configuration", + "dir-logmode.sh", + map[string]string{ + fileparser.NameConfigurationKey: "test-dir-logmode", + fileparser.VerbConfigurationKey: "test", + fileparser.DirConfigurationKey: "./subdir", + fileparser.LogModeConfigurationKey: "text", + }), + ) + + DescribeTable("should correctly parse parameter configurations", + func(file string, expectedFields map[string]string, expectedParams executable.ParameterList) { + filePath := filepath.Join("testdata", file) + fileBytes, err := os.ReadFile(filepath.Clean(filePath)) + Expect(err).ToNot(HaveOccurred()) + + result, err := fileparser.ExtractExecConfig(string(fileBytes), "# ") + Expect(err).ToNot(HaveOccurred()) + Expect(result.SimpleFields).To(Equal(expectedFields)) + Expect(result.Params).To(Equal(expectedParams)) + Expect(result.Args).To(BeEmpty()) + }, + Entry( + "parameters configuration", + "params.sh", + map[string]string{ + fileparser.NameConfigurationKey: "test-params", + fileparser.VerbConfigurationKey: "test", + }, + executable.ParameterList{ + {SecretRef: "my-secret", EnvKey: "SECRET_VAR"}, + {Prompt: "Enter name", EnvKey: "NAME_VAR"}, + {Text: "default-value", EnvKey: "DEFAULT_VAR"}, + }), + ) + + DescribeTable("should correctly parse argument configurations", + func(file string, expectedFields map[string]string, expectedArgs executable.ArgumentList) { + filePath := filepath.Join("testdata", file) + fileBytes, err := os.ReadFile(filepath.Clean(filePath)) + Expect(err).ToNot(HaveOccurred()) + + result, err := fileparser.ExtractExecConfig(string(fileBytes), "# ") + Expect(err).ToNot(HaveOccurred()) + Expect(result.SimpleFields).To(Equal(expectedFields)) + Expect(result.Params).To(BeEmpty()) + for _, arg := range expectedArgs { + Expect(slices.ContainsFunc(result.Args, func(argument executable.Argument) bool { + return argument.EnvKey == arg.EnvKey + })).To(BeTrue()) + } + }, + Entry( + "arguments configuration", + "args.sh", + map[string]string{ + fileparser.NameConfigurationKey: "test-args", + fileparser.VerbConfigurationKey: "test", + }, + executable.ArgumentList{ + {Flag: "verbose", EnvKey: "VERBOSE"}, + {Pos: 1, EnvKey: "FILENAME"}, + {Flag: "count", EnvKey: "COUNT"}, + }), + ) + + Describe("should handle parsing errors gracefully", func() { + It("should return error for malformed params", func() { + content := "# f:params=invalid:format" + _, err := fileparser.ExtractExecConfig(content, "# ") + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("param 1 requires at least 3 fields")) + }) + + It("should return error for malformed args", func() { + content := "# f:args=invalid:format" + _, err := fileparser.ExtractExecConfig(content, "# ") + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("arg 1 requires exactly 3 fields")) + }) + + It("should return error for invalid param type", func() { + content := "# f:params=invalid:value:ENV_KEY" + _, err := fileparser.ExtractExecConfig(content, "# ") + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("invalid parameter type")) + }) + + It("should return error for invalid arg type", func() { + content := "# f:args=invalid:value:ENV_KEY" + _, err := fileparser.ExtractExecConfig(content, "# ") + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("invalid argument type")) + }) + + It("should return error for invalid position in pos arg", func() { + content := "# f:args=pos:notanumber:ENV_KEY" + _, err := fileparser.ExtractExecConfig(content, "# ") + Expect(err).To(HaveOccurred()) + Expect(err.Error()).To(ContainSubstring("invalid position number")) + }) + }) + + Describe("should handle singular and plural key forms", func() { + It("should normalize singular forms to plural", func() { + content := `# f:tag=production f:alias=prod-deploy f:param=text:value:ENV f:arg=flag:test:TEST` + result, err := fileparser.ExtractExecConfig(content, "# ") + Expect(err).ToNot(HaveOccurred()) + + Expect(result.SimpleFields).To(HaveKey(fileparser.TagConfigurationKey)) + Expect(result.SimpleFields).To(HaveKey(fileparser.AliasConfigurationKey)) + Expect(result.SimpleFields[fileparser.TagConfigurationKey]).To(Equal("production")) + Expect(result.SimpleFields[fileparser.AliasConfigurationKey]).To(Equal("prod-deploy")) + + Expect(result.Params).To(HaveLen(1)) + Expect(result.Args).To(HaveLen(1)) + }) + + It("should handle mixed singular/plural usage", func() { + content := `# f:tag=production f:tags=deployment|staging f:alias=prod f:aliases=deploy` + result, err := fileparser.ExtractExecConfig(content, "# ") + Expect(err).ToNot(HaveOccurred()) + + Expect(result.SimpleFields[fileparser.TagConfigurationKey]).To(Equal("production|deployment|staging")) + Expect(result.SimpleFields[fileparser.AliasConfigurationKey]).To(Equal("prod|deploy")) + }) + }) +}) + +const expectedMultiLineDescription = `first line +second line + third line, with commas and 'quotes' +fourth line +fifth line +sixth line +seventh line +eighth line` diff --git a/internal/fileparser/configmap.go b/internal/fileparser/configmap.go deleted file mode 100644 index 5970bd48..00000000 --- a/internal/fileparser/configmap.go +++ /dev/null @@ -1,229 +0,0 @@ -package fileparser - -import ( - "fmt" - "regexp" - "strings" - "time" - - "github.com/pkg/errors" - - "github.com/flowexec/flow/types/executable" -) - -const ( - TimeoutConfigurationKey = "timeout" - VerbConfigurationKey = "verb" - NameConfigurationKey = "name" - AliasConfigurationKey = "alias" - DescriptionConfigurationKey = "description" - VisibilityConfigurationKey = "visibility" - TagConfigurationKey = "tag" - - InternalListSeparator = "," - - commentPrefix = "# " - keyPrefix = "f:" - multiLineKeyPrefix = "f|" - descriptionSeparator = "\n" - descriptionAlias = "desc" -) - -var multiLineDescriptionTag = fmt.Sprintf("<%s%s>", multiLineKeyPrefix, DescriptionConfigurationKey) - -func ExtractExecConfigMap(data, prefix string) (map[string]string, error) { - configMap := make(map[string]string) - processingMultiLineDescription := false - for _, line := range strings.Split(data, "\n") { - isComment := strings.HasPrefix(line, strings.TrimSpace(prefix)) - if trimmedLine := strings.TrimSpace(line); !isComment && trimmedLine != "" { - // If the line is not a comment or empty, break out of the loop. - // All flow executable configuration should be at the top of the file. - break - } - - line = strings.TrimPrefix(line, commentPrefix) - if processingMultiLineDescription = processMultiLineDescription( - line, configMap, processingMultiLineDescription, - ); processingMultiLineDescription { - continue - } - - cfg, err := parseConfigurations(line) - if err != nil { - return nil, fmt.Errorf("unable to extract executable configurations: %w", err) - } - - for key, value := range cfg { - switch key { - case TimeoutConfigurationKey, VerbConfigurationKey, NameConfigurationKey, VisibilityConfigurationKey: - configMap[key] = strings.TrimSpace(value) - case DescriptionConfigurationKey, descriptionAlias: - if existingValue, ok := configMap[DescriptionConfigurationKey]; ok { - configMap[DescriptionConfigurationKey] = - fmt.Sprintf("%s%s%s", existingValue, descriptionSeparator, value) - } else { - configMap[DescriptionConfigurationKey] = value - } - case AliasConfigurationKey, TagConfigurationKey: - value = strings.TrimSpace(value) - if existingValue, ok := configMap[key]; ok { - configMap[key] = fmt.Sprintf("%s%s%s", existingValue, InternalListSeparator, value) - } else { - configMap[key] = value - } - } - } - } - return configMap, nil -} - -func applyConfig(exec *executable.Executable, key, value string) error { - switch key { - case TimeoutConfigurationKey: - dur, err := time.ParseDuration(value) - if err != nil { - return errors.Wrapf(err, "unable to parse timeout duration %s", value) - } - exec.Timeout = &dur - case VerbConfigurationKey: - exec.Verb = executable.Verb(value) - case NameConfigurationKey: - exec.Name = value - case VisibilityConfigurationKey: - v := executable.ExecutableVisibility(value) - exec.Visibility = &v - case DescriptionConfigurationKey: - exec.Description = value - case AliasConfigurationKey: - values := make([]string, 0) - for _, v := range strings.Split(value, InternalListSeparator) { - values = append(values, strings.TrimSpace(v)) - } - exec.Aliases = values - case TagConfigurationKey: - values := make([]string, 0) - for _, v := range strings.Split(value, InternalListSeparator) { - values = append(values, strings.TrimSpace(v)) - } - exec.Tags = values - } - return nil -} - -// This regex is used to extract all flow configurations from a shell script. -// The regex matches the following: -// - f:= -// - f:="" -// - f:='' -// - f:="",='',key= -// - f:exampleKey='Example value\, with comma' -// - f:exampleKey="Example value\, with comma" -// - f:exampleKey="\'Example value with escaped quotes\'" -// - f:exampleKey='\"Example value with escaped quotes\"' -// and so on. -var flowConfigRegex = regexp.MustCompile(`f:\w+=(?:"(?:[^"\\]|\\.)*"|'(?:[^'\\]|\\.)*'|[^, ]+)`) - -func processMultiLineDescription(line string, configMap map[string]string, processing bool) bool { - multiLineDescPrefixed := strings.HasPrefix(line, multiLineDescriptionTag) - multiLineDescSuffixed := strings.HasSuffix(line, multiLineDescriptionTag) - existingValue := configMap[DescriptionConfigurationKey] - switch { - case processing && multiLineDescPrefixed: - processing = false - case processing && multiLineDescSuffixed: - processing = false - line = strings.TrimSuffix(line, multiLineDescriptionTag) - if existingValue != "" { - configMap[DescriptionConfigurationKey] = fmt.Sprintf( - "%s%s%s", - existingValue, - descriptionSeparator, - line, - ) - } else { - configMap[DescriptionConfigurationKey] = line - } - case processing: - if existingValue != "" { - configMap[DescriptionConfigurationKey] = fmt.Sprintf( - "%s%s%s", - existingValue, - descriptionSeparator, - line, - ) - } else { - configMap[DescriptionConfigurationKey] = line - } - case multiLineDescPrefixed: - processing = true - line = strings.TrimPrefix(line, multiLineDescriptionTag) - if trimmedLine := strings.TrimSpace(line); trimmedLine == "" { - return processing - } else if existingValue != "" { - configMap[DescriptionConfigurationKey] = fmt.Sprintf( - "%s%s%s", - existingValue, - descriptionSeparator, - line, - ) - } else { - configMap[DescriptionConfigurationKey] = line - } - } - return processing -} - -func parseConfigurations(line string) (map[string]string, error) { - configMap := make(map[string]string) - matches := flowConfigRegex.FindAllString(line, -1) - for _, match := range matches { - split := strings.SplitN(match, "=", 2) - key := strings.TrimSpace(strings.ToLower(strings.TrimPrefix(split[0], keyPrefix))) - value := split[1] - // Removing quotes if present - if (value[0] == '"' && value[len(value)-1] == '"') || (value[0] == '\'' && value[len(value)-1] == '\'') { - value = value[1 : len(value)-1] - } - // Replace escaped characters - value = strings.ReplaceAll(value, `\"`, `"`) - value = strings.ReplaceAll(value, `\'`, `'`) - value = strings.ReplaceAll(value, `\\`, `\`) - value = strings.ReplaceAll(value, `\,`, `,`) - - if !validateKey(key) { - return nil, fmt.Errorf("invalid key (%s)", key) - } - - switch key { - case AliasConfigurationKey, TagConfigurationKey: - value = strings.TrimSpace(value) - if existingValue, ok := configMap[key]; ok { - configMap[key] = fmt.Sprintf("%s%s%s", existingValue, InternalListSeparator, value) - } else { - configMap[key] = value - } - case DescriptionConfigurationKey, descriptionAlias: - if existingValue, ok := configMap[DescriptionConfigurationKey]; ok { - configMap[DescriptionConfigurationKey] = - fmt.Sprintf("%s%s%s", existingValue, descriptionSeparator, value) - } else { - configMap[DescriptionConfigurationKey] = value - } - default: - configMap[key] = strings.TrimSpace(value) - } - } - return configMap, nil -} - -func validateKey(key string) bool { - switch key { - case VerbConfigurationKey, NameConfigurationKey, DescriptionConfigurationKey, descriptionAlias, - AliasConfigurationKey, VisibilityConfigurationKey, TagConfigurationKey, - TimeoutConfigurationKey: - return true - default: - return false - } -} diff --git a/internal/fileparser/configmap_test.go b/internal/fileparser/configmap_test.go deleted file mode 100644 index d82f4fbc..00000000 --- a/internal/fileparser/configmap_test.go +++ /dev/null @@ -1,87 +0,0 @@ -package fileparser_test - -import ( - "os" - "path/filepath" - - "github.com/flowexec/flow/internal/fileparser" - - . "github.com/onsi/ginkgo/v2" - . "github.com/onsi/gomega" -) - -var _ = Describe("ExecConfigMapFromFile", func() { - DescribeTable("should correctly parse configurations", - func(file string, expected map[string]string) { - filePath := filepath.Join("testdata", file) - fileBytes, err := os.ReadFile(filepath.Clean(filePath)) - Expect(err).ToNot(HaveOccurred()) - - result, err := fileparser.ExtractExecConfigMap(string(fileBytes), "# ") - Expect(err).ToNot(HaveOccurred()) - Expect(result).To(Equal(expected)) - }, - Entry( - "simple key-value pairs", - "simple.sh", - map[string]string{ - fileparser.NameConfigurationKey: "hello", - fileparser.VerbConfigurationKey: "show", - }), - Entry( - "values with spaces in quotes", - "quoted.sh", - map[string]string{ - fileparser.NameConfigurationKey: "value 1", - fileparser.VerbConfigurationKey: "value2", - fileparser.DescriptionConfigurationKey: "value 3", - }), - Entry( - "mixed separators", - "mixed.sh", - map[string]string{ - fileparser.NameConfigurationKey: "value1", - fileparser.VerbConfigurationKey: "value2", - fileparser.DescriptionConfigurationKey: "value 3", - }), - Entry( - "values with escaped characters", - "escaped.sh", - map[string]string{ - fileparser.NameConfigurationKey: "value 1, one", - fileparser.DescriptionConfigurationKey: "'value two'", - fileparser.TagConfigurationKey: "tag1,tag2", - }), - Entry( - "repeated key configurations", - "repeated.sh", - map[string]string{ - fileparser.TagConfigurationKey: "tag1,tag2,tag3,tag4,tag5", - fileparser.AliasConfigurationKey: "alias", - fileparser.DescriptionConfigurationKey: "first line\nsecond line\nthird line", - }), - Entry( - "complex configuration parsing", - "complex.sh", - map[string]string{ - fileparser.NameConfigurationKey: "name", - fileparser.VerbConfigurationKey: "verb", - fileparser.DescriptionConfigurationKey: "first line\nsecond line\nthird line with 'quotes', and commas\nclosin'", - }), - Entry( - "multi-line description", - "multiline.sh", - map[string]string{ - fileparser.DescriptionConfigurationKey: expectedMultiLineDescription, - }), - ) -}) - -const expectedMultiLineDescription = `first line -second line - third line, with commas and 'quotes' -fourth line -fifth line -sixth line -seventh line -eighth line` diff --git a/internal/fileparser/makefile_parser.go b/internal/fileparser/makefile_parser.go index d24a4995..316b4f32 100644 --- a/internal/fileparser/makefile_parser.go +++ b/internal/fileparser/makefile_parser.go @@ -76,17 +76,15 @@ func ExecutablesFromMakefile(wsPath, path string) (executable.ExecutableList, er }, } - configMap, err := ExtractExecConfigMap(t.description, "") + cfg, err := ExtractExecConfig(t.description, "") if err != nil { return nil, err } - if len(configMap) != 0 { + if len(cfg.SimpleFields) > 0 || len(cfg.Params) > 0 || len(cfg.Args) > 0 { e.Description = "" - for key, value := range configMap { - if err := applyConfig(e, key, value); err != nil { - return nil, err - } + if err := ApplyExecConfig(e, cfg); err != nil { + return nil, err } } diff --git a/internal/fileparser/shell_file_parser.go b/internal/fileparser/shell_file_parser.go index a0ce87d1..be2f0031 100644 --- a/internal/fileparser/shell_file_parser.go +++ b/internal/fileparser/shell_file_parser.go @@ -25,15 +25,12 @@ func ExecutablesFromShFile(wsPath, filePath string) (*executable.Executable, err if err != nil { return nil, err } - configMap, err := ExtractExecConfigMap(string(fileBytes), "# ") + cfg, err := ExtractExecConfig(string(fileBytes), "# ") if err != nil { return nil, err } - - for key, value := range configMap { - if err := applyConfig(exec, key, value); err != nil { - return nil, err - } + if err := ApplyExecConfig(exec, cfg); err != nil { + return nil, err } exec.Tags = append(exec.Tags, generatedTag) diff --git a/internal/fileparser/testdata/args.sh b/internal/fileparser/testdata/args.sh new file mode 100644 index 00000000..eeed3a7b --- /dev/null +++ b/internal/fileparser/testdata/args.sh @@ -0,0 +1,8 @@ +#!/bin/bash + +# f:name=test-args f:verb=test +# f:args=flag:verbose:VERBOSE|pos:1:FILENAME|flag:count:COUNT + +echo "Verbose: $VERBOSE" +echo "Filename: $FILENAME" +echo "Count: $COUNT" \ No newline at end of file diff --git a/internal/fileparser/testdata/complex.sh b/internal/fileparser/testdata/complex.sh index 7fd73ab9..62a64fe6 100644 --- a/internal/fileparser/testdata/complex.sh +++ b/internal/fileparser/testdata/complex.sh @@ -8,10 +8,10 @@ # unrelated text # f:desc="second line" # -# f:desc="third line with \'quotes\'\, and commas" +# f:desc="third line with \'quotes\', and commas" # f:description=closin' -# f:name=name followed by unrelated text +# f:name=name echo "Hello, world!" diff --git a/internal/fileparser/testdata/dir-logmode.sh b/internal/fileparser/testdata/dir-logmode.sh new file mode 100644 index 00000000..696dc2b9 --- /dev/null +++ b/internal/fileparser/testdata/dir-logmode.sh @@ -0,0 +1,8 @@ +#!/bin/bash + +# f:name=test-dir-logmode f:verb=test +# f:dir=./subdir +# f:logmode=text + +echo "Running in directory: $(pwd)" +echo "Log mode configured" \ No newline at end of file diff --git a/internal/fileparser/testdata/escaped.sh b/internal/fileparser/testdata/escaped.sh index 429c2d1c..6518addc 100644 --- a/internal/fileparser/testdata/escaped.sh +++ b/internal/fileparser/testdata/escaped.sh @@ -1,5 +1,5 @@ #!/bin/sh -# f:name="value 1\, one" f:description="\'value two\'" f:tag='tag1\,tag2' +# f:name="value 1\' one" f:description="\'value two\'" f:tags=tag1|tag2 echo "Hello, world!" diff --git a/internal/fileparser/testdata/mixed.sh b/internal/fileparser/testdata/mixed.sh deleted file mode 100644 index cb58b28b..00000000 --- a/internal/fileparser/testdata/mixed.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/sh - -# f:name=value1, f:verb=value2 f:description="value 3" - -echo "Hello, world!" diff --git a/internal/fileparser/testdata/params.sh b/internal/fileparser/testdata/params.sh new file mode 100644 index 00000000..4a154d15 --- /dev/null +++ b/internal/fileparser/testdata/params.sh @@ -0,0 +1,8 @@ +#!/bin/bash + +# f:name=test-params f:verb=test +# f:params=secretRef:my-secret:SECRET_VAR|prompt:"Enter name":NAME_VAR|text:default-value:DEFAULT_VAR + +echo "Secret: ${SECRET_VAR:0:8}..." +echo "Name: $NAME_VAR" +echo "Default: $DEFAULT_VAR" \ No newline at end of file diff --git a/internal/fileparser/testdata/repeated.sh b/internal/fileparser/testdata/repeated.sh index 7448d458..c4665104 100644 --- a/internal/fileparser/testdata/repeated.sh +++ b/internal/fileparser/testdata/repeated.sh @@ -2,6 +2,6 @@ # f:tag=tag1 f:tag=tag2 f:alias=alias # f:tag=tag3 f:description="first line" f:description="second line" -# f:description="third line" f:tag='tag4\,tag5' +# f:description="third line" f:tag="tag4\|tag5" echo "Hello, world!"