From 3509d9cd10bee908e57993d6b4a83c4f76abf06f Mon Sep 17 00:00:00 2001 From: Jahvon Dockery Date: Fri, 11 Jul 2025 10:08:10 -0400 Subject: [PATCH] docs: revamp all documentation for v1 --- .github/FUNDING.yml | 1 + README.md | 91 +++++- docs/README.md | 32 +- docs/_coverpage.md | 4 +- docs/_sidebar.md | 24 +- docs/cli/_sidebar.md | 12 +- docs/development.md | 141 ++++++--- docs/guide/README.md | 35 ++ docs/guide/_sidebar.md | 24 +- docs/guide/advanced.md | 457 +++++++++++++++++++++++++++ docs/guide/concepts.md | 262 +++++++++++++++ docs/guide/conditional.md | 126 -------- docs/guide/executable.md | 420 ------------------------ docs/guide/executables.md | 448 ++++++++++++++++++++++++++ docs/guide/first-workflow.md | 209 ++++++++++++ docs/guide/integrations.md | 65 ++-- docs/guide/interactive.md | 226 ++++++++++--- docs/guide/{secret.md => secrets.md} | 211 +++++++++---- docs/guide/state.md | 159 ---------- docs/guide/templating.md | 414 ++++++++++++------------ docs/guide/workspace.md | 78 ----- docs/guide/workspaces.md | 170 ++++++++++ docs/index.html | 7 + docs/installation.md | 50 ++- docs/quickstart.md | 127 +++++--- docs/types/README.md | 39 ++- docs/types/_sidebar.md | 17 +- 27 files changed, 2559 insertions(+), 1290 deletions(-) create mode 100644 .github/FUNDING.yml create mode 100644 docs/guide/README.md create mode 100644 docs/guide/advanced.md create mode 100644 docs/guide/concepts.md delete mode 100644 docs/guide/conditional.md delete mode 100644 docs/guide/executable.md create mode 100644 docs/guide/executables.md create mode 100644 docs/guide/first-workflow.md rename docs/guide/{secret.md => secrets.md} (54%) delete mode 100644 docs/guide/state.md delete mode 100644 docs/guide/workspace.md create mode 100644 docs/guide/workspaces.md diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 00000000..c2e41efd --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1 @@ +github: jahvon diff --git a/README.md b/README.md index 046b719f..ff72d49d 100644 --- a/README.md +++ b/README.md @@ -1,16 +1,95 @@

flow

-
-

GitHub release Go Reference + Join Discord + GitHub Repo stars

- flow is a customizable and interactive CLI automation tool designed to streamline how you manage and run local - development and operations workflows. It uses executable YAML configurations, organized across workspaces and namespaces, - for tasks like running scripts, transforming APIs, or generating file systems from templates. Charming, interactive TUI included! + flow is your personal workflow hub - organize automation across all your projects with built-in secrets, templates, and cross-project composition. + Define workflows in YAML, discover them visually, and run them anywhere.

-

Learn more at flowexec.io

+--- + +## Quick Start + +```bash +# Install +curl -sSL https://raw.githubusercontent.com/flowexec/flow/main/scripts/install.sh | bash + +# Create your first workflow +flow workspace add my-project . --set +echo 'executables: + - verb: run + name: hello + exec: + cmd: echo "Hello from flow!"' > hello.flow + +# Run it +flow sync +flow run hello +``` + +## Key Features + +flow complements existing CLI tools by adding multi-project organization, built-in security, and visual discovery to your automation toolkit. + +- **Workspace organization** - Group and manage workflows across multiple projects +- **Encrypted secret vaults** - Multiple backends (AES, Age, external tools) +- **Interactive discovery** - Browse, search, and filter workflows visually +- **Flexible execution** - Serial, parallel, conditional, and interactive workflows +- **Workflow generation** - Create projects and workflows from reusable templates +- **Composable workflows** - Reference and chain workflows within and across projects + +

flow

+ +## Example Workflows + +```yaml +# api.flow +executables: + - verb: deploy + name: staging + serial: + execs: + - cmd: npm run build + - cmd: docker build -t api:staging . + - ref: shared-tools/k8s:deploy-staging + - cmd: curl -f https://api-staging.example.com/health + + - verb: backup + name: database + exec: + params: + - secretRef: database-url + envKey: DATABASE_URL + cmd: pg_dump $DATABASE_URL > backup-$(date +%Y%m%d).sql +``` + +```bash +# Run workflows +flow deploy staging +flow backup database + +# Visual discovery +flow browse +``` + +## Documentation + +**Complete documentation at [flowexec.io](https://flowexec.io)** + +- [Installation](https://flowexec.io/#/installation) - Multiple installation methods +- [Quick Start](https://flowexec.io/#/quickstart) - Get up and running in 5 minutes +- [Core Concepts](https://flowexec.io/#/guide/concepts) - Understand workspaces, executables, and vaults +- [User Guides](https://flowexec.io/#/guide/README) - Comprehensive guides for all features + +## Community + +- [Discord Community](https://discord.gg/CtByNKNMxM) - Get help and share workflows +- [Issue Tracker](https://github.com/flowexec/flow/issues) - Report bugs and request features +- [Examples Repository](https://github.com/flowexec/examples) - Real-world workflow patterns +- [Contributing Guide](https://flowexec.io/#/development) - Help make flow better diff --git a/docs/README.md b/docs/README.md index 5dffe71f..edafa27e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -3,24 +3,34 @@

GitHub release Go Reference + Join Discord + GitHub Repo stars

-flow is a customizable and interactive CLI tool designed to streamline how you manage and run local development and -operations workflows. +flow is your personal **workflow hub** - organize automation across all your projects with built-in secrets, templates, +and cross-project composition. Go beyond task running to workflow management that scales with your development ecosystem. -**This project is currently in beta and documentation is a work in progress.** Contributions and feedback are welcome. +#### _Why flow?_ -#### _Features_ +- **Unified automation hub**: Organize workflows across all your projects in one place +- **Built-in security**: Encrypted vaults keep your secrets safe and shareable +- **Cross-project power**: Reference workflows from other projects and share common tools +- **Template-driven**: Generate new projects and workflows from reusable templates +- **Visual discovery**: Browse and search workflows across your entire development ecosystem -- **Task Runner**: Easily define, manage, and run your tasks (called [executables](guide/executable.md)) from the command line. -- **Secret Vault**: Store sensitive secrets in a secure local [vault](guide/secret.md#vault-setup). -- **Template Generator**: Generate executables and workspace scaffolding with [flow file templates](guide/templating.md). -- **TUI Library**: Explore and run executables from the interactive and searchable TUI [library](cli/flow_browse.md). -- **Executable Organizer**: Group, reference, and search for executables by workspace, namespace, verbs, and tags. -- **Input Handler**: Pass values into executables with environment variables defined by secrets, command-line args, or interactive prompts. -- **Customizable TUI**: Personalize your [TUI experience](guide/interactive.md) with settings for log formatting, log archiving, and execution notifications, and more. +#### _Core Capabilities_ + +- **Workflow Management**: Define, organize, and run workflows across multiple projects +- **Secure Secret Storage**: Built-in encrypted vaults with multiple backend options +- **Project Generation**: Create new projects and workflows from customizable templates +- **Cross-Project Composition**: Reference and share workflows between different projects +- **Visual Workflow Browser**: Discover and run workflows with powerful filtering and search +- **Flexible Configuration**: YAML-based definitions with arguments, secrets, and conditional logic + +**Ready to organize your automation?** → [Install flow](installation.md) → [Quick start guide](quickstart.md) ---

+Browse examples in the [examples repo](https://github.com/flowexec/examples) and [flow project](https://github.com/flowexec/flow/tree/main/.execs) \ No newline at end of file diff --git a/docs/_coverpage.md b/docs/_coverpage.md index 7819cc22..ce04f1db 100644 --- a/docs/_coverpage.md +++ b/docs/_coverpage.md @@ -1,11 +1,11 @@ flow -# flow beta +# flow > _Interactive CLI automation that flows with you._ - Define your _executables_ in YAML -- Run and manage them from the TUI +- Browse and execute with ease [GitHub](https://github.com/flowexec/flow) [Get Started](README.md) diff --git a/docs/_sidebar.md b/docs/_sidebar.md index 2d263895..61fef469 100644 --- a/docs/_sidebar.md +++ b/docs/_sidebar.md @@ -1,22 +1,6 @@ -- [Home](README.md "flow documentation") - [Install](installation.md "Installation guide") - -- Getting Started - - [Quick start](quickstart.md "Quick start guide") - - [Workspaces](guide/workspace.md "Managing workspaces") - - [Executables](guide/executable.md "Managing executables") - - [Secrets vault](guide/secret.md "Using the secrets vault") - -- Advanced Features - - [Templating](guide/templating.md "Using flowfile templates") - - [Managing state](guide/state.md "Managing executable state") - - [Conditional execution](guide/conditional.md "Conditional execution") - - [Customizing the TUI](guide/interactive.md "Customizing the interactive UI") - - [Integrations](guide/integrations.md "Integration") - -- Reference - - - [CLI Commands](cli/README.md "Command line interface reference") - - [Configuration Files](types/README.md "Configuration file reference") - +- [Quick Start](quickstart.md "Quick start guide") +- [User Guides](guide/README.md "Complete guides for mastering flow automation") +- [CLI Reference](cli/README.md "Command line interface reference") +- [Configuration Reference](types/README.md "YAML configuration file reference") - [Contributing](development.md "Development guide") diff --git a/docs/cli/_sidebar.md b/docs/cli/_sidebar.md index 82859dbd..3f4feaa4 100644 --- a/docs/cli/_sidebar.md +++ b/docs/cli/_sidebar.md @@ -1,15 +1,13 @@ -- [Home](../README.md "flow documentation") -- [Install](../installation.md "Installation guide") -- [Quick Start](../quickstart.md "Quick start guide") +- [← Back to Home](../README.md "flow documentation") -- Reference - - [CLI commands](README.md "Command line interface reference") +- CLI Reference + - [flow](README.md "Command line interface reference") - [flow exec](flow_exec.md) - - [flow library](flow_library.md) + - [flow browse](flow_browse.md) - [flow template](flow_template.md) - [flow config](flow_config.md) - [flow workspace](flow_workspace.md) - [flow secret](flow_secret.md) + - [flow vault](flow_vault.md) - [flow sync](flow_sync.md) - [flow logs](flow_logs.md) - - [Configuration files](../types/README.md "Configuration file reference") diff --git a/docs/development.md b/docs/development.md index 12da14d6..b01c0413 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,76 +1,135 @@ -# flow Development +# Contributing to flow [![Go Report Card](https://goreportcard.com/badge/github.com/flowexec/flow)](https://goreportcard.com/report/github.com/flowexec/flow) [![Go Reference](https://pkg.go.dev/badge/github.com/flowexec/flow.svg)](https://pkg.go.dev/github.com/flowexec/flow) -[![GitHub branch status](https://img.shields.io/github/checks-status/flowexec/flow/main)](https://github.com/flowexec/flow/actions?query=branch%3Amain) +[![GitHub branch check runs](https://img.shields.io/github/check-runs/flowexec/flow/main)](https://github.com/flowexec/flow/actions?query=branch%3Amain) [![Codecov](https://img.shields.io/codecov/c/github/flowexec/flow)](https://app.codecov.io/gh/flowexec/flow) -Before getting started, please read the [Code of Conduct](../.github/CODE_OF_CONDUCT.md) and [Contributing Guidelines](../.github/CONTRIBUTING.md). +This document provides an overview of how to contribute to the flow project, including setting up your development environment, understanding the project structure, and running tests. -flow is written in [Go](https://golang.org/). See the [go.mod](../go.mod) file for the current Go version used in -building the project. +Before getting started, please read our [Code of Conduct](https://github.com/flowexec/flow/blob/main/.github/CODE_OF_CONDUCT.md) and [Contributing Guidelines](https://github.com/flowexec/flow/blob/main/.github/CONTRIBUTING.md). -## Getting Started +**Ways to Contribute** -Before developing on this project, you will need to make sure you have the latest `flow` version installed. -Refer to the [Installation](installation.md) section for more information. +- **Report bugs** - [Open an issue](https://github.com/flowexec/flow/issues/new) with reproduction steps +- **Suggest features** - Share ideas for new functionality +- **Improve documentation** - Fix typos, add examples, or clarify explanations +- **Write code** - Fix bugs, implement features, or optimize performance +- **Share examples** - Contribute to the [examples repository](https://github.com/flowexec/examples) -After cloning the repository, you can start using the below commands after registering the repo workspace: +## Quick Start + +**Prerequisites** + +- **Go** - See [go.mod](https://github.com/flowexec/flow/blob/main/go.mod) for the required version +- **flow CLI** - Install the [latest version](installation.md) before developing ```sh -flow workspace create flow +# Clone and set up the repository +git clone https://github.com/flowexec/flow.git +cd flow + +# Register the repo as a flow workspace +flow workspace add flow . --set + +# Install development dependencies +flow install tools + +# Verify everything works +flow validate ``` -### Development Executables +## Development Executables -The `flow` project contains a few development executables that can be run locally. After registering the repo -workspace, you can run the following commands: +The flow project uses flow itself for development! Here are the key commands: ```sh -# Install Go tool dependencies -flow install tools - # Build the CLI binary -flow build binary +flow build binary ./bin/flow -# Validate code changes (runs tests, linters, codegen, etc) +# Run all validation (tests, linting, code generation) flow validate -# Only generate code -flow generate +# Run specific checks +flow test all # All tests +flow generate # Code generation +flow run lint # Linting only -# Only run tests -flow test all +# Install/update Go tools +flow install tools ``` -### Working with generated types +## Project Structure -The `flow` project uses [go-jsonschema](github.com/atombender/go-jsonschema) with [go generate](https://blog.golang.org/generate) -to generate Go types from JSON schema files (defined in YAML). If you need to make changes to the generated types -(found in the `types` package), you should update the associated `*schema.yaml` file and run the flow `run generate` executable -or go generate directly. +``` +flow/ +├── .execs/ # Development workflows +├── cmd/ # CLI entry point +├── docs/ # Documentation +├── internal/ # Core application logic +│ ├── cache/ # Executable and workspace caching logic +│ ├── context/ # Global application context +│ ├── io/ # Terminal user interface and I/O +│ ├── runner/ # Executable execution engine +│ ├── services/ # Business logic services +│ ├── templates/ # Templating system for workflows +│ └── vault/ # Secret management +├── tests/ # CLI end-to-end test suite +└── types/ # Generated types from schemas +``` -Note that go generate alone does not update generated documentation. -Be sure to regenerate the JSON schema files and markdown documentation before submitting a PR. +_Some directories are omitted for brevity._ -### Working with tuikit +## Working with Generated Code -The `flow` project uses the [tuikit](tuikit.md) framework for building the terminal UI. -Contributions to the components and helpers in `tuikit` are welcome. +flow uses code generation extensively: + +### Go CLI Type Generation + +Types are generated from YAML schemas using [go-jsonschema](https://github.com/atombender/go-jsonschema): -_You should test all tuikit changes with a local flow build before submitting a PR._ - ```sh - go mod edit -replace github.com/flowexec/tuikit=../tuikit +# Regenerate types after schema changes +flow generate cli +``` + +**Important**: When modifying types, edit the `schemas/*.yaml` files, not the generated Go files in `types/`. + +### Documentation Generation + +CLI and type documentation is generated automatically: + +```sh +# Updates both CLI docs and type reference docs +flow generate docs +``` + +## TUI Development + +flow uses [tuikit](tuikit.md) for terminal interface development: + +**Local Development** + +```sh +# Link to local tuikit for development +go mod edit -replace github.com/flowexec/tuikit=../tuikit + +# Test TUI changes +flow build binary ./bin/flow-dev +./bin/flow-dev browse ``` ## Development Tools -Required tools for development: +### Required Tools + +These are installed automatically by `flow install tools`: + +- [mockgen](https://github.com/uber-go/mock) - Generate test mocks +- [golangci-lint](https://golangci-lint.run/) - Code linting +- [go-jsonschema](https://github.com/atombender/go-jsonschema) - Generate Go types from YAML schemas -- [mockgen](https://github.com/uber-go/mock) for generating test mocks -- [golangci-lint](https://golangci-lint.run/) for linting +### Additional Tools -Other tools used in the project: -- [goreleaser](https://goreleaser.com/) for releasing the project -- [ginkgo](https://onsi.github.io/ginkgo/) and [gomega](https://onsi.github.io/gomega/) for testing +- [goreleaser](https://goreleaser.com/) - Release automation +- [ginkgo](https://onsi.github.io/ginkgo/) - BDD testing framework diff --git a/docs/guide/README.md b/docs/guide/README.md new file mode 100644 index 00000000..71b1a95f --- /dev/null +++ b/docs/guide/README.md @@ -0,0 +1,35 @@ +# User Guides + +Welcome to the flow user guides! These guides will help you master flow automation, from basic concepts to advanced workflows. + +## Getting Started + +New to flow? Start here to understand the fundamentals: + +- **[Core Concepts](concepts.md)** - Learn about workspaces, executables, and vaults +- **[Your First Workflow](first-workflow.md)** - Build a complete automation from scratch + +## Essential Guides + +Master the core features of flow: + +- **[Workspaces](workspaces.md)** - Organize your automation across projects +- **[Executables](executables.md)** - Complete reference for all executable types and configuration +- **[Working with Secrets](secrets.md)** - Secure your workflows with encrypted vaults + +## Advanced Topics + +Take your automation to the next level: + +- **[Templates & Code Generation](templating.md)** - Generate workflows and project scaffolding +- **[Advanced Workflows](advanced.md)** - Complex automation with conditionals, state, and composition +- **[Interactive UI](interactive.md)** - Customize and use the terminal interface +- **[Integrations](integrations.md)** - Run flow in CI/CD and containerized environments + +## Other Resources + +- **[CLI Reference](../cli/README.md)** - Complete command documentation +- **[Configuration Reference](../types/README.md)** - YAML file schemas and options +- **[Examples Repository](https://github.com/flowexec/examples)** - Real-world workflow patterns + +Ready to get started? Begin with [Core Concepts](concepts.md) to understand how flow works, then build your first automation with [Your First Workflow](first-workflow.md)! \ No newline at end of file diff --git a/docs/guide/_sidebar.md b/docs/guide/_sidebar.md index e2bd94c5..1f1e9f2d 100644 --- a/docs/guide/_sidebar.md +++ b/docs/guide/_sidebar.md @@ -1,16 +1,16 @@ -- [Home](../README.md "flow documentation") -- [Install](../installation.md "Installation guide") +- [← Back to Home](../README.md "flow documentation") - Getting Started - - [Quick start](../quickstart.md "Quick start guide") - - [Workspaces](workspace.md "Managing workspaces") - - [Executables](executable.md "Managing executables") - - [Secrets vault](secret.md "Using the secrets vault") + - [Core Concepts](concepts.md "Understand workspaces, executables, and vaults") + - [Your First Workflow](first-workflow.md "Build a complete automation from scratch") -- Advanced Features - - [Templating](templating.md "Using flowfile templates") - - [Managing state](state.md "Managing executable state") - - [Conditional execution](conditional.md "Conditional execution") - - [Customizing the TUI](interactive.md "Customizing the interactive UI") - - [Integrations](integrations.md "Integration") +- Essentials + - [Workspaces](workspaces.md "Organize your automation across projects") + - [Executables](executables.md "Complete reference for all executable types") + - [Working with Secrets](secrets.md "Secure your workflows with encrypted vaults") +- Advanced + - [Templates & Workflow Generation](templating.md "Generate workflows and project scaffolding") + - [Advanced Workflows](advanced.md "Complex automation with conditionals and state") + - [Interactive UI](interactive.md "Customize and use the terminal interface") + - [Integrations](integrations.md "Run flow in CI/CD and containerized environments") \ No newline at end of file diff --git a/docs/guide/advanced.md b/docs/guide/advanced.md new file mode 100644 index 00000000..88f29796 --- /dev/null +++ b/docs/guide/advanced.md @@ -0,0 +1,457 @@ +# Advanced Workflows + +Once you're comfortable with basic executables, flow's real power comes from building sophisticated workflows that adapt +to conditions, maintain state, and compose multiple operations into powerful automations. + +## Conditional Execution + +Make your workflows smart by running different steps based on runtime conditions. + +### Basic Operators + +The expression language supports standard comparison and logical operators: + +- Comparison: `==`, `!=`, `<`, `>`, `<=`, `>=` +- Logical: `and`, `or`, `not` +- String: `+` (concatenation), `matches` (regex matching) +- Length: `len()` + +### Basic Conditions + +Use the `if` field to control when executables run: + +```yaml +executables: + - verb: deploy + name: app + serial: + execs: + # Only run on macOS + - if: os == "darwin" + cmd: brew install kubectl + + # Only run on Linux + - if: os == "linux" + cmd: apt-get install kubectl + + # Always run the deployment + - cmd: kubectl apply -f deployment.yaml +``` + +### Environment-Based Conditions + +Check environment variables to customize behavior: + +```yaml +executables: + - verb: build + name: app + serial: + execs: + # Development build + - if: env["NODE_ENV"] == "development" + cmd: npm run build:dev + + # Production build + - if: env["NODE_ENV"] == "production" + cmd: npm run build:prod + + # Run tests in CI + - if: env["CI"] == "true" + cmd: npm test +``` + +### Data-Driven Conditions + +Use stored data to control execution flow: + +```yaml +executables: + - verb: setup + name: feature + serial: + execs: + # Enable a feature flag + - cmd: flow cache set feature-x enabled + + # Later workflows can check this flag + - if: store["feature-x"] == "enabled" + cmd: echo "Feature X is enabled" + + # Complex conditions + - if: len(store["build-id"]) > 0 and os == "linux" + cmd: echo "Valid build on Linux" +``` + +### Available Context + +Conditions have access to extensive runtime information: + +**System Information:** +- `os` - Operating system ("linux", "darwin", "windows") +- `arch` - System architecture ("amd64", "arm64") + +**Environment Variables:** +- `env["VAR_NAME"]` - Any environment variable + +**Stored Data:** +- `store["key"]` - Values from the cache/data store + +**Flow Context:** +- `ctx.workspace` - Current workspace name +- `ctx.namespace` - Current namespace +- `ctx.workspacePath` - Full path to workspace root +- `ctx.flowFilePath` - Path to current flow file +- `ctx.flowFileDir` - Directory containing current flow file + +## Managing State + +Persist data across executions and share information between workflow steps. + +### Cache Basics + +The cache stores key-value data with different persistence scopes: + +```yaml +executables: + - verb: build + name: with-cache + serial: + execs: + # Set data in cache + - cmd: | + build_id=$(date +%Y%m%d_%H%M%S) + flow cache set current-build $build_id + echo "Build ID: $build_id" + + # Use cached data in later steps + - cmd: | + build_id=$(flow cache get current-build) + echo "Deploying build: $build_id" + docker tag myapp:latest myapp:$build_id +``` + +### Cache Persistence Scopes + +Understanding cache persistence is crucial for complex workflows: + +**Global Scope** - Set outside executables, persists until manually cleared: +```shell +# Set globally (persists across all executions) +flow cache set api-endpoint "https://api.prod.com" +flow cache set feature-enabled true + +# Available in all executables until cleared +flow run any-executable # Can access api-endpoint and feature-enabled +flow cache clear # Removes all global data +``` + +**Execution Scope** - Set within executables, automatically cleared when parent completes: +```yaml +executables: + - verb: test + name: integration + serial: + execs: + # This data persists across ALL sub-executables + - cmd: flow cache set test-db-url "localhost:5432" + - cmd: flow cache set test-session-id $(uuidgen) + + # All these steps can access the test data + - ref: setup database # Can use test-db-url + - ref: run tests # Can use test-session-id + - ref: cleanup database # Can use test-db-url + + # Data automatically cleared when 'test integration' completes +``` + +**Cache Management Commands:** +```shell +# View all cached data +flow cache list + +# Get specific value +flow cache get key-name + +# Set value globally +flow cache set key-name value + +# Remove specific key +flow cache remove key-name + +# Clear all data +flow cache clear + +# Clear all data (including different scopes) +flow cache clear --all +``` + +See the [cache command reference](../cli/flow_cache.md) for detailed commands and options. + +### Temporary Directories + +Use isolated temporary directories for complex operations: + +```yaml +executables: + - verb: build + name: container + exec: + dir: f:tmp # Creates temporary directory + cmd: | + # All commands run in isolated temp directory + git clone https://github.com/user/repo . + docker build -t myapp . + # Directory automatically cleaned up +``` + +When defined for a `serial` or `parallel` executable, the temporary directory is created at the start and cleaned up after all steps complete. + +## Workflow Composition + +Build complex automations by combining executables in sophisticated ways. + +### Serial vs Parallel Execution + +**Serial** - Steps run one after another: +```yaml +executables: + - verb: deploy + name: backend + serial: + failFast: true # Stop on first failure + execs: + - cmd: npm run build + - cmd: docker build -t api . + - cmd: kubectl apply -f deployment.yaml + - cmd: kubectl rollout status deployment/api +``` + +**Parallel** - Steps run simultaneously: +```yaml +executables: + - verb: test + name: all + parallel: + maxThreads: 3 # Limit concurrent operations + failFast: false # Run all tests even if some fail + execs: + - cmd: npm run test:unit + - cmd: npm run test:integration + - cmd: npm run test:e2e + - cmd: npm run lint +``` + +### Mixed Execution Patterns + +Combine serial and parallel for sophisticated workflows by defining separate executables and referencing them: + +```yaml +executables: + # Individual build steps + - verb: build + name: api-image + exec: + cmd: docker build -t api ./api + + - verb: build + name: web-image + exec: + cmd: docker build -t web ./web + + - verb: build + name: worker-image + exec: + cmd: docker build -t worker ./worker + + # Parallel builds + - verb: build + name: all-images + parallel: + execs: + - ref: build api-image + - ref: build web-image + - ref: build worker-image + + # Full deployment workflow + - verb: deploy + name: microservices + serial: + execs: + # Parallel preparation + - ref: build all-images + + # Serial deployment (order matters) + - cmd: kubectl apply -f database.yaml + - cmd: kubectl wait --for=condition=ready pod -l app=database + - cmd: kubectl apply -f api.yaml + - cmd: kubectl apply -f web.yaml + - cmd: kubectl apply -f worker.yaml +``` + +> [!NOTE] +> **Cross-workspace references**: To reference executables from other workspaces, they must have `visibility: public` in their configuration. Private, internal, and hidden executables cannot be referenced from other workspaces. + +### Error Handling and Retries + +Build resilient workflows that handle failures gracefully: + +```yaml +executables: + - verb: deploy + name: resilient + serial: + failFast: false # Continue on failures + execs: + # Retry flaky operations + - cmd: curl -f https://api.example.com/health + retries: 3 + + # Continue even if optional steps fail + - cmd: ./optional-cleanup.sh || true + + # Critical step with custom error handling + - cmd: | + if ! kubectl apply -f deployment.yaml; then + echo "Deployment failed, rolling back..." + kubectl rollout undo deployment/app + exit 1 + fi +``` + +### Review Gates + +Add human approval steps for critical operations: + +```yaml +executables: + - verb: deploy + name: production + serial: + execs: + - cmd: ./build-production.sh + - cmd: ./run-smoke-tests.sh + + # Pause for human review + - reviewRequired: true + cmd: echo "Review deployment artifacts before continuing" + + - cmd: ./deploy-to-production.sh + - cmd: ./notify-team.sh +``` + +## Environment Variable Handling + +Understanding how environment variables are resolved and prioritized in flow executables. + +### Resolution Order + +Environment variables are resolved in this order (highest to lowest priority): + +1. **Command-line overrides** (`--param`) +2. **Executable `params`** (secretRef, prompt, text) +3. **Executable `args`** (positional and flag arguments) +4. **Shell environment** (inherited from your terminal) + +```yaml +executables: + - verb: deploy + name: app + exec: + params: + - text: "staging" + envKey: ENVIRONMENT + - secretRef: api-key + envKey: API_KEY + args: + - flag: verbose + envKey: VERBOSE + type: bool + default: false + cmd: ./deploy.sh +``` + +**Resolution example:** +```shell +# Shell environment +export ENVIRONMENT=development +export API_KEY=shell-key +export VERBOSE=true + +# Command execution +flow deploy app verbose=false --param ENVIRONMENT=production + +# Final environment variables: +# ENVIRONMENT=production (--param override wins) +# API_KEY= (params wins over shell) +# VERBOSE=false (args wins over shell) +``` + +### Environment Variable Expansion + +Environment variables are expanded in certain contexts: + +**Directory paths:** +```yaml +executables: + - verb: backup + name: logs + exec: + dir: "$HOME/backups" # Expands to /home/user/backups + cmd: cp /var/log/app.log . +``` + +**Command strings:** +```yaml +executables: + - verb: deploy + name: app + exec: + params: + - prompt: "Environment?" + envKey: ENV + cmd: | + echo "Deploying to $ENV" + kubectl config use-context $ENV-cluster + kubectl apply -f k8s/$ENV/ +``` + +### Special Environment Variables + +flow provides special environment variables automatically: + +- `FLOW_CURRENT_WORKSPACE` - Current workspace name +- `FLOW_CURRENT_NAMESPACE` - Current namespace +- `FLOW_WORKSPACE_PATH` - Full path to workspace +- `FLOW_EXECUTABLE_NAME` - Name of current executable +- `FLOW_DEFINITION_DIR` - Directory containing the current flow file +- `FLOW_TMP_DIR` - Temporary directory for current execution, if `f:tmp` is set + +### Environment Inheritance + +Child executables inherit environment variables from their parents: + +```yaml +executables: + - verb: deploy + name: full-stack + serial: + params: + - prompt: "Environment?" + envKey: DEPLOY_ENV + - secretRef: "${DEPLOY_ENV}/api-key" + envKey: API_KEY + execs: + # These inherit DEPLOY_ENV and API_KEY + - ref: deploy backend + - ref: deploy frontend + - cmd: echo "Deployed to $DEPLOY_ENV" + + - verb: deploy + name: backend + exec: + # Automatically has access to DEPLOY_ENV and API_KEY + cmd: | + echo "Deploying backend to $DEPLOY_ENV" + ./deploy-backend.sh --env $DEPLOY_ENV --key $API_KEY +``` diff --git a/docs/guide/concepts.md b/docs/guide/concepts.md new file mode 100644 index 00000000..d13e0144 --- /dev/null +++ b/docs/guide/concepts.md @@ -0,0 +1,262 @@ +# Core Concepts + +flow is built around three key concepts that work together to organize and run your automation workflows. + +## Workspaces + +A **workspace** is a directory containing your flow files and executables. Think of it as a project folder that flow knows about. + +```shell +# Register any directory as a workspace +flow workspace add my-project ~/code/my-project --set + +# Switch between workspaces +flow workspace switch my-project +``` + +Each workspace has a `flow.yaml` config file that defines workspace-specific settings like which directories to search for executables. + +**Key points:** +- Workspaces can be anywhere on your system +- You can have multiple workspaces for different projects +- flow automatically discovers executables within registered workspaces +- Workspaces can be configured to customize behavior and discovery + +> **Learn more**: See the [Workspaces guide](workspaces.md) for complete workspace management, configuration, and organization patterns. + +## Executables + +An **executable** is a task or workflow defined in a flow file (`.flow`, `.flow.yaml`, or `.flow.yml`). Executables are the building blocks of your automation. + +```yaml +# hello.flow +executables: + - verb: run + name: hello + exec: + cmd: echo "Hello, world!" + + - verb: deploy + name: app + serial: + execs: + - cmd: npm run build + - cmd: docker build -t app . + - cmd: kubectl apply -f deployment.yaml +``` + +**Key points:** +- Executables are defined in YAML files within your workspaces +- They can be simple commands or complex multi-step workflows +- Each executable has a verb (like `run`, `build`, `deploy`) and optional name +- You can compose executables by referencing other executables + +> **Learn more**: See the [Executables guide](executables.md) for complete configuration details and all executable types. + +## Secrets Vault + +The **vault** securely stores sensitive information like API keys, passwords, and tokens that your executables need. + +```shell +# Create a vault +flow vault create my-vault + +# Add secrets +flow secret set api-key +flow secret set database-url +``` + +```yaml +# Use in executables +executables: + - verb: deploy + name: app + exec: + params: + - secretRef: api-key + envKey: API_KEY + - secretRef: database-url + envKey: DATABASE_URL + cmd: ./deploy.sh +``` + +**Key points:** +- Secrets are encrypted and stored locally +- Multiple vault types supported (AES, Age, external tools _coming soon_) +- Secrets are passed to executables as environment variables +- Vaults can be switched for different environments or projects + +> **Learn more**: See the [Working with secrets guide](secrets.md) for complete vault setup and secret management. + +## How They Work Together + +Here's how these concepts connect: + +1. **Workspaces** contain your flow files and organize your projects +2. **Executables** defined in those files automate your tasks +3. **Secrets** from the vault provide secure configuration for those executables + +``` +Workspace (my-project/) +├── flow.yaml # Workspace config +├── api.flow # API-related executables +├── deploy.flow # Deployment executables +└── scripts/ + └── deploy.sh + +Vault (encrypted) +├── api-key +├── database-url +└── ssl-cert +``` + +## Running Executables + +### Basic Execution + +The main command for running executables is `flow exec`, but you can use any verb: + +```shell +# These are equivalent +flow exec my-task +flow run my-task + +# Use the configured verb +flow build my-app +flow test integration +``` + +### Executable References + +Executables are identified by their unique verb and ID using the format `workspace/namespace:name`: + +```shell +# Full reference +flow send my-workspace/api:request + +# Current workspace assumed +flow send api:request + +# Current workspace and namespace assumed +flow send request + +# Nameless executable (verb only) +flow build +``` + +**Verbs** + +The "verb" is a single word that describes the operation being executed. It can be configured in the flowfile for +each executable. + +When running in the CLI, the configured verb can be replaced with any synonym/alias that describes the operation. + +For instance, `flow test my-app` is equivalent to `flow validate my-app`. This allows for a more natural language-like +interaction with the CLI, making it easier to remember and use. +*See the [verb reference](../types/flowfile.md#verb-groups) for a list the default verbs and their synonym mappings.* + +> [!TIP] +> Create shell aliases for commonly used verbs to make running executables easier. For example: +> ```shell +> alias build="flow build" +> ``` +> This allows you to run `build my-app` instead of `flow build my-app` or the alias `flow package my-app`. + + +### Command-Line Overrides + +Override or set additional environment variables using the `--param` flag: + +```shell +flow deploy app --param API_KEY=override-value --param DRY_RUN=true --param VERBOSE=true +``` + +### Custom Verb Aliases + +Workspaces can customize which verb aliases are available. This allows you to: + +- **Use custom aliases**: Define your own preferred aliases for verbs +- **Disable default aliases**: Set an empty map `{}` to disable all verb aliases +- **Selective aliases**: Only enable specific aliases for certain verbs + +```yaml +# In workspace flow.yaml +verbAliases: + run: ["exec", "start"] # `run` executables can use `exec` or `start` + build: ["compile"] # `build` executables can use `compile` + # No entry for `test` means no aliases for test executables + +# To disable all verb aliases: +verbAliases: {} +``` + +### Discovery and Sync + +When you create, move, or delete flow files, update the executable index: + +```shell +# Sync executables +flow sync + +# Or sync automatically before running +flow exec my-task --sync +``` + +## Organization Features + +flow provides several ways to organize and find your executables: + +**Workspaces** - Organize projects and configure discovery: +```yaml +# In workspace flow.yaml +displayName: "My API Project" +description: "REST API and deployment tools" +tags: ["api", "production"] +``` + +**Namespaces** - Group related executables within a flow file: +```yaml +namespace: api +executables: + - name: start + - name: stop + - name: restart +``` + +**Tags** - Label executables for easy filtering: +```yaml +executables: + - name: deploy + tags: [production, critical] +``` + +**Verbs** - Describe what an executable does (run, build, test, deploy, etc.) + +**Visibility** - Control who can see and run executables (public, private, internal, hidden) + +> **Learn more**: See the [Executables guide](executables.md) for complete configuration options and the [Interactive UI guide](interactive.md) for filtering and search features. + +## Discovery and Execution + +Once you understand these concepts, using flow becomes straightforward: + +```shell +# Discover executables +flow browse # Interactive browser +flow browse --list # Interactive simple list +flow browse --workspace api # Filter by workspace +flow browse --tag production # Filter by tags + +# Run executables +flow run hello # By name +flow build my-project/app # By full reference +``` + +## What's Next? + +Now that you understand the core concepts: + +- **Build something** → [Your first workflow](first-workflow.md) +- **Secure your automation** → [Working with secrets](secrets.md) +- **Explore advanced features** → [Advanced workflows](advanced.md) +- **Customize your experience** → [Interactive UI](interactive.md) diff --git a/docs/guide/conditional.md b/docs/guide/conditional.md deleted file mode 100644 index 84edc885..00000000 --- a/docs/guide/conditional.md +++ /dev/null @@ -1,126 +0,0 @@ -## Conditional Expressions - -flow CLI uses conditional expressions to control [executable](executable.md) behavior based on runtime conditions. These expressions are written -using a simple expression language that provides access to system information, environment variables, and stored data. - -### Expression Language - -flow uses the [Expr](https://expr-lang.org) language for evaluating conditions. The language supports common -operators and functions while providing access to flow executable-specific context data. - -**See the [Expr language documentation](https://expr-lang.org/docs/language-definition) for more information on the -expression syntax.** - -#### Basic Operators - -The expression language supports standard comparison and logical operators: - -- Comparison: `==`, `!=`, `<`, `>`, `<=`, `>=` -- Logical: `and`, `or`, `not` -- String: `+` (concatenation), `matches` (regex matching) -- Length: `len()` - -#### Available Context - -When writing conditions, you have access to several context variables: - -- `os`: Operating system (e.g., "linux", "darwin", "windows") -- `arch`: System architecture (e.g., "amd64", "arm64") -- `ctx`: Flow context information - - `workspace`: Current workspace name - - `namespace`: Current namespace - - `workspacePath`: Path to current workspace - - `flowFilePath`: Path to current flow file - - `flowFileDir`: Directory containing current flow file -- `store`: Key-value map of data store contents -- `env`: Map of environment variables - -### Writing Conditions - -Conditions can be used in various places within flow, most commonly in the `if` field of executable configurations. Here are -some examples of common conditional patterns: - -#### Operating System and Architecture Checks - -Check for specific operating systems or architectures: - -```yaml -executables: - - verb: install - name: system-specific - serial: - execs: - - if: os == "darwin" - cmd: brew install myapp - - if: os == "linux" - cmd: apt-get install myapp - - if: arch == "amd64" - cmd: make build-amd64 - - if: arch == "arm64" - cmd: make build-arm64 -``` - -#### Environment Variable Checks - -Make decisions based on environment variables: - -```yaml -executables: - - verb: deploy - name: env-check - serial: - execs: - - if: env["ENVIRONMENT"] == "production" - cmd: echo "Deploying to production" - - if: env["DEBUG"] == "true" - cmd: echo "Debug mode enabled" -``` - -#### Data Store Conditions - -Use stored data to control execution: - -```yaml -executables: - - verb: run - name: data-check - serial: - execs: - - cmd: flow store set feature-flag enabled - - if: store["feature-flag"] == "enabled" - cmd: echo "Feature is enabled" - - if: len(store["optional-key"]) > 0 - cmd: echo "Optional key exists" -``` - -#### Complex Conditions - -Combine multiple conditions using logical operators: - -```yaml -executables: - - verb: build - name: complex-check - serial: - execs: - - if: os == "linux" and env["CI"] == "true" - cmd: echo "Running in Linux CI environment" - - if: len(store["build-id"]) > 0 and (os == "darwin" or os == "linux") - cmd: echo "Valid build on Unix-like system" -``` - -#### Path and Location Checks - -Use context information to make path-based decisions: - -```yaml -executables: - - verb: setup - name: path-check - serial: - execs: - - if: ctx.workspace == "development" - cmd: echo "development workspace is active" - - if: ctx.flowFileDir matches ".*/scripts$" - cmd: echo "In scripts directory" -``` diff --git a/docs/guide/executable.md b/docs/guide/executable.md deleted file mode 100644 index e0622c10..00000000 --- a/docs/guide/executable.md +++ /dev/null @@ -1,420 +0,0 @@ -## Finding Executables - -Executables are customizable actions defined in a YAML [flowfile](#flowfile). There are a few [flow library](../cli/flow_browse.md) -command that can be used to find executables: - -```shell -flow browse # Multi-pane view for browsing executables -flow browse --list # Single-pane view for browsing executables -flow browse VERB ID # View documentation for a single executable -``` - -The `flow browse` command accept optional command-line flags to filter the list of -executables by workspace, namespace, verb, or tag: - -```shell -flow browse --workspace ws --namespace ns --verb exec --tag my-tag -# additionally, the --all flag can be used to show executables from all namespaces and the -# --filter flag can be used to search the executable names and descriptions -flow browse --all --filter "search string" -``` - -## Running Executables - -The [flow exec](../cli/flow_exec.md) command is used -to run executables, which can be individual tasks or workflows ([serial](#serial) & [parallel](#parallel)). - -```shell -flow VERB EXECUTABLE_ID [flags] [args] -# by default, the verb is 'exec' -flow exec EXECUTABLE_ID [flags] [args] -``` - -**Verbs** - -The "verb" is a single word that describes the operation being executed. It can be configured in the flowfile for -each executable. - -When running in the CLI, the configured verb can be replaced with any synonym/alias that describes the operation. - -For instance, `flow test my-app` is equivalent to `flow validate my-app`. This allows for a more natural language-like -interaction with the CLI, making it easier to remember and use. -*See the [verb reference](../types/flowfile.md#verb-groups) for a list all verbs and their synonyms.* - -**Custom Verb Aliases** - -You can customize which verb aliases are available in your workspace by configuring `verbAliases` in your [workspace configuration](workspace.md#workspace-configuration). This allows you to: - -- **Use custom aliases**: Define your own preferred aliases for verbs -- **Disable default aliases**: Set an empty map `{}` to disable all verb aliases -- **Selective aliases**: Only enable specific aliases for certain verbs - -```yaml -# In your workspace flow.yaml -verbAliases: - run: ["exec", "start"] # `run` executables can be called with `exec` or `start` - build: ["compile"] # `build` executables can be called with `compile` - # No entry for `test` means no aliases for test executables - -# To disable all verb aliases: -verbAliases: {} -``` - -With the above configuration, `flow start my-app` would run a `run` executable, but `flow execute my-app` would fail because `execute` is no longer a valid alias. - -> [!TIP] -> Create shell aliases for commonly used verbs to make running executables easier. For example: -> ```shell -> alias build="flow build" -> ``` -> This allows you to run `build my-app` instead of `flow build my-app` or the synonym `flow package my-app`. - -**Executable IDs** - -Executables are identified by their unique ID, which is a combination of the workspace, namespace, and name - using the -format `workspace/namespace:name`. If the workspace and namespace are omitted (`ws/name`, `ns:name`, or just `name`), -the current workspace and namespace are assumed. See the [workspace guide](workspace.md) for more information on workspaces and namespaces. - -The name of an executable can also be replaced with an alias if one is defined in the flowfile. - -**Nameless Executables** - -If an executable is defined without a namespace and name, it is considered "nameless" and can be run by its verb alone. -Nameless executables cannot exist within a namespace. - -```shell -# run the current workspaces nameless executable with the verb 'validate' -flow validate -# run the (non-current) 'my-project' workspaces nameless executable with the verb 'build' -flow build my-project/ -``` - -## Flowfile - -The flowfile is the primary configuration file that defines what an executable should do. The file is written in YAML but -should have a `.flow`, `.flow.yaml`, or `.flow.yml` extension. - -The [flow sync](../cli/flow_sync.md) command is used to trigger a discovery of executables within workspaces. This -command should be run whenever an executable is created, moved, or deleted from a flowfile. - -```shell -flow sync -# alternatively, use the --sync flag with any other command to sync before running -flow exec my-task --sync -``` - -On sync, the CLI reads all flowfiles in your workspaces and updates the index of executables. You can configure where -the CLI should look for flowfiles in your [workspace configuration](workspace.md). - -**Example Structure** - -Below is an example of a flowfile. It contains a single executable named `my-task` that prints a message to the console. -_See the [flowfile reference](../types/flowfile.md) for a more detailed explanation of the flowfile schema._ - -```yaml -visibility: internal -namespace: example -executables: - - name: my-task - exec: - cmd: echo "Hello, world!" -``` - -### Executables - -The only required field in an executable's configuration is the `name` (if the `namespace` is unset). - -Additionally, you can define the following fields: - -- **visibility**: The visibility of the executable. Can be `public`, `private`, `internal`, or `hidden`. - - `public`: viewable in the library and can be run from any workspace. - - `private`: viewable in the library but can only be run if your current workspace is the same as the executable's workspace. - - `internal`: not viewable in the library and can only be run from the same workspace. - - `hidden` not viewable in the library and cannot be run from the CLI. -- **description**: Markdown description of the executable to display in the library. -- **tags**: A list of tags to categorize the executable. -- **aliases**: A list of alternative names for the executable. -- **timeout**: The maximum time the executable is allowed to run before being terminated. - -One of the following executable types must be defined: - -- [exec](#exec): Execute a command directly in the shell. -- [serial](#serial): Run a list of executables sequentially. -- [parallel](#parallel): Run a list of executables concurrently. -- [launch](#launch): Open a service or application. -- [request](#request): Make HTTP requests to APIs. -- [render](#render): Generate and view markdown created dynamically with templates or configurations. - -#### Environment variables - -Environment variables are used to customize executable behavior without modifying the flowfile itself. -In addition to inheriting environment variables from the shell, you can define custom environment variables by -setting `params` or `args` in an executable's configuration. Here's an overview of the different options: - -**Params** - -```yaml -executables: - - verb: "deploy" - name: "devbox" - exec: - file: "dev-deploy.sh" - params: - # secret param - - secretRef: "dev/api-token" - envKey: "API_TOKEN" - - secretRef: "admin-user" - envKey: "ADMIN_USER" - # prompt / form param - - prompt: "What application are you deploying?" - envKey: "APP_NAME" - # static param - - text: "false" - envKey: "DRY_RUN" -``` - -In the example above, the `devbox` executable has four parameters: `API_TOKEN`, `ADMIN_USER` ,`APP_NAME`, and `PORT`. -The `secretRef` parameter type is used to reference a secret stored in the vault (see the [secret vault](secret.md) guide -for more information). The `prompt` parameter type prompts the user for input when the executable is run. The `text` -parameter type sets a static value for the environment variable. - -_This example used the `exec` type, but the `params` field can be used with any executable type._ - -You can override any environment variable defined in the `params` or provide additional ones by using the `--param` flag -when running the executable: - -```shell -flow deploy devbox --param API_TOKEN=token --param DRY_RUN=true --param VERBOSE=value -``` - -**Args** - - -```yaml -executables: - - verb: "build" - name: "container" - exec: - file: "build-image.sh" - args: - # positional argument - - pos: 1 - envKey: "TAG" - required: true - # flag arguments - - flag: "publish" - envKey: "PUBLISH" - type: "bool" - - flag: "builder" - envKey: "BUILDER" - default: "podman" -``` - -In the example above, the `container` executable has three arguments: `TAG`, `PUBLISH`, and `BUILDER`. -The `pos` argument type is a positional CLI argument that must be provided when running the executable. The `flag` argument -type is a "flag" that can be set when running the executable. The `type` field can be used to specify the type to validate -the argument against. - -Here is an example of how this would be run: - -```shell -flow build container 1.0.0 publish=true builder=docker -``` - -_This example used the `exec` type, but the `args` field can be used with any executable type._ - -#### Changing directories - -You can use the `dir` field in the executable configuration to specify the working directory for the executable. By default, -the working directory is the directory where the flowfile is located. The path can include environment variables in the -form of `$VAR`. Additionally, the following prefixes can be used to reference specific directories: - -- `//`: workspace root directory -- `~/`: user home directory -- `./`: current working directory - -Other values are assumed to be relative to the flowfile's directory. - -Specifying the value `f:tmp` will create a temporary directory that is automatically cleaned up after the executable finishes. - -```yaml -executables: - - verb: "clean" - name: "downloads" - exec: - file: "cleanup.sh" - dir: "$HOME/downloads" - - verb: "build" - name: "app" - exec: - file: "build.sh" - dir: "//" - - verb: "test" - name: "unit" - exec: - cmd: "cp $HOME/unit-tests.sh . && ./unit-tests.sh" - dir: "f:tmp" -``` - -_This example used the `exec` type, but the `dir` field can be used with the `serial` and `parallel` types as well._ - -### Executable Type Examples - -> [!TIP] -> Check out the [flowfile examples](https://github.com/flowexec/examples) found on GitHub for more examples -> of executable configurations. - -##### exec - -The `exec` type is used to run a command directly in the shell. The command can be a single command or a script file. - -```yaml -fromFile: - - "generated.sh" -executables: - - verb: "init" - name: "chezmoi" - exec: - file: "init-chezmoi.sh" - - verb: "apply" - name: "dotfiles" - exec: - cmd: "chezmoi apply" -``` - -**Generated Executable** - -Executables can also be generated from comments in a `.sh` file. Include the file name in the `fromFile` field of the flowfile -and add flow comments to the file. The comments should be in the format `f:key=value` to define the executable properties. - -The following keys are supported: - -- `name` -- `verb` -- `description` -- `tags` -- `aliases` -- `visibility` -- `timeout` - -Here is an example of valid script with flow comments. Note that any comments after the first non-comment line will be -ignored. - -```shell -#!/bin/sh - -# f:name=generated f:verb=run -# f:description="start of the description" -# f:tags=example,generated -# I'm ignoring this comment -# -# -# continued description -# - -echo "Hello from a generated executable!" -```` - -##### serial - -The `serial` type is used to run a list of executables sequentially. For each `exec` in the list, you must define -either a `ref` to another executable or a `cmd` to run directly in the shell. - -The [executable environment variables](#environment-variables) and [executable directory](#changing-directories) -of the parent executable are inherited by the child executables. - -```yaml -executables: - - verb: "setup" - name: "flow-system" - serial: - failFast: true # Setting `failFast` will stop the execution if any of the executables fail - execs: - - ref: "upgrade flow:cli" - - ref: "install flow:workspaces" - args: ["all"] # When referring to another executable that requires arguments, you can pass them in the `args` field - retries: 2 # retry the executable up to 2 times if it fails - - ref: "apply flow:config" - reviewRequired: true # serial execution will pause here for review (user input) - - cmd: "flow sync" -``` - -##### parallel - -The `parallel` type is used to run a list of executables concurrently. For each `exec` in the list, you must define -either a `ref` to another executable or a `cmd` to run directly in the shell. - -The [executable environment variables](#environment-variables) and [executable directory](#changing-directories) -of the parent executable are inherited by the child executables. - -```yaml -executables: - - verb: "deploy" - name: "apps" - parallel: - failFast: true # Setting `failFast` will stop the execution if any of the executables fail - maxThreads: 2 # Setting `maxThreads` will limit the number of concurrent threads that can be run at once - execs: - - ref: "deploy helm-app" - args: ["tailscale"] # When referring to another executable that requires arguments, you can pass them in the `args` field - - ref: "deploy helm-app" - args: ["metallb"] - - ref: "setup gloo-gateway" - retries: 1 # retry the executable up to 1 time if it fails - - cmd: "kubectl apply -f external-services.yaml" -``` - -##### launch - -The `launch` type is used to open a service or application. The `uri` field is required and can include environment variables -(including those resolved from params and args) - -```yaml -executables: - - verb: "open" - name: "workspace" - launch: - uri: "$FLOW_WORKSPACE_PATH" - app: "Visual Studio Code" # optional application to open the URI with - wait: true # wait for the application to close before continuing -``` - -##### request - -The `request` type is used to make HTTP requests to APIs. The `url` field is required, and the `method` field defaults -to `GET`. - -Additionally, you can define the `body` field to include a request body and the `headers` field to include custom headers. - -```yaml -executables: - - verb: "pause" - name: "pihole" - request: - method: "POST" - url: "http://pi.hole/admin/api.php?disable=$DURATION&auth=$PWHASH" - logResponse: true # log the response body - validStatusCodes: [200] # only consider the execution successful if the status code is 200 - # transform the response body with a Expr expression - transformResponse: | - "paused: " + string(fromJSON(body)["status"] == "disabled") -``` - -##### render - -The `render` type is used to generate and view markdown created dynamically with templates or configurations. -The `templateFile` field is required and can include environment variables (including those resolved from params and args). - -The markdown template can include [Go template](https://pkg.go.dev/text/template) syntax to dynamically generate content. -[Sprig functions](https://masterminds.github.io/sprig/) are also available for use in the template. - -```yaml -executables: - - verb: "show" - name: "cluster-summary" - render: - templateFile: "cluster-template.md" - # Optionally, you can define a data file to use with the template - # It can be a JSON or YAML file. - templateDataFile: "kubectl-out.json" -``` diff --git a/docs/guide/executables.md b/docs/guide/executables.md new file mode 100644 index 00000000..78dcb987 --- /dev/null +++ b/docs/guide/executables.md @@ -0,0 +1,448 @@ +# Executables + +Executables are the building blocks of flow automation. They can be simple commands, complex multi-step workflows, HTTP requests, or even GUI applications. +This guide covers all executable types and configuration options. + +## Finding Executables + +Use the `flow browse` command to discover executables across your workspaces: + +```shell +flow browse # Interactive multi-pane browser +flow browse --list # Simple list view +flow browse VERB ID # View specific executable details +``` + +Filter executables by workspace, namespace, verb, or tag: + +```shell +flow browse --workspace api --namespace v1 --verb deploy --tag production +flow browse --all --filter "database" # Search names and descriptions +``` + +## Executable Configuration + +### Basic Structure + +Every executable needs a verb and optionally a name: + +```yaml +executables: + - verb: run + name: my-task + description: "Does something useful" + tags: [development, automation] + aliases: [task, job] + timeout: 5m + visibility: public + exec: + cmd: echo "Hello, world!" +``` + +### Common Fields + +- **verb**: Action type (run, build, test, deploy, etc.) +- **name**: Unique identifier within the namespace +- **description**: Markdown documentation for the executable +- **tags**: Labels for categorization and filtering +- **aliases**: Alternative names for the executable +- **timeout**: Maximum execution time (e.g., 30s, 5m, 1h) +- **visibility**: Access control (public, private, internal, hidden) + +### Visibility Levels + +- **public**: Available from any workspace +- **private**: Only available within the same workspace but shown in browse lists (default) +- **internal**: Available within workspace but hidden from browse lists +- **hidden**: Cannot be run or listed + +## Environment Variables + +Customize executable behavior with environment variables using `params` or `args`. + +### Parameters (`params`) + +Set environment variables from various sources: + +```yaml +executables: + - verb: deploy + name: app + exec: + file: deploy.sh + params: + # From secrets + - secretRef: api-token + envKey: API_TOKEN + - secretRef: production/database-url + envKey: DATABASE_URL + + # Interactive prompts + - prompt: "Which environment?" + envKey: ENVIRONMENT + + # Static values + - text: "production" + envKey: DEPLOY_ENV +``` + +**Parameter types:** +- `secretRef`: Reference to vault secret +- `prompt`: Interactive user input +- `text`: Static value + +### Arguments (`args`) + +Handle command-line arguments: + +```yaml +executables: + - verb: build + name: container + exec: + file: build.sh + args: + # Positional argument + - pos: 1 + envKey: IMAGE_TAG + required: true + + # Flag arguments + - flag: publish + envKey: PUBLISH + type: bool + default: false + + - flag: registry + envKey: REGISTRY + default: "docker.io" +``` + +**Run with arguments:** +```shell +flow build container v1.2.3 publish=true registry=my-registry.com +``` + +**Argument types:** +- `pos`: Positional argument (by position number, starting from 1) +- `flag`: Named flag argument +- `type`: Validation type (string, int, float, bool) + +### Command-Line Overrides + +Override any environment variable with `--param`: + +```shell +flow deploy app --param API_TOKEN=override --param ENVIRONMENT=staging +``` + +## Working Directories + +Control where executables run with the `dir` field: + +```yaml +executables: + - verb: build + name: frontend + exec: + cmd: npm run build + dir: "./frontend" # Relative to flowfile + + - verb: clean + name: downloads + exec: + cmd: rm -rf downloads/* + dir: "~/Downloads" # User home directory + + - verb: deploy + name: from-root + exec: + cmd: kubectl apply -f k8s/ + dir: "//" # Workspace root + + - verb: test + name: isolated + exec: + cmd: | + echo "Running in temporary directory" + ls -la + dir: "f:tmp" # Temporary directory (auto-cleaned) +``` + +**Directory prefixes:** +- `//`: Workspace root directory +- `~/`: User home directory +- `./`: Current working directory +- `f:tmp`: Temporary directory (auto-cleaned) +- `$VAR`: Environment variable expansion + +## Executable Types + +### exec - Shell Commands + +Run commands or scripts directly: + +```yaml +executables: + - verb: build + name: app + exec: + cmd: npm run build && npm test + + - verb: deploy + name: app + exec: + file: deploy.sh + logMode: json # text, logfmt, json, or hidden +``` + +**Options:** +- `cmd`: Inline command to run +- `file`: Script file to execute +- `logMode`: How to format command output + +### serial - Sequential Execution + +Run multiple steps in order: + +```yaml +executables: + - verb: deploy + name: full-stack + serial: + failFast: true # Stop on first failure + execs: + - cmd: docker build -t api . + - cmd: docker build -t web ./frontend + - ref: test api + - cmd: kubectl apply -f k8s/ + retries: 3 + - cmd: kubectl rollout status deployment/api + reviewRequired: true # Pause for user confirmation +``` + +The [executable environment variables](#environment-variables) and [executable directory](#working-directories) +of the parent executable are inherited by the child executables. + +**Options:** +- `failFast`: Stop execution on first failure (default: true) +- `retries`: Number of times to retry failed steps +- `reviewRequired`: Pause for user confirmation + +### parallel - Concurrent Execution + +Run multiple steps simultaneously: + +```yaml +executables: + - verb: test + name: all-suites + parallel: + maxThreads: 4 # Limit concurrent operations + failFast: false # Run all tests even if some fail + execs: + - cmd: npm run test:unit + - cmd: npm run test:integration + - cmd: npm run test:e2e + - ref: lint code + retries: 1 +``` + +The [executable environment variables](#environment-variables) and [executable directory](#working-directories) +of the parent executable are inherited by the child executables. + +**Options:** +- `maxThreads`: Maximum concurrent operations (default: 5) +- `failFast`: Stop all operations on first failure (default: true) +- `retries`: Number of times to retry failed operations + +### launch - Open Applications + +Open files, URLs, or applications: + +```yaml +executables: + - verb: open + name: workspace + launch: + uri: "$FLOW_WORKSPACE_PATH" + app: "Visual Studio Code" + + - verb: open + name: docs + launch: + uri: "https://flowexec.io" + + - verb: open + name: note + launch: + uri: "./note.md" + app: "Obsidian" +``` + +**Options:** +- `uri`: File path or URL to open (required) +- `app`: Specific application to use + +### request - HTTP Requests + +Make HTTP requests to APIs: + +```yaml +executables: + - verb: deploy + name: webhook + request: + method: POST + url: "https://api.example.com/deploy" + headers: + Authorization: "Bearer $API_TOKEN" + Content-Type: "application/json" + body: | + { + "environment": "$ENVIRONMENT", + "version": "$VERSION" + } + timeout: 30s + validStatusCodes: [200, 201] + logResponse: true + transformResponse: | + "Deployment " + fromJSON(data)["status"] + responseFile: + filename: "deploy-response.json" +``` + +**Options:** +- `method`: HTTP method (GET, POST, PUT, PATCH, DELETE) +- `url`: Request URL (required) +- `headers`: Custom headers +- `body`: Request body +- `timeout`: Request timeout +- `validStatusCodes`: Acceptable status codes +- `logResponse`: Log response body +- `transformResponse`: Transform response with Expr +- `responseFile`: Save response to file + +### render - Dynamic Documentation + +Generate and display markdown with templates: + +```yaml +executables: + - verb: show + name: status + render: + templateFile: "status-template.md" + templateDataFile: "status-data.json" +``` + +**Template file example:** +```markdown +# System Status + +Current time: {{ .timestamp }} + +## Services +{{- range .services }} +- **{{ .name }}**: {{ .status }} +{{- end }} + +## Metrics +- CPU: {{ .cpu }}% +- Memory: {{ .memory }}% +``` + +**Options:** +- `templateFile`: Markdown template file (required) +- `templateDataFile`: JSON/YAML data file + +## Generated Executables + +Generate executables from shell scripts with special comments: + +```yaml +# In flowfile +fromFile: + - "scripts/deploy.sh" + - "scripts/backup.sh" +``` + +```bash +#!/bin/bash +# scripts/deploy.sh + +# f:name=production f:verb=deploy +# f:description="Deploy to production environment" +# f:tags=production,critical +# f:aliases=prod-deploy +# f:visibility=internal +# f:timeout=10m + +echo "Deploying to production..." +kubectl apply -f k8s/ +``` + +**Supported comment keys:** +- `name`, `verb`, `description`, `tags`, `aliases`, `visibility`, `timeout` + +**Multi-line descriptions:** +```bash +# f:name=backup f:verb=run +# +# Creates a backup of the database +# and uploads it to S3 storage +# +``` + +## Executable References + +Reference other executables to build modular workflows: + +```yaml +executables: + # Reusable components + - verb: build + name: api + exec: + cmd: docker build -t api . + + - verb: test + name: api + exec: + cmd: npm test + + # Composite workflows + - verb: deploy + name: full + serial: + execs: + - ref: build api + - ref: test api + - cmd: kubectl apply -f api.yaml + + # Cross-workspace references (requires public visibility) + - verb: deploy + name: with-monitoring + serial: + execs: + - ref: deploy full + - ref: trigger monitoring/slack:deployment-complete +``` + +**Reference formats:** +- `ref: build api` - Current workspace/namespace +- `ref: build workspace/namespace:api` - Full reference +- `ref: build workspace/api` - Specific workspace +- `ref: build namespace:api` - Specific namespace + +**Cross-workspace requirements:** +- Referenced executables must have `visibility: public` +- Private, internal, and hidden executables cannot be cross-referenced + +## What's Next? + +Now that you understand all executable types and options: + +- **Build complex workflows** → [Advanced workflows](advanced.md) +- **Secure your automation** → [Working with secrets](secrets.md) +- **Generate project templates** → [Templates & code generation](templating.md) \ No newline at end of file diff --git a/docs/guide/first-workflow.md b/docs/guide/first-workflow.md new file mode 100644 index 00000000..1cd3ae93 --- /dev/null +++ b/docs/guide/first-workflow.md @@ -0,0 +1,209 @@ +# Your First Workflow + +Now that you understand flow's [core concepts](concepts.md), let's build a real workflow that shows how executables, workspaces, and secrets work together. + +We'll create a simple web project deployment workflow that: +- Builds a static site +- Runs tests +- Deploys to a server (simulated) + +## Setup + +Create a new workspace for this tutorial: + +```shell +mkdir ~/flow-tutorial +cd ~/flow-tutorial +flow workspace add tutorial . --set +``` + +## Step 1: Create the Project Structure + +Let's simulate a simple web project: + +```shell +# Create project files +mkdir -p src tests +echo "

My Website

" > src/index.html +echo "console.log('Testing...');" > tests/test.js +echo "build/" > .gitignore +``` + +## Step 2: Build Your First Workflow + +Create a `deploy.flow` file: + +```yaml +# deploy.flow +executables: + - verb: build + name: site + exec: + cmd: | + echo "Building website..." + mkdir -p build + cp -r src/* build/ + echo "✅ Build complete" + + - verb: test + name: site + exec: + cmd: | + echo "Running tests..." + node tests/test.js + echo "✅ Tests passed" + + - verb: deploy + name: full + serial: + execs: + - ref: build site + - ref: test site + - cmd: | + echo "Deploying to server..." + echo "📦 Deployment complete!" +``` + +## Step 3: Run Individual Steps + +Sync and try each step: + +```shell +flow sync + +# Try each step individually +flow build site +flow test site +``` + +## Step 4: Run the Full Workflow + +Now run the complete deployment: + +```shell +flow deploy full +``` + +You'll see each step run in sequence. This is your first multi-step workflow! + +## Step 5: Add Configuration with Secrets + +Real deployments need configuration. Let's add some secrets: + +```shell +# Create a vault for this project +flow vault create tutorial-vault + +# Set the generated key in the default environment variable +export FLOW_VAULT_KEY="" + +# Add some deployment secrets +flow secret set server-url "https://my-server.com" +flow secret set api-key "your-secret-key-here" +``` + +## Step 6: Use Secrets in Your Workflow + +Update your `deploy.flow` to use secrets: + +```yaml +# deploy.flow +executables: + # Build and test steps remain the same + + # Update deploy step to use secrets + - verb: deploy + name: full + serial: + execs: + - ref: build site + - ref: test site + - cmd: | + echo "Deploying to $SERVER_URL..." + echo "Using API key: ${API_KEY:0:8}..." + echo "📦 Deployment complete!" + params: + - secretRef: server-url + envKey: SERVER_URL + - secretRef: api-key + envKey: API_KEY +``` + +Run it again: + +```shell +flow deploy full +``` + +Now your workflow uses secure configuration! + +## Step 7: Add Interactive Elements + +Let's make the workflow more interactive by adding prompts: + +```yaml +# Add this executable to deploy.flow + - verb: deploy + name: interactive + exec: + params: + - prompt: "Which environment? (dev/staging/prod)" + envKey: ENVIRONMENT + - prompt: "Run tests first? (y/n)" + envKey: RUN_TESTS + - secretRef: server-url + envKey: SERVER_URL + cmd: | + echo "Deploying to $ENVIRONMENT environment..." + + if [ "$RUN_TESTS" = "y" ]; then + echo "Running tests first..." + node tests/test.js + fi + + echo "Deploying to $SERVER_URL..." + echo "🚀 $ENVIRONMENT deployment complete!" +``` + +Try the interactive version: + +```shell +flow sync +flow deploy interactive +``` + +## Step 8: Browse Your Workflows + +Use the TUI to explore what you've built: + +```shell +flow browse +``` + +Navigate through your executables, view their details, and run them directly from the interface. + +## Recap + +Congratulations! You've built a workflow that demonstrates: + +✅ **Multi-step workflows** with `serial` executables +✅ **Executable references** with `ref` to reuse steps +✅ **Secret management** with secure configuration +✅ **Interactive prompts** for runtime customization + +### More Examples + +Want to see more workflow patterns? Check out: +- [flow examples repo](https://github.com/flowexec/examples) - Collection of workflow patterns and scaffolding +- [flow project itself](https://github.com/flowexec/flow/tree/main/.execs) - Real development workflows for building flow + +Both contain flow files you can explore and adapt for your own projects. + +## Next Steps + +Ready to level up your flow skills? + +- **Secure more workflows** → [Working with secrets](secrets.md) +- **Build complex automations** → [Advanced workflows](advanced.md) +- **Generate new projects** → [Templates & code generation](templating.md) +- **Customize your experience** → [Interactive UI](interactive.md) diff --git a/docs/guide/integrations.md b/docs/guide/integrations.md index ce9eafff..7d6a3564 100644 --- a/docs/guide/integrations.md +++ b/docs/guide/integrations.md @@ -1,60 +1,77 @@ # Integrations -> **Note:** Integrations are still considered experimental and may change in future releases. Please provide feedback on your experience. +flow integrates with popular CI/CD platforms and containerized environments to bring your automation anywhere. ## GitHub Actions -flow provides seamless integration with GitHub Actions through the [flow-execute](https://github.com/marketplace/actions/flow-execute) marketplace action. This enables you to execute -your flow executables directly within your CI/CD pipelines. - -#### Quick Start +Execute flow workflows directly in your GitHub Actions pipelines with the official action. ```yaml -- uses: flowexec/action@v1 - with: - executable: 'build app' +name: Build and Deploy +on: [push] + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: flowexec/action@v1 + with: + executable: 'build app' ``` -For complete documentation, configuration options, and examples, visit the [Flow Execute Action](https://github.com/marketplace/actions/flow-execute) on the GitHub Marketplace. +> **Complete documentation**: Visit the [Flow Execute Action](https://github.com/marketplace/actions/flow-execute) on GitHub Marketplace. ## Docker -flow can run in containerized environments, making it useful for CI/CD pipelines or isolated execution environments. +Run flow in containerized environments for CI/CD pipelines or isolated execution. -#### Basic Usage +### Basic Usage ```shell +# Run with default workspace docker run -it --rm ghcr.io/flowexec/flow + +# Execute specific executable +docker run -it --rm ghcr.io/flowexec/flow validate ``` -This runs the container with the default flow workspace and shows the version. +**Environment Variables** +- `REPO`: Repository URL to clone (defaults to flow's repo) +- `BRANCH`: Git branch to checkout (optional) +- `WORKSPACE`: Workspace name to use (defaults to "flow") + -#### With Workspace Configuration +### Workspace from Git -You can automatically clone and configure a flow workspace by setting environment variables: +Automatically clone and configure a workspace: ```shell docker run -it --rm \ -e REPO=https://github.com/your-org/your-workspace \ -e BRANCH=main \ -e WORKSPACE=my-workspace \ - ghcr.io/flowexec/flow exec "your-executable" + ghcr.io/flowexec/flow exec "deploy app" ``` -#### Environment Variables - -- `REPO`: Repository URL to clone (defaults to flow's own repo) -- `BRANCH`: Git branch to checkout (optional, defaults to default branch) -- `WORKSPACE`: Workspace name to use (defaults to "flow") +### Local Workspace -#### Volume Mounting - -For persistent data or to use local workspaces: +Mount your local workspace: ```shell docker run -it --rm \ -v $(pwd):/workspaces/my-workspace \ -w /workspaces/my-workspace \ -e WORKSPACE=my-workspace \ - ghcr.io/flowexec/flow "exec your-executable" + ghcr.io/flowexec/flow exec "build app" ``` + +### In CI/CD Pipelines + +Any CI/CD platform that supports Docker can run flow. The key is: + +1. **Use the Docker image**: `ghcr.io/flowexec/flow` +2. **Set environment variables**: `REPO`, `WORKSPACE`, `BRANCH` as needed +3. **Execute your flow commands**: `flow exec "your-executable"` + +> **Note**: While this should work, the Docker integration hasn't been extensively tested. If you try flow with other CI/CD platforms, we'd love to hear about your experience! diff --git a/docs/guide/interactive.md b/docs/guide/interactive.md index 8c5994a1..a2347bc6 100644 --- a/docs/guide/interactive.md +++ b/docs/guide/interactive.md @@ -1,92 +1,218 @@ -## Interactive Configurations +# Interactive UI -The interactive TUI can be customized in the [flow config file](../types/config.md). Additionally, -there are several [flow config commands](../cli/flow_config.md) that can be used to change the TUI settings. +flow provides both a powerful terminal user interface (TUI) and flexible command-line options to fit your workflow. +This guide covers using the interactive browser, customizing the experience, and working with different output formats. -> [!TIP] -> You can view your current settings with the config view command: -> ```shell -> flow config get -> ``` +## Using the TUI Browser -### Changing the TUI theme +The `flow browse` command launches an interactive browser for discovering and running executables. -There are several themes available in the TUI: -- `default` (everforest) -- `light` -- `dark` -- `dracula` -- `tokyo-night` +### Basic Navigation -Use the following command to change the theme: +```shell +flow browse # Launch the interactive browser +``` + +**Keyboard shortcuts:** +- ↑ / ↓ - Move up/down through the list +- ← / → - Navigate between panels (workspaces, executables) +- Enter - Select the highlighted workspace or executable +- Space - Toggle the namespace list for the selected workspace +- Tab - Toggle the executable detail viewer +- R - Run the selected executable (when applicable) +- H - Show help menu with all shortcuts +- Q - Quit the browser + +### Filtering and Search + +Filter executables by various criteria: + +```shell +# Filter by workspace +flow browse --workspace api-service +# Filter by namespace +flow browse --namespace deployment + +# Filter by verb +flow browse --verb deploy + +# Filter by tags +flow browse --tag production --tag critical + +# Search by name or description +flow browse --filter "database backup" + +# Show executables from all namespaces (not just current) +flow browse --all +``` + +**Combine filters for precise results:** ```shell -flow config set theme (default|light|dark|dracula|tokyo-night) +flow browse --workspace api --verb deploy --tag production ``` -**Overriding the theme's colors** +### Running Executables + +**From the browser:** +- Select an executable and press R to run it +- Arguments and prompts will be handled interactively + +**Direct execution:** +```shell +# View specific executable details +flow browse deploy api:production + +# Run without browsing +flow deploy api:production +``` + +## Output Formats + +Control how flow displays information with output format options. + +### TUI vs Non-Interactive + +```shell +# Interactive TUI (default) +flow browse +flow workspace list +flow secret list + +# Simple list output +flow browse --output json +flow workspace list --output json +flow secret list --output yaml +``` + +### Disabling the TUI + +For scripts, CI/CD, or personal preference: + +```shell +# Permanently disable TUI +flow config set tui false + +# Temporarily disable with environment variable +DISABLE_FLOW_INTERACTIVE=true flow browse +``` -Additionally, you can override the theme colors by setting the `colorOverride` field in the config file. Any color not -set in the `colorOverride` field will use the default color for the set theme. -See the [config file reference](../types/config.md#ColorPalette) for more information. +## Customization -### Changing desktop notification settings +### Themes -Desktop notifications can be sent when executables are completed. Use the following command to enable or disable desktop notifications: +Choose from several built-in themes: ```shell -flow config set notifications (true|false) # --sound +# Available themes +flow config set theme default # Everforest (default) +flow config set theme light # Light theme +flow config set theme dark # Dark theme +flow config set theme dracula # Dracula +flow config set theme tokyo-night # Tokyo Night +``` + +### Custom Colors + +Override theme colors by editing your config file: + +```yaml +# In your flow config file +colorOverride: + primary: "#83C092" + secondary: "#D699B6" + background: "#2D353B" + border: "#7FBBB3" + # See config reference for all options ``` -### Changing the log mode +> **Complete reference**: See the [config file reference](../types/config.md#ColorPalette) for all color options. -There are 4 log modes available in the TUI: -- `logfmt`: Includes the log level, timestamp, and log message. -- `text`: Includes just the log message. -- `json`: Includes the log level, timestamp, and log message in JSON format. -- `hidden`: Hides the log messages. +### Notifications -The default log mode is `logfmt`. Use the following command to change the log mode: +Get notified when long-running executables complete: ```shell -flow config set log-mode (logfmt|text|json|hidden) +# Enable desktop notifications +flow config set notifications true + +# Enable notification sound +flow config set notifications true --sound + +# Disable notifications +flow config set notifications false ``` -`exec` executables can be also configured to use a specific log mode. See the [flowfile configuration](../types/flowfile.md#executableexecexecutabletype) for more information. +### Log Display +Control how command output is displayed: + +```shell +# Set global log mode +flow config set log-mode logfmt # Structured logs (default) +flow config set log-mode text # Plain text output +flow config set log-mode json # JSON format +flow config set log-mode hidden # Hide output +``` + +**Per-executable log modes:** ```yaml executables: - - name: my-task + - name: debug-task exec: - logMode: text - cmd: echo "Hello, world!" + logMode: text # Override global setting + cmd: echo "Debug output" ``` -Note: the [flow logs](../cli/flow_logs.md) command will always display logs in `json` mode. +### Workspace Modes + +Control how flow determines your current workspace: -### Changing the workspace mode +```shell +# Dynamic mode - auto-switch based on directory +flow config set workspace-mode dynamic -There are 2 workspace modes available in the TUI: -- `fixed`: The current workspace is fixed to the one you've set with [flow workspace set](../cli/flow_workspace_set.md). -- `dynamic`: The current workspace is determined by your current working directory. If you're in a workspace directory, the TUI will automatically switch to that workspace. Otherwise, the TUI will use the workspace you've set with [flow workspace set](../cli/flow_workspace_set.md). +# Fixed mode - always use set workspace +flow config set workspace-mode fixed +``` -See the [workspace guide](workspace.md) for more information on workspaces. +> **Learn more**: See the [Workspaces guide](workspaces.md) for detailed workspace mode explanations. -### Changing the default executable timeout +### Timeouts -The global default executable timeout is 30 minutes. Use the following command to change the default executable timeout: +Set default timeout for all executables: ```shell -flow config set timeout +# Set global timeout +flow config set timeout 45m + +# Examples: 30s, 5m, 2h +flow config set timeout 10m ``` -### Disable the TUI +## Configuration Management -In some cases, you may want to disable the interactive TUI (in CI/CD pipelines and containers, for example). -Use the following command will switch all TUI commands to their non-interactive counterparts: +### View Current Settings ```shell -flow config set tui false +# View all settings +flow config get + +# View specific setting +flow config get --output json | jq '.theme' ``` -Alternatively, you can set the `DISABLE_FLOW_INTERACTIVE` environment variable to `true` to disable the TUI. +### Reset to Defaults + +```shell +# Reset all configuration +flow config reset + +# Warning: This overwrites all customizations +``` + +### Configuration File Location + +Your config is stored in: +- **Linux**: `~/.config/flow/config.yaml` +- **macOS**: `~/Library/Application Support/flow/config.yaml` diff --git a/docs/guide/secret.md b/docs/guide/secrets.md similarity index 54% rename from docs/guide/secret.md rename to docs/guide/secrets.md index 9b8541e6..46b382c7 100644 --- a/docs/guide/secret.md +++ b/docs/guide/secrets.md @@ -1,22 +1,38 @@ -## Vault Setup +# Working with Secrets -The flow CLI has an integrated vault system that can be used to store secrets securely. -Vaults support multiple encryption backends and can be easily switched between different environments or teams. +flow's built-in vault keeps your sensitive data secure while making it easy to use in your workflows. +Whether you're managing API keys, database passwords, or deployment tokens, the vault has you covered. -#### Creating a New Vault +## Quick Start -To create a new vault, use the `flow vault create` command with a name: +Create your first vault and add a secret: ```shell -flow vault create development +# Create a vault (generates a key and shows it in output) +flow vault create my-vault + +# Set the generated key in the default environment variable +export FLOW_VAULT_KEY="" + +# Add a secret (you'll be prompted for the value) +flow secret set database-password ``` -This creates an AES256-encrypted vault with a randomly generated key that will be displayed in the output. -Store this key securely - if you lose it, you won't be able to access your secrets. +```yaml +# Use it in an executable +executables: + - verb: backup + name: database + exec: + params: + - secretRef: database-password + envKey: DB_PASSWORD + cmd: pg_dump -h localhost -U admin mydb +``` -#### Vault Types +## Vault Types -flow supports multiple vault types: +flow supports multiple vault backends for different security needs: @@ -31,6 +47,9 @@ flow vault create myapp flow vault create myapp --type aes256 ``` +This creates an AES256-encrypted vault with a randomly generated key that will be displayed in the output. +Store this key securely - if you lose it, you won't be able to access your secrets. + **Key Management Options:** ```shell # Store key in environment variable @@ -44,11 +63,11 @@ flow vault create myapp --key-file ~/mykeys/myapp.key If you specify a `--key-env` and that environment variable already contains a valid encryption key, the vault will use that existing key instead of generating a new one: ```shell -# Set a shared key in environment -export SHARED_VAULT_KEY="your-existing-valid-key" - -# Create multiple vaults using the same key +# Create first vault and set the generated key flow vault create dev --key-env SHARED_VAULT_KEY +export SHARED_VAULT_KEY="" + +# Create additional vaults using the same key flow vault create staging --key-env SHARED_VAULT_KEY ``` @@ -83,11 +102,9 @@ flow vault create team --type age --recipients key1,key2,key3 --identity-env MY_ -See the [vault command reference](../cli/flow_vault.md) for more details on managing vaults, including listing existing vaults, switching between them, and deleting vaults. - #### Authentication -The environment variable or file that you provide at setup will be used to resolve the encryption key when accessing the vault. +The environment variable or file that you provide at setup will be used to resolve the encryption key when accessing the vault. If you did not provide a key or file, these default environment variables will be used: - For AES256 vaults: `FLOW_VAULT_KEY` environment variable @@ -106,17 +123,6 @@ flow vault create myapp --key-file ~/mykeys/myapp.key flow vault create team --type age --identity-file ~/identities/identity.txt --identity-env MY_IDENTITY ``` -#### Custom Vault Storage Location - -You can specify a custom storage location for the encrypted data when creating a vault: - -```shell -flow vault create myapp --path /storage/myapp -``` - -This data is encrypted, so you can safely store it as-is without worrying about plaintext secrets being exposed. - - #### Pre-v1 Migration If you have a (pre-v1.0) legacy vault, you can migrate it to a v1 vault: @@ -127,53 +133,84 @@ flow vault create new-vault --key-env MY_NEW_VAULT_KEY flow vault migrate new-vault ``` -This migrates secrets from the old vault format to the new named vault system. Note that this requires the old vault to -be accessible with its key set in the `FLOW_VAULT_KEY` environment variable. - -## Managing Secrets +This migrates secrets from the old vault format to the new named vault system. Note that this requires the old vault to be accessible with its key set in the `FLOW_VAULT_KEY` environment variable. -#### Adding Secrets +## Using Secrets in Workflows -To add a secret to the current vault, use the `flow secret set` command: +### Basic Usage -```shell -# Set a secret with a value -flow secret set api-key "my-secret-value" +```yaml +executables: + - verb: deploy + name: app + exec: + params: + - secretRef: api-key + envKey: API_KEY + - secretRef: database-url + envKey: DATABASE_URL + cmd: ./deploy.sh +``` -# Set a secret interactively (you'll be prompted for the value) -flow secret set api-key +### Cross-Vault References + +Reference secrets from different vaults: + +```yaml +executables: + - verb: sync + name: environments + exec: + params: + - secretRef: production/api-key + envKey: PROD_API_KEY + - secretRef: staging/api-key + envKey: STAGING_API_KEY + cmd: ./sync-environments.sh ``` -Secrets are stored in the currently active vault. Use `flow vault switch` to change which vault receives new secrets. +## Secret Management -#### Retrieving Secrets +### Adding Secrets -**List all secrets in the current vault:** ```shell -flow secret list +# Interactive prompt (recommended) +flow secret set my-secret + +# From command line (less secure) +flow secret set my-secret "secret-value" + +# From file +cat secret.txt | flow secret set my-secret ``` -**Get a specific secret value:** +### Viewing Secrets + ```shell -flow secret get api-key --plaintext -``` +# List all secrets (values hidden) +flow secret list -By default, secret commands don't display actual secret values for security. Use the `--plaintext` flag to view the actual values. +# Get specific secret (obfuscated) +flow secret get my-secret -**Using secrets in executables:** -See the [executable guide](executable.md#environment-variables) for information on how to include secrets as executable environment variables. The `secretRef` provided is equivalent to the key you used when adding the secret to the vault. +# Get plaintext value +flow secret get my-secret --plaintext -#### Removing Secrets +# Copy to clipboard +flow secret get my-secret --copy +``` -To remove a secret from the current vault: +### Updating and Removing ```shell -flow secret remove api-key -``` +# Update a secret (prompts for new value) +flow secret set existing-secret -You can also delete secrets interactively when using the TUI views for listing secrets. +# Remove a secret +flow secret remove old-secret +``` -### Working with Multiple Vaults +### Working with Multiple Vaults When working with multiple vaults, secrets are isolated per vault but the vault's name can be used to reference secrets across vaults. You can retrieve secrets from a specific vault without switching to it by using the vault name as a prefix: @@ -184,13 +221,65 @@ flow secret get production/db-password flow secret get development/api-key ``` -### Backup and Restore +## Vault Management + +See the [vault command reference](../cli/flow_vault.md) for detailed commands and options. -By default, vault data is stored in the flow cache directory, with each vault having its own directory: +### Vault Configuration + +```shell +# View the current vault +flow vault get + +# View specific vault details +flow vault get my-vault + +# Edit vault settings +flow vault edit my-vault --key-env NEW_KEY_VAR + +# Remove vault (data remains, just unlinks) +flow vault remove old-vault +``` + +#### Custom Vault Storage Location + +You can specify a custom storage location for the encrypted data when creating a vault: + +```shell +flow vault create myapp --path /storage/myapp +``` -- **Linux**: `~/.cache/flow/vaults/` or `$XDG_CACHE_HOME/flow/vaults/` -- **MacOS**: `~/Library/Caches/flow/vaults/` +This data is encrypted, so you can safely store it as-is without worrying about plaintext secrets being exposed. + +### Managing Multiple Vaults + +Switch between vaults for different projects or environments: + +```shell +# List all vaults +flow vault list + +# Switch to a different vault +flow vault switch production + +# Work with secrets in current vault +flow secret set api-key +flow secret list +``` + +### Backup and Recovery + +Vault data is stored in your flow config directory: + +```shell +# Find your vaults +ls ~/.config/flow/vaults/ # Linux +ls ~/Library/Caches/flow/vaults/ # macOS + +# Backup (encrypted data is safe to copy) +cp -r ~/.config/flow/vaults/ ~/backups/ +``` -Each vault you create gets its own configuration file and data file. -You can back up these directories to ensure you have a copy of your vaults. +Each vault you create gets its own configuration file and data file. +You can back up these directories to ensure you have a copy of your vaults. Note that if you are using a custom storage path, you should include that in your backup strategy. diff --git a/docs/guide/state.md b/docs/guide/state.md deleted file mode 100644 index 6f0323a5..00000000 --- a/docs/guide/state.md +++ /dev/null @@ -1,159 +0,0 @@ -## State Management - -flow provides several mechanisms for managing state across [executable](executable.md) runs. This guide explores how to use the data store -and temporary directories to maintain state, as well as how to make execution decisions based on that state. - -### Cache - -The cache is a key-value store that persists data across executions. It provides a simple way to share information -between executables and maintain state between runs. - -#### Store Persistence Scope - -Values in the cache have different persistence scopes depending on where they are set: - -- Values set outside an executable (using the CLI directly) persist across all executions until explicitly cleared -- Values set within an executable persist only across that executable's sub-executables (both serial and parallel) -- All values set within an executable are automatically cleared when the parent executable completes - -#### Managing Cache Data - -The cache can be managed at a global level in the CLI and within an executable's script. Here are the key operations: - -**Setting Values** - -```shell -# Direct CLI usage -flow cache set KEY VALUE - -# for example: -flow cache set my-key "my value" -# or pipe a value from a command -echo "my value" | flow cache set my-key -``` - -**Getting Values** - -```shell -# Direct CLI usage -flow cache get KEY - -# for example: -value=$(flow cache get my-key) -``` - -**Clearing Values** - -```shell -# Clear all values -flow cache clear -# Clear with all flag to remove all stored data -flow cache clear --all -``` - -#### Using the Cache in Executables - -Here's an example of how to use the data store within an executable: - -```yaml -executables: - - verb: run - name: data-store-demo - serial: - params: - execs: - # Set some values in the store - - cmd: | - flow cache set user-preference dark-mode - flow cache set last-run "$(date)" - # Use those values in a subsequent step - - cmd: | - preference=$(flow cache get user-preference) - echo "User preference is: $preference" - echo "Last run: $(flow cache get last-run)" -``` - -#### Cache-Based Conditional Execution - -The data store's contents can be accessed in executable `if` conditions using the `data` context variable. This allows for -dynamic execution paths based on stored values: - -```yaml -executables: - - verb: run - name: conditional-demo - serial: - execs: - - cmd: flow cache set feature-enabled true - # This will execute because feature-enabled is set to "true" - - if: data["feature-enabled"] == "true" - cmd: echo "Feature is enabled" - # This will not execute because test-key is not set - - if: len(data["test-key"]) > 0 - cmd: echo "Test key exists" -``` - -See the [Conditional Execution](conditional.md) guide for more examples of using conditions in Flow. - -### Temporary Directories - -Flow provides a special directory reference `f:tmp` that creates an isolated temporary directory for an executable. This -directory is automatically cleaned up when the executable completes. - -#### Using Temporary Directories - -To use a temporary directory, set the `dir` field in your executable configuration: - -```yaml -executables: - - verb: build - name: temp-workspace - exec: - dir: f:tmp # Creates and uses a temporary directory - cmd: | - # All commands run in an isolated temp directory - git clone https://github.com/user/repo . - make build -``` - -#### Sharing Temporary Files - -While temporary directories are isolated, you can share files between steps in a serial or parallel executable by using -the same temporary directory: - -```yaml -executables: - - verb: process - name: shared-temp - serial: - dir: f:tmp # All sub-executables share this temp directory - execs: - - cmd: echo "Step 1" > output.txt - - cmd: cat output.txt && echo "Step 2" >> output.txt - - cmd: cat output.txt -``` - -### Combining State Management Approaches - -The data store and temporary directories can be used together for more complex state management: - -```yaml -executables: - - verb: build - name: complex-state - serial: - dir: f:tmp - execs: - # Generate a build ID and store it - - cmd: | - build_id=$(date +%Y%m%d_%H%M%S) - flow store set current-build $build_id - # Use the stored build ID for conditional execution - - if: len(data["current-build"]) > 0 - cmd: | - echo "Building artifacts for ${build_id}" - make build - # Clean up based on stored state - - if: data["cleanup-enabled"] == "true" - cmd: make clean -``` diff --git a/docs/guide/templating.md b/docs/guide/templating.md index 4b0f88d9..51e25e98 100644 --- a/docs/guide/templating.md +++ b/docs/guide/templating.md @@ -1,271 +1,283 @@ -Templates are a powerful feature in flow that allow you to define reusable flowfile and workspace structures. -This guide will walk you through the process of creating and using templates in flow. It will use a simple example -of create a set of executables for managing a Kubernetes deployment. +# Templates & Workflow Generation -## Registering templates +Templates let you generate new workflows and project scaffolding with interactive forms. +Perfect for creating consistent project structures, operation workflows, or any repeatable automation pattern. -Templates are registered with flow using the [flow template add](../cli/flow_template_add.md) command. This command accepts a -the name to be given to the template and the path to the template file. +## Quick Start -```shell -# Register the k8s-deployment template -flow template add k8s-deployment --file /path/to/k8s-deployment.flow.tmpl -``` - -## Generating scaffolding from a template - -Templates are rendered using the [flow template generate](../cli/flow_template_generate.md) command. This command accepts a -the name to be given to the flowfile (if applicable) when rendering its template and several flags to control the rendering process. +Let's create a simple web app template: ```shell -# Run the kes-deployment template generation in the mealie directory of the homelab workspace. -# The rendered flowfile will be given the name mealie.flow -flow template generate mealie --output mealie --template k8s-deployment --workspace homelab +# Create a template file +touch webapp.flow.tmpl ``` -Alternatively, you can reference a flowfile template directly from a file using the `--file` flag. +```yaml +# webapp.flow.tmpl +form: + - key: "name" + prompt: "What's your app name?" + required: true + - key: "port" + prompt: "Which port should it run on?" + default: "3000" -```shell -flow template generate mealie --output mealie --file /path/to/k8s-deployment.flow.tmpl --workspace homelab +template: | + executables: + - verb: start + name: "{{ name }}" + exec: + cmd: "npm start -- --port {{ form["port"] }}" + - verb: build + name: "{{ name }}" + exec: + cmd: "npm run build" ``` -## Viewing templates - -Individual templates can be viewed using the [flow template get](../cli/flow_template_get.md) command. This command -accepts either the registered name of the template or the path to the template file. +Register and use it: ```shell -# View the k8s-deployment template by its registered name -flow template get --template k8s-deployment -# View the k8s-deployment template by its file path -flow template get --file /path/to/k8s-deployment.flow.tmpl -``` +# Register the template +flow template add webapp ./webapp.flow.tmpl -You can also list all registered templates using the [flow template list](../cli/flow_template_list.md) command. - -```shell -flow template list +# Generate from template +flow template generate my-app --template webapp ``` -## Creating a template file - -Templates are defined as YAML files that contain a string template of a [flowfile](executable.md#flowfile), artifacts to be copied, -executables to be run during the templating process, and form fields that can be used throughout the template's fields. - -Check out the [template configurations](../types/template.md) for more details on the structure the template file. +## Template Components -### Form inputs +Templates have four main parts: -The form section defines the fields that will be prompted to the user when the template is rendered. Each field has a key, -prompt, and optional default value, type, and validation. It's the first step in the template rendering process and is what -provides the data for the [Go text templating](https://pkg.go.dev/text/template) that is used throughout the template file. +### 1. Forms - Collect User Input -For instance, the following form section will prompt the user for the namespace, image, replicas, and -whether the app should be deployed immediately. +Forms define interactive prompts shown during generation: ```yaml form: - - key: "Namespace" - prompt: "What namespace should the deployment be created in?" - default: "apps" - - key: "Deploy" - prompt: "Should the app be deployed immediately?" - type: "confirm" # This will prompt the user with a yes/no question - - key: "Image" - prompt: "What image should be used for the deployment?" - required: true # The template will not render if this field is not provided - - key: "Replicas" - prompt: "How many replicas should be created?" - default: "1" - validate: "^[0-9]+$" # This will validate that the input is a number - - key: "Type" - prompt: "Should the deployment be a Helm chart or a Kubernetes manifest?" - default: "Helm" - validate: "^(Helm|K8s)$" # This will validate that the input is either "Helm" or "K8s" + - key: "namespace" + prompt: "Which namespace?" + default: "default" + - key: "replicas" + prompt: "How many replicas?" + default: "3" + validate: "^[0-9]+$" # Numbers only + - key: "deploy" + prompt: "Deploy immediately?" + type: "confirm" # Yes/no question + - key: "image" + prompt: "Container image?" + required: true # Must provide value ``` -### Artifacts +**Form field types:** +- `text` - Single line input (default) +- `multiline` - Multi-line text +- `masked` - Hidden input for passwords +- `confirm` - Yes/no question -The artifacts section defines the files that will be copied to the output directory when the template is rendered. -Each artifact has a source, destination, and optional template flag that will render the file as a Go text template. +### 2. Templates - Generate Flow Files -In the following example, the template will copy the `helm-deploy.sh`, `deploy.sh`, `values.yaml.tmpl`, and `resources.yaml.tmpl` files. -The `values.yaml.tmpl` and `resources.yaml.tmpl` files will be rendered as Go text templates (using the data provided through the form). - -The `if` field can be used to conditionally copy the file based on the value of a form field. Below the `helm-deploy.sh` and `values.yaml.tmpl` -files will only be copied if the `Type` field is set to `Helm`. The `deploy.sh` and `resources.yaml.tmpl` files will only be copied if the -`Type` field is set to `K8s`. - -```yaml -artifacts: - - srcName: "helm-deploy.sh" - srcDir: "scripts" # By default, the file will be copied from the template directory. This field can be used to specify a different directory. - if: form["Type"] == "Helm" # This will only copy the file if the Helm field is true - - srcName: "deploy.sh" - srcDir: "scripts" - if: form["Type"] == "K8s" # This will only copy the file if the K8s field is true - - srcName: "values.yaml.tmpl" - asTemplate: true - dstName: "values.yaml" - if: form["Type"] == "Helm" # This will only copy the file if the Helm field is true - - srcName: "resources.yaml.tmpl" - asTemplate: true - dstName: "resources.yaml" - if: form["Type"] == "K8s" # This will only copy the file if the K8s field is true -``` - -### flowfile template string - -The template section defines the string template of the flowfile that will be rendered. The template can be as simple or -complex as needed and can include Go text templating to reference the form fields provided by the user. - -In the following example, the template will create a set flowfile with executables for deploying, restarting, and opening the app. +The main template creates your flow file: ```yaml template: | - tags: [k8s] executables: - verb: deploy name: "{{ name }}" exec: - file: "{{ if form["Type"] == 'Helm' }}helm-deploy.sh{{ else }}deploy.sh{{ end }}" params: - - envKey: "NAMESPACE" - text: "{{ form["Namespace"] }}" - - envKey: "APP_NAME" - text: "{{ name }}" - - verb: restart + - envKey: "REPLICAS" + text: "{{ form["replicas"] }}" + cmd: kubectl apply -f deployment.yaml + + - verb: scale name: "{{ name }}" exec: - cmd: "kubectl rollout restart deployment/{{ name }} -n {{ form["Namespace"] }}" - - verb: open - name: "{{ name }}" - launch: - uri: "https://{{ name }}.my.haus" + cmd: kubectl scale deployment {{ name }} --replicas={{ form["replicas"] }} ``` -### Pre- and post- run executables +### 3. Artifacts - Copy Supporting Files + +Copy and optionally template additional files: + +```yaml +artifacts: + # Copy static files + - srcName: "docker-compose.yml" + dstName: "docker-compose.yml" + + # Template files (process with form data) + - srcName: "deployment.yaml.tmpl" + dstName: "deployment.yaml" + asTemplate: true + + # Conditional copying + - srcName: "helm-values.yaml" + if: form["type"] == "helm" +``` -The preRun and postRun sections define the executables that will be run before and after the template is rendered. -These executables can be used to extend the templating process by running additional commands. +### 4. Hooks - Run Commands -In the following example, the template will run a validation executable before copying artifacts and rendering the flowfile. -Before exiting, it will also run a simple command and either open the flowfile in vscode or deploy the app based on the user's input. +Execute commands before/after generation: ```yaml preRun: - - ref: "validate k8s/validation:context" # You can reference other executables that you have on your system - args: ["homelab"] - if: form["Deploy"] + - cmd: mkdir -p config + - ref: validate environment + postRun: - - cmd: | - echo 'Rendered {{ if form["Helm"] }}Helm values{{ else }}k8s manifest{{end}}'; ls -al - - ref: "edit vscode" - args: ["{{ flowFilePath }}"] - if: not form["Deploy"] - - ref: "deploy {{ name }}" - if: form["Deploy"] + - cmd: chmod +x scripts/*.sh + - ref: "deploy {{ .name }}" + if: form["deploy"] ``` -**Note**: preRun executables are run from the template directory, while postRun executables are run from the output directory. - -### Full template example +## Real-World Example -Bringing it all together, the following is a full example of the k8s deployment template. It's not required to have all the sections -in a template, but it's a good starting point for creating your own templates. +Here's a complete Kubernetes deployment template: ```yaml form: - - key: "Namespace" - prompt: "What namespace should the deployment be created in?" - default: "apps" - - key: "Deploy" - prompt: "Should the app be deployed immediately?" + - key: "namespace" + prompt: "Deployment namespace?" + default: "default" + - key: "image" + prompt: "Container image?" + required: true + - key: "replicas" + prompt: "Number of replicas?" + default: "3" + validate: "^[1-9][0-9]*$" + - key: "expose" + prompt: "Expose via LoadBalancer?" type: "confirm" - - key: "Image" - prompt: "What image should be used for the deployment?" - - key: "Replicas" - prompt: "How many replicas should be created?" - default: "1" - validate: "^[0-9]+$" - - key: "Type" - prompt: "Should the deployment be a Helm chart or a Kubernetes manifest?" - default: "Helm" - validate: "^(Helm|K8s)$" + artifacts: - - srcName: "helm-deploy.sh" - srcDir: "scripts" - if: "{{ eq .Type 'Helm' }}" - - srcName: "deploy.sh" - srcDir: "scripts" - if: "{{ eq .Type 'K8s' }}" - - srcName: "values.yaml.tmpl" + - srcName: "k8s-deployment.yaml.tmpl" + dstName: "deployment.yaml" asTemplate: true - dstName: "values.yaml" - if: "{{ eq .Type 'Helm' }}" - - srcName: "resources.yaml.tmpl" + - srcName: "k8s-service.yaml.tmpl" + dstName: "service.yaml" asTemplate: true - dstName: "resources.yaml" - if: "{{ eq .Type 'K8s' }}" -preRun: - - ref: "validate k8s/validation:context" - args: ["homelab"] - if: "{{ .Deploy }}" + if: form["expose"] + postRun: - - cmd: "echo 'Rendered {{ if .Helm }}Helm values{{ else }}k8s manifest{{end}}'; ls -al" - - ref: "edit vscode" - args: ["{{ .FlowFilePath }}"] - if: "{{ not .Deploy }}" - - ref: "deploy {{ .name }}" - if: "{{ .Deploy }}" + - cmd: echo "Generated Kubernetes manifests" + - cmd: kubectl apply -f . + if: form["deploy"] + template: | - tags: [k8s] executables: - verb: deploy name: "{{ name }}" exec: - file: "{{ if eq .Type 'Helm' }}helm-deploy.sh{{ else }}deploy.sh{{ end }}" - params: - - envKey: "NAMESPACE" - text: "{{ .form.namespace }}" - - envKey: "APP_NAME" - text: "{{ name }}" - - verb: restart + cmd: kubectl apply -f deployment.yaml -f service.yaml + + - verb: scale name: "{{ name }}" exec: - cmd: "kubectl rollout restart deployment/{{ .FlowFileName }} -n {{ .Namespace }}" - - verb: open + params: + - prompt: "New replica count?" + envKey: "REPLICAS" + cmd: kubectl scale deployment {{ name }} --replicas=$REPLICAS + + - verb: logs name: "{{ name }}" - launch: - uri: "https://{{ name }}.my.haus" + exec: + cmd: kubectl logs -l app={{ name }} -f +``` + +## Template Management + +See the [template command reference](../cli/flow_template.md) for all detailed commands and options. + +### Register Templates + +```shell +# From file +flow template add webapp ./templates/webapp.flow.tmpl + +# List registered templates +flow template list + +# View template details +flow template get -t webapp +``` + +### Generate from Templates + +```shell +# Using registered template +flow template generate my-app --template webapp + +# Using file directly +flow template generate my-app --file ./webapp.flow.tmpl + +# Specify workspace and output directory +flow template generate my-app \ + --template webapp \ + --workspace my-workspace \ + --output ./apps/my-app ``` -### Templating language -flow uses a hybrid of [Go text templating](https://pkg.go.dev/text/template) and [Expr](https://expr-lang.org) language for -rendering templates. +## Template Language + +flow uses [Expr](https://expr-lang.org) language for all template evaluation, but with Go template syntax. +You write templates using familiar `{{ }}` syntax, but the expressions inside are evaluated using Expr. + +**Available Variables:** + +| Variable | Description | Example | +|----------|-------------|---------| +| `name` | Generated file name | `{{ name }}` | +| `workspace` | Target workspace | `{{ workspace }}` | +| `form` | Form input values | `{{ form["replicas"] }}` | +| `env` | Environment variables | `{{ env["USER"] }}` | +| `os` | Operating system | `{{ os }}` | +| `arch` | System architecture | `{{ arch }}` | +| `workspacePath` | Full path to workspace | `{{ workspacePath }}` | +| `flowFilePath` | Full path to target flow file | `{{ flowFilePath }}` | +| `templatePath` | Path to template file | `{{ templatePath }}` | +| `directory` | Target directory | `{{ directory }}` | + +**Template Examples:** +```yaml +# Basic variable access +template: | + executables: + - name: "{{ name }}" + exec: + cmd: echo "Hello from {{ name }}" + +# Form data access + - verb: deploy + exec: + cmd: kubectl apply -f {{ form["manifest"] }} -Aside from the `if` field in the artifacts, preRun, and postRun sections, all other fields that use templating -will need to include the `{{` and `}}` delimiters to indicate that the text should be rendered as a template. +# Conditionals +{{ if form["type"] == "web" }} + - verb: start + exec: + cmd: npm start +{{ end }} -**Template Variables** +# String functions +{{ upper(name) }} +{{ form["image"] | replace(":", "-") }} +``` -The following variables are automatically available in all template expressions: +**`if` fields in artifacts/hooks:** +For `if` fields in artifacts, preRun, and postRun sections, use Expr directly (no `{{ }}` needed): -| Variable | Type | Description | -|-----------------|------------------------|------------------------------------------------------------------| -| `os` | string | Operating system identifier (e.g., "linux", "darwin", "windows") | -| `arch` | string | System architecture (e.g., "amd64", "arm64") | -| `workspace` | string | Target workspace name | -| `workspacePath` | string | Full path to the target workspace root directory | -| `name` | string | Name provided for the newly rendered flow file | -| `directory` | string | Target directory that the template will be render in to | -| `flowFilePath` | string | Full path to the target flow file | -| `templatePath` | string | Path to the template file being rendered | -| `env` | map (string -> string) | Environment variables accessible to the template | -| `form` | map (string -> any) | Values provided through template form inputs | +```yaml +artifacts: + - srcName: "web.conf" + if: form["type"] == "web" + - srcName: "api.conf" + if: form["type"] == "api" and len(form["endpoints"]) > 0 -**See the [Expr language documentation](https://expr-lang.org/docs/language-definition) for more information on the -additional expression functions and syntax.** +postRun: + - cmd: ./deploy.sh + if: form["deploy"] and form["environment"] == "production" +``` -> [!NOTE] -> The `env` map contains environment variables that were present when the template was rendered. The `form` map contains values from any form inputs defined in the template configuration. +See the [Expr language documentation](https://expr-lang.org/docs/language-definition) for more information on Expr syntax and functions. diff --git a/docs/guide/workspace.md b/docs/guide/workspace.md deleted file mode 100644 index b1a65ec5..00000000 --- a/docs/guide/workspace.md +++ /dev/null @@ -1,78 +0,0 @@ -Flow is built around [executables](executable.md), which are defined in [flowfiles](../types/flowfile.md) and organized into -directory structures called **workspaces**. Executables can also be grouped across **namespaces** nested within the flowfile trees of these workspaces. - -## Workspace Registration - -You can register multiple workspaces with Flow. To do so, use the [flow workspace add](../cli/flow_workspace_add.md) command: - -```shell -# Create a workspace named "my-workspace" in the current directory -flow workspace add my-workspace . -# Create a workspace named "my-workspace" in a specific directory and set it as the current workspace -flow workspace add my-workspace /path/to/directory --set -``` - -When a workspace is created, a [configuration](#workspace-configuration) file is added to the root of the workspace if one does not already exist. - -### Workspace Configuration - -The workspace configuration file is a YAML file that contains the configuration options for the workspace. This file is located in the root directory of the workspace and is named `flow.yaml`. - -```yaml -# Example workspace configuration -displayName: "My Project" -description: "A sample Flow workspace" -descriptionFile: README.md -tags: ["development", "web"] -verbAliases: - run: ["exec", "start"] - build: ["compile"] -executables: - included: ["scripts/", "tools/"] - excluded: ["node_modules/", ".git/"] -``` - -**Key Configuration Options:** - -- **verbAliases**: Customize which verb aliases are available when running executables. Set to `{}` to disable all aliases. See [custom verb aliases](executable.md#custom-verb-aliases) for more details. -- **executables**: Configure which directories to include or exclude when discovering executables. -- **displayName**, **description**, and **descriptionFile**: Used in the interactive UI and library views. -- **tags**: Used for filtering workspaces. - -For more details about workspace configuration options, see [Workspace](../types/workspace.md). - - -## Changing the Current Workspace - -To change the current workspace, use the [flow workspace switch](../cli/flow_workspace_switch.md) command: - -```shell -# Set the current workspace to "my-workspace" -flow workspace switch my-workspace -``` - -Also see the [workspace mode](interactive.md#changing-the-workspace-mode) documentation for more information on workspace modes. - -## Deleting a Workspace - -To delete a workspace, use the [flow workspace remove](../cli/flow_workspace_remove.md) command: - -```shell -# Delete the workspace named "my-workspace" -flow workspace remove my-workspace -``` - -## Workspace Viewer - -The [flow workspace](../cli/flow_workspace.md) command provides various viewing and management options for workspaces: - -```shell -# View the current workspace -flow workspace get -# View a specific workspace -flow workspace get my-workspace -# List all registered workspaces -flow workspace list -# List all workspaces with a specific tag -flow workspace list --tag my-tag -``` diff --git a/docs/guide/workspaces.md b/docs/guide/workspaces.md new file mode 100644 index 00000000..90d8e0d4 --- /dev/null +++ b/docs/guide/workspaces.md @@ -0,0 +1,170 @@ +# Workspaces + +Workspaces organize your flow files and executables into logical projects. Think of them as containers for related automation. + +## Workspace Management + +### Adding Workspaces + +Register any directory as a workspace: + +```shell +# Create workspace in current directory +flow workspace add my-project . --set + +# Basic registration in specific directory +flow workspace add my-project /path/to/project + +# Register and switch to it +flow workspace add my-project /path/to/project --set +``` + +When you add a workspace, flow creates a `flow.yaml` configuration file in the root directory if one doesn't exist. + +### Switching Workspaces + +Change your current workspace: + +```shell +# Switch to a workspace +flow workspace switch my-project + +# Switch with fixed mode (see workspace modes below) +flow workspace switch my-project --fixed +``` + +### Listing and Viewing + +Explore your registered workspaces: + +```shell +# List all workspaces +flow workspace list + +# List workspaces with specific tags +flow workspace list --tag production + +# View current workspace details +flow workspace get + +# View specific workspace +flow workspace get my-project +``` + +### Removing Workspaces + +Unregister a workspace: + +```shell +# Remove workspace registration +flow workspace remove old-project +``` + +> [!NOTE] +> Removing a workspace only unlinks it from flow - your files and directories remain unchanged. + +## Workspace Configuration + +Configure workspace behavior in the `flow.yaml` file: + +```yaml +# flow.yaml +displayName: "API Service" +description: "REST API and deployment automation" +descriptionFile: README.md +tags: ["api", "production", "backend"] + +# Customize verb aliases +verbAliases: + run: ["start", "exec"] + build: ["compile", "make"] + # Set to {} to disable all aliases + +# Control executable discovery +executables: + included: ["api/", "scripts/", "deploy/"] + excluded: ["node_modules/", ".git/", "tmp/"] +``` + +### Configuration Options + +**Display and Documentation:** +- `displayName`: Human-readable name for the workspace +- `description`: Markdown description shown in the UI +- `descriptionFile`: Path to markdown file with workspace documentation +- `tags`: Labels for filtering and categorization + +**Executable Discovery:** +- `included`: Directories to search for flow files +- `excluded`: Directories to skip during discovery + +**Behavior Customization:** +- `verbAliases`: Customize which verb synonyms are available + +> **Complete reference**: See the [workspace configuration schema](../types/workspace.md) for all available options. + +## Workspace Modes + +Control how flow determines your current workspace: + +### Dynamic Mode (Default) +flow automatically switches to the workspace containing your current directory: + +```shell +# Configure dynamic mode +flow config set workspace-mode dynamic + +# Now flow automatically uses the right workspace +cd ~/code/api-service # Uses api-service workspace +cd ~/code/frontend # Uses frontend workspace +``` + +### Fixed Mode +flow always uses the workspace you've explicitly set: + +```shell +# Configure fixed mode +flow config set workspace-mode fixed + +# Set the fixed workspace +flow workspace switch my-project + +# Now flow always uses my-project, regardless of directory +``` + +## Multi-Workspace Workflows + +### Cross-Workspace References + +Reference executables from other workspaces (requires `visibility: public`): + +```yaml +executables: + - verb: deploy + name: full-stack + serial: + execs: + - ref: build frontend/app + - ref: build backend/api + - ref: deploy infrastructure/k8s:services +``` + +### Shared Workspaces + +Create workspaces for shared tools and utilities: + +```shell +# Create shared workspace +flow workspace add shared-tools ~/shared + +# Reference from other workspaces +flow send shared-tools/slack:notification "Deployment complete" +``` + +## What's Next? + +Now that you can organize your automation with workspaces: + +- **Define your tasks** → [Executables](executables.md) +- **Build sophisticated workflows** → [Advanced workflows](advanced.md) +- **Customize your interface** → [Interactive UI](interactive.md) \ No newline at end of file diff --git a/docs/index.html b/docs/index.html index 27adb451..361d6b33 100644 --- a/docs/index.html +++ b/docs/index.html @@ -14,6 +14,13 @@ --theme-color: #83C092; --base-font-size: 14px; } + + h1 { font-size: 1.8em !important; } + h2 { font-size: 1.5em !important; } + h3 { font-size: 1.3em !important; } + h4 { font-size: 1.1em !important; } + h5 { font-size: 1em !important; } + h6 { font-size: 0.9em !important; } diff --git a/docs/installation.md b/docs/installation.md index 8b412a30..b584f88b 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,35 +1,61 @@ -> **System Requirements:** At this time, flow is only supported on Linux and MacOS systems. On Linux, you will need `xclip` installed to use the clipboard feature. +# Installation -### Pre-compiled binary +> **System Requirements:** flow supports Linux and macOS systems. On Linux, you'll need `xclip` installed to use clipboard features. -Run the following command to install the latest version of flow: +## Quick Install + +Run this command to install the latest version of flow: ```shell curl -sSL https://raw.githubusercontent.com/flowexec/flow/main/scripts/install.sh | bash ``` -Alternatively, you can download the latest release from the [releases page](https://github.com/flowexec/flow/releases) and -add the binary to your `$PATH`. A checksum is provided for each release to verify the download. +## Alternative Install Methods -### Homebrew +### Homebrew (macOS/Linux) ```shell -brew install jahvon/tap/flow +brew install flowexec/tap/flow ``` -### Go +### Go Install ```bash go install github.com/flowexec/flow@latest ``` -For CI/CD integrations and containerized environments, see the [integrations guide](guide/integrations.md). +### Manual Download + +Download the latest release from the [releases page](https://github.com/flowexec/flow/releases) and add the binary to your `$PATH`. +Each release includes checksums for verification. + +## Verify Installation + +Check that flow is installed correctly: + +```shell +flow --version +``` -### Autocompletion +## Shell Completion -flow supports autocompletion for bash, zsh, and fish shells. Setup depends on the shell you are using. For instance, if -you are using zsh with oh-my-zsh, you can run the following command to generate the autocompletion script: +Enable tab completion for your shell: ```bash +# Bash +flow completion bash > /etc/bash_completion.d/flow + +# Zsh (oh-my-zsh) flow completion zsh > ~/.oh-my-zsh/completions/_flow + +# Fish +flow completion fish > ~/.config/fish/completions/flow.fish ``` + +## Next Steps + +Ready to start automating? → [Quick start guide](quickstart.md) + +## CI/CD & Containers + +For GitHub Actions, Docker, and other integrations, see the [integrations guide](guide/integrations.md). \ No newline at end of file diff --git a/docs/quickstart.md b/docs/quickstart.md index f8cda3c8..9a501138 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -1,93 +1,132 @@ +# Quick Start + > [!NOTE] -> Before getting started, install the latest `flow` version using one of the methods described in the +> Before getting started, install the latest `flow` version using one of the methods described in the > [installation guide](installation.md). -This guide will walk you through the basic steps of creating and running an executable with `flow`. - -**Create a new workspace** +This guide will walk you through creating your first workspace and executable with `flow` in about 5 minutes. -A workspace can be created anywhere on your system but must be registered in order to have executables discovered by flow. +## 1. Create Your First Workspace -To create a new workspace, run the following command in the directory where you want the workspace to be created: +A workspace is where flow looks for your executables. Create one in any directory: ```shell flow workspace add my-workspace . --set ``` -You can replace `my-workspace` with any name you want to give your workspace and `.` with the path to the root directory -of the new workspace. +This registers the workspace and creates a `flow.yaml` config file. The `--set` flag makes it your current workspace. -This command will register the workspace and create a `flow.yaml` file in the root directory. This file contains the new -workspace's configurations. For more information on workspaces see the [workspace guide](guide/workspace.md). +## 2. Create Your First Executable -**Create an executable** +Executables are defined in flow files (`.flow`, `.flow.yaml`, or `.flow.yml`). Let's create one: -Executables are the core of flow. Each executable is driven by its definition within a flow file (`*.flow`, `*.flow.yaml`, or `*.flow.yml`). -There are several types of executables that can be defined. For more information on executables and the flow file, see the [executable guide](guide/executable.md). - -To get started, create a new flow file in the workspace directory. - ```shell -touch executables.flow +touch hello.flow ``` -Open the file in your favorite text editor and add the following content: +Open the file and add this content: ```yaml executables: - verb: run - name: my-task + name: hello exec: params: - - prompt: What is your favorite color? - envKey: COLOR - cmd: echo "Your favorite color is $COLOR" + - prompt: What is your name? + envKey: NAME + cmd: echo "Hello, $NAME! Welcome to flow 🎉" ``` -This flow file defines a single executable named `my-task`. When run, it will prompt the user for their favorite color -and then echo that color back to the console. +This creates an executable that prompts for your name and greets you. -**Add another workspace** +## 3. Sync and Run -If you're new to flow, try adding the `flow` workspace and explore the executables it contains. +Update flow's index of executables: ```shell -git clone github.com/flowexec/flow -flow workspace add flow flow +flow sync ``` -When you have multiple workspaces, you can switch between them using a command like the following: +Now run your executable: ```shell -flow workspace set flow +flow run hello ``` -**Running an executable** +You'll be prompted for your name, then see your personalized greeting! + +## 4. Try the Interactive Browser -Whenever you create, move, or delete executables and flow files, you will need to update the index of executables before running them. +flow's TUI makes it easy to discover and run executables: ```shell -flow sync +flow browse ``` -The main command for running executables is `flow exec`. This command will execute the workflow with the provided -executable ID. `exec` can be replaced with any verb known to flow but should match the verb defined in the flow file -configurations or an alias of that verb. +Use arrow keys to navigate press R to run an executable that you have selected. -In our case, we will use the `run` verb: +## 5. Add More Executables -```shell -flow run my-task +Try adding different types of executables to your `hello.flow` file: + +```yaml +executables: + - verb: run + name: hello + exec: + params: + - prompt: What is your name? + envKey: NAME + cmd: echo "Hello, $NAME! Welcome to flow 🎉" + + - verb: open + name: docs + launch: + uri: https://flowexec.io + + - verb: test + name: system + exec: + cmd: | + echo "Testing system info..." + echo "OS: $(uname -s)" + echo "User: $(whoami)" + echo "Date: $(date)" ``` -> [!TIP] -> You can also run the executable by its full name, `run my-workspace/my-task` or an alias if one is defined. +Run `flow sync` then try: +- `flow open docs` - Opens the flow documentation +- `flow test system` - Shows system information -Try adding more executables to the workspace! You can create multiple flow files anywhere in the workspace. As you add more -executables, try viewing them from the interactive UI: +## 6. Explore a Real Workspace + +Want to see more examples? Add the flow project itself as a workspace: + +```shell +git clone https://github.com/flowexec/flow.git +flow workspace add flow flow +flow workspace switch flow +``` + +Then browse the executables: ```shell flow browse ``` -When in the library, you can press the R key on a selected executable to run it. +You'll see real-world examples of builds, tests, and development workflows used for developing flow. + +## What's Next? + +Now that you've got the basics: + +- **Learn the fundamentals** → [Core concepts](guide/concepts.md) +- **Secure your workflows** → [Working with secrets](guide/secrets.md) +- **Build complex automations** → [Advanced workflows](guide/advanced.md) +- **Customize your experience** → [Interactive UI](guide/interactive.md) + +## Getting Help + +- **Browse the docs** → Explore the guides and reference sections +- **Join the community** → [Discord server](https://discord.gg/CtByNKNMxM) +- **Report issues** → [GitHub issues](https://github.com/flowexec/flow/issues) diff --git a/docs/types/README.md b/docs/types/README.md index 8b20d22a..81363397 100644 --- a/docs/types/README.md +++ b/docs/types/README.md @@ -5,15 +5,44 @@ - [Workspace](workspace.md) - [Config](config.md) -## Schemas +## IDE Integration -All flow configuration files also have YAML schemas available for use in IDEs with the Language Server Protocol (LSP) to perform intelligent suggestions. -You can add the following comment to the top of your flow files to enable this: +All flow configuration files have YAML schemas available for intelligent suggestions and validation in your IDE. + +### Enable Schema Validation + +Add this comment to the top of your flow files: ```yaml # yaml-language-server: $schema=https://flowexec.io/schemas/flowfile_schema.json + +executables: + - verb: run + name: my-task + exec: + cmd: echo "Hello, world!" ``` -See the [schemas directory on GitHub](https://github.com/flowexec/flow/tree/main/docs/schemas) for all available schemas. +### Available Schemas + +- **FlowFile**: `https://flowexec.io/schemas/flowfile_schema.json` +- **Template**: `https://flowexec.io/schemas/template_schema.json` +- **Workspace**: `https://flowexec.io/schemas/workspace_schema.json` +- **Config**: `https://flowexec.io/schemas/config_schema.json` + +### IDE Setup + +**VS Code**: Install the YAML extension and configure file associations: + +```json +// settings.json +{ + "files.associations": { + "*.flow": "yaml", + "*.flow.yaml": "yaml", + "*.flow.yml": "yaml" + } +} +``` -Note: If using the flow file schema, you will need to make sure your IDE is configured to treat `*.flow`, `*.flow.yaml`, and `*.flow.yml` files as YAML files. +**Other IDEs**: Configure your IDE to treat `*.flow`, `*.flow.yaml`, and `*.flow.yml` files as YAML files. diff --git a/docs/types/_sidebar.md b/docs/types/_sidebar.md index b6a58015..be61b987 100644 --- a/docs/types/_sidebar.md +++ b/docs/types/_sidebar.md @@ -1,12 +1,7 @@ -- [Home](../README.md "flow documentation") -- [Install](../installation.md "Installation guide") -- [Quick start](../quickstart.md "Quick start guide") +- [← Back to Home](../README.md "flow documentation") -- Reference - - - [CLI commands](../cli/README.md "Command line interface reference") - - [Configuration files](README.md "Configuration file reference") - - [flow (executable) file](flowfile.md) - - [template file](template.md) - - [workspace file](workspace.md) - - [user configuration file](config.md) +- [Configuration Reference](README.md "Configuration file reference") + - [flow file (executables)](flowfile.md) + - [template file](template.md) + - [workspace file](workspace.md) + - [user configuration file](config.md)