refactor: consolidate and streamline documentation pipeline

- Create unified pipeline script that runs steps in sequence
- Move pipeline scripts to /scripts/publish directory
- Add new /scripts/ingest directory for processing new patterns
- Update documentation:
  - Add detailed scripts/README.md
  - Update CONTRIBUTING.md with new workflow
  - Improve template.mdx with better guidance
- Remove deprecated scripts:
  - extract-examples.ts
  - restore-examples.ts
  - run-all-ts.sh

Author: PaulJPhilp

CONTRIBUTING.md

@@ -3,14 +3,102 @@
 Thank you for your interest in contributing! This project aims to be the best
 community-driven knowledge base for Effect-TS patterns. Every contribution helps.
 
-## How to Add a New Pattern
-
-1.  **Create a new file** in the `/content` directory.
-2.  **Name the file** using a descriptive, kebab-case name, like
-    `use-gen-for-business-logic.mdx`.
-3.  **Copy the contents** from the `template.mdx` file into your new file.
-4.  **Fill out the template** with the details of your pattern.
-5.  **Submit a Pull Request.**
+## Documentation Pipeline
+
+This project uses a two-stage pipeline system:
+
+### 1. Ingest Stage
+
+New patterns start in `/content/new` and go through validation:
+
+- **Source Files**
+  - TypeScript examples in `/content/new/src/*.ts`
+  - MDX documentation in `/content/new/*.mdx`
+  - Run `npm run ingest` to process
+
+### 2. Publishing Pipeline
+
+The main pipeline has five sequential steps:
+
+1. **Test** (`/content/src`)
+   - Run all TypeScript examples
+   - Verify code correctness
+
+2. **Publish** (`/content/raw` → `/content/published`)
+   - Convert raw MDX to published format
+   - Expand TypeScript examples inline
+
+3. **Validate** (`/content/published`)
+   - Check frontmatter and sections
+   - Verify code block consistency
+
+4. **Generate** (`README.md`)
+   - Create main README
+   - Group patterns by use case
+
+5. **Rules** (`/rules`)
+   - Generate AI coding rules
+   - Multiple output formats
+
+### Available Commands
+
+- `npm run ingest` - Process new patterns from `/content/new`
+- `npm run pipeline` - Run all publishing steps in sequence
+- `npm run all` - Alias for pipeline
+
+Individual pipeline steps:
+- `npm run test` - Run TypeScript examples
+- `npm run publish` - Convert raw to published MDX
+- `npm run validate` - Validate published files
+- `npm run generate` - Generate README
+- `npm run rules` - Generate AI rules
+
+### Validation Rules
+
+All patterns must have:
+1. Valid frontmatter with required fields
+2. A Good Example section with TypeScript code
+3. An Anti-Pattern section
+4. Either an Explanation or Rationale section
+5. TypeScript code that matches the source file
+
+## Adding New Patterns
+
+### Step 1: Create New Pattern Files
+
+Create your new pattern in the `/content/new` directory:
+
+1. Create the TypeScript example in `/content/new/src/{pattern-id}.ts`
+2. Create the MDX documentation in `/content/new/{pattern-id}.mdx`
+
+### Step 2: Run the Ingest Process
+
+Run `npm run ingest` to process your new pattern. This will:
+1. Validate your pattern files
+2. Move the TypeScript file to `/content/src`
+3. Move the MDX file to `/content/raw`
+
+If validation fails, fix the issues and try again.
+
+### Step 3: Run the Publishing Pipeline
+
+1. **Fill out your pattern**
+   - Add your TypeScript code to the `.ts` file
+   - Fill out the MDX template with your pattern details
+   - Make sure to include all required sections
+
+2. **Run the pipeline**
+   ```bash
+   npm run pipeline
+   ```
+   This will:
+   - Run your TypeScript example
+   - Convert and validate your MDX
+   - Update README and rules
+
+3. **Submit a Pull Request**
+   - Verify all pipeline steps passed
+   - Include both your source files and generated files
 
 ## The Pattern Structure
 

README.md

@@ -4,12 +4,14 @@
 	"private": true,
 	"description": "A community-driven knowledge base for Effect-TS patterns.",
 	"scripts": {
-		"all-ts": "bash scripts/run-all-ts.sh",
-		"dev": "echo 'Development server not yet implemented'",
-		"build": "echo 'Build script not yet implemented'",
-		"start": "echo 'Start script not yet implemented'",
-		"readme": "bun scripts/generate_readme.ts",
-		"rules": "bun scripts/generate_rules.ts",
+		"publish": "bun run scripts/publish/publish.ts",
+		"validate": "bun run scripts/publish/validate.ts",
+		"generate": "bun run scripts/publish/generate.ts",
+		"test": "bun run scripts/publish/test.ts",
+		"rules": "bun run scripts/publish/rules.ts",
+		"ingest": "bun run scripts/ingest/process.ts",
+		"pipeline": "bun run scripts/publish/pipeline.ts",
+		"all": "bun run pipeline",
 		"lint:examples": "eslint \"content/src/**/*.ts\"",
 		"typecheck:examples": "tsc --noEmit"
 	},