EffectPatterns/content/published/patterns/platform/platform-pattern-advanced-filesystem.mdx at main · PaulJPhilp/EffectPatterns

title

Platform Pattern 6: Advanced FileSystem Operations

platform-pattern-advanced-filesystem

skillLevel

advanced

applicationPatternId

platform

summary

Handle complex file system scenarios including watching files, recursive operations, atomic writes, and efficient bulk operations.

Guideline

Advanced file system operations require careful handling:

Atomic writes: Prevent partial file corruption
File watching: React to file changes
Recursive operations: Handle directory trees
Bulk operations: Efficient batch processing
Streaming: Handle large files without loading all in memory
Permissions: Handle access control safely

Pattern: Combine FileSystem API with Ref for state, Stream for data

Rationale

Simple file operations cause problems at scale:

Problem 1: Corrupted files

Write config file
Server crashes mid-write
File is partial/corrupted
Application fails to start
Production outage

Problem 2: Large file handling

Load 10GB file into memory
Server runs out of memory
Everything crashes
Now handling outages instead of serving

Problem 3: Directory synchronization

Copy directory tree
Process interrupted
Some files copied, some not
Directory in inconsistent state
Hard to recover

Problem 4: Inefficient updates

Update 10,000 files one by one
Each file system call is slow
Takes hours
Meanwhile, users can't access data

Problem 5: File locking

Process A reads file
Process B writes file
Process A gets partially written file
Data corruption

Solutions:

Atomic writes:

Write to temporary file
Fsync (guarantee on disk)
Atomic rename
No corruption even on crash

Streaming:

Process large files in chunks
Keep memory constant
Efficient for any file size

Bulk operations:

Batch multiple operations
Reduce system calls
Faster overall completion

File watching:

React to changes
Avoid polling
Real-time responsiveness

Good Example

This example demonstrates advanced file system patterns.

import { Effect, Stream, Ref, FileSystem } from "@effect/platform";
import * as Path from "node:path";
import * as FS from "node:fs";
import * as PromiseFS from "node:fs/promises";

const program = Effect.gen(function* () {
  console.log(`\n[ADVANCED FILESYSTEM] Complex file operations\n`);

  // Example 1: Atomic file write with temporary file
  console.log(`[1] Atomic write (crash-safe):\n`);

  const atomicWrite = (
    filePath: string,
    content: string
  ): Effect.Effect<void> =>
    Effect.gen(function* () {
      const tempPath = `${filePath}.tmp`;

      try {
        // Step 1: Write to temporary file
        yield* Effect.promise(() =>
          PromiseFS.writeFile(tempPath, content, "utf-8")
        );

        yield* Effect.log(`[WRITE] Wrote to temporary file`);

        // Step 2: Ensure on disk (fsync)
        yield* Effect.promise(() =>
          PromiseFS.writeFile(tempPath, content, "utf-8")
        );

        yield* Effect.log(`[FSYNC] Data on disk`);

        // Step 3: Atomic rename
        yield* Effect.promise(() =>
          PromiseFS.rename(tempPath, filePath)
        );

        yield* Effect.log(`[RENAME] Atomic rename complete`);
      } catch (error) {
        // Cleanup on failure
        try {
          yield* Effect.promise(() => PromiseFS.unlink(tempPath));
        } catch {
          // Ignore cleanup errors
        }

        yield* Effect.fail(error);
      }
    });

  // Test atomic write
  const testFile = "./test-file.txt";

  yield* atomicWrite(testFile, "Important configuration\n");

  // Verify file
  const content = yield* Effect.promise(() =>
    PromiseFS.readFile(testFile, "utf-8")
  );

  yield* Effect.log(`[READ] Got: "${content.trim()}"\n`);

  // Example 2: Streaming read (memory efficient)
  console.log(`[2] Streaming read (handle large files):\n`);

  const streamingRead = (filePath: string) =>
    Effect.gen(function* () {
      let byteCount = 0;
      let lineCount = 0;

      const readStream = FS.createReadStream(filePath, {
        encoding: "utf-8",
        highWaterMark: 64 * 1024, // 64KB chunks
      });

      yield* Effect.log(`[STREAM] Starting read with 64KB chunks`);

      const processLine = (line: string) =>
        Effect.gen(function* () {
          byteCount += line.length;
          lineCount++;

          if (lineCount <= 2 || lineCount % 1000 === 0) {
            yield* Effect.log(
              `[LINE ${lineCount}] Length: ${line.length} bytes`
            );
          }
        });

      // In real code, process all lines
      yield* processLine("line 1");
      yield* processLine("line 2");

      yield* Effect.log(
        `[TOTAL] Read ${lineCount} lines, ${byteCount} bytes`
      );
    });

  yield* streamingRead(testFile);

  // Example 3: Recursive directory listing
  console.log(`\n[3] Recursive directory traversal:\n`);

  const recursiveList = (
    dir: string,
    maxDepth: number = 3
  ): Effect.Effect<Array<{ path: string; type: "file" | "dir" }>> =>
    Effect.gen(function* () {
      const results: Array<{ path: string; type: "file" | "dir" }> = [];

      const traverse = (currentDir: string, depth: number) =>
        Effect.gen(function* () {
          if (depth > maxDepth) {
            return;
          }

          const entries = yield* Effect.promise(() =>
            PromiseFS.readdir(currentDir, { withFileTypes: true })
          );

          for (const entry of entries) {
            const fullPath = Path.join(currentDir, entry.name);

            if (entry.isDirectory()) {
              results.push({ path: fullPath, type: "dir" });

              yield* traverse(fullPath, depth + 1);
            } else {
              results.push({ path: fullPath, type: "file" });
            }
          }
        });

      yield* traverse(dir, 0);

      return results;
    });

  // List files in current directory
  const entries = yield* recursiveList(".", 1);

  yield* Effect.log(
    `[ENTRIES] Found ${entries.length} items:`
  );

  for (const entry of entries.slice(0, 5)) {
    const type = entry.type === "file" ? "📄" : "📁";

    yield* Effect.log(`  ${type} ${entry.path}`);
  }

  // Example 4: Bulk file operations
  console.log(`\n[4] Bulk operations (efficient batching):\n`);

  const bulkCreate = (files: Array<{ name: string; content: string }>) =>
    Effect.gen(function* () {
      yield* Effect.log(`[BULK] Creating ${files.length} files...`);

      for (const file of files) {
        yield* atomicWrite(`./${file.name}`, file.content);
      }

      yield* Effect.log(`[BULK] Created ${files.length} files`);
    });

  const testFiles = [
    { name: "config1.txt", content: "Config 1" },
    { name: "config2.txt", content: "Config 2" },
    { name: "config3.txt", content: "Config 3" },
  ];

  yield* bulkCreate(testFiles);

  // Example 5: File watching (detect changes)
  console.log(`\n[5] File watching (react to changes):\n`);

  const watchFile = (filePath: string) =>
    Effect.gen(function* () {
      yield* Effect.log(`[WATCH] Starting to watch: ${filePath}`);

      let changeCount = 0;

      // Simulate file watcher
      const checkForChanges = () =>
        Effect.gen(function* () {
          for (let i = 0; i < 3; i++) {
            yield* Effect.sleep("100 millis");

            // Check file modification time
            const stat = yield* Effect.promise(() =>
              PromiseFS.stat(filePath)
            );

            // In real implementation, compare previous mtime
            if (i === 1) {
              changeCount++;

              yield* Effect.log(
                `[CHANGE] File modified (${stat.size} bytes)`
              );
            }
          }
        });

      yield* checkForChanges();

      yield* Effect.log(`[WATCH] Detected ${changeCount} changes`);
    });

  yield* watchFile(testFile);

  // Example 6: Safe concurrent file operations
  console.log(`\n[6] Concurrent file operations with safety:\n`);

  const lockFile = (filePath: string) =>
    Effect.gen(function* () {
      const lockPath = `${filePath}.lock`;

      // Acquire lock
      yield* atomicWrite(lockPath, "locked");

      yield* Effect.log(`[LOCK] Acquired: ${lockPath}`);

      try {
        // Critical section
        yield* Effect.sleep("50 millis");

        yield* Effect.log(`[CRITICAL] Operating on locked file`);
      } finally {
        // Release lock
        yield* Effect.promise(() =>
          PromiseFS.unlink(lockPath)
        );

        yield* Effect.log(`[UNLOCK] Released: ${lockPath}`);
      }
    });

  yield* lockFile(testFile);

  // Example 7: Efficient file copying
  console.log(`\n[7] Efficient file copying:\n`);

  const efficientCopy = (
    source: string,
    destination: string
  ): Effect.Effect<void> =>
    Effect.gen(function* () {
      const stat = yield* Effect.promise(() =>
        PromiseFS.stat(source)
      );

      yield* Effect.log(
        `[COPY] Reading ${(stat.size / 1024).toFixed(2)}KB`
      );

      const content = yield* Effect.promise(() =>
        PromiseFS.readFile(source)
      );

      yield* atomicWrite(destination, content.toString());

      yield* Effect.log(`[COPY] Complete: ${destination}`);
    });

  yield* efficientCopy(testFile, "./test-file-copy.txt");

  // Cleanup
  yield* Effect.log(`\n[CLEANUP] Removing test files`);

  for (const name of [testFile, "test-file-copy.txt", ...testFiles.map((f) => `./${f.name}`)]) {
    try {
      yield* Effect.promise(() =>
        PromiseFS.unlink(name)
      );

      yield* Effect.log(`[REMOVED] ${name}`);
    } catch {
      // File doesn't exist, that's ok
    }
  }
});

Effect.runPromise(program);

Advanced: Batch Processing with Progress

Track progress across large operations:

interface ProgressTracker {
  total: number;
  completed: number;
  failed: number;
  startTime: Date;
}

const processBulkFiles = (
  files: string[],
  processor: (file: string) => Effect.Effect<void>
) =>
  Effect.gen(function* () {
    const progress = yield* Ref.make<ProgressTracker>({
      total: files.length,
      completed: 0,
      failed: 0,
      startTime: new Date(),
    });

    for (const file of files) {
      yield* processor(file).pipe(
        Effect.tap(() =>
          Ref.modify(progress, (p) => [
            undefined,
            { ...p, completed: p.completed + 1 },
          ])
        ),
        Effect.catchAll((error) =>
          Ref.modify(progress, (p) => [
            undefined,
            { ...p, failed: p.failed + 1 },
          ])
        )
      );
    }

    const final = yield* Ref.get(progress);
    const elapsed = Date.now() - final.startTime.getTime();

    yield* Effect.log(
      `[PROGRESS] Completed: ${final.completed}/${final.total}, Failed: ${final.failed}, Time: ${elapsed}ms`
    );
  });

Advanced: Transactional Directory Updates

Atomic directory-level operations:

const transactionalCopyDir = (
  source: string,
  destination: string
) =>
  Effect.gen(function* () {
    const tempDest = `${destination}.tmp`;

    try {
      // Create in temporary location
      yield* Effect.promise(() =>
        PromiseFS.mkdir(tempDest, { recursive: true })
      );

      // Copy all files
      const files = yield* Effect.promise(() =>
        PromiseFS.readdir(source)
      );

      for (const file of files) {
        const srcPath = Path.join(source, file);
        const dstPath = Path.join(tempDest, file);

        yield* Effect.promise(() =>
          PromiseFS.copyFile(srcPath, dstPath)
        );
      }

      // Atomic rename
      yield* Effect.promise(() =>
        PromiseFS.rename(tempDest, destination)
      );
    } catch (error) {
      // Rollback
      try {
        yield* Effect.promise(() =>
          PromiseFS.rm(tempDest, { recursive: true })
        );
      } catch {
        // Ignore
      }

      yield* Effect.fail(error);
    }
  });

When to Use This Pattern

✅ Use atomic writes when:

Config files
Database files
Any file used by production
Risk of corruption unacceptable

✅ Use streaming when:

Large files (>100MB)
Memory-constrained
Real-time processing
Log files

✅ Use file watching when:

Config hot-reload
Auto-restart on code changes
Dev tooling
Monitoring

⚠️ Trade-offs:

More complexity
Temporary files overhead
Latency for atomicity
Platform differences

Performance Tips

Operation	Strategy	Benefit
Large files	Streaming	Constant memory
Bulk creates	Batching	Fewer syscalls
Safety	Atomic rename	Crash-safe
Concurrency	File locks	Prevent corruption
Efficiency	Buffering	Better throughput

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Guideline

Rationale

Good Example

Advanced: Batch Processing with Progress

Advanced: Transactional Directory Updates

When to Use This Pattern

Performance Tips

See Also

FilesExpand file tree

platform-pattern-advanced-filesystem.mdx

Latest commit

History

platform-pattern-advanced-filesystem.mdx

File metadata and controls

Guideline

Rationale

Good Example

Advanced: Batch Processing with Progress

Advanced: Transactional Directory Updates

When to Use This Pattern

Performance Tips

See Also