Skip to content

New Components - RenderIO#21181

Merged
michelle0927 merged 9 commits into
masterfrom
renderio-new-components
Jun 22, 2026
Merged

New Components - RenderIO#21181
michelle0927 merged 9 commits into
masterfrom
renderio-new-components

Conversation

@michelle0927

@michelle0927 michelle0927 commented Jun 16, 2026

Copy link
Copy Markdown
Collaborator

Resolves #20895

Replaces #20896

Summary by CodeRabbit

Release Notes

  • New Features

    • Added RenderIO integration with 14 new actions for stored file management, video download/processing (yt-dlp), preset listing/execution, and FFmpeg job submission (single, chained, or multiple).
    • Added 2 new event sources to emit newly created FFmpeg commands and stored files using incremental polling.
  • Enhancements

    • Improved action input validation and result messaging (including informative $summary output).
    • Added optional job-completion tracking, and optional temporary downloads of produced files.

@vercel

vercel Bot commented Jun 16, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
pipedream-docs-redirect-do-not-edit Ignored Ignored Jun 22, 2026 9:07pm

Request Review

@coderabbitai

coderabbitai Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Important

Review skipped

Review was skipped due to path filters

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml

CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including **/dist/** will override the default block on the dist directory, by removing the pattern from both the lists.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: fa8b9999-dadf-4dd7-a9be-d622f5d6ebaa

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds a complete RenderIO Pipedream integration: expands renderio.app.mjs with an authenticated HTTP client and 17 API methods, introduces common/utils.mjs with shared parsing and validation helpers, adds 13 action modules covering FFmpeg, file management, yt-dlp, and presets, and adds two timestamp-checkpointed polling sources for new FFmpeg commands and stored files.

Changes

RenderIO Integration

Layer / File(s) Summary
App client, propDefinitions, HTTP helpers, and API methods
components/renderio/package.json, components/renderio/renderio.app.mjs
package.json bumped to 0.1.0 and adds @pipedream/platform and form-data dependencies. renderio.app.mjs gains propDefinitions (inputFiles, inputUrls, outputFiles, metadata), _baseUrl/_makeRequest HTTP helpers that inject X-API-KEY authentication, and 15+ API client methods (listCommands, getFfmpegCommand, runFfmpegCommand, runChainedFfmpegCommands, runMultipleFfmpegCommands, downloadVideoWithYtdlp, downloadAndProcessVideoWithYtdlp, listFiles, getFile, storeFile, uploadFile, deleteFile, listPresets, getPreset, executePreset).
Shared parsing, validation, and normalization utilities
components/renderio/common/utils.mjs
Introduces parseObject, parseRequiredObject, and parseArray for flexible JSON/native input handling with ConfigurationError on invalid types. Adds validateKeys to enforce object key prefixes, normalizeList for array response fallback, getCommandId to extract command identifiers, getRealTimestamp and getTimestamp to select timestamp fields, and isTerminalStatus for command status checking.
Simple file and preset CRUD actions
components/renderio/actions/delete-stored-file/*, components/renderio/actions/get-stored-file/*, components/renderio/actions/get-ffmpeg-command-status/*, components/renderio/actions/get-preset/*, components/renderio/actions/list-stored-files/*, components/renderio/actions/list-presets/*, components/renderio/actions/store-file/*, components/renderio/actions/upload-file/*
Eight straightforward actions that delegate to app methods and export a $summary. delete-stored-file is marked destructive, get-ffmpeg-command-status is read-only, list actions use normalizeList for pluralized summaries, and upload-file uses FormData multipart for streaming file content.
FFmpeg command actions with validation, polling, and completion handling
components/renderio/actions/run-ffmpeg-command/*, components/renderio/actions/run-chained-ffmpeg-commands/*, components/renderio/actions/run-multiple-ffmpeg-commands/*
run-ffmpeg-command validates command string and key prefixes (in_ for inputs, out_ for outputs), submits the job, optionally polls every 3 seconds until terminal status or timeout, optionally downloads output HTTP URLs to STASH_DIR//tmp, verifies success/completion, and exports $summary. run-chained-ffmpeg-commands and run-multiple-ffmpeg-commands validate their respective command payload shapes before submission.
yt-dlp download and preset execution actions
components/renderio/actions/download-video-with-ytdlp/*, components/renderio/actions/download-and-process-video-with-ytdlp/*, components/renderio/actions/execute-preset/*
download-video-with-ytdlp validates in_-prefixed input URL keys before submission. download-and-process-video-with-ytdlp additionally validates optional FFmpeg command and out_-prefixed output aliases. execute-preset parses inputFiles, optional metadata, and webhookUrl before calling renderio.executePreset.
Polling sources for new FFmpeg commands and stored files
components/renderio/sources/common/base.mjs, components/renderio/sources/new-ffmpeg-command/*, components/renderio/sources/new-stored-file/*
Base polling component (common/base.mjs) maintains persisted lastTs checkpoint in db, paginates list API responses, filters by timestamp, sorts and optionally truncates items, emits per-item events with generated metadata, and updates lastTs. Two concrete sources extend the base: new-ffmpeg-command polls listCommands and new-stored-file polls listFiles. Both run processEvent(25) on deploy for backfill and processEvent() on scheduled polling. Test event fixtures with realistic command and file payloads included.

Sequence Diagrams

sequenceDiagram
  participant User
  participant RunFFmpegAction as run-ffmpeg-command action
  participant RenderioApp as renderio.app.mjs
  participant RenderioAPI as RenderIO API
  participant Filesystem

  User->>RunFFmpegAction: invoke with inputFiles, outputFiles, command, options
  RunFFmpegAction->>RunFFmpegAction: validate command, key prefixes (in_/out_), maxCommandRunSeconds
  RunFFmpegAction->>RenderioApp: runFfmpegCommand(data)
  RenderioApp->>RenderioAPI: POST /commands/run-ffmpeg-command (X-API-KEY)
  RenderioAPI-->>RenderioApp: response with command_id
  RenderioApp-->>RunFFmpegAction: initial response

  alt waitForCompletion = true
    loop poll every 3s until isTerminalStatus or timeout
      RunFFmpegAction->>RenderioApp: getFfmpegCommand(commandId)
      RenderioApp->>RenderioAPI: GET /commands/{commandId}
      RenderioAPI-->>RenderioApp: status update
      RenderioApp-->>RunFFmpegAction: updated response
    end
    alt downloadFilesToTmp = true
      loop for each output HTTP URL
        RunFFmpegAction->>RenderioAPI: GET storage URL (arraybuffer)
        RenderioAPI-->>RunFFmpegAction: binary data
        RunFFmpegAction->>Filesystem: writeFile to STASH_DIR or /tmp
      end
    end
    RunFFmpegAction->>RunFFmpegAction: check error_message and terminal status
  end

  RunFFmpegAction-->>User: export $summary, return final response
Loading
sequenceDiagram
  participant Timer
  participant NewFFmpegCommandSource as new-ffmpeg-command source
  participant RenderioApp as renderio.app.mjs
  participant RenderioAPI as RenderIO API
  participant PipedreamDB as db (lastTs)

  Timer->>NewFFmpegCommandSource: run() or deploy()
  NewFFmpegCommandSource->>PipedreamDB: _getLastTs()
  PipedreamDB-->>NewFFmpegCommandSource: lastTs

  loop paginated batches (offset, limit) until maxPages
    NewFFmpegCommandSource->>RenderioApp: listCommands(limit, offset)
    RenderioApp->>RenderioAPI: GET /commands (X-API-KEY)
    RenderioAPI-->>RenderioApp: commands page
    RenderioApp-->>NewFFmpegCommandSource: commands array
  end

  NewFFmpegCommandSource->>NewFFmpegCommandSource: filter ts >= lastTs, sort by timestamp, optionally cap count
  loop for each qualifying command
    NewFFmpegCommandSource->>NewFFmpegCommandSource: generateMeta(command)
    NewFFmpegCommandSource->>NewFFmpegCommandSource: $emit(command, metadata)
  end
  NewFFmpegCommandSource->>PipedreamDB: _setLastTs(newest emitted timestamp)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Suggested labels

User submitted

Suggested reviewers

  • mariano-pd
🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning PR description references linked issue #20895 but lacks required checklist items (versioning, app integration status, CodeRabbit review). Template sections remain unfilled. Complete the PR description template by filling Summary section details and checking all required checklist items regarding versioning, app integration, and CodeRabbit review acknowledgment.
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed Title 'New Components - RenderIO' directly and clearly summarizes the main change: adding new RenderIO integration components to Pipedream.
Linked Issues check ✅ Passed All requested components are implemented with 100% pass rate confirmed by MCP evaluation: 13 actions (FFmpeg, file management, presets, video download) and 2 sources cover all #20895 requirements.
Out of Scope Changes check ✅ Passed All changes directly support RenderIO integration: app definition, actions for FFmpeg/files/presets, sources for monitoring, and shared utilities. No out-of-scope modifications detected.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch renderio-new-components

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@michelle0927 michelle0927 marked this pull request as ready for review June 16, 2026 21:35

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/renderio/common/utils.mjs`:
- Around line 52-55: The normalizeList function currently returns a non-array
value when response[key] is truthy but not an array (e.g., an object), which
will cause downstream for...of loops to crash. Fix this by adding an additional
Array.isArray check on response?.[key] before returning it. Only return
response[key] if it is actually an array, otherwise return an empty array. This
ensures normalizeList always returns either the original response if it is an
array, an array from response[key] if that property is an array, or an empty
array as a fallback.
- Around line 22-24: The parseRequiredObject function currently returns an empty
object when the input is falsy or invalid, which skips pre-flight validation.
Instead of returning {} as a fallback, throw a ConfigurationError with an
appropriate message when parseObject returns a falsy value. This ensures that
missing or invalid required configuration is caught immediately during
validation rather than deferring the failure to API calls.

In `@components/renderio/sources/new-ffmpeg-command/new-ffmpeg-command.mjs`:
- Around line 60-63: The comparison operator in the condition checking
timestamps within the pageCommands loop uses strict greater-than (ts > lastTs),
which causes commands sharing the same timestamp as the saved checkpoint to be
skipped on subsequent pages or runs. Change the comparison from strict
greater-than to greater-than-or-equal (ts >= lastTs) to ensure commands with
matching checkpoint timestamps are preserved and not lost during pagination,
maintaining reliable deduplication behavior as required by polling source
guidelines.

In `@components/renderio/sources/new-stored-file/new-stored-file.mjs`:
- Around line 58-61: The condition in the loop within new-stored-file.mjs that
checks `if (ts > lastTs)` uses a strict greater-than comparison, which can miss
files that share the same timestamp as the last checkpoint when records are
returned later through pagination or eventual consistency. Change the comparison
operator from `>` to `>=` in this condition to include files with a timestamp
equal to the last checkpoint timestamp, ensuring robust deduplication while
paginating through results.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2293dbb2-9666-46cc-995e-a7edaa5431f7

📥 Commits

Reviewing files that changed from the base of the PR and between 99ba6a6 and 1a030be.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (21)
  • components/renderio/actions/delete-stored-file/delete-stored-file.mjs
  • components/renderio/actions/download-and-process-video-with-ytdlp/download-and-process-video-with-ytdlp.mjs
  • components/renderio/actions/download-video-with-ytdlp/download-video-with-ytdlp.mjs
  • components/renderio/actions/execute-preset/execute-preset.mjs
  • components/renderio/actions/get-ffmpeg-command-status/get-ffmpeg-command-status.mjs
  • components/renderio/actions/get-preset/get-preset.mjs
  • components/renderio/actions/get-stored-file/get-stored-file.mjs
  • components/renderio/actions/list-presets/list-presets.mjs
  • components/renderio/actions/list-stored-files/list-stored-files.mjs
  • components/renderio/actions/run-chained-ffmpeg-commands/run-chained-ffmpeg-commands.mjs
  • components/renderio/actions/run-ffmpeg-command/run-ffmpeg-command.mjs
  • components/renderio/actions/run-multiple-ffmpeg-commands/run-multiple-ffmpeg-commands.mjs
  • components/renderio/actions/store-file/store-file.mjs
  • components/renderio/actions/upload-file/upload-file.mjs
  • components/renderio/common/utils.mjs
  • components/renderio/package.json
  • components/renderio/renderio.app.mjs
  • components/renderio/sources/new-ffmpeg-command/new-ffmpeg-command.mjs
  • components/renderio/sources/new-ffmpeg-command/test-event.mjs
  • components/renderio/sources/new-stored-file/new-stored-file.mjs
  • components/renderio/sources/new-stored-file/test-event.mjs

Comment thread components/renderio/common/utils.mjs
Comment thread components/renderio/common/utils.mjs
Comment thread components/renderio/sources/new-ffmpeg-command/new-ffmpeg-command.mjs Outdated
Comment thread components/renderio/sources/new-stored-file/new-stored-file.mjs Outdated
@michelle0927

Copy link
Copy Markdown
Collaborator Author

MCP Eval Results — RenderIO ✅

14 / 14 passing | 100% tool coverage

Ran evals against all 14 actions in this PR using Pipedream's remote MCP server with the privately published components.

# Eval Status Tools Called
1 Store A Public File URL In RenderIO ✅ PASS renderio-store-file
2 List All Stored Files In RenderIO ✅ PASS renderio-list-stored-files
3 Get Details Of A Specific Stored File ✅ PASS renderio-list-stored-files, renderio-get-stored-file
4 Delete A Stored File From RenderIO ✅ PASS renderio-list-stored-files, renderio-delete-stored-file
5 Upload A File Directly To RenderIO ✅ PASS renderio-upload-file
6 Run A Single FFmpeg Command ✅ PASS renderio-run-ffmpeg-command
7 Check Status Of An FFmpeg Command ✅ PASS renderio-run-ffmpeg-command, renderio-get-ffmpeg-command-status
8 Run Chained FFmpeg Commands ✅ PASS renderio-run-chained-ffmpeg-commands
9 Run Multiple Independent FFmpeg Commands ✅ PASS renderio-run-multiple-ffmpeg-commands
10 List Available RenderIO Presets ✅ PASS renderio-list-presets
11 Get Details Of A Specific Preset ✅ PASS renderio-list-presets, renderio-get-preset
12 Execute A RenderIO Preset ✅ PASS renderio-list-presets, renderio-execute-preset
13 Download A Video With yt-dlp ✅ PASS renderio-download-video-with-ytdlp
14 Download And Process Video With yt-dlp ✅ PASS renderio-download-and-process-video-with-ytdlp

All 14 tools were exercised. Avg 1.4 tool calls per eval. Total run time: ~60s.

Evals tracked in pd-connect-eval-monster

@GTFalcao GTFalcao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, left a couple comments, one is a structural change for sources that we need to address.

Comment thread components/renderio/actions/run-ffmpeg-command/run-ffmpeg-command.mjs Outdated
Comment thread components/renderio/sources/new-ffmpeg-command/new-ffmpeg-command.mjs Outdated
Comment thread components/renderio/sources/new-stored-file/new-stored-file.mjs Outdated
Comment thread components/renderio/sources/new-stored-file/new-stored-file.mjs Outdated
@GTFalcao GTFalcao moved this from Ready for PR Review to Changes Required in Component (Source and Action) Backlog Jun 22, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@components/renderio/sources/common/base.mjs`:
- Around line 43-46: The code is advancing lastTs with synthetic fallback
timestamps from getTimestamp() which returns Date.now() when API timestamps are
missing or invalid. This can cause legitimate events with older real timestamps
to be skipped. Modify the logic where items are checked and added (in the
condition checking ts >= lastTs and the items.push(item) section, as well as the
similar code at lines 64-66) to distinguish between real API timestamps and
synthetic fallback values, then only update lastTs with real timestamps from the
API, not with the Date.now() fallback values returned by getTimestamp().
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 30aaa4d9-7891-4d3e-a3eb-659f730050f8

📥 Commits

Reviewing files that changed from the base of the PR and between 991173a and 7eb253b.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (4)
  • components/renderio/actions/run-ffmpeg-command/run-ffmpeg-command.mjs
  • components/renderio/sources/common/base.mjs
  • components/renderio/sources/new-ffmpeg-command/new-ffmpeg-command.mjs
  • components/renderio/sources/new-stored-file/new-stored-file.mjs

Comment thread components/renderio/sources/common/base.mjs
@michelle0927 michelle0927 moved this from Changes Required to Ready for PR Review in Component (Source and Action) Backlog Jun 22, 2026

@GTFalcao GTFalcao left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@GTFalcao GTFalcao moved this from Ready for PR Review to Ready for QA in Component (Source and Action) Backlog Jun 22, 2026
@michelle0927

Copy link
Copy Markdown
Collaborator Author

All evals passed! Ready for Release!

@michelle0927 michelle0927 merged commit 837fcb9 into master Jun 22, 2026
9 checks passed
@michelle0927 michelle0927 deleted the renderio-new-components branch June 22, 2026 21:10
@michelle0927 michelle0927 mentioned this pull request Jun 22, 2026
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

[APP] RenderIO

3 participants