Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -77,3 +77,6 @@ Thumbs.db
/.playwright-mcp
.azure
/src/BlazorDataOrchestrator.Core/BlazorDataOrchestrator.Core.csproj.lscache

# Auto-generated job template zip (built by scripts/Package-JobTemplate.ps1)
/src/BlazorOrchestrator.Web/JobTemplate/BlazorDataOrchestrator.JobCreatorTemplate.zip
191 changes: 191 additions & 0 deletions docs/JobTemplateZipAutomationPlan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
# Job Template Zip Automation Plan

## Problem

The `BlazorDataOrchestrator.JobCreatorTemplate` project is shipped to users as a zip file located at `src/BlazorOrchestrator.Web/JobTemplate/BlazorDataOrchestrator.JobCreatorTemplate.zip`. Today this zip is created **manually** with error-prone steps:

1. Copy the entire `src/BlazorDataOrchestrator.JobCreatorTemplate` directory
2. Delete `bin/`, `obj/`, `Properties/` directories
3. Delete `BlazorDataOrchestrator.JobCreatorTemplate.csproj.user`
4. Include the root-level `JobTemplate.slnx` file
5. Zip everything and place it in `src/BlazorOrchestrator.Web/JobTemplate/`

This is tedious, easy to forget, and can lead to stale zip files being deployed.

---

## Goals

- **Zero manual steps** — the zip is regenerated automatically on every build of `BlazorOrchestrator.Web`.
- **Always in sync** — any change to the template project is immediately reflected in the zip.
- **No extra tools** — use only MSBuild targets and built-in .NET SDK capabilities (or a small PowerShell/shell script called from MSBuild).
- **CI-friendly** — works in both local dev (`dotnet build` / `aspire run`) and CI/CD pipelines.

---

## Proposed Solution

### Option A — MSBuild Target in `BlazorOrchestrator.Web.csproj` (Recommended)

Add a `<Target>` to `BlazorOrchestrator.Web.csproj` that runs **before** the build, invokes a PowerShell/shell script (or inline MSBuild tasks) to produce the zip, and places it in the `JobTemplate/` folder so it gets included as content.

#### Implementation Steps

##### 1. Create the packaging script

Create `scripts/Package-JobTemplate.ps1`:

```powershell
<#
.SYNOPSIS
Packages the JobCreatorTemplate project into a zip for distribution.
#>
param(
[Parameter(Mandatory)]
[string]$SourceDir, # Path to src/BlazorDataOrchestrator.JobCreatorTemplate

[Parameter(Mandatory)]
[string]$SlnxFile, # Path to JobTemplate.slnx

[Parameter(Mandatory)]
[string]$OutputZip # Full path for the output .zip file
)

$ErrorActionPreference = 'Stop'

# Create a temp staging directory
$stagingDir = Join-Path ([System.IO.Path]::GetTempPath()) "JobTemplateStaging_$([guid]::NewGuid().ToString('N'))"
$templateDir = Join-Path $stagingDir "BlazorDataOrchestrator.JobCreatorTemplate"

try {
# Copy the template project to staging
Copy-Item -Path $SourceDir -Destination $templateDir -Recurse

# Remove directories that should not be shipped
$dirsToRemove = @('bin', 'obj', 'Properties')
foreach ($dir in $dirsToRemove) {
$target = Join-Path $templateDir $dir
if (Test-Path $target) {
Remove-Item $target -Recurse -Force
}
}

# Remove user-specific files
$filesToRemove = @(
'*.csproj.user',
'execution_errors.log'
)
foreach ($pattern in $filesToRemove) {
Get-ChildItem -Path $templateDir -Filter $pattern -Recurse | Remove-Item -Force
}

# Copy the .slnx file into the staging root (sibling to the project folder)
Copy-Item -Path $SlnxFile -Destination $stagingDir

# Ensure the output directory exists
$outputDir = Split-Path $OutputZip -Parent
if (-not (Test-Path $outputDir)) {
New-Item -ItemType Directory -Path $outputDir -Force | Out-Null
}

# Remove old zip if it exists
if (Test-Path $OutputZip) {
Remove-Item $OutputZip -Force
}

# Create the zip
Compress-Archive -Path "$stagingDir\*" -DestinationPath $OutputZip -Force

Write-Host "Created template zip: $OutputZip"
}
finally {
# Clean up staging
if (Test-Path $stagingDir) {
Remove-Item $stagingDir -Recurse -Force
}
}
```

##### 2. Add MSBuild target to `BlazorOrchestrator.Web.csproj`

```xml
<!-- Auto-package the JobCreatorTemplate zip before build -->
<Target Name="PackageJobTemplate" BeforeTargets="BeforeBuild"
Inputs="$(MSBuildThisFileDirectory)..\BlazorDataOrchestrator.JobCreatorTemplate\**\*.*"
Outputs="$(MSBuildThisFileDirectory)JobTemplate\BlazorDataOrchestrator.JobCreatorTemplate.zip">
<Exec Command="pwsh -NoProfile -ExecutionPolicy Bypass -File &quot;$(MSBuildThisFileDirectory)..\..\scripts\Package-JobTemplate.ps1&quot; -SourceDir &quot;$(MSBuildThisFileDirectory)..\BlazorDataOrchestrator.JobCreatorTemplate&quot; -SlnxFile &quot;$(MSBuildThisFileDirectory)..\..\JobTemplate.slnx&quot; -OutputZip &quot;$(MSBuildThisFileDirectory)JobTemplate\BlazorDataOrchestrator.JobCreatorTemplate.zip&quot;" />
</Target>
```

Key details:
- **`Inputs` / `Outputs`** — MSBuild incremental build support. The target only re-runs when template source files are newer than the zip, avoiding unnecessary work on every build.
- **`BeforeTargets="BeforeBuild"`** — ensures the zip is ready before the Web project compiles and copies content to output.

##### 3. Ensure the zip is included as content

Verify that `BlazorOrchestrator.Web.csproj` already includes the zip as content (it likely does via a glob). If not, add:

```xml
<ItemGroup>
<Content Include="JobTemplate\BlazorDataOrchestrator.JobCreatorTemplate.zip">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
```

##### 4. Add the zip to `.gitignore`

Since the zip is now a build artifact, it should not be committed to source control:

```
# Auto-generated job template zip
src/BlazorOrchestrator.Web/JobTemplate/BlazorDataOrchestrator.JobCreatorTemplate.zip
```

---

### Option B — Standalone Script Only (Simpler, Less Integrated)

If MSBuild integration is not desired, the same `Package-JobTemplate.ps1` script can be run manually or as a CI step. This is simpler but requires developers to remember to run it.

Usage:
```powershell
.\scripts\Package-JobTemplate.ps1 `
-SourceDir "src\BlazorDataOrchestrator.JobCreatorTemplate" `
-SlnxFile "JobTemplate.slnx" `
-OutputZip "src\BlazorOrchestrator.Web\JobTemplate\BlazorDataOrchestrator.JobCreatorTemplate.zip"
```

---

### Option C — CI-Only Automation

Add a step to the CI/CD pipeline (GitHub Actions / Azure DevOps) that runs the script before building the Web project. The zip remains committed in the repo for local dev but is always regenerated in CI.

---

## Recommendation

**Option A** is recommended because:
- It is fully automatic — no manual steps, no forgotten updates.
- Incremental builds prevent performance impact (the zip is only regenerated when template files change).
- Works identically in local dev and CI.
- The script is cross-platform compatible (PowerShell 7 / `pwsh` runs on Windows, Linux, macOS).

---

## Files Changed

| File | Action | Purpose |
|------|--------|---------|
| `scripts/Package-JobTemplate.ps1` | **Create** | Packaging script |
| `src/BlazorOrchestrator.Web/BlazorOrchestrator.Web.csproj` | **Edit** | Add `PackageJobTemplate` MSBuild target |
| `.gitignore` | **Edit** | Exclude the generated zip |

## Risks & Mitigations

| Risk | Mitigation |
|------|------------|
| `pwsh` not installed on build agent | .NET 10 SDK images include PowerShell 7. Add a check/fallback in CI. Alternatively, rewrite the script as a cross-platform shell script or pure MSBuild tasks. |
| Incremental build `Inputs` glob is too broad | Refine the glob or add `Condition` checks if build times are affected. |
| Zip contents diverge from expectations | Add a smoke test that extracts the zip and verifies expected files are present (e.g., `.csproj`, `Program.cs`, `.slnx`, no `bin/`). |
92 changes: 92 additions & 0 deletions scripts/Package-JobTemplate.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
<#
.SYNOPSIS
Packages the JobCreatorTemplate project into a zip for distribution.
.DESCRIPTION
Copies the JobCreatorTemplate source, removes build artifacts and user-specific files,
includes the JobTemplate.slnx, and produces a zip ready for the Web project.
.PARAMETER SourceDir
Path to the BlazorDataOrchestrator.JobCreatorTemplate project directory.
.PARAMETER SlnxFile
Path to the JobTemplate.slnx file.
.PARAMETER OutputZip
Full path for the output .zip file.
#>
param(
[Parameter(Mandatory)]
[string]$SourceDir,

[Parameter(Mandatory)]
[string]$SlnxFile,

[Parameter(Mandatory)]
[string]$OutputZip
)

$ErrorActionPreference = 'Stop'

# Create a temp staging directory
$stagingDir = Join-Path ([System.IO.Path]::GetTempPath()) "JobTemplateStaging_$([guid]::NewGuid().ToString('N'))"
$templateDir = Join-Path $stagingDir "BlazorDataOrchestrator.JobCreatorTemplate"

try {
Write-Host "Packaging JobCreatorTemplate..."

# Copy the template project to staging
Copy-Item -Path $SourceDir -Destination $templateDir -Recurse

# Remove directories that should not be shipped
$dirsToRemove = @('bin', 'obj', 'Properties')
foreach ($dir in $dirsToRemove) {
$target = Join-Path $templateDir $dir
if (Test-Path $target) {
Remove-Item $target -Recurse -Force
Write-Host " Removed $dir/"
}
}

# Remove __pycache__ directories anywhere in the tree
Get-ChildItem -Path $templateDir -Directory -Filter '__pycache__' -Recurse -ErrorAction SilentlyContinue |
ForEach-Object {
Remove-Item $_.FullName -Recurse -Force
Write-Host " Removed $($_.FullName | Split-Path -Leaf)/ (pycache)"
}

# Remove user-specific and transient files
$filesToRemove = @(
'*.csproj.user',
'execution_errors.log'
)
foreach ($pattern in $filesToRemove) {
Get-ChildItem -Path $templateDir -Filter $pattern -Recurse -ErrorAction SilentlyContinue |
ForEach-Object {
Remove-Item $_.FullName -Force
Write-Host " Removed $($_.Name)"
}
}

# Copy the .slnx file into the staging root (sibling to the project folder)
Copy-Item -Path $SlnxFile -Destination $stagingDir
Write-Host " Included $(Split-Path $SlnxFile -Leaf)"

# Ensure the output directory exists
$outputDir = Split-Path $OutputZip -Parent
if (-not (Test-Path $outputDir)) {
New-Item -ItemType Directory -Path $outputDir -Force | Out-Null
}

# Remove old zip if it exists
if (Test-Path $OutputZip) {
Remove-Item $OutputZip -Force
}

# Create the zip
Compress-Archive -Path "$stagingDir\*" -DestinationPath $OutputZip -Force

Write-Host "Created template zip: $OutputZip"
}
finally {
# Clean up staging
if (Test-Path $stagingDir) {
Remove-Item $stagingDir -Recurse -Force
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
<PackageReference Include="Anthropic.SDK" Version="5.10.0" />
<!-- Google AI packages -->
<PackageReference Include="Mscc.GenerativeAI" Version="3.1.0" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.15.3" />
<PackageReference Include="Radzen.Blazor" Version="*" />
</ItemGroup>

Expand Down
10 changes: 6 additions & 4 deletions src/BlazorDataOrchestrator.Core/JobManager.cs
Original file line number Diff line number Diff line change
Expand Up @@ -494,12 +494,14 @@ public async Task RunJobAsync(int jobInstanceId)

var evaluator = CSScript.Evaluator;

// Add references to all DLLs found in the package
// Security: Reject packages that contain .dll files.
// Only NuGet package dependencies (resolved via .nuspec) are allowed.
var dlls = Directory.GetFiles(tempDir, "*.dll", SearchOption.AllDirectories);
foreach (var dll in dlls)
if (dlls.Length > 0)
{
evaluator.ReferenceAssembly(dll);
await LogAsync("RunJob", $"Added reference: {Path.GetFileName(dll)}", "Debug");
var dllNames = string.Join(", ", dlls.Select(System.IO.Path.GetFileName));
await LogAsync("RunJob", $"Security: Rejected package containing .dll files: {dllNames}", "Error");
throw new InvalidOperationException($"Package contains .dll files which are not allowed. Use NuGet packages instead. Found: {dllNames}");
}

// Load and execute
Expand Down
37 changes: 8 additions & 29 deletions src/BlazorDataOrchestrator.Core/Services/CodeExecutorService.cs
Original file line number Diff line number Diff line change
Expand Up @@ -177,19 +177,16 @@ private async Task<CodeExecutionResult> ExecuteCSharpAsync(string extractedPath,
}
}

// Add references to DLLs found in the package
// Security: Reject packages that contain .dll files.
// Only NuGet package dependencies (resolved via .nuspec) are allowed.
var dlls = Directory.GetFiles(extractedPath, "*.dll", SearchOption.AllDirectories);
foreach (var dll in dlls)
if (dlls.Length > 0)
{
try
{
evaluator.ReferenceAssembly(dll);
result.Logs.Add($"Added reference: {Path.GetFileName(dll)}");
}
catch
{
// Ignore DLLs that can't be loaded
}
var dllNames = string.Join(", ", dlls.Select(Path.GetFileName));
result.Success = false;
result.ErrorMessage = $"Package contains .dll files which are not allowed. Use NuGet packages instead. Found: {dllNames}";
result.Logs.Add($"Security: Rejected package containing .dll files: {dllNames}");
return result;
}

// Add common references
Expand Down Expand Up @@ -234,24 +231,6 @@ private async Task<CodeExecutionResult> ExecuteCSharpAsync(string extractedPath,
}
}

// Also load DLLs from the package
foreach (var dll in dlls)
{
try
{
var loadedAsm = Assembly.LoadFrom(dll);
var asmName = loadedAsm.GetName().Name;
if (asmName != null && !loadedAssemblies.ContainsKey(asmName))
{
loadedAssemblies[asmName] = loadedAsm;
}
}
catch
{
// Ignore DLLs that can't be loaded
}
}

result.Logs.Add($"Pre-loaded {loadedAssemblies.Count} assemblies for runtime resolution.");

// Set up assembly resolve handler for the compiled script
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,21 @@ public async Task<PackageBuildResult> BuildPackageAsync(PackageBuildConfiguratio
var csharpFolder = Path.Combine(config.CodeRootPath, "CodeCSharp");
var pythonFolder = Path.Combine(config.CodeRootPath, "CodePython");

// Security: Reject source directories that contain .dll files.
// Users must use NuGet package dependencies declared in .nuspec instead.
if (Directory.Exists(config.CodeRootPath))
{
var dllFiles = Directory.GetFiles(config.CodeRootPath, "*.dll", SearchOption.AllDirectories);
if (dllFiles.Length > 0)
{
var dllNames = string.Join(", ", dllFiles.Select(Path.GetFileName));
result.Success = false;
result.ErrorMessage = $"Source directory contains .dll files which are not allowed in packages. Use NuGet packages instead. Found: {dllNames}";
result.Logs.Add($"Security: Rejected build due to .dll files: {dllNames}");
return result;
}
}

// Copy JSON files from the root Code folder
if (Directory.Exists(config.CodeRootPath))
{
Expand Down
Loading
Loading