feat(script): Add logic to generate a Nuget.config and skip already uploaded packages.

Author: spbsoluble

.github/workflows/sync-nuget.yml

@@ -54,32 +54,8 @@ jobs:
           python -m pip install --upgrade pip
           pip install -e .
 
-      - name: Create NuGet config
-        run: |
-          mkdir -p ~/.nuget/NuGet
-          echo '<?xml version="1.0" encoding="utf-8"?>' > ~/.nuget/NuGet/NuGet.Config
-          echo '<configuration>' >> ~/.nuget/NuGet/NuGet.Config
-          echo '  <packageSources>' >> ~/.nuget/NuGet/NuGet.Config
-          echo '    <add key="keyfactor" value="https://pkgs.dev.azure.com/Keyfactor/_packaging/KeyfactorPackages/nuget/v3/index.json" />' >> ~/.nuget/NuGet/NuGet.Config
-          echo '    <add key="github" value="https://nuget.pkg.github.com/Keyfactor/index.json" />' >> ~/.nuget/NuGet/NuGet.Config
-          echo '  </packageSources>' >> ~/.nuget/NuGet/NuGet.Config
-          echo '  <packageSourceCredentials>' >> ~/.nuget/NuGet/NuGet.Config
-          echo '    <github>' >> ~/.nuget/NuGet/NuGet.Config
-          echo '      <add key="Username" value="spbsoluble" />' >> ~/.nuget/NuGet/NuGet.Config
-          echo '      <add key="ClearTextPassword" value="${{ secrets.GH_NUGET_TOKEN }}" />' >> ~/.nuget/NuGet/NuGet.Config
-          echo '    </github>' >> ~/.nuget/NuGet/NuGet.Config
-          echo '    <keyfactor>' >> ~/.nuget/NuGet/NuGet.Config
-          echo '      <add key="Username" value="AzureDevOps" />' >> ~/.nuget/NuGet/NuGet.Config
-          echo '      <add key="ClearTextPassword" value="${{ secrets.AZ_DEVOPS_PAT }}" />' >> ~/.nuget/NuGet/NuGet.Config
-          echo '    </keyfactor>' >> ~/.nuget/NuGet/NuGet.Config
-          echo '  </packageSourceCredentials>' >> ~/.nuget/NuGet/NuGet.Config
-          echo '</configuration>' >> ~/.nuget/NuGet/NuGet.Config
-          cat ~/.nuget/NuGet/NuGet.Config
-        env:
-          GITHUB_TOKEN: ${{ secrets.GH_NUGET_TOKEN }}
-
       - name: Run NuGet sync script
         run: python scripts/sync_nuget.py
         env:
           GITHUB_TOKEN: ${{ secrets.GH_NUGET_TOKEN }}
-          # Add any other environment variables needed by your script
+          AZ_DEVOPS_PAT: ${{ secrets.AZ_DEVOPS_PAT }}

.gitignore

@@ -4,4 +4,8 @@ _site
 .jekyll-metadata
 vendor
 
-.vs
+.vs
+*.env
+.claude
+CLAUDE.md
+nupkgs/*

README.md

@@ -2,36 +2,49 @@
 
 Publicly available NuGet packages useful for building Keyfactor integrations
 
-To access these packages add https://nuget.pkg.github.com/Keyfactor/index.json as a NuGet package source to Visual
-Studio or other development tools.
+To access these packages add `https://nuget.pkg.github.com/Keyfactor/index.json` as a NuGet package source in Visual Studio or other development tools.
 
-Some package availability changes from time to time. Please view the `CHANGELOG.md` for further details on changes to
-available libraries.
+Some package availability changes from time to time. Please view the `CHANGELOG.md` for further details on changes to available libraries.
 
-## Adding a Nuget Package and/or Version
+## Adding a NuGet Package and/or Version
 
 ### Via GitHub Actions
 
-To add a Nuget package modify the [packages.yml](./packages.yml) file in the root of this repository.
+1. Add the package and version(s) to [`packages.yml`](./packages.yml).
+2. Trigger the [Sync NuGet Packages](https://github.com/Keyfactor/public-nuget-packages/actions/workflows/sync-nuget.yml) workflow.
+
+Once complete, the package will be available in the Keyfactor Public GitHub Package Registry (GPR). The workflow runs automatically on a weekly schedule and skips any versions already published.
 
 > [!IMPORTANT]
-> The package and version must exist in this Nuget repository before it can be added to the `packages.yml` file.
+> The package and version must already exist in the Azure DevOps feed before adding it to `packages.yml`:
 > https://dev.azure.com/Keyfactor/Engineering/_artifacts/feed/KeyfactorPackages@Local
 
-The to trigger the sync run the
-workflow [Sync NuGet Packages](https://github.com/Keyfactor/public-nuget-packages/actions/workflows/sync-nuget.yml). Once
-it's complete, the package will be available in the Keyfactor Public GitHub Package Registry (GPR).
+### Running Locally
 
-### Manually using dotnet CLI
+**Prerequisites:** Python 3.8+, `nuget` CLI (with Mono on Linux), `dotnet` CLI
+
+1. Create a `.env` file in the repo root:
+
+```bash
+export AZ_DEVOPS_PAT=<Azure DevOps PAT with package read permissions>
+export GITHUB_TOKEN=<GitHub PAT with write:packages permission>
+export GITHUB_ORG=keyfactor
+```
 
-In order to add a nuget package to the Keyfactor Public GitHub Package Registry (GPR), you must download from the 
-[DevOps Artifacts site](https://dev.azure.com/Keyfactor/Engineering/_artifacts/feed/KeyfactorPackages@Local) 
-and then push it to the GPR using a Personal Access Token (PAT).
+2. Install dependencies and run:
+
+```bash
+python -m venv venv && source venv/bin/activate
+pip install -e .
+source .env && python scripts/sync_nuget.py
+```
+
+The script queries the GitHub Package Registry first and skips any versions already published.
+
+### Manually using dotnet CLI
 
-Steps:
-1. Create a new PAT with access with `write:packages` access
-2. Grant SSO to the PAT
-3. Run the following command to push the package to the GitHub Package Registry (GPR):
+1. Create a GitHub PAT with `write:packages` access and grant it SSO.
+2. Push the package:
 
 ```bash
 dotnet nuget push ./Your.Package.1.0.0.nupkg \

pyproject.toml

@@ -13,7 +13,8 @@ readme = "README.md"
 requires-python = ">=3.8"
 license = {file = "LICENSE"}
 dependencies = [
-    "pyyaml>=6.0"
+    "pyyaml>=6.0",
+    "requests>=2.28"
 ]
 
 [project.optional-dependencies]

scripts/sync_nuget.py

@@ -1,16 +1,21 @@
 import os
 import shutil
 import subprocess
+import tempfile
+import requests
 import yaml
 
 class NuGetSyncer:
     def __init__(self):
         self.NUGET_FEED_URL = "https://pkgs.dev.azure.com/Keyfactor/_packaging/KeyfactorPackages/nuget/v3/index.json"
         self.GITHUB_NUGET_URL = "https://nuget.pkg.github.com/keyfactor/index.json"
         self.GITHUB_TOKEN = os.getenv("GH_NUGET_TOKEN", os.getenv("GITHUB_TOKEN"))
+        self.AZ_DEVOPS_PAT = os.getenv("AZ_DEVOPS_PAT")
         self.TMP_DIR = "nupkgs"
         self.PACKAGES_YML = "packages.yml"
+        self.GITHUB_NUGET_BASE = "https://nuget.pkg.github.com/keyfactor"
         self.allowed_packages = self.load_allowed_packages()
+        self._github_versions_cache = {}
         os.makedirs(self.TMP_DIR, exist_ok=True)
 
     def load_allowed_packages(self):
@@ -34,44 +39,84 @@ def get_all_packages_and_versions(self):
             return self.allowed_packages
         return {pkg: [] for pkg in self.allowed_packages}
 
+    def get_github_published_versions(self, name):
+        """Fetch the list of versions already published to GitHub Packages for a given package."""
+        if name in self._github_versions_cache:
+            return self._github_versions_cache[name]
+        url = f"{self.GITHUB_NUGET_BASE}/download/{name.lower()}/index.json"
+        try:
+            resp = requests.get(url, auth=("token", self.GITHUB_TOKEN), timeout=15)
+            if resp.status_code == 200:
+                versions = set(resp.json().get("versions", []))
+            else:
+                versions = set()
+        except Exception as e:
+            print(f"Warning: could not fetch published versions for {name}: {e}")
+            versions = set()
+        self._github_versions_cache[name] = versions
+        return versions
+
     def download_package(self, name, version):
         filename = f"{name}.{version}.nupkg".replace("/", "_")
         filepath = os.path.join(self.TMP_DIR, filename)
         if os.path.exists(filepath):
             print(f"Already downloaded: {filename}")
             return filepath
         print(f"Downloading {name} {version} from NuGet v3 feed...")
-        config_path = os.path.expanduser("~/.nuget/NuGet/NuGet.Config")
-        cmd = [
-            "nuget", "install", name,
-            "-Source", self.NUGET_FEED_URL,
-            "-Version", version,
-            "-OutputDirectory", self.TMP_DIR,
-            "-DirectDownload",
-            "-ConfigFile", config_path
-        ]
-        subprocess.run(cmd, check=True)
-        # Find the downloaded .nupkg file
-        pkg_dir = os.path.join(self.TMP_DIR, f"{name}.{version}")
-        for file in os.listdir(pkg_dir):
-            if file.endswith(".nupkg"):
-                src = os.path.join(pkg_dir, file)
-                os.rename(src, filepath)
-                print(f"Downloaded: {filename}")
-                break
-        # Remove all non-nupkg files in the self.TMP_DIR
-        files = os.listdir(self.TMP_DIR)
-        for file in files:
-            try:
-                # Check if file is a directory
-                full_path = os.path.join(self.TMP_DIR, file)
-                if os.path.isdir(full_path):
-                    shutil.rmtree(full_path)  # Recursively deletes directory and contents
-                elif not file.endswith(".nupkg"):
-                    os.remove(full_path)
-            except Exception as e:
-                print(f"Failed to remove {file} in {self.TMP_DIR}. It may not be a directory or file.")
-                print(e)
+        tmp_config_path = None
+        if self.AZ_DEVOPS_PAT:
+            nuget_config = f"""<?xml version="1.0" encoding="utf-8"?>
+<configuration>
+  <packageSources>
+    <add key="KeyfactorPackages" value="{self.NUGET_FEED_URL}" />
+  </packageSources>
+  <packageSourceCredentials>
+    <KeyfactorPackages>
+      <add key="Username" value="any" />
+      <add key="ClearTextPassword" value="{self.AZ_DEVOPS_PAT}" />
+    </KeyfactorPackages>
+  </packageSourceCredentials>
+</configuration>"""
+            with tempfile.NamedTemporaryFile(mode='w', suffix='.config', delete=False) as tmp:
+                tmp.write(nuget_config)
+                tmp_config_path = tmp.name
+            config_path = tmp_config_path
+        else:
+            config_path = os.path.expanduser("~/.nuget/NuGet/NuGet.Config")
+        try:
+            cmd = [
+                "nuget", "install", name,
+                "-Source", self.NUGET_FEED_URL,
+                "-Version", version,
+                "-OutputDirectory", self.TMP_DIR,
+                "-DirectDownload",
+                "-ConfigFile", config_path,
+            ]
+            subprocess.run(cmd, check=True)
+            # Find the downloaded .nupkg file
+            pkg_dir = os.path.join(self.TMP_DIR, f"{name}.{version}")
+            for file in os.listdir(pkg_dir):
+                if file.endswith(".nupkg"):
+                    src = os.path.join(pkg_dir, file)
+                    os.rename(src, filepath)
+                    print(f"Downloaded: {filename}")
+                    break
+            # Remove all non-nupkg files in the self.TMP_DIR
+            files = os.listdir(self.TMP_DIR)
+            for file in files:
+                try:
+                    # Check if file is a directory
+                    full_path = os.path.join(self.TMP_DIR, file)
+                    if os.path.isdir(full_path):
+                        shutil.rmtree(full_path)  # Recursively deletes directory and contents
+                    elif not file.endswith(".nupkg"):
+                        os.remove(full_path)
+                except Exception as e:
+                    print(f"Failed to remove {file} in {self.TMP_DIR}. It may not be a directory or file.")
+                    print(e)
+        finally:
+            if tmp_config_path:
+                os.unlink(tmp_config_path)
 
         return filepath
 
@@ -133,17 +178,34 @@ def sync_packages(self):
             return
         print(f"Will sync the following packages: {self.allowed_packages}")
         packages_and_versions = self.get_all_packages_and_versions()
+        skipped = 0
+        successful = 0
+        failed = 0
         for pkg in packages_and_versions:
             pkg_name = pkg.get('name', pkg)
             versions = pkg.get('versions', [])
+            published = self.get_github_published_versions(pkg_name)
             for version in versions:
+                if version in published:
+                    print(f"Already published: {pkg_name} {version} — skipping")
+                    skipped += 1
+                    continue
                 try:
                     package_file = self.download_package(pkg_name, version)
                     # Upload to GitHub after successful download
                     if package_file and os.path.exists(package_file):
-                        self.upload_to_github(package_file)
+                        if self.upload_to_github(package_file):
+                            successful += 1
+                        else:
+                            failed += 1
                 except Exception as e:
-                    print(f"Failed to download {pkg_name} {version}: {e}")
+                    print(f"Failed to sync {pkg_name} {version}: {e}")
+                    failed += 1
+
+        print("\nSync summary:")
+        print(f"  Uploaded:  {successful}")
+        print(f"  Skipped:   {skipped}")
+        print(f"  Failed:    {failed}")
 
 if __name__ == "__main__":
     syncer = NuGetSyncer()