Merge pull request #1 from ziadhany/mine-cargo

Add cargo packageURL mining in CI with scancode-action

Author: AyanSinhaMahapatra

.github/workflows/mine-cargo-packageurls.yml

@@ -0,0 +1,16 @@
+on: [workflow_dispatch]
+
+jobs:
+  mine-pypi-purls:
+    runs-on: ubuntu-24.04
+    name: Mine cargo PackageURLs
+    steps:
+    - uses: aboutcode-org/scancode-action@beta
+      with:
+        scancodeio-repo-branch: "collect-purl-metadata#egg=scancodeio[mining]"
+        pipelines: "mine_cargo"
+      env:
+        FEDERATEDCODE_GIT_ACCOUNT_URL: https://github.com/aboutcode-data/minecode-data-cargo-test
+        FEDERATEDCODE_GIT_SERVICE_TOKEN: ${{ secrets.MINING_GITHUB_TOKEN }}
+        FEDERATEDCODE_GIT_SERVICE_NAME: "AboutCode Automation"
+        FEDERATEDCODE_GIT_SERVICE_EMAIL: "automation@aboutcode.org"

.github/workflows/mine-pypi-packageurls.yml

@@ -12,5 +12,5 @@ jobs:
       env:
         FEDERATEDCODE_GIT_ACCOUNT_URL: https://github.com/aboutcode-data/minecode-data-pypi-test
         FEDERATEDCODE_GIT_SERVICE_TOKEN: ${{ secrets.MINING_GITHUB_TOKEN }}
-        FEDERATEDCODE_GIT_SERVICE_NAME: "the AboutCode bot"
+        FEDERATEDCODE_GIT_SERVICE_NAME: "AboutCode Automation"
         FEDERATEDCODE_GIT_SERVICE_EMAIL: "automation@aboutcode.org"

README.rst

@@ -21,22 +21,28 @@ Configuration format
 
   * last serial number processed (used in indexes at pypi, npm etc)
   * last processed commit (where the data is stored in git repos)
-  * directory to store las fetched index data (like the JSON fetched from pypi simple with package names and last updated info)
+  * directory to store las fetched index data
+    (like the JSON fetched from pypi simple with package names and last updated info)
   * state information in ``state``:
 
     * ``null``: mining has not started.
-    * ``initital-sync`` : at the start of mining we need to mine a huge amount of packages for packageURL to catch up.
-      This is typically very large and could take several hours to several days dependening on the ecosystem size.
-      We fetch and save an index state and mine all packageURLs till there. Once we reach a state where remaining
-      new packageURLs can be mined in a couple hours, we can move on to the next state where we mine new packageURLs
-      added in a periodic manner.  
-    * ``periodic-sync`` : This is a periodic update of new packageURLs added in the index in a period, and typically this
+    * ``initital-sync`` : at the start of mining we need to mine a huge
+      amount of packages for packageURL to catch up.
+      This is typically very large and could take several hours to several days
+      dependening on the ecosystem size.
+      We fetch and save an index state and mine all packageURLs till there.
+      Once we reach a state where remaining
+      new packageURLs can be mined in a couple hours, we can move on to
+      the next state where we mine new packageURLs
+      added in a periodic manner.
+    * ``periodic-sync`` : This is a periodic update of new packageURLs
+      added in the index in a period, and typically this
       should not take more than a couple hours.
 
   * optional elements to improve readability/debugging:
 
     * ``last_updated``: date and time of last checkpoint update
 
 * ``packages_checkpoints.json``: stores checkpoint related to:
-  
+
   * ``packages_mined``: which packages have been mined in the ``initital-sync`` state.

cargo/checkpoints.json

@@ -0,0 +1 @@
+{}

docs/source/index.rst

@@ -1,5 +1,5 @@
 Welcome to miencode-pipelines documentation!
-=========================================
+=============================================
 
 This is released at pypi: https://pypi.org/project/minecode-pipelines/