Feat/distribution check 2 stage

.clinerules/workflows/examine_logs_from_workflow.md

@@ -0,0 +1,40 @@
+Help the user figure out what happened in a Github Action test that failed.
+<detailed_sequence_of_steps>
+## 1. Identify the workflow to analyze
+<ask_followup_question>
+<question>What Tests do you want to analyze?</question>
+</ask_followup_question>
+The user should describe the Tests they want to analyze. Unless otherwise specified, assume they mean the "Integration Tests" workflow.
+
+They could specify a run for a specific branch or commit or PR, or simply whatever the latest results are.
+## 2. Load the workflow run overview
+Use the GitHub CLI tool to find the workflow run specified by the user. Do NOT download the entire logs, they are too large. Instead, use the GitHub CLI tool to get the workflow run overview. This will give you a summary of the workflow run, including the status of each job and step. 
+
+## 3. Matrix job
+"Integration Tests" runs a matrix job, so each Test Class is run in it's own job, with it's own result (the job will fail if there are any test failures) and logs.
+
+## 4. Download the logs
+The logs will be Artifacts attached to the workflow run. Download them to a directory that has some idenifying information about the workflow run, such as the commit hash or PR number. Download the FILES locally, make sure they don't end up in the LLM context. Only download the logs for jobs that failed.
+
+## 5. Identify the failing tests
+There will be a file in the logs directory called `failures.log`. This file will contain a list of the tests that failed, along with the reason for the failure. You can use this file to identify which tests to analyze.
+
+## 6. Summarize what has failed, ask for next steps
+Now simply summarize the failures (how many tests, which tests, which classes)
+<ask_followup_question>
+<question>Which tests do you want to analyze?</question>
+</ask_followup_question>
+The user may want to analyze all the tests, or just a specific test.
+
+## 6. Use the testermintlogs tool to analyze the log
+Look at `examine_test_log.md` for details on how to analyze the logs. The test log for a failure will be in the logs directory, and will be named after the test case, with `ClassName-test name might have spaces.log` as the name.
+
+## 7. Summarize all the findings and include CLI commands for further analysis
+Focus on next steps and the likely cause of the failures. If a failure is clearly a known failure, be sure and emphasize that.
+Format a copy-pasteable cmd to run lnav on the log file so the user can grab that and look at the log file themselves.
+Fairly simple, just `lnav <log file>`, with the full path and quotes if needed.
+
+## 8. Be prepared for a drill down request
+The user may want you to drill down into a specific test failure. Use the Resources provided by the `testermintlogs` tool to understand how to drill down and look. You can also look at the test file itself to understand the test and what it is supposed to do. If you have questions about the test, ask the user for clarification.
+
+</detailed_sequence_of_steps>

.clinerules/workflows/examine_test_log.md

@@ -0,0 +1,41 @@
+# Examine a test log
+Your job is to use the testermintlogs tool to analyze a test log file. The log file is MUCH too large to open directly, and will either be too big for most models
+or will cost a TON of tokens, so you should use the testermintlogs tool to sift through the logs.
+<detailed_sequence_of_steps>
+## 1. Identify the test to analyze
+Look up the most recent test runs in ./testermint/logs/failures.log.
+Present the user with a list of the most recent failures, and ask them to pick either one or describe which ones to analyze.
+## 2. Load the test log file
+The test log for a failure will be in the logs directory, and will be named after the test case, with `ClassName-test name might have spaces.log` as the name.
+
+Load it into the testermintlogs tool by passing in the full path to the file.
+## 3. Use the testermintlogs tool to analyze the log
+Start by loading the Step by Step instructions resource. This will give you an overview of the approach to use for examining the log. There are other critical resources, but load them as needed, not before hand to reduce token usage. USE THE TOOL AND THE RESOURCES TO UNDERSTAND THE LOGS!
+
+## 4. Expand the context as needed
+The source code for the tests themselves are in the `testermint` folder in this project, and the product code is in `inference-chain` and `decentralized-api`. Use these (especially the test code) to help you understand what the test is doing and what might be going wrong.
+
+## 5. Summarize all the findings as a final step.
+Focus on next steps and the likely cause of the failures. If a failure is clearly a known failure, be sure and emphasize that.
+
+Format a copy-pasteable cmd to run lnav on the log file so the user can grab that and look at the log file themselves.
+Fairly simple, just `lnav <log file>`, with the full path and quotes if needed.
+
+## 6. Offer to rerun certain tests
+<ask_followup_question>
+Ask the user if they want to rerun any test or tests based on the findings.
+<question>Would you like to rerun any tests based on these findings?</question>
+</ask_followup_question>
+
+## 7. Backup the original log file for comparson
+
+## 8. Rerun tests if requested
+Use gradle for the testermint project. You can run by test names, class names or tags. For example:
+```bash
+cd testermint && ./gradlew test --tests "TestClassName.*"
+```
+
+## 9. Repeat the analysis if tests were rerun
+Go back to step 1 based on the new test results. Pay special attention to if the failure happened again in the exact same way.
+
+</detailed_sequence_of_steps>

.clinerules/workflows/run_test_and_examine_logs.md

@@ -0,0 +1,33 @@
+<detailed_sequence_of_steps>
+# 1. Identify the test to run
+<ask_followup_question>
+<question>What test (or tests) would you like to run?</question>
+</ask_followup_question>
+After the user has selected the test, verify that it is a test,tag or class by looking in the testermint project. Ask follow up questions if unnable to find the test.
+
+# 2. Build and start the chain
+```bash
+cd local-test-net
+./stop-rebuild-launch.sh
+```
+Do NOT include the output of this command in the context! If it fails, immediately stop and ask the user to fix it.
+
+# 4. Run the test(s)
+Use gradle for the testermint project. You can run by test names, class names or tags. For example:
+```bash
+cd testermint && ./gradlew test --tests "TestClassName.*"
+```
+or, for tags:
+```bash
+cd testermint && ./gradlew test --tests "*" -DexcludeTags=unstable,exclude
+```
+
+# 5. Examine the logs
+First, look at the ./testermint/logs/failures.log file to see if there are any failures. If there are, you can use the testermintlogs tool to analyze the logs. The logs will be in the logs directory, and will be named after the test case, with `ClassName-test name might have spaces.log` as the name.
+
+Refer to examine_test_log.md for full details on how to examine the logs. You can use the testermintlogs tool to analyze the logs, and you can also use the test code and product code to help you understand what is going wrong.
+
+# 6. Summarize the findings
+Report back failure/success as well as an analysis of any failures. If a failure is clearly a known failure, be sure and emphasize that.
+
+</detailed_sequence_of_steps>

.dockerignore

@@ -0,0 +1,50 @@
+# VCS and IDE
+.git
+.gitignore
+.gitattributes
+**/.git
+**/.DS_Store
+**/.idea/
+**/.vscode/
+
+# Logs and transient files
+**/*.log
+direct.log
+test-logs-*/
+
+# Dependency caches
+**/.gradle/
+**/node_modules/
+
+# Build outputs and caches
+**/build/
+**/dist/
+**/output/
+**/.cache
+**/.pytest_cache
+**/__pycache__
+
+# Environment files
+**/.venv
+**/*.env
+
+# Large/unrelated directories to api/node Docker builds
+local-test-net/
+testermint/
+public-html/
+data/
+genesis/
+prod-genesis/
+prod-local/
+prod-sim/
+deploy/
+docs/
+proposals/
+mcp-obsidian/
+logdy/
+jq_test/
+test-net-cloud/
+upgrade-build-files/
+16504109747-artifacts/
+demo/
+mlnode/

.github/actions/gcp-auth-and-gcloud-setup/action.yml

@@ -0,0 +1,14 @@
+name: Authenticate to GCP and setup gcloud
+description: Authenticate to GCP using identity workload federation approach and setup gcloud CLI
+runs:
+  using: "composite"
+  steps:
+  - id: 'gcp-auth'
+    name: 'Authenticate to Google Cloud'
+    uses: 'google-github-actions/auth@v1.1.1'
+    with:
+      workload_identity_provider: 'projects/552613898602/locations/global/workloadIdentityPools/github-pool/providers/github'
+      service_account: 'test-net-deploy@decentralized-ai.iam.gserviceaccount.com'
+  - id: 'gcloud-setup'
+    name: Set up gcloud Cloud SDK environment
+    uses: google-github-actions/setup-gcloud@v1.1.1