Merge pull request #29 from thisisqubika/DC-353-fix

fix on output folder logic to include eval results

Author: FacuSentena

README.md

@@ -138,3 +138,27 @@ Notes:
 - `make translate` invokes the script at `scripts/run_translation.sh` and is the cleanest option.
 - The script exports `PYTHONPATH=src` so you can run it from the repository root.
 - Output is written to `src/artifact_translation_package/out_sql_examples_<timestamp>`.
+
+Local vs Databricks output paths
+--------------------------------
+
+This project uses Databricks-style paths (for example `dbfs:/...`) as the canonical configuration. To make the same code run locally without changing the canonical config, the runner maps Databricks paths to a local directory when it detects a non-Databricks runtime.
+
+- **Local mapping environment variable**: set `LOCAL_DBFS_MOUNT` to the local directory that should act as the root for `dbfs:/` paths. Default: `./ddl_output`.
+- **Per-run `results_dir`**: when running the translation job locally (for example via `make translate`), the job pre-creates a timestamped `results_dir` and propagates it to the translation context. Evaluation results are written under `<results_dir>/evaluation_results/`. Translation outputs such as `translation_results.json` and `results_summary.json` are written into the same `results_dir`.
+- **Fallback behavior**: the runner also exports `DDL_OUTPUT_DIR=<results_dir>` so code paths that read the `DDL_OUTPUT_DIR` environment variable will also target the per-run folder.
+
+Examples
+
+```bash
+# run with default local mapping (output under ./ddl_output by default)
+make translate
+
+# override where dbfs:/ maps to locally
+export LOCAL_DBFS_MOUNT=/tmp/my_local_dbfs
+make translate
+
+# inspect latest run outputs
+ls -la src/artifact_translation_package/out_sql_examples_*/
+ls -la <that-run-folder>/evaluation_results/
+```

scripts/run_translation.sh

@@ -9,13 +9,15 @@ BATCH_SIZE=${2:-2}
 FORMAT=${3:-sql}
 
 TIMESTAMP=$(date +%Y%m%d_%H%M%S)
-OUT_DIR="src/artifact_translation_package/out_sql_examples_${TIMESTAMP}"
+# Prefer an explicit DDL_OUTPUT_DIR if provided (e.g. from .env). Otherwise
+# fall back to creating a timestamped out_sql_examples_<timestamp> folder.
+OUT_DIR="${DDL_OUTPUT_DIR:-src/artifact_translation_package/out_sql_examples_${TIMESTAMP}}"
 
 echo "-> Running translation job"
 echo "   examples: ${EXAMPLES_GLOB}"
 echo "   batch size: ${BATCH_SIZE}"
 echo "   output format: ${FORMAT}"
-echo "   output dir: ${OUT_DIR}"
+echo "   output dir: ${OUT_DIR} (DDL_OUTPUT_DIR=${DDL_OUTPUT_DIR:-unset})"
 
 mkdir -p "$OUT_DIR"
 

src/artifact_translation_package/README.md

@@ -164,6 +164,28 @@ DATABRICKS_TOKEN=your-databricks-token
 DDL_OUTPUT_DIR=./output
 ```
 
+### Local vs Databricks output paths
+
+- Databricks is the canonical runtime and configuration may reference DBFS paths like `dbfs:/some/base`.
+- When running locally (for example via `make translate` or the provided run scripts), the code will automatically map `dbfs:/...` paths to a local directory so outputs are written to your machine.
+- Control the local mapping with the `LOCAL_DBFS_MOUNT` environment variable (default: `./ddl_output`). Example:
+
+```bash
+# use default local mapping
+make translate
+
+# override local mapping directory
+export LOCAL_DBFS_MOUNT=./my_local_dbfs_mount
+make translate
+```
+
+- Evaluation/validation JSON files are written into the per-run timestamped results folder under an `evaluation_results/` subdirectory (not into a global `ddl_output/evaluation_results` folder) when an output path is used. This keeps SQL, JSON, and evaluation artifacts grouped by run for easier inspection.
+
+Note: the included `scripts/run_translation.sh` helper prefers the `DDL_OUTPUT_DIR` environment
+variable when present (for example from your `.env` file). If `DDL_OUTPUT_DIR` is unset the helper
+falls back to creating `src/artifact_translation_package/out_sql_examples_<timestamp>` as before.
+
+
 ## Architecture
 
 ### Translation Graph

src/artifact_translation_package/config/constants.py

@@ -14,7 +14,8 @@ class LangGraphConfig(Enum):
     #DDL Settings
     DDL_BATCH_SIZE = 8
     # Default outputs for Databricks should go to a Volume mounted path
-    DDL_OUTPUT_DIR = "/Volumes/qubika_partner_solutions.migration_accelerator.outputs"
+    # Use segmented path segments: /Volumes/<catalog>/<schema>/<volume_name>/
+    DDL_OUTPUT_DIR = "/Volumes/qubika_partner_solutions/migration_accelerator/outputs"
 
     # Optional: DDL Generation Settings
     #DDL_CATALOG_NAME= "demo_catalog"