Remove submodule, add proto file download script (#72)

* Initial plan

* Enhance proto generation: Remove submodule, add auto-download script

Co-authored-by: YunchuWang <12449837+YunchuWang@users.noreply.github.com>

* Improve download-proto.sh: Use jq when available, make commit hash fetch failure explicit

Co-authored-by: YunchuWang <12449837+YunchuWang@users.noreply.github.com>

* Clarify README documentation about proto file update process

Co-authored-by: YunchuWang <12449837+YunchuWang@users.noreply.github.com>

* Run proto download script automatically during build

Co-authored-by: YunchuWang <12449837+YunchuWang@users.noreply.github.com>

* fix

* update readme

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: YunchuWang <12449837+YunchuWang@users.noreply.github.com>

Author: Copilot

.github/workflows/codeQL.yml

@@ -46,8 +46,6 @@ jobs:
         # queries: security-extended,security-and-quality
 
     - uses: actions/checkout@v3
-      with:
-        submodules: true
 
     # Autobuild attempts to build any compiled languages  (C/C++, C#, Go, or Java).
     # If this step fails, then you should remove it and run the build manually (see below)

.gitmodules

@@ -6,6 +6,4 @@ dist
 # don't lint nyc coverage output
 coverage
 # don't lint proto files and output
-proto
-# don't lint submodules
-submodules
+proto

.prettierignore

@@ -1,3 +1,9 @@
+## Upcoming
+
+### New
+
+- Add automatic proto file download and commit hash tracking during build ([#XX](https://github.com/microsoft/durabletask-js/pull/XX))
+
 ## v0.1.0-alpha.2
 
 ### New

CHANGELOG.md

@@ -145,30 +145,22 @@ See the [examples](./examples) directory for a list of sample orchestrations and
 
 ### Generating protobufs
 
-Protobuf definitions are stored in the [./submodules/durabletask-proto](./submodules/durabletask-proto) directory, which is a submodule. To update the submodule, run the following command from the project root:
+Protobuf definitions are downloaded from the [durabletask-protobuf](https://github.com/microsoft/durabletask-protobuf) repository. To download the latest proto files, run:
 
 ```sh
-git submodule update --init
+npm run download-proto
 ```
 
-Once the submodule is available, the corresponding source code can be regenerated using the following command from the project root:
+This will download the proto files to `internal/durabletask-protobuf/protos/`.
+
+Once the proto files are available, the corresponding TypeScript source code can be regenerated using the following command from the project root:
 
 ```sh
-npm install grpc_tools_node_protoc_ts --save-dev
-
-# generate js codes via grpc-tools
-grpc_tools_node_protoc \
---js_out=import_style=commonjs,binary:src/proto \
---grpc_out=grpc_js:src/proto \
---plugin=protoc-gen-grpc=`which grpc_tools_node_protoc_plugin` \
--I ./submodules/durabletask-protobuf/protos orchestrator_service.proto
-
-protoc \
---plugin=protoc-gen-ts=./node_modules/.bin/protoc-gen-ts \
---ts_out=grpc_js:src/proto \
--I ./submodules/durabletask-protobuf/protos orchestrator_service.proto
+./tools/generate-grpc-javascript.sh ./src/proto
 ```
 
+Note: You need `grpc-tools` installed globally (`npm install -g grpc-tools`).
+
 ### Running unit tests
 
 Unit tests can be run using the following command from the project root. Unit tests _don't_ require a sidecar process to be running.

README.md

@@ -10,7 +10,6 @@ pool:
 
 steps:
 - checkout: self
-  submodules: true
 
 - task: NodeTool@0
   inputs:

azure-pipelines.yml

@@ -11,7 +11,6 @@ jobs:
                 sbomBuildComponentPath: "$(Build.SourcesDirectory)/node_modules"
       steps:
           - checkout: self
-            submodules: true
           - task: NodeTool@0
             inputs:
                 versionSpec: 20.x

eng/templates/build.yml

@@ -0,0 +1 @@
+026329c53fe6363985655857b9ca848ec7238bd2

internal/durabletask-protobuf/PROTO_SOURCE_COMMIT_HASH

@@ -0,0 +1,25 @@
+# Durable Task Protobuf Files
+
+This directory contains the protocol buffer definitions used by the Durable Task Framework JavaScript SDK. The files in this directory are automatically downloaded and updated during the build process from the [microsoft/durabletask-protobuf](https://github.com/microsoft/durabletask-protobuf) repository.
+
+## Directory Structure
+
+- `protos/` - Contains the proto files
+- `PROTO_SOURCE_COMMIT_HASH` - Contains the commit hash of the proto file version
+
+## Auto-Update Process
+
+The proto files are automatically downloaded and updated when running `npm run build`. This is handled by the `scripts/download-proto.sh` script. The script:
+
+1. Downloads the latest version of `orchestrator_service.proto`
+2. Saves the current commit hash for tracking purposes
+
+## Manual Update
+
+To manually update the proto files, you can run:
+
+```bash
+npm run download-proto
+# or
+./scripts/download-proto.sh [branch-name]
+```