diff --git a/docs/contribution-guidelines/_index.md b/docs/contribution-guidelines/_index.md index 72017aa635a..be4a7eaad10 100644 --- a/docs/contribution-guidelines/_index.md +++ b/docs/contribution-guidelines/_index.md @@ -86,16 +86,15 @@ Avoid including: ``` 3. Execute tests: ```bash - cd core sbt test ``` -> For IntelliJ users: ensure the working directory matches the module (`amber` for engine tests, `core` for services). +> For IntelliJ users: ensure the working directory matches the module (`amber` for engine tests, the repo root for services). ### Frontend (Angular) 1. Run unit tests: ```bash - cd core/gui + cd frontend ng test --watch=false ``` 2. Format code: diff --git a/docs/contribution-guidelines/guide-for-developers.md b/docs/contribution-guidelines/guide-for-developers.md index 02e6a924234..ac73828e28d 100644 --- a/docs/contribution-guidelines/guide-for-developers.md +++ b/docs/contribution-guidelines/guide-for-developers.md @@ -263,7 +263,7 @@ This command will optimize the frontend code to make it run faster. This step wi ## 3. Email Notification (Optional) -1. Set `smtp` in `config/src/main/resources/user-system.conf`. You need an App password if the account has 2FA. +1. Set `smtp` in `common/config/src/main/resources/user-system.conf`. You need an App password if the account has 2FA. 2. Log in to Texera with an admin account. 3. Open the Gmail dashboard under the admin tab. 5. Send a test email. @@ -286,10 +286,10 @@ This part is optional; you only need to do this if you are working on a specific Note: Jooq creates DAO for simple operations if the requested SQL query is complex, then the developer can use the generated Table classes to implement the operation ### Disable password login -Edit `config/src/main/resources/gui.conf`, change `local-login` to `false`. +Edit `common/config/src/main/resources/gui.conf`, change `local-login` to `false`. ### Enforce invite only -Edit `config/src/main/resources/user-system.conf`, change `invite-only` to `true`. +Edit `common/config/src/main/resources/user-system.conf`, change `invite-only` to `true`. ### Backend endpoints Role Annotation There are two types of permissions for the backend endpoints: diff --git a/docs/contribution-guidelines/guide-to-implement-java-operator.md b/docs/contribution-guidelines/guide-to-implement-java-operator.md index 15a4dc8c578..34ac4d00861 100644 --- a/docs/contribution-guidelines/guide-to-implement-java-operator.md +++ b/docs/contribution-guidelines/guide-to-implement-java-operator.md @@ -8,7 +8,7 @@ In this page, we'll explain the basic concepts in Texera and use examples to sho ### Code structure of every operator: -Every operator ideally has three classes that are found in each operator package in `core\workflow-operator\src\main\scala\edu\uci\ics\amber\operator` +Every operator ideally has three classes that are found in each operator package in `common/workflow-operator/src/main/scala/org/apache/texera/amber/operator` * LogicalOp * OperatorExecutor * OperatorExecutorConfig @@ -49,7 +49,7 @@ schema.getAttributes().get(1) // Attribute("tweet", AttributeType.String) A regular expression operator matches a regular expression (regex) on each input tuple. For example, if we search the regex "weather" on the `tweet` attribute, then only tuple 2 will be the result. In other words, the regular expression operator is a kind of `filter()` operation in many programming languages. -To implement a regular expression operator, you will first need to write an `LogicalOp`. The following code is part of class [`RegexOpDesc`](https://github.com/apache/texera/blob/main/core/workflow-operator/src/main/scala/edu/uci/ics/amber/operator/regex/RegexOpDesc.scala) . +To implement a regular expression operator, you will first need to write an `LogicalOp`. The following code is part of class [`RegexOpDesc`](https://github.com/apache/texera/blob/main/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/regex/RegexOpDesc.scala) . ```scala class RegexOpDesc extends FilterOpDesc { @@ -127,7 +127,7 @@ abstract class LogicalOp extends PortDescriptor with Serializable { Now this operator will be automatically available in the frontend. We can now start the system and test this operator. -To add an image for this operator, go to `core/gui/src/assets/operator_images`, then add an image with the _**SAME NAME**_ as what's specified in the operator registration. The image file should be in `png` format, with a transparent background, black and white, and should be square. +To add an image for this operator, go to `frontend/src/assets/operator_images`, then add an image with the _**SAME NAME**_ as what's specified in the operator registration. The image file should be in `png` format, with a transparent background, black and white, and should be square. For example, for the regex operator, the code `new Type(value = classOf[RegexOpDesc], name = "Regex")` specified a name `Regex`, then the image file name should be `Regex.png`. @@ -146,7 +146,7 @@ id tweet sentiment ``` -The following code is the implementation of class [`SentimentAnalysisOpDesc`](https://github.com/apache/texera/blob/main/core/workflow-operator/src/main/scala/edu/uci/ics/amber/operator/huggingFace/HuggingFaceSentimentAnalysisOpDesc.scala) in Java. +The following code is the implementation of class [`SentimentAnalysisOpDesc`](https://github.com/apache/texera/blob/main/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/huggingFace/HuggingFaceSentimentAnalysisOpDesc.scala) in Java. ```java public class SentimentAnalysisOpDesc extends MapOpDesc { @@ -202,7 +202,7 @@ In Texera, currently we have 4 pre-defined operations you can extend. - `flatmap()`: for each input tuple, transforms it to a list of output tuples. - `aggregate()`: performs an aggregation, such as sum, count, average, etc. -To implement an operator, you can first check if your operator can be implemented using the 4 pre-defined operations. You can find these pre-defined operations under [`texera/workflow/common/operators`](https://github.com/Texera/texera/tree/master/core/amber/src/main/scala/edu/uci/ics/texera/workflow/common/operators). Your own operator implementation should be in [`texera/workflow/operators/youroperator`](https://github.com/Texera/texera/tree/master/core/amber/src/main/scala/edu/uci/ics/texera/workflow/operators). +To implement an operator, you can first check if your operator can be implemented using the 4 pre-defined operations. You can find these pre-defined operations under [`common/workflow-operator`](https://github.com/apache/texera/tree/main/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator) (e.g., the `map`, `filter`, `flatmap`, and `aggregate` packages). Your own operator implementation should be in its own package under [the same directory](https://github.com/apache/texera/tree/main/common/workflow-operator/src/main/scala/org/apache/texera/amber/operator). ### Low-level OperatorExecutor API For more complicated operators, if they cannot be implemented using these operations, then you need to implement `OperatorExecutor` using the following low-level interface. @@ -278,6 +278,6 @@ Texera's backend is responsible for determining the UI information to the fronte ``` ### Registration and icon -In the file `amber/src/main/scala/edu/uci/ics/texera/workflow/common/operators/LogicalOp.scala`, you will find a list of all registered operators, complete with their descriptor classes and names. After adding an operator's information, you can assign an icon to it. All operator icons are stored in the `/core/new-gui/src/assets/operator_images` directory. It's essential to ensure that the icon filename matches its respective operator descriptor name. +In the file `common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/LogicalOp.scala`, you will find a list of all registered operators, complete with their descriptor classes and names. After adding an operator's information, you can assign an icon to it. All operator icons are stored in the `frontend/src/assets/operator_images` directory. It's essential to ensure that the icon filename matches its respective operator descriptor name. diff --git a/docs/contribution-guidelines/micro-services-local-dev.md b/docs/contribution-guidelines/micro-services-local-dev.md deleted file mode 100644 index 7cb76b43f1a..00000000000 --- a/docs/contribution-guidelines/micro-services-local-dev.md +++ /dev/null @@ -1,58 +0,0 @@ ---- -title: "Build, Run and Configure micro‐services in local development environment" -weight: 60 ---- - -This Document is aim to provide a instruction on how to setup the local development environment for developing and deploying the `core/micro-services`. - -## Prerequisite - -This document requires you to finish all the setup of Texera local development environment described in `https://github.com/Texera/texera/wiki`. - -## What is `micro-services`? - -`core/micro-services` is a sbt-managed project added by the PR https://github.com/Texera/texera/pull/2922. The ongoing code separation effort will gradually migrate all the services in `core/amber` to `core/micro-services`. - -## How to directly build and run the micro-services directly - -If you just want to run some services under `micro-services`, you can use some provided shell scripts. - -### `WorkflowCompilingService` - -```shell -cd texera/core - -# make sure to give scripts the execution permission -chmod +x scripts/build-workflow-compiling-service.sh -chmod +x scripts/workflow-compiling-service.sh - -# Build the WorkflowCompilingService -scripts/build-workflow-compiling-service.sh - -# Run the WorkflowCompilingService -scripts/workflow-compiling-service.sh -``` - -## How to set up the development environment - -As there are many sub sbt projects under `micro-services`, Intellij is the most suitable IDE for setting up the whole environment - -### Use Intellij (Most Recommended) - -1. Open the folder `texera/core/micro-services` through `Open Project` in Intellij -Screenshot 2024-11-19 at 6 00 08 PM - -Once you open it, Intellij will auto-detect the sbt setting and start to load the project. After loading you should see the sbt tab, which has the `micro-services` as the root project and several other services as the sub-projects: -Screenshot 2024-11-19 at 6 05 15 PM - - -2. Run `sbt clean compile` command in folder `core/micro-services`. This command will compile everything under `micro-services` and generate proto-specified codes. - - - - - - - - - diff --git a/docs/tutorials/guide-to-use-python-udf.md b/docs/tutorials/guide-to-use-python-udf.md index 6cd5e536a34..e9de246c8cf 100644 --- a/docs/tutorials/guide-to-use-python-udf.md +++ b/docs/tutorials/guide-to-use-python-udf.md @@ -137,7 +137,7 @@ class GenerateOperator(UDFSourceOperator): This `produce()` API returns an iterator of `TupleLike`, `TableLike`, or simply `None`. -See [Generator Operator](https://github.com/Texera/texera/blob/master/core/amber/src/main/python/pytexera/udf/examples/generator_operator.py) for an example of 1-out UDF. +See [Generator Operator](https://github.com/apache/texera/blob/main/amber/src/main/python/pytexera/udf/examples/generator_operator_integer.py) for an example of 1-out UDF. #### 2-in UDF