Use async function across tests (#190)

* remove unused code

Signed-off-by: kerthcet <kerthcet@gmail.com>

* use async function across tests

Signed-off-by: kerthcet <kerthcet@gmail.com>

* update tests

Signed-off-by: kerthcet <kerthcet@gmail.com>

* fix lint

Signed-off-by: kerthcet <kerthcet@gmail.com>

* update test

Signed-off-by: kerthcet <kerthcet@gmail.com>

* Add comments

Signed-off-by: kerthcet <kerthcet@gmail.com>

* update readme.md

* fix lint

Signed-off-by: kerthcet <kerthcet@gmail.com>

* release v0.2.0

Signed-off-by: kerthcet <kerthcet@gmail.com>

* fix test

Signed-off-by: kerthcet <kerthcet@gmail.com>

---------

Signed-off-by: kerthcet <kerthcet@gmail.com>

Author: kerthcet

README.md

@@ -6,123 +6,121 @@
 </p>
 
 <h3 align="center">
-Open, modular framework to build GenAI applications.
+Open, modular framework to build and optimize GenAI applications
 </h3>
 
 [![stability-alpha](https://img.shields.io/badge/stability-alpha-f4d03f.svg)](https://github.com/mkenney/software-guides/blob/master/STABILITY-BADGES.md#alpha)
 [![Latest Release](https://img.shields.io/github/v/release/inftyai/alphatrion?include_prereleases)](https://github.com/inftyai/alphatrion/releases/latest)
 
-**AlphaTrion** is an open-source framework to help build GenAI applications, including experiment tracking, adaptive model routing, prompt optimization, performance evaluation and so on. The name comes after the oldest and wisest Transformer - AlphaTrion.
+**AlphaTrion** is an open-source framework for building and optimizing GenAI applications. Track experiments, monitor performance, analyze model usage, and manage artifacts—all through an intuitive dashboard. Named after the oldest and wisest Transformer.
 
-*Still under active development.*
+*Currently in active development.*
 
-## Concepts
+## Features
 
-- **Team**: A Team is the highest-level organizational unit in AlphaTrion. It represents a group of users collaborating on experiments.
-- **Experiment**: An Experiment is a logic-level abstraction for organizing and managing a series of related runs. It allows users to group runs that share a common purpose or configuration. Experiments can be organized using labels.
-- **Run**: A Run is a real execution instance of an experiment. It represents the actual execution of the code with the specified configuration and hyperparameters defined in the experiment.
+- **🔬 Experiment Tracking** - Organize and manage ML experiments with hierarchical teams, experiments, and runs
+- **📊 Performance Monitoring** - Track metrics, visualize trends, and monitor experiment status in real-time
+- **🔍 Distributed Tracing** - Automatic OpenTelemetry integration for LLM calls with detailed span analysis
+- **💰 Token Usage Analytics** - Monitor daily token consumption across input/output with historical trends
+- **🤖 Model Distribution** - Analyze request patterns and usage across different AI models
+- **📦 Artifact Management** - Store and version execution results, checkpoints, and model outputs
+- **🎯 Interactive Dashboard** - Modern web UI for exploring experiments, metrics, and traces
+- **🔌 Easy Integration** - Simple Python API with async/await support
+
+## Core Concepts
+
+- **Team** - Top-level organizational unit for user collaboration
+- **Experiment** - Logical grouping of runs with shared purpose, organized by labels
+- **Run** - Individual execution instance with configuration and metrics
 
 ## Quick Start
 
-### Install from PyPI
+### 1. Installation
 
 ```bash
+# From PyPI
 pip install alphatrion
-```
-
-### Install from Source
-
-* Git clone the repository
-* Run `source start.sh` to activate the virtual environment.
-
 
-### Initialize the Environment
-
-Run the following command for setup:
-
-```bash
-cp .env.example .env & make up
+# Or from source
+git clone https://github.com/inftyai/alphatrion.git && cd alphatrion
+source start.sh
 ```
-You can login to pgAdmin at `http://localhost:8081` to see the Postgres database with following credentials. Remember to register the server first.
-
-```shell
-Email:       alphatrion@inftyai.com
-Password:    alphatr1on
-ServerName:  alphatrion
-HostName:    postgres
-ServerPWD:   alphatr1on
-```
-
-You can also visit the Docker Registry UI at `http://localhost:80` to see the local registry where the built images are stored.
 
-Next, init the environment with a user and team:
+### 2. Setup Infrastructure
 
 ```bash
-alphatrion init  # see -h for options to specify username, email and team name
+# Start PostgreSQL, ClickHouse, and Registry
+cp .env.example .env
+make up
+
+# Initialize your team and user
+alphatrion init  # Use -h for custom options
 ```
 
-You will see the generated user ID in the console. Use this ID to initialize the AlphaTrion environment in your code later.
+Save the generated user ID—you'll need it to track experiments.
 
-### Run a Simple Experiment
+**Optional Tools:**
+- pgAdmin: `http://localhost:8081` (alphatrion@inftyai.com / alphatr1on)
+- Registry UI: `http://localhost:80`
 
-Below is a simple example with two approaches demonstrating how to create an experiment and log performance metrics.
+### 3. Track Your First Experiment
 
 ```python
 import alphatrion as alpha
 from alphatrion import experiment
 
-# Use the user ID generated from the `alphatrion init` command.
-alpha.init(user_id=<user_id>)
+# Initialize with your user ID
+alpha.init(user_id="<your_user_id>")
 
-async def your_task():
-  # Run your code here then log metrics.
-  await alpha.log_metrics({"accuracy": 0.95})
+async def my_task():
+    # Your ML code here
+    await alpha.log_metrics({"accuracy": 0.95, "loss": 0.12})
 
 async with experiment.CraftExperiment.start(name="my_experiment") as exp:
-  task = exp.run(your_task) # use lambda or partial if you need to pass arguments to your_task
-  await task.wait()
+    task = exp.run(my_task)
+    await task.wait()
 ```
 
-### View Dashboard
-
-![dashboard](./site/images/dashboard.png)
-
-The dashboard provides a web interface to explore experiments, runs, and metrics through an intuitive UI.
-
-#### Launch Dashboard
+### 4. Launch Dashboard
 
 ```bash
-# Start the backend server (in one terminal)
+# Start backend server (terminal 1)
 alphatrion server
 
-# Launch the dashboard (in another terminal)
+# Launch dashboard (terminal 2)
 alphatrion dashboard
 ```
 
-The dashboard will automatically open in your browser at `http://127.0.0.1:5173`.
-
-**Options:**
-- `--port <PORT>`: Run on a custom port (default: 5173)
+Access the dashboard at `http://127.0.0.1:5173` to explore experiments, visualize metrics, and analyze traces.
 
-**Documentation:**
-- [Dashboard Setup Guide](./docs/dashboard/setup.md) - Complete setup and troubleshooting guide
-- [Dashboard CLI Guide](./docs/dashboard/dashboard-cli.md) - Using the dashboard CLI command
-- [Dashboard Architecture](./docs/dashboard/dashboard-architecture.md) - Technical architecture and deployment patterns
+![dashboard](./site/images/dashboard.png)
 
-### Tracing
+### 5. View Traces
 
-AlphaTrion automatically captures tracing data for all runs, including spans for each run and associated metadata. You can query this data to analyze model performance, latency, and token usage.
+AlphaTrion automatically captures distributed tracing data for all LLM calls, including latency, token usage, and span relationships.
 
 ![tracing](./site/images/trace.png)
 
+### 6. Other APIs
+
+- **log_params**: Track hyperparameters and configuration settings
+- **log_metrics**: Record performance metrics and visualize trends
+- **log_artifacts**: Store and manage files, checkpoints, and model outputs
+
+
 ### Cleanup
 
 ```bash
 make down
 ```
 
+## Documentation
+
+- **Dashboard**: [Setup Guide](./docs/dashboard/setup.md) | [CLI Reference](./docs/dashboard/dashboard-cli.md) | [Architecture](./docs/dashboard/dashboard-architecture.md)
+- **Development**: [Contributing Guide](./docs/dev/development.md)
+
 ## Contributing
 
-We welcome contributions! Please refer to [developer.md](./docs/dev/development.md) for more information on how to set up your development environment and contribute to the project.
+We welcome contributions! Check out our [development guide](./docs/dev/development.md) to get started.
 
 [![Star History Chart](https://api.star-history.com/svg?repos=inftyai/alphatrion&type=Date)](https://www.star-history.com/#inftyai/alphatrion&Date)

alphatrion/experiment/base.py

@@ -199,6 +199,8 @@ async def __aexit__(self, exc_type, exc_val, exc_tb):
             current_exp_id.reset(self._token)
             self._runtime.current_experiment = None
 
+        self._cleanup_files()
+
     def _start(
         self,
         name: str,
@@ -366,13 +368,8 @@ async def wait(self):
     def is_done(self) -> bool:
         return self._context.cancelled()
 
-    # done function should be called manually as a pair of start
-    # FIXME: watch for system signals to cancel the Experiment gracefully,
-    # or it could lead to experiment not being marked as completed.
-    # TODO: Should we distinguish done and cancel?
     def done(self):
         self._cancel()
-        self._cleanup()
 
     def done_with_err(self):
         self._end_status = "Err"
@@ -462,7 +459,7 @@ def start(
     ) -> "Experiment":
         raise NotImplementedError
 
-    def _cleanup(self):
+    def _cleanup_files(self):
         # remove the whole folder once the experiment is done.
         if (
             os.path.exists(team_path())

alphatrion/log/log.py

@@ -30,6 +30,7 @@ async def log_artifact(
         If a folder is provided, all files in the folder will be logged.
         Don't support nested folders currently, only files in the first level
         of the folder will be logged.
+    :param repo_name: the name of the repository to log the artifact to.
     :param version: the version (tag) to log the files
     :param pre_save_hook: a callable function to be called before saving the artifact.
            If want to save something, make sure it's under the paths.
@@ -63,9 +64,13 @@ async def log_artifact(
     )
 
 
-# log_params is used to save a set of parameters, which is a dict of key-value pairs.
-# should be called after starting a Experiment.
 async def log_params(params: dict):
+    """
+    Log parameters to the database.
+    Support in Experiment level currently, should be called after starting a Experiment.
+
+    :param params: a dict of key-value pairs to log as parameters.
+    """
     exp_id = current_exp_id.get()
     if exp_id is None:
         raise RuntimeError("log_params must be called inside a Experiment.")
@@ -78,14 +83,14 @@ async def log_params(params: dict):
     )
 
 
-# log_metrics is used to log a set of metrics at once,
-# metric key must be string, value must be float.
-# If save_on_best is enabled in the experiment config, and the metric is the best metric
-# so far, the experiment will checkpoint the current data.
-#
-# Note: log_metrics can only be called inside a Run, because it needs a run_id.
-# Return bool indicates whether the metric is the best metric.
 async def log_metrics(metrics: dict[str, float]) -> bool:
+    """
+    Log metrics to the database.
+    Support in Run level currently, should be called after starting a Run.
+
+    :param metrics: a dict of key-value pairs to log as metrics.
+    :return: a bool indicating whether the metric is the best metric.
+    """
     run_id = current_run_id.get()
     if run_id is None:
         raise RuntimeError("log_metrics must be called inside a Run.")
@@ -150,6 +155,7 @@ async def log_metrics(metrics: dict[str, float]) -> bool:
 # including both input and output, e.g. you want to save the code snippet.
 # It will be stored in the object storage as a JSON file if object storage
 # is enabled or locally otherwise.
+# NOTE: will be deprecated in the v0.3.0, use log_dataset instead.
 async def log_result(
     output: dict[str, Any],
     input: dict[str, Any] | None = None,
@@ -196,3 +202,28 @@ async def log_result(
                 }
             },
         )
+
+
+# log_records is used to log a list of records, which is similar to log_metrics
+# but for tracing the execution of the code.
+# async def log_records():
+
+# log_dataset will store sometime in the artifacts als record in the database.
+# async def log_dataset(
+#     name: str,
+#     paths: str | list[str],
+#     version: str | None = None,
+# ):
+#     path = await log_artifact(
+#         paths=paths,
+#         repo_name="dataset",
+#         version=version,
+#     )
+
+#     runtime = global_runtime()
+#     runtime.metadb.create_dataset(
+#         name=name,
+#         team_id=runtime._team_id,
+#         path=path,
+#         version=version,
+#     )

alphatrion/server/graphql/types.py

@@ -69,12 +69,6 @@ def exps_by_timeframe(
             end_time=end_time,
         )
 
-    # @strawberry.field
-    # def labels(self) -> list["Label"]:
-    #     from .resolvers import GraphQLResolvers
-
-    #     return GraphQLResolvers.list_labels_by_team_id(team_id=self.id)
-
 
 @strawberry.type
 class User:

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "alphatrion"
-version = "0.1.0"
+version = "0.2.0"
 description = "⚒️ AlphaTrion is an open-source framework to help build GenAI applications, including experiment tracking, adaptive model routing, prompt optimization and performance evaluation."
 license = {text = "Apache-2.0"}
 readme = "README.md"