Deploy PR #23 preview

Author: github-actions[bot]

pr-23/_sources/algorithms.rst.txt

@@ -167,7 +167,7 @@ Hyperparameter Tuning
 General Guidelines
 ~~~~~~~~~~~~~~~~~~
 
-1. **Start with defaults**: See ``twisterl/defaults.py`` for sensible default parameters
+1. **Start with defaults**: See ``src/twisterl/defaults.py`` for sensible default parameters
 2. **Adjust learning rate first**: This usually has the biggest impact
 3. **Monitor training curves**: Use TensorBoard to track progress (logs saved to ``runs/`` by default)
 

pr-23/_sources/api/environments.rst.txt

@@ -66,12 +66,19 @@ The ``PyEnv`` class wraps Python environments for use with TwisteRL's Rust train
        }
    }
 
-The Python environment class should implement:
-
-- ``reset() -> observation``
-- ``step(action) -> (observation, reward, done, info)``
-- ``obs_shape() -> list[int]``
-- ``num_actions() -> int``
+The Python environment class must implement:
+
+- ``reset(difficulty: int)``: Reset the environment to initial state with given difficulty
+- ``next(action: int)``: Execute an action (advances the environment state)
+- ``observe() -> list[int]``: Return the current observation
+- ``obs_shape() -> list[int]``: Return observation dimensions
+- ``num_actions() -> int``: Return number of valid actions
+- ``is_final() -> bool``: Return True if current state is terminal
+- ``success() -> bool``: Return True if the goal was achieved
+- ``value() -> float``: Return the reward value for current state
+- ``masks() -> list[bool]``: Return action mask (True if action is valid)
+- ``set_state(state: list[int])``: Set environment to specific state
+- ``copy()``: Return a copy of the environment (for parallel collection)
 - ``twists() -> (obs_perms, act_perms)`` (optional, for symmetry-aware training)
 
 Creating Custom Environments
@@ -87,25 +94,31 @@ For best performance, implement environments in Rust. See the ``examples/grid_wo
 
 See :doc:`../examples` for detailed instructions.
 
-Environment Interface
----------------------
+Environment Interface (Rust Trait)
+-----------------------------------
+
+Rust environments implement the ``twisterl::rl::env::Env`` trait. The required methods are:
+
+- ``num_actions() -> usize``: Return number of possible actions
+- ``obs_shape() -> Vec<usize>``: Return observation dimensions
+- ``set_state(state: Vec<i64>)``: Set environment to a specific state
+- ``reset()``: Reset to a random initial state
+- ``step(action: usize)``: Execute an action (evolve the state)
+- ``is_final() -> bool``: Return True if current state is terminal
+- ``success() -> bool``: Return True if the goal was achieved
+- ``reward() -> f32``: Return the reward value for current state
+- ``observe() -> Vec<usize>``: Return current state as sparse observation
 
-All environments must provide these methods (called from Rust or Python):
+Optional methods with default implementations:
 
-- ``reset()``: Reset to initial state, return observation
-- ``step(action)``: Take action, return (obs, reward, done, info)
-- ``obs_shape()``: Return observation dimensions
-- ``num_actions()``: Return number of valid actions
-- ``is_final()``: Return True if current state is terminal
-- ``success()``: Return True if the goal was achieved (episode ended successfully)
-- ``reward()``: Return the reward value for the current state
-- ``twists()``: Return permutation symmetries (optional)
-- ``set_state(state)``: Set environment to specific state (for inference)
-- ``difficulty``: Property to get/set difficulty level
+- ``set_difficulty(difficulty: usize)``: Set difficulty level (default: no-op)
+- ``get_difficulty() -> usize``: Get current difficulty (default: 1)
+- ``masks() -> Vec<bool>``: Return action mask (default: all True)
+- ``twists() -> (Vec<Vec<usize>>, Vec<Vec<usize>>)``: Return permutation symmetries (default: empty)
 
 Permutation Symmetries (Twists)
 -------------------------------
 
 TwisteRL supports symmetry-aware training through "twists" - permutations of observations and actions that represent equivalent states.
 
-See ``docs/twists.md`` for detailed documentation on implementing twists in your environments.
+See `twists.md <https://github.com/AI4quantum/twisteRL/blob/main/docs/twists.md>`_ for detailed documentation on implementing twists in your environments.

pr-23/_sources/docs-guide.rst.txt

@@ -368,8 +368,6 @@ Troubleshooting Deployment
 - Click "Run workflow"
 - Choose the branch and click "Run workflow"
 
-The documentation will be available at http://localhost:8000
-
 Custom Domain (Optional)
 ------------------------
 

pr-23/_sources/examples.rst.txt

@@ -76,8 +76,8 @@ TwisteRL supports custom environments implemented in Rust. The ``examples/grid_w
    crate-type = ["cdylib"]
 
    [dependencies]
-   pyo3 = { version = "0.20", features = ["extension-module"] }
-   twisterl = { path = "path/to/twisterl/rust", features = ["python_bindings"] }
+   pyo3 = { version = "0.24", features = ["extension-module"] }
+   twisterl = { path = "../../rust", features = ["python_bindings"] }
 
 3. **Implement the environment** by implementing the ``twisterl::rl::env::Env`` trait.
 
@@ -86,20 +86,27 @@ TwisteRL supports custom environments implemented in Rust. The ``examples/grid_w
 .. code-block:: rust
 
    use pyo3::prelude::*;
+   use twisterl::rl::env::Env;
    use twisterl::python_interface::env::PyBaseEnv;
 
-   #[pyclass(name = "MyEnv", extends = PyBaseEnv)]
+   #[pyclass(name="MyEnv", extends=PyBaseEnv)]
    struct PyMyEnv;
 
    #[pymethods]
    impl PyMyEnv {
        #[new]
-       fn new(...) -> (Self, PyBaseEnv) {
-           let env = MyEnv::new(...);
+       fn new(/* your params */) -> (Self, PyBaseEnv) {
+           let env = MyEnv::new(/* ... */);
            (PyMyEnv, PyBaseEnv { env: Box::new(env) })
        }
    }
 
+   #[pymodule]
+   fn my_env(_py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> {
+       m.add_class::<PyMyEnv>()?;
+       Ok(())
+   }
+
 5. **Build and install** the module:
 
 .. code-block:: bash
@@ -124,7 +131,9 @@ TwisteRL also supports Python environments through the ``PyEnv`` wrapper:
        }
    }
 
-Note that Python environments may be slower than native Rust environments.
+Your Python environment class must implement the required interface (``reset``, ``next``, ``observe``, etc.). See :doc:`api/environments` for the complete list of required methods.
+
+Note that Python environments may be slower than native Rust environments due to the Python-Rust interop overhead.
 
 Use Cases
 ---------

pr-23/_sources/index.rst.txt

@@ -45,17 +45,19 @@ This example trains a model to play the popular "8 puzzle" where numbers have to
 
 This model can be trained on a single CPU in under 1 minute (no GPU required!).
 
-🏗️ Current State (PoC)
------------------------
+Current State (Proof of Concept)
+---------------------------------
 
-- Hybrid rust-python implementation:
+- Hybrid Rust-Python implementation:
     - Data collection and inference in Rust
     - Training in Python (PyTorch)
 - Supported algorithms:
     - PPO (Proximal Policy Optimization)
     - AlphaZero
 - Focus on discrete observation and action spaces
-- Support for native Rust environments and for Python environments through a wrapper
+- Support for native Rust environments and Python environments through a wrapper
+
+**Repository:** `GitHub <https://github.com/AI4quantum/twisteRL>`_
 
 Getting Started
 ---------------

pr-23/_sources/quickstart.rst.txt

@@ -56,7 +56,12 @@ The training configuration is specified in JSON format. Here's an example based
                "num_epochs": 10,
                "vf_coef": 0.8,
                "ent_coef": 0.01,
-               "clip_ratio": 0.1
+               "clip_ratio": 0.1,
+               "normalize_advantage": true
+           },
+           "learning": {
+               "diff_threshold": 0.85,
+               "diff_max": 32
            },
            "optimizer": {
                "lr": 0.00015

pr-23/_sources/twists.md.txt

@@ -11,10 +11,10 @@ lightweight form of regularization because the agent sees equivalent states unde
 - Every environment implements the `twisterl::rl::env::Env` trait. The trait includes a `twists`
   method that returns `(Vec<Vec<usize>>, Vec<Vec<usize>>)` representing valid permutations on the
   flattened observation array and matching permutations on the discrete action space
-  (`rust/src/rl/env.rs:33`).
+  (`rust/src/rl/env.rs:59`).
 - When an environment is instantiated from Python via `prepare_algorithm`, twisteRL immediately calls
   `env.twists()` and forwards the returned permutations to the policy constructor
-  (`src/twisterl/utils.py:126`). The policy can then symmetrize logits, average values, or augment
+  (`src/twisterl/utils.py:194`). The policy can then symmetrize logits, average values, or augment
   rollouts without extra environment queries.
 
 ## Data Contract

pr-23/algorithms.html

@@ -292,7 +292,7 @@ <h2>Hyperparameter Tuning<a class="headerlink" href="#hyperparameter-tuning" tit
 <section id="general-guidelines">
 <h3>General Guidelines<a class="headerlink" href="#general-guidelines" title="Link to this heading"></a></h3>
 <ol class="arabic simple">
-<li><p><strong>Start with defaults</strong>: See <code class="docutils literal notranslate"><span class="pre">twisterl/defaults.py</span></code> for sensible default parameters</p></li>
+<li><p><strong>Start with defaults</strong>: See <code class="docutils literal notranslate"><span class="pre">src/twisterl/defaults.py</span></code> for sensible default parameters</p></li>
 <li><p><strong>Adjust learning rate first</strong>: This usually has the biggest impact</p></li>
 <li><p><strong>Monitor training curves</strong>: Use TensorBoard to track progress (logs saved to <code class="docutils literal notranslate"><span class="pre">runs/</span></code> by default)</p></li>
 </ol>

pr-23/api/environments.html

@@ -62,7 +62,7 @@
 </li>
 <li class="toctree-l2"><a class="reference internal" href="#python-environment-wrapper">Python Environment Wrapper</a></li>
 <li class="toctree-l2"><a class="reference internal" href="#creating-custom-environments">Creating Custom Environments</a></li>
-<li class="toctree-l2"><a class="reference internal" href="#environment-interface">Environment Interface</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#environment-interface-rust-trait">Environment Interface (Rust Trait)</a></li>
 <li class="toctree-l2"><a class="reference internal" href="#permutation-symmetries-twists">Permutation Symmetries (Twists)</a></li>
 </ul>
 </li>
@@ -154,12 +154,19 @@ <h2>Python Environment Wrapper<a class="headerlink" href="#python-environment-wr
 <span class="p">}</span>
 </pre></div>
 </div>
-<p>The Python environment class should implement:</p>
+<p>The Python environment class must implement:</p>
 <ul class="simple">
-<li><p><code class="docutils literal notranslate"><span class="pre">reset()</span> <span class="pre">-&gt;</span> <span class="pre">observation</span></code></p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">step(action)</span> <span class="pre">-&gt;</span> <span class="pre">(observation,</span> <span class="pre">reward,</span> <span class="pre">done,</span> <span class="pre">info)</span></code></p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">obs_shape()</span> <span class="pre">-&gt;</span> <span class="pre">list[int]</span></code></p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">num_actions()</span> <span class="pre">-&gt;</span> <span class="pre">int</span></code></p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">reset(difficulty:</span> <span class="pre">int)</span></code>: Reset the environment to initial state with given difficulty</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">next(action:</span> <span class="pre">int)</span></code>: Execute an action (advances the environment state)</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">observe()</span> <span class="pre">-&gt;</span> <span class="pre">list[int]</span></code>: Return the current observation</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">obs_shape()</span> <span class="pre">-&gt;</span> <span class="pre">list[int]</span></code>: Return observation dimensions</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">num_actions()</span> <span class="pre">-&gt;</span> <span class="pre">int</span></code>: Return number of valid actions</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">is_final()</span> <span class="pre">-&gt;</span> <span class="pre">bool</span></code>: Return True if current state is terminal</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">success()</span> <span class="pre">-&gt;</span> <span class="pre">bool</span></code>: Return True if the goal was achieved</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">value()</span> <span class="pre">-&gt;</span> <span class="pre">float</span></code>: Return the reward value for current state</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">masks()</span> <span class="pre">-&gt;</span> <span class="pre">list[bool]</span></code>: Return action mask (True if action is valid)</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">set_state(state:</span> <span class="pre">list[int])</span></code>: Set environment to specific state</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">copy()</span></code>: Return a copy of the environment (for parallel collection)</p></li>
 <li><p><code class="docutils literal notranslate"><span class="pre">twists()</span> <span class="pre">-&gt;</span> <span class="pre">(obs_perms,</span> <span class="pre">act_perms)</span></code> (optional, for symmetry-aware training)</p></li>
 </ul>
 </section>
@@ -174,26 +181,32 @@ <h2>Creating Custom Environments<a class="headerlink" href="#creating-custom-env
 </ol>
 <p>See <a class="reference internal" href="../examples.html"><span class="doc">Examples</span></a> for detailed instructions.</p>
 </section>
-<section id="environment-interface">
-<h2>Environment Interface<a class="headerlink" href="#environment-interface" title="Link to this heading"></a></h2>
-<p>All environments must provide these methods (called from Rust or Python):</p>
+<section id="environment-interface-rust-trait">
+<h2>Environment Interface (Rust Trait)<a class="headerlink" href="#environment-interface-rust-trait" title="Link to this heading"></a></h2>
+<p>Rust environments implement the <code class="docutils literal notranslate"><span class="pre">twisterl::rl::env::Env</span></code> trait. The required methods are:</p>
 <ul class="simple">
-<li><p><code class="docutils literal notranslate"><span class="pre">reset()</span></code>: Reset to initial state, return observation</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">step(action)</span></code>: Take action, return (obs, reward, done, info)</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">obs_shape()</span></code>: Return observation dimensions</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">num_actions()</span></code>: Return number of valid actions</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">is_final()</span></code>: Return True if current state is terminal</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">success()</span></code>: Return True if the goal was achieved (episode ended successfully)</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">reward()</span></code>: Return the reward value for the current state</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">twists()</span></code>: Return permutation symmetries (optional)</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">set_state(state)</span></code>: Set environment to specific state (for inference)</p></li>
-<li><p><code class="docutils literal notranslate"><span class="pre">difficulty</span></code>: Property to get/set difficulty level</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">num_actions()</span> <span class="pre">-&gt;</span> <span class="pre">usize</span></code>: Return number of possible actions</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">obs_shape()</span> <span class="pre">-&gt;</span> <span class="pre">Vec&lt;usize&gt;</span></code>: Return observation dimensions</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">set_state(state:</span> <span class="pre">Vec&lt;i64&gt;)</span></code>: Set environment to a specific state</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">reset()</span></code>: Reset to a random initial state</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">step(action:</span> <span class="pre">usize)</span></code>: Execute an action (evolve the state)</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">is_final()</span> <span class="pre">-&gt;</span> <span class="pre">bool</span></code>: Return True if current state is terminal</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">success()</span> <span class="pre">-&gt;</span> <span class="pre">bool</span></code>: Return True if the goal was achieved</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">reward()</span> <span class="pre">-&gt;</span> <span class="pre">f32</span></code>: Return the reward value for current state</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">observe()</span> <span class="pre">-&gt;</span> <span class="pre">Vec&lt;usize&gt;</span></code>: Return current state as sparse observation</p></li>
+</ul>
+<p>Optional methods with default implementations:</p>
+<ul class="simple">
+<li><p><code class="docutils literal notranslate"><span class="pre">set_difficulty(difficulty:</span> <span class="pre">usize)</span></code>: Set difficulty level (default: no-op)</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">get_difficulty()</span> <span class="pre">-&gt;</span> <span class="pre">usize</span></code>: Get current difficulty (default: 1)</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">masks()</span> <span class="pre">-&gt;</span> <span class="pre">Vec&lt;bool&gt;</span></code>: Return action mask (default: all True)</p></li>
+<li><p><code class="docutils literal notranslate"><span class="pre">twists()</span> <span class="pre">-&gt;</span> <span class="pre">(Vec&lt;Vec&lt;usize&gt;&gt;,</span> <span class="pre">Vec&lt;Vec&lt;usize&gt;&gt;)</span></code>: Return permutation symmetries (default: empty)</p></li>
 </ul>
 </section>
 <section id="permutation-symmetries-twists">
 <h2>Permutation Symmetries (Twists)<a class="headerlink" href="#permutation-symmetries-twists" title="Link to this heading"></a></h2>
 <p>TwisteRL supports symmetry-aware training through “twists” - permutations of observations and actions that represent equivalent states.</p>
-<p>See <code class="docutils literal notranslate"><span class="pre">docs/twists.md</span></code> for detailed documentation on implementing twists in your environments.</p>
+<p>See <a class="reference external" href="https://github.com/AI4quantum/twisteRL/blob/main/docs/twists.md">twists.md</a> for detailed documentation on implementing twists in your environments.</p>
 </section>
 </section>
 

pr-23/docs-guide.html

@@ -458,7 +458,6 @@ <h2>Troubleshooting Deployment<a class="headerlink" href="#troubleshooting-deplo
 - Select “Build and Deploy Documentation”
 - Click “Run workflow”
 - Choose the branch and click “Run workflow”</p>
-<p>The documentation will be available at <a class="reference external" href="http://localhost:8000">http://localhost:8000</a></p>
 </section>
 <section id="custom-domain-optional">
 <h2>Custom Domain (Optional)<a class="headerlink" href="#custom-domain-optional" title="Link to this heading"></a></h2>