docs: document how to build a dev environment with poetry and tox

Signed-off-by: Vassilis Vassiliadis <vassilis.vassiliadis@ibm.com>

Author: VassilisVassiliadis

CONTRIBUTING.md

@@ -82,12 +82,31 @@ The following tools are required:
 - [git](https://git-scm.com)
 - [python](https://www.python.org) (v3.8+)
 - [pip](https://pypi.org/project/pip/) (v23.0+)
+- [poetry](https://python-poetry.org/docs/#installation) (v1.8.3+)
+  - Poetry should always be installed in a dedicated virtual environment to isolate it from the rest of your system. It should in no case be installed in the environment of the project that is to be managed by Poetry. This ensures that Poetry’s own dependencies will not be accidentally upgraded or uninstalled.
+- [tox](https://tox.wiki/en/4.15.1/installation.html) (v4.15.1+)
+  - Just like `poetry` install `tox` in an isolated virtual environment
 
 Installation:
-``` 
-pip install -U datasets
-pip install -e .
+
+```bash 
+: Install poetry and tox in an isolated virtual environment
+python3 -m venv isolated
+./isolated/bin/pip install -U pip setuptools
+./isolated/bin/pip install poetry tox
+
+: Ensure you can access poetry and tox without activating the
+: the isolated virtual environment
+export PATH=$PATH:`pwd`/isolated/bin
+
+: Create your development virtual environment
+python3 -m venv venv
+. venv/bin/activate
+
+: Install a dev version (similar to pip -e ".[dev]") of fms-hf-tuning
+poetry install --with dev
 ```
+
 <details>
 <summary>Linting</summary>
 

README.md

@@ -7,11 +7,6 @@ This repo provides basic tuning scripts with support for specific models. The re
 
 ## Installation
 
-
-### Most recent versions of python dependencies
-
-If you want to use the most recent versions of python dependencies then use `pip`.
-
 ```
 pip install -e .
 ```
@@ -34,37 +29,6 @@ pip install -e ".[fms-accel]"
 ```
 `fms-acceleration` is a collection of plugins that packages that accelerate fine-tuning / training of large models, as part of the `fms-hf-tuning` suite. For more details on see [this section below](#fms-acceleration).
 
-
-### Using versions of python dependencies that are known to be compatible with fms-hf-tuning
-
-This repository uses [`poetry`](https://python-poetry.org/) for reproducible builds. You can use poetry to create the same virtual environment that we use for our CI/CD tests like so:
-
-```
-python -m venv venv
-. ./venv/bin/activate
-pip install poetry
-
-poetry install
-```
-
-> Note: After installing, if you wish to use [FlashAttention](https://github.com/Dao-AILab/flash-attention), then you need to install these requirements:
- 
-```
-poetry install --with dev,flash-attn
-```
-
-If you wish to use [aim](https://github.com/aimhubio/aim), then you need to install it:
-```
-poetry install --with aim
-```
-
-If you wish to use [fms-acceleration](https://github.com/foundation-model-stack/fms-acceleration), you need to install it. 
-```
-poetry install --with fms-accel
-```
-`fms-acceleration` is a collection of plugins that packages that accelerate fine-tuning / training of large models, as part of the `fms-hf-tuning` suite. For more details on see [this section below](#fms-acceleration).
-
-
 ## Data format
 We support two data formats:
 

build/README.md

@@ -1,6 +1,6 @@
 # Building fms-hf-tuning as an Image
 
-The Dockerfile provides a way of running fms-hf-tuning SFT Trainer. It installs the dependencies needed and adds two additional scripts that helps to parse arguments to pass to SFT Trainer. The accelerate_launch.py script is run by default when running the image to trigger SFT trainer for single or multi GPU by parsing arguments and running `accelerate launch launch_training.py`. 
+The Dockerfile provides a way of running fms-hf-tuning SFT Trainer. It installs the dependencies needed and adds two additional scripts that helps to parse arguments to pass to SFT Trainer. The `accelerate_launch.py` script is run by default when running the image to trigger SFT trainer for single or multi GPU by parsing arguments and running `accelerate launch launch_training.py`. 
 
 ## Configuration