Add badges, Classifiers, CI/CD Testing, Improved TWST Collect, opensampl-server2 (#7)

This adds badges to the README.md for quick knowledge of the Python
software's current status, as well as classifiers to better enable
package discovery, and a CI/CD workflow for running pytest. 

Extends the Microchip TWST collect script to collect all data entries by
default and moves the filtering of which measurements should be sent to the db to 
the MicrochipTWST Probe Object. 

An additional entrypoint opensampl-server2 was added which more directly wraps 
docker compose while providing the correct compose file and env file. 
---------

Co-authored-by: Grant, Josh <grantjn@ornl.gov>
Co-authored-by: MacFarland, Midgie <macfarlandmj@ornl.gov>

Author: 3 people

.github/workflows/tests.yml

@@ -0,0 +1,26 @@
+name: Testing with Pytest
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+  workflow_dispatch:
+
+jobs:
+  uv-pytest:
+    name: python
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v6
+
+      - name: Install the project
+        run: uv sync --all-extras --dev
+
+      - name: Run tests
+        run: uv run pytest tests/

CHANGELOG.md

@@ -37,6 +37,24 @@ This project adheres to [Semantic Versioning](https://semver.org/).
 *Unreleased* versions radiate potential—-and dread. Once you merge an infernal PR, move its bullet under a new version heading with the actual release date.*
 
 -->
+
+## [1.1.2] - 2025-07-24
+### Added
+- 🔥 README badges 
+- 🧪 Added testing CI/CD
+- 🏛️ Added classifiers for easier discovery
+- 🔥 opensampl-server2 which passes everything directly to docker compose with correct compose and env
+- 🔥 opensampl-collect entry point for accessing the collection scripts
+
+### Changed
+- ⚡ MicrochipTWST collection script takes all data by default rather than specific readings
+- ⚡ MicrochipTWST collection script can take optional server and control ports 
+- ⚡ MicrochipTWST probe object measurement logic updated to reflect collection
+
+### Fixed
+- 🩹 Dataframe insertion of time data more durable against single row collisions
+
+
 ## [1.1.1] - 2025-07-09
 ### Fixed
 - 🩹 Added tabulate dependency to pyproject.toml

README.md

@@ -1,5 +1,25 @@
 # OpenSAMPL
 
+<div align="center">
+<!-- PyPI → version -->
+<a href="https://pypi.org/project/opensampl/"><img src="https://img.shields.io/pypi/v/opensampl?logo=pypi" alt="PyPI"></a>
+<!-- MIT licence -->
+<a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-lightgrey.svg" alt="MIT licence"></a>
+<!-- Supported Python versions (will show “missing” until you add the trove classifiers) -->
+<a href="https://pypi.org/project/opensampl/"><img src="https://img.shields.io/pypi/pyversions/opensampl?logo=python" alt="python versions"></a>
+<!-- Universal wheel? -->
+<a href="https://pypi.org/project/opensampl/"><img src="https://img.shields.io/pypi/wheel/opensampl" alt="wheel"></a>
+<!-- Monthly downloads -->
+<a href="https://pypistats.org/packages/opensampl"><img src="https://img.shields.io/pypi/dm/opensampl?label=downloads%20%28month%29" alt="downloads per month"></a>
+<!-- GitHub Actions CI -->
+<a href="https://github.com/ORNL/OpenSAMPL/actions/workflows/publish.yml"><img src="https://github.com/ORNL/OpenSAMPL/actions/workflows/publish.yml/badge.svg" alt="PyPi Publishing"></a>
+<a href="https://github.com/ORNL/OpenSAMPL/actions/workflows/lint.yml"><img src="https://github.com/ORNL/OpenSAMPL/actions/workflows/lint.yml/badge.svg" alt="ruff Formating and Linting"></a>
+<a href="https://github.com/ORNL/OpenSAMPL/actions/workflows/tests.yml"><img src="https://github.com/ORNL/OpenSAMPL/actions/workflows/tests.yml/badge.svg" alt="PyTest Testing"></a>
+<!-- Docs on GitHub Pages -->
+<a href="https://ornl.github.io/OpenSAMPL/"><img src="https://img.shields.io/website?url=https%3A%2F%2Fornl.github.io%2FOpenSAMPL%2F&label=docs&logo=github" alt="docs"></a>
+</div>
+
+
 OpenSAMPL was created to provide a set of Python tools for managing clock data in a TimescaleDB database, specifically designed for synchronization analytics and monitoring.
 This project came out of [**CAST**](https://cast.ornl.gov), the **C**enter for **A**lternative **S**yncrhonization and **T**iming, a research group at Oak Ridge National Laboratory (ORNL).
 The name OpenSAMPL stands for **O**pen **S**ynchronization **A**nalytics and **M**onitoring **PL**atform, and provides the code and logic for uploading, managing, and visualizing clock data from various sources, including ADVA probes and Microchip TWST data files,

opensampl/collect/cli.py

@@ -0,0 +1,53 @@
+"""Consolidated CLI entry point for opensampl.collect tools."""
+
+import click
+
+from opensampl.collect.microchip.twst.generate_twst_files import collect_files as collect_twst_files
+
+
+@click.group()
+def cli():
+    """OpenSAMPL data collection tools."""
+    pass
+
+
+@cli.group()
+def microchip():
+    """Microchip device collection tools."""
+    pass
+
+
+@microchip.command()
+@click.option("--ip", required=True, help="IP address of the modem")
+@click.option("--control-port", required=False, default=1700, help="Control port of the modem (default: 1700)")
+@click.option("--status-port", required=False, default=1900, help="Status port of the modem (default: 1900)")
+@click.option("--dump-interval", default=300, help="Duration between file dumps in seconds (default: 300 = 5 minutes)")
+@click.option(
+    "--total-duration", default=None, type=int, help="Total duration to run in seconds (default: run indefinitely)"
+)
+@click.option("--output-dir", default="./output", help="Output directory for CSV files (default: ./output)")
+def twst(ip: str, control_port: int, status_port: int, dump_interval: int, total_duration: int, output_dir: str):
+    """
+    Collect data from Microchip TWST modems.
+
+    This command connects to TWST modems via IP address to collect measurement data including
+    offset and EBNO tracking values, along with contextual information. Data is saved to
+    CSV files with YAML metadata headers for comprehensive data logging.
+
+    Examples:
+        opensampl-collect microchip twst --ip 192.168.1.100
+        opensampl-collect microchip twst --ip 192.168.1.100 --dump-interval 600 --total-duration 3600
+
+    """
+    collect_twst_files(
+        host=ip,
+        control_port=control_port,
+        status_port=status_port,
+        dump_interval=dump_interval,
+        total_duration=total_duration,
+        output_dir=output_dir,
+    )
+
+
+if __name__ == "__main__":
+    cli()

opensampl/collect/microchip/twst/context.py

@@ -25,18 +25,19 @@ class ModemContextReader(ModemReader):
     including local station details and remote station tracking data.
     """
 
-    def __init__(self, host: str, prompt: str = "ATS 6502>"):
+    def __init__(self, host: str, prompt: str = "ATS 6502>", port: int = 1700):
         """
         Initialize ModemContextReader.
 
         Args:
             host: IP address or hostname of the ATS6502 modem.
             prompt: Command prompt string expected from the modem.
+            port: what port to connect to for commands (default 1700).
 
         """
         self.result = SimpleNamespace()
         self.prompt = prompt
-        super().__init__(host=host, port=1700)
+        super().__init__(host=host, port=port)
 
     @staticmethod
     def finished_ok(line: str) -> bool:

opensampl/collect/microchip/twst/generate_twst_files.py

@@ -1,20 +1,22 @@
 """
-Microchip TWST modem data collection and CSV generation tool.
+Microchip TWST modem data collection and CSV generation library.
 
 This module provides functionality to collect measurements from Microchip TWST modems
-and save them to timestamped CSV files. It includes both programmatic and command-line
-interfaces for flexible data collection with configurable intervals and durations.
+and save them to timestamped CSV files. It provides a programmatic interface for
+flexible data collection with configurable intervals and durations.
 
 The tool connects to TWST modems via IP address to collect measurement data including
 offset and EBNO tracking values, along with contextual information. Data is saved to
 CSV files with YAML metadata headers for comprehensive data logging.
 
 Example:
-    Run from command line with required IP address:
-        $ python generate_twst_files.py --ip 192.168.1.100
+    Import and use programmatically:
+        from opensampl.collect.microchip.twst.generate_twst_files import collect_files
+        collect_files(host="192.168.1.100", dump_interval=600, total_duration=3600)
 
-    Run with custom intervals and duration:
-        $ python generate_twst_files.py --ip 192.168.1.100 --dump-interval 600 --total-duration 3600
+    Use via CLI (recommended):
+        $ opensampl-collect microchip twst --ip 192.168.1.100
+        $ opensampl-collect microchip twst --ip 192.168.1.100 --dump-interval 600 --total-duration 3600
 
 """
 
@@ -24,7 +26,7 @@
 from pathlib import Path
 from typing import Optional
 
-import click
+from loguru import logger
 
 from opensampl.collect.microchip.twst.context import ModemContextReader
 from opensampl.collect.microchip.twst.readings import ModemStatusReader
@@ -38,16 +40,32 @@ async def collect_data(status_reader: ModemStatusReader, context_reader: ModemCo
         status_reader: ModemStatusReader instance for collecting measurements.
         context_reader: ModemContextReader instance for collecting context data.
 
-    """
-    await asyncio.gather(status_reader.collect_readings(), context_reader.get_context())
-
+    Raises:
+        Exception: Re-raises any exception from the collection process.
 
-def collect_files(host: str, output_dir: str, dump_interval: int, total_duration: Optional[int] = None):
+    """
+    try:
+        await asyncio.gather(status_reader.collect_readings(), context_reader.get_context())
+    except Exception as e:
+        logger.error(f"Error during data collection: {e}")
+        raise
+
+
+def collect_files(
+    host: str,
+    control_port: int = 1700,
+    status_port: int = 1900,
+    output_dir: str = "./output",
+    dump_interval: int = 300,
+    total_duration: Optional[int] = None,
+):
     """
     Continuously collect blocks of modem measurements and save to timestamped CSV files.
 
     Args:
         host: IP address or hostname of the modem.
+        control_port: Control port for modem (default: 1700)
+        status_port: Status port for modem (default: 1900)
         output_dir: Directory path where CSV files will be saved.
         dump_interval: Duration in seconds between each data collection cycle.
         total_duration: Optional total runtime in seconds. If None, runs indefinitely.
@@ -61,66 +79,90 @@ def collect_files(host: str, output_dir: str, dump_interval: int, total_duration
     output_path.mkdir(parents=True, exist_ok=True)
 
     start_time = time.time()
+    consecutive_failures = 0
+    max_consecutive_failures = 5
+    base_retry_delay = 30  # seconds
+    max_retry_delay = 300  # 5 minutes
+
+    logger.info(f"Starting data collection from {host}, saving to {output_path}")
 
     while True:
         if total_duration and (time.time() - start_time) >= total_duration:
+            logger.info("Total duration reached, stopping collection")
             break
 
-        status_reader = ModemStatusReader(host=host, duration=dump_interval, keys=["meas:offset", "tracking:ebno"])
-        context_reader = ModemContextReader(host=host, prompt="TWModem-32>")
+        try:
+            status_reader = ModemStatusReader(host=host, duration=dump_interval, port=status_port)
+            context_reader = ModemContextReader(host=host, prompt="TWModem-32>", port=control_port)
+
+            logger.debug(f"Starting data collection cycle for {host}")
+            asyncio.run(collect_data(status_reader, context_reader))
+
+            # Write to CSV file
+            timestamp_str = context_reader.result.timestamp
+            output_file = output_path / f"{host}_6502-Modem_{timestamp_str}.csv"
+
+            with output_file.open("w", newline="") as f:
+                f.write(context_reader.get_result_as_yaml_comment())
+                f.write("\n")
+
+                writer = csv.writer(f)
+                writer.writerow(["timestamp", "reading", "value"])
+                writer.writerows(status_reader.readings)
+
+            logger.info(f"Wrote {len(status_reader.readings)} readings to {output_file}")
+
+            # Reset failure counter on successful collection
+            consecutive_failures = 0
 
-        asyncio.run(collect_data(status_reader, context_reader))
+        except Exception as e:
+            consecutive_failures += 1
+            logger.error(f"Collection failed (attempt {consecutive_failures}): {e}")
 
-        # Write to CSV file
-        timestamp_str = context_reader.result.timestamp
-        output_file = output_path / f"{host}_6502-Modem_{timestamp_str}.csv"
+            if consecutive_failures >= max_consecutive_failures:
+                logger.critical(
+                    f"Maximum consecutive failures ({max_consecutive_failures}) reached. Stopping collection."
+                )
+                break
 
-        with output_file.open("w", newline="") as f:
-            f.write(context_reader.get_result_as_yaml_comment())
-            f.write("\n")
+            # Calculate exponential backoff delay
+            retry_delay = min(base_retry_delay * (2 ** (consecutive_failures - 1)), max_retry_delay)
+            logger.warning(f"Retrying in {retry_delay} seconds...")
 
-            writer = csv.writer(f)
-            writer.writerow(["timestamp", "reading", "value"])
-            writer.writerows(status_reader.readings)
+            try:
+                time.sleep(retry_delay)
+            except KeyboardInterrupt:
+                logger.info("Keyboard interrupt received, stopping collection")
+                break
 
 
-def main(ip_address: str, dump_interval: int, total_duration: int, output_dir: str):
+def main(
+    ip_address: str, control_port: int, status_port: int, dump_interval: int, total_duration: int, output_dir: str
+):
     """
     Start modem data collection.
 
     Args:
         ip_address: IP address of the modem.
+        control_port: Control port for modem (default: 1700)
+        status_port: Status port for modem (default: 1900)
         dump_interval: Duration between file dumps in seconds.
         total_duration: Total duration to run in seconds, or None for indefinite.
         output_dir: Output directory for CSV files.
 
     """
-    collect_files(host=ip_address, dump_interval=dump_interval, total_duration=total_duration, output_dir=output_dir)
-
-
-@click.command()
-@click.option("--ip", required=True, help="IP address of the modem")
-@click.option("--dump-interval", default=300, help="Duration between file dumps in seconds (default: 300 = 5 minutes)")
-@click.option(
-    "--total-duration", default=None, type=int, help="Total duration to run in seconds (default: run indefinitely)"
-)
-@click.option("--output-dir", default="./output", help="Output directory for CSV files (default: ./output)")
-def main_click(ip: str, dump_interval: int, total_duration: int, output_dir: str):
-    """
-    Click command-line interface for modem data collection.
-
-    Args:
-        ip: IP address of the modem (required).
-        dump_interval: Duration between file dumps in seconds (default: 300).
-        total_duration: Total duration to run in seconds (default: None for indefinite).
-        output_dir: Output directory for CSV files (default: './output').
-
-    This function serves as the entry point for the click CLI, collecting
-    modem measurements and saving them to timestamped CSV files.
-
-    """
-    collect_files(host=ip, dump_interval=dump_interval, total_duration=total_duration, output_dir=output_dir)
+    collect_files(
+        host=ip_address,
+        control_port=control_port,
+        status_port=status_port,
+        dump_interval=dump_interval,
+        total_duration=total_duration,
+        output_dir=output_dir,
+    )
 
 
 if __name__ == "__main__":
-    main_click()
+    import sys
+
+    logger.error("This module is now used as a library. Use 'opensampl-collect microchip twst' for CLI access.")
+    sys.exit(1)