merge models.yml

Author: ThomasWarford

.pre-commit-config.yaml

@@ -2,7 +2,7 @@
 # pre-commit install
 repos:
 - repo: https://github.com/pre-commit/pre-commit-hooks
-  rev: v5.0.0
+  rev: v6.0.0
   hooks:
     - id: end-of-file-fixer
     - id: mixed-line-ending
@@ -11,7 +11,7 @@ repos:
 
 - repo: https://github.com/astral-sh/ruff-pre-commit
   # Ruff version.
-  rev: v0.11.13
+  rev: v0.15.5
   hooks:
     # Run the linter.
     - id: ruff-check
@@ -20,7 +20,7 @@ repos:
     - id: ruff-format
 
 - repo: https://github.com/numpy/numpydoc
-  rev: v1.8.0
+  rev: v1.10.0
   hooks:
     - id: numpydoc-validation
       files: ^ml_peg/

README.md

@@ -40,7 +40,7 @@ python3 -m pip install git+https://github.com/ddmms/ml-peg.git
 
 ## Features
 
-Coming soon!
+More details coming soon!
 
 
 ## Development
@@ -65,6 +65,12 @@ pytest -v  # Discover and run all tests
 Please refer to the [online documentation](https://ddmms.github.io/ml-peg/developer_guide/index.html)
 for information about contributing new benchmarks and models.
 
+### Tutorials
+
+We encourage developers new to the ML-PEG framework to work through the detailed
+step-by-step guides provided by our Jupyter Notebook tutorials:
+- [Adding a New Benchmark](docs/source/tutorials/python/adding_benchmark.ipynb) [![badge](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ddmms/ml-peg/blob/main/docs/source/tutorials/python/adding_benchmark.ipynb)
+
 
 ## Docker/Podman images
 

docs/source/developer_guide/index.rst

@@ -6,6 +6,7 @@ Developer guide
     :maxdepth: 3
 
     get_started
+    llm_agent_integration
     add_benchmarks
     add_category
     running

docs/source/developer_guide/llm_agent_integration.rst

@@ -0,0 +1,79 @@
+=========================
+LLM-Assistant Integration
+=========================
+
+When using LLM coding assistants (e.g. `Claude Code <https://code.claude.com>`_) for development or usage of this package, please see some tips and tricks here.
+
+Docs on Context7
+----------------
+
+ML-PEG documentation is indexed on `Context7 <https://context7.com>`_, which supplies up-to-date parsed documentation for the package as an MCP server.
+
+**Library ID**: ``/ddmms/ml-peg``
+
+Getting started
++++++++++++++++
+
+Install the MCP server, e.g. for Claude Code:
+
+.. code-block:: text
+
+    claude mcp add context7 -- npx -y @upstash/context7-mcp
+
+See `installation instructions <https://github.com/upstash/context7#installation>`_ for other tools as well.
+
+Afterwards can ask the agent to load the library, or you can add this to your agent's standard instructions, e.g. `CLAUDE.md` file.
+
+.. code-block:: text
+
+    @context7 load library /ddmms/ml-peg
+
+Claude Code will use the ``/ddmms/ml-peg`` library ID to fetch relevant documentation and code examples.
+
+Common Use Cases
+++++++++++++++++
+
+Asking to query the documentation directly:
+
+.. code-block:: text
+
+    @context7 How do I add a new benchmark calculation script?
+
+Asking about further specifics:
+
+.. code-block:: text
+
+   How do I structure a calculation script using pytest parametrization? use context7
+
+Related Documentation
+---------------------
+
+For related libraries on Context7:
+
+- **mlipx** (``/basf/mlipx``): Base calculator abstraction library used by ML-PEG
+- **janus-core** (``/stfc/janus-core``: another back-end for ML-PEG
+
+
+API Keys and Rate Limits
+-------------------------
+
+Context7 provides free access with rate limits at the time of writing.
+For higher rate limits and being able to submit new documentation sources to index:
+
+1. Visit https://context7.com
+2. Create an account
+3. Generate an API key from the dashboard
+4. Configure your AI assistant with the API key
+
+.. note::
+
+    ML-PEG is a public repository and does not require an API key for basic usage through Context7.
+
+Keeping Documentation Updated
+-----------------------------
+
+The ML-PEG library on Context7 is automatically updated when changes are pushed to the main branch. Documentation typically updates within 24 hours of changes being merged. If you notice outdated information you can click to refresh on the site, `you need to log in to do this`.
+
+.. tip::
+
+    When contributing new benchmarks or features, ensure your docstrings and documentation updates are merged to main before using Context7 to help others understand your additions.

docs/source/index.rst

@@ -1,5 +1,5 @@
 Welcome to ML-PEG's documentation!
-========================================
+==================================
 
 ML-PEG is an ML Performance and Extrapolation Guide.
 

docs/source/user_guide/benchmarks/conformers.rst

@@ -0,0 +1,42 @@
+==========
+Conformers
+==========
+
+ACONFL
+======
+
+Summary
+-------
+
+Performance in predicting relative conformer energies of 12 C12H26,
+16 C16H34 and 20 C20H42 conformers. Reference data from PNO-LCCSD(T)-F12/ AVQZ calculations.
+
+Metrics
+-------
+
+1. Conformer energy error
+
+For each complex, the the relative energy is calculated by taking the difference in energy
+between the given conformer and the reference (zero-energy) conformer. This is
+compared to the reference conformer energy, calculated in the same way.
+
+Computational cost
+------------------
+
+Low: tests are likely to take minutes to run on CPU.
+
+Data availability
+-----------------
+
+Input structures:
+
+* Conformational Energy Benchmark for Longer n-Alkane Chains
+  Sebastian Ehlert, Stefan Grimme, and Andreas Hansen
+  The Journal of Physical Chemistry A 2022 126 (22), 3521-3535
+  DOI: 10.1021/acs.jpca.2c02439
+
+Reference data:
+
+* Same as input data
+* :math:`PNO-LCCSD(T)-F12/ AVQZ` level of theory: a local, explicitly
+  correlated coupled cluster method.

docs/source/user_guide/benchmarks/index.rst

@@ -12,3 +12,7 @@ Benchmarks
     molecular_crystal
     molecular
     bulk_crystal
+    lanthanides
+    non_covalent_interactions
+    tm_complexes
+    conformers

docs/source/user_guide/benchmarks/lanthanides.rst

@@ -0,0 +1,45 @@
+===========
+Lanthanides
+===========
+
+Isomer complexes
+================
+
+Summary
+-------
+
+Performance in predicting relative isomer energies for lanthanide complexes
+compared to r2SCAN-3c DFT reference data.
+
+
+Metrics
+-------
+
+1. Relative isomer energy MAE
+
+Accuracy of relative isomer energy predictions.
+
+For each complex, the relative isomer energies are computed with respect to the
+lowest-energy isomer in the r2SCAN-3c reference set and compared to the r2SCAN-3c
+relative energies reported in the reference dataset.
+
+
+Computational cost
+------------------
+
+Low: tests are likely to take less than a minute to run on CPU.
+
+
+Data availability
+-----------------
+
+Input structures:
+
+* T. Rose, M. Bursch, J.-M. Mewes, and S. Grimme, Fast and Robust Modeling of
+  Lanthanide and Actinide Complexes, Biomolecules, and Molecular Crystals with
+  the Extended GFN-FF Model, Inorganic Chemistry 63 (2024) 19364-19374.
+
+Reference data:
+
+* Relative isomer energies from r2SCAN-3c (see Supporting Information of the
+  above reference).

docs/source/user_guide/benchmarks/molecular.rst

@@ -124,3 +124,53 @@ Reference data:
 
 * Same as input data
 * DLPNO-CCSD(T)/CBS
+
+
+BMIM Cl RDF
+===========
+
+Summary
+-------
+
+Tests whether MLIPs incorrectly predict covalent bond formation between chloride
+anions (Cl⁻) and carbon atoms in 1-butyl-3-methylimidazolium (BMIM⁺) cations.
+Such Cl-C bonds should NOT form in the ionic liquid under normal conditions.
+
+This benchmark runs NVT molecular dynamics simulations of BMIM Cl at
+353.15 K and analyses the Cl-C RDF to detect any unphysical bond formation.
+
+
+Metrics
+-------
+
+1. Cl-C Bonds Formed
+
+Binary metric indicating whether unphysical Cl-C bonds formed during the MD simulation.
+
+The Cl-C RDF is computed from the MD trajectory. If the RDF shows a peak (g(r) > 0.1)
+at distances below 2.5 Å, this indicates bond formation and the model fails the test.
+
+* 0 = no bonds formed (correct physical behaviour)
+* 1 = bonds formed (unphysical, model failure)
+
+
+Computational cost
+------------------
+
+Medium: tests require running 10,000 steps of Langevin MD for a system of 10 ion
+pairs, which may take tens of minutes on GPU.
+
+
+Data availability
+-----------------
+
+Input structures:
+
+* Generated using molify from SMILES representations of BMIM⁺ (CCCCN1C=C[N+](=C1)C)
+  and Cl⁻ ions, packed to experimental density of 1052 kg/m³ at 353.15 K.
+* Zills, F. molify: Molecular Structure Interface. Journal of Open Source Software
+  10, 8829 (2025). https://doi.org/10.21105/joss.08829
+* Density from: Yang, F., Wang, D., Wang, X. & Liu, Z. Volumetric Properties of
+  Binary and Ternary Mixtures of Bis(2-hydroxyethyl)ammonium Acetate with Methanol,
+  N,N-Dimethylformamide, and Water at Several Temperatures. J. Chem. Eng. Data 62,
+  3958-3966 (2017). https://doi.org/10.1021/acs.jced.7b00654

docs/source/user_guide/benchmarks/molecular_crystal.rst

@@ -86,3 +86,53 @@ Reference data:
 
 * Same as input data
 * PBE-D3(BJ)
+
+
+CPOSS209
+=========
+
+Summary
+-------
+
+Performance in predicting lattice energies of 209 organic molecular crystals from the CPOSS209 dataset.
+
+
+Metrics
+-------
+
+1. Absolute lattice energy MAE
+
+Accuracy of the absolute lattice energy predictions.
+
+For each molecular crystal, lattice energy is calculated by taking the difference between the energy of the solid molecular crystal divided by the number of molecules it comprises,
+and the energy of the isolated molecule. This is compared to the reference lattice energy.
+
+2. Relative lattice energy MAE
+
+Accuracy of the reltive lattice energy predictions.
+
+We compute the lattice energies in the same way as before, but this time we compute the relative error.
+The relative error is computed by identifying the most stable polymorphs and, for each crystal, subtracting the lattice energy of the most stable polymorph.
+This metric cares more about the ranking of the crystal polymorphs rather than reproducing the reference values of the lattice energies.
+
+
+Computational cost
+------------------
+
+Low: tests should take a few minutes on a CPU
+
+
+Data availability
+-----------------
+
+Input structures:
+
+* Structures are optimized at the PBE+TS level.
+* Louise S. Price, Matteo Paloni, Matteo Salvalaglio, and Sarah L. Price
+  Crystal Growth & Design 2025 25 (9), 3186-3209
+  DOI: 10.1021/acs.cgd.5c00255
+
+Reference data:
+
+* Same as input data
+* wb97md3 with 1b CCSD(T) corrections