Update implementation notes

Author: znicholls

IMPLEMENTATION-NOTES.md

@@ -1,11 +1,184 @@
 @Marco please move these into `docs/further-background/wrapping-derived-types.md` (and do any other clean up and formatting fixes you'd like)
 
-Wrapping derived types is tricky.
-Notably, [f2py](@Marco please add link) does not provide direct support for it.
+## What is the goal?
+
+The goal is to be able to run MAGICC, a model written in Fortran, from Python.
+This means we need to be able to instantiate MAGICC's inputs in memory in Python,
+pass them to Fortran to solve the model and get them back as results in Python.
+
+Our data is not easily represented as primitive types (floats, ints, strings, arrays)
+because we want to have more robust data handling, e.g. attaching units to arrays.
+As a result, we need to pass objects to Fortran and return Fortran derived types to Python.
+It turns out that wrapping derived types is tricky.
+Notably, [f2py](https://numpy.org/doc/stable/f2py/)
+does not provide direct support for it.
 As a result, we need to come up with our own solution.
 
 ## Our solution
 
+Our solution is based on a key simplifying assumption.
+Once we have passed data across the Python-Fortran interface,
+there is no way to modify it again from the other side of the interface.
+In other words, our wrappers are not views,
+instead they are independent instantiations of the same (or as similar as possible) data models.
+
+For example, if I have an object in Python
+and I pass this to a wrapped Fortran function which alters some attribute of this object,
+that modification will only happen on the Fortran side,
+the original Python object will remain unchanged
+(as a note, to see the result, we must return a new Python object from the Fortran wrapper).
+
+This assumption makes ownership and memory management clear.
+We do not need to keep instances around as views
+and worry about consistency across the Python-Fortran interface.
+Instead, we simply pass data back and forth,
+and the normal rules of data consistency within each programming language apply.
+
+To actually pass derived types back and forth across the Python-Fortran interface,
+we introduce a 'manager' module for all derived types.
+
+The manager module has two key components:
+
+1. an allocatable array of instances of the derived type it manages,
+   call this `instance_array`.
+   The array of instances are instances which the manager owns.
+   In practice, they are essentially temporary variables.
+1. an allocatable array of logical (boolean) values,
+   call this `available_array`.
+   The convention is that, if `available_array(i)`, where `i` is an integer,
+   is `.true.` then the instance at `instance_array(i)` is available for the manager to use,
+   otherwise the manager assumes that the instance is already being used for some purpose
+   and therefore cannot be used for whatever operation is currently being performed.
+
+This setup allows us to effectively pass derived types back and forth between Python and Fortran.
+
+Whenever we need to return a derived type to Python, we:
+
+[TODO think about retrieving multiple derived types at once]
+
+1. get the derived type from whatever Fortran function or subroutine created it,
+   call this `derived_type_original`
+1. find an index, `idx`, in `available_array` such that `available_array(idx)` is `.true.`
+1. set `instance_array(idx)` equal to `derived_type_original`
+1. we return `idx` to Python
+    - `idx` is an integer, so we can return this easily to Python using `f2py`
+1. we then create a Python object with an API that mirrors `derived_type_original`
+   using the class method `from_instance_index`.
+   This class method is [TODO or will be] auto-generated via `pyfgen`
+   and handles retrieval of all the attribute values of `derived_type_original`
+   from Fortran and sets them on the Python object that is being instantiated
+    - we can do this as, if you dig down deep enough, all attributes eventually
+      become primitive types which can be passed back and forth using `f2py`,
+      it can just be that multiple levels of recursion are needed
+      if you have derived types that themselves have derived type attributes
+1. we then call the manager [TODO I think this will end up being wrapper, we can tighten the language later]
+   module's `finalise_instance_index` function to free the (temporary) instance
+   that was used by the manager
+    - this instance is no longer needed because all the data has been transferred to Python
+1. we end up with a Python instance that has the result
+   and no extra/leftover memory footprint in Fortran
+   (and leave Fortran to decide whether to clean up `derived_type_original` or not)
+
+Whenever we need to pass a derived type to Fortran, we:
+
+[TODO think about passing multiple derived types at once]
+
+1. call the manager [TODO I think this will end up being wrapper, we can tighten the language later]
+   module's `get_free_instance_index` function to get an available index to use for the passing
+1. call the manager [TODO I think this will end up being wrapper, we can tighten the language later]
+   module's `build_instance` function with the index we just received
+   plus all of the Python object's attribute values
+    - on the Fortran side, there is now an instantiated derived type, ready for use
+1. call the wrapped Fortran function of interest,
+   except we pass the instance index instead of the derived type
+1. on the Fortran side, retrieve the instantiated index from the manager module
+   and use this to call the Fortran function/subroutine of interest
+1. return the result from Fortran back to Python
+1. call the manager [TODO I think this will end up being wrapper, we can tighten the language later]
+   module's `finalise_instance_index` function to free the (temporary) instance
+   that was used to pass the instance in the first place
+    - this instance is no longer needed because all the data has been transferred and used by Fortran
+1. we end up with the result of the Fortran callable back in Python
+   and no extra/leftover memory footprint in Fortran from the instance created by the manager module
+
+## Further background
+
+We initially started this project and took quite a different route.
+The reason was that we were actually solving a different problem.
+What we were trying to do was to provide views into underlying Fortran instances.
+For example, we wanted to enable the following:
+
+```python
+>>> from some_fortran_wrapper import SomeWrappedFortranDerivedType
+
+
+>>> inst = SomeWrappedFortranDerivedType(value1=2, value2="hi")
+>>> inst2 = inst
+>>> inst.value1 = 5
+>>> # Updating the view via `inst` also affects `inst2`
+>>> inst2.value1
+5
+```
+
+Supporting views like this introduces a whole bunch of headaches,
+mainly due to consistency and memory management.
+
+A first headache is consistency.
+Consider the following, which is a common gotcha with numpy
+
+```python
+>>> import numpy as np
+>>>
+>>> a = np.array([1.2, 2.2, 2.5])
+>>> b = a
+>>> a[2] = 0.0
+>>> # b has been updated too - many users don't expect this
+>>> b
+array([1.2, 2.2, 0. ])
+```
+
+The second is memory management.
+For example, in the example above, if I delete variable `a`,
+what should variable `b` become?
+
+With numpy, it turns out that the answer is that `b` is unaffected
+
+```python
+>>> del a
+>>> a
+Traceback (most recent call last):
+  File "<stdin>", line 1, in <module>
+NameError: name 'a' is not defined
+>>> b
+array([1.2, 2.2, 0. ])
+```
+
+However, we would argue that this is not the only possibility.
+It could also be that `b` should become undefined,
+as the underlying array it views has been deleted.
+Doing it like this must also be very complicated for numpy,
+as they need to keep track of how many references
+there are to the array underlying the Python variables
+to know whether to actually free the memory or not.
+
+We don't want to solve these headaches,
+which is why our solution does not support views,
+instead only supporting the passing of data across the Python-Fortran interface
+(which ensures that ownership is clear at all times
+and normal Python rules apply in Python
+(which doesn't mean there aren't gotchas, just that we won't introduce any new gotchas)).
+
+## Other solutions we rejected
+
+### Provide views rather than passing data
+
+Note: this section was never properly finished.
+Once we started trying to write it,
+we realised how hard it would be to avoid weird edge cases
+so we stopped and changed to [our current solution][Our solution]
+(@Marco please check that this internal cross-reference works
+once the docs are built).
+
 To pass derived types back and forth across the Python-Fortran interface,
 we introduce a 'manager' module for all derived types.
 This manager module is responsible for managing derived type instances
@@ -15,14 +188,12 @@ and is needed because we can't pass them directly using f2py.
 The manager module has two key components:
 
 1. an allocatable array of instances of the derived type it manages
-   (@Marco note that this isn't how it is implemented now,
-   but this is how we will end up implementing it)
 1. an allocatable array of logical (boolean) values
 
 The array of instances are instances which the manager owns.
 It holds onto these: can instantiate them, can make them have the same values
 as results from Fortran functions etc.
-(@Marco I think we need to decide whether this is an array of instances
+(I think we need to decide whether this is an array of instances
 or an array of pointers to instances (although I don't think that's a thing https://fortran-lang.discourse.group/t/arrays-of-pointers/4851/6,
 so doing something like this might require yet another layer of abstraction).
 Array of instances means we have to do quite some data copying
@@ -79,8 +250,6 @@ and have slow reallocation calls sometimes (when we need to increase the number
 There is no perfect solution, and we think this way strikes the right balance of
 'just works' for most users while also offering access to fine-grained memory control for 'power users'.
 
-## Other solutions we rejected
-
 ### Pass pointers back and forth
 
 Example repository: https://github.com/Nicholaswogan/f2py-with-derived-types