Lots of docs

Added extra documentation for much of simlib.

Author: willgebhardt

docs/compartments.md

@@ -0,0 +1,49 @@
+# Compartments
+
+Inside ngcsimlib there is the global state as the backbone of any given model.
+This global state is the culmination of all the dynamic or changing parts of the
+model. Each value that builds this state is stored in a special container that
+helps track these changes known as a Compartment.
+
+## Practical Information
+
+Practically when working with compartments there are a few simple things to keep
+in mind and the rest is all behind the scenes bookkeeping. The first piece to
+keep in mind is that each compartment holds a value and thus setting a
+compartment with `myCompartment = newValue` will not function as intended as
+this will overwrite the python object that is the compartment with `newValue`.
+Instead, make use of the `.set()` method to update the value stored inside a
+compartment so `myCompartment = newValue` becomes `myCompartment.set(newValue)`.
+The second piece of information is that to retrieve a value from the compartment
+use `myCompartment.get()`. These methods to get and set data inside a
+compartment are the two main pieces to remember.
+
+## Technical information
+
+The follow sections are devoted to more technical information regarding how a
+compartment functions in the broader scope of ngcsimlib and how to leverage that
+information.
+
+### How data is stored
+
+The data stored inside a compartment is not actually stored inside a
+compartment. Instead, it is stored inside the global state and each compartment
+just holds the path or `key`, in the global state to pull a specific piece of
+information. As such it is technically possible to manipulate the value of a
+compartment without actually touching the compartment object created in a
+component. By default, compartments have safeguards to prevent this from happening
+accidentally but directly addressing the compartment in the global state has no
+such safeguards.
+
+### What is targeting
+
+As discussed in the model building section there is a concept of wiring together
+different compartments of different components. These wires are created through
+the concept of targeting. In esscence targeting is just updating the path stored
+in a compartment with the path of a different compartment. This means that if
+the targeted compartment goes to retireve the value stored in it, it will
+actually retrieve the value for the a different compartment. When a compartment
+is in this state where it is targeted at another compartment it is set to read
+only meaning that it can not modify a different compartment.
+
+

docs/compiling.md

@@ -0,0 +1,35 @@
+# Compiling
+The term compiling for ngcsimlib refers to automatic step that happens inside a context that produces a transformed method for all of its components. This step is the most complecated part of the library and generally does not need to be touched or interacted with. Nevertheless this section will cover all the steps that it does at a high level. There is an expectation that the reader has an understanding of python abstract syntax trees, globals, and how to dynamically compile python code to run.
+
+## The decorator
+In ngcsimlib there is a decorator marked as `@compilable` that is used to add a flag to methods that the user wants to compile. On its own it doesn't do anything but this lets the parser distinguish between method that should be compiled and methods that should be ignored.
+
+## Step by Step
+The process started by telling the parser to compile a specific object.
+
+### Step 1: Compile Children
+The first step to compile any object is to make sure that all the compilable objects of the top level object are compiled. So it loops through all the whole object and compiles each part that it finds that is flagged as compilable and is an instance of a class.
+
+### Step 2: Extract Methods to Compile
+While the parser is looping through all the parts of the top level object it is also extracting the methods on the object that are flagged as compilable with the decorator. It stores them for later but this lets the parser only loop over the object once.
+
+### Step 3: Parse Each Method
+As each method is its own entry point into the compiler the step will run for each method in the top level object
+
+### Step 3a: Set up Transformer
+The step sets up a ContextTransformer which is a ask.NodeTransformer and will convert the method from a class method with the use of `self` and other methods that need to be removed into their more context friendly counterpart.
+
+### Step 3b: Transform the function
+There are quite a few pieces of common python that need to be transformed. This step happens with the overall goal of replacing all object focused parts with a more global view. This means compartment's `.get` and `.set` calls are replaced with direct setting and getting from the global state based on the compartment's target. It also means that all temporally constant values such as `batch_size` are moved into the globals space for that specific file and replaced with the naming convention of `object_path_constant`. One more step that this does is ensure that there is no branching in the code. Specifically if there is a brach such as an if statement it will evaluate it and only keep the branch it will traverse down. This means that there can not be any branch logic based on inputs or computed values (this is a common restriction for just in time compiling). 
+
+### Step 3c: Parse Sub-Methods
+As it is possible to have other class methods that are not marked as entry points for compilation but need to be compiled as step 3b happens it tracks all the sub-methods needed, this step goes through and repeats steps 3a and 3b for each of them with a naming convention similar to the temporally constant values for each method.
+
+### Step 3d: Compile the AST
+Once we have all the needed namespace and globals needed to execute the method properly transformed the method is compiled with python and executed.
+
+### Step 3e: Binding
+The final step per method is to bind it to the original method, this replaces the method with an object which when called with act like the normal uncompiled version but has the addition of the `.compiled` attribute which contains all the compiled information to be used later. This allows for the end user to call `myComponent.myMethod.compiled()` and have it run. The exact type for `compiled` value can be found in `ngcsimlib._src.parser.utils:CompiledMethod`.
+
+### Step 4: Finishing Up
+Some objects such as the processes have more steps to modify themselves or their compiled methods to align themselves with needed functionality but that is found in each class's expanded `compile` method and should be referred to by looking at those methods specifically.    

docs/components.md

@@ -0,0 +1,31 @@
+# Components
+
+Living one step above compartments in a model hierachy rests the component. The
+components hold a collection of both temporally constant values and dynamic
+compartments, in addition they are the lowest place where logic governing the
+dynamics of a system are defined. Generally components are going to be the
+building blocks that are reused multiple times throughout a model to create a
+final dynmical system.
+
+## Temporally Constant vs Dynamic Compartments
+
+One important distinction that needs to be highlighted inside a component is the
+difference between a temporally constant value and a dynamic compartment.
+Starting with compartments these values that change over time, generally they
+will have the type `ngcsimlib.Compartment` and be used to track internal values
+of the component. These values can be ones such inputs, decaying values,
+counters, etc. The second catagory of values are temporally constant, these are
+values that will remain fixed on the model is constructed. These values tend to
+include ones such as matrix shapes and coefficients.
+
+## Defining Compilable Methods
+
+Inside a component it is expected that there are methods used to govern the
+temporal dynamics of a system. These compilable methods are decorated
+with `@compilable` and are defined like any regular method. Inside a compilable
+method there will be access to `self`, meaning that to reference a compartment's
+value it is `self.myCompartment.get()`. The only requirement is that any method
+decorated can not have a return value, values should be stored inside their
+respective compartments. In an external step from defining the component a
+transformer will change all of these methods into ones that function with the
+rest of ngcsimlib.

docs/context.md

@@ -0,0 +1,66 @@
+# Contexts
+
+Contexts in ngcsimlib are the top level container that holds everything used to
+define the model. On their own they have no runtime logic, they rely on their
+internal processes and component to build a working model.
+
+## Defining a Context
+
+To define a context ngcsimlib leverages the `with` block meaning that to create
+a new context simply start with `with Context("myContext") as ctx:` and a new
+context will be created. (Important note: names are unique, if a context is
+created with the same name they will be the same context and thus there might be
+conflicts). This defined context does not do anything on its own.
+
+## Adding Components
+
+To add components to the context simply initialize components while inside
+the `with` block of the context. Any component defined while inside the block
+will automatically be added and tacked by the context object.
+
+## Wiring Components
+
+Inside a model components will need to pass data to one another that setup is
+done here inside the context. To connect the compartments of two components
+follow the pattern `mySource.output >> myDestination.input` where output and
+input are compartments inside their respective components. This will ensure that
+when processes are being run the value will flow properly from component to
+component.
+
+### Operators
+
+There is a special type of wire called an operator that performs a simple
+operation on the compartment values as the data flows from one component to
+another. Generally these are use for simple math operations such as
+negation `Negate(mySource.output) >> myDestination.input` or the summation of
+multiple compartments into
+one `Summation(mySource1.output, mySource2.output, ...) >> myDestination.input`.
+These can be chained, so it would be possible to negate one or more of the inputs
+to the summation.
+
+## Adding Processes
+
+To add processes to the context simple initialize the process and add all of its
+steps while inside the `with` block of the process.
+
+## Exiting the `with` block
+
+When the context exits the `with` block it will recompile the entire model. This
+means that all parts of the model that needed to be compiled before views (such
+as the compiled methods of a process) needs to happen after the `with` block is
+exited. Behind the scenes this is calling `recompile` on the context, it is
+possible to manually trigger the recompile step but that can break certain
+connections so use this functionality sparingly.
+
+## Saving and Loading
+
+The context's one unique job is the handling of the saving and loading of models
+to disk. By default, calling `save_to_json(...)` will create the correct file
+structure and files needed and load the context in the future. To load a model
+calling `Context.load(...)` will load the context in from a directory, something
+important to note while loading in a context is that it is effectively
+recreating the components with their initial values using their arguments and
+keywords arguments (excluding those that can't be serialized). This means that
+if you have a trained model ensure that your components have a save method
+defined that will handle the saving and loading of all values in their
+compartments.  

docs/global_state.md

@@ -0,0 +1,48 @@
+# The Global State
+
+As ngcsimlib is a simulation library focused on temporal models; or models that
+change over time there, all models have some concept of a state. These states
+might be comprised of a single value that changes or of a complex set of values
+that when combined all together make up the dynamical system for the model. In
+both cases these sets of values are stored in what is known as the global state.
+
+## Interacting with the global state
+
+Since the global state will contain a large amount of information about a given
+model there is a need to be able to interact with and modify the values in the
+global state. In most use cases this is not done directly. The most common way
+to interact with the global state is through the use of the state manager. The
+state manager exists to provide a set of helper methods for interacting with the
+global state. While the manager is there to assist you it is not going to stop
+you from changing the state, or breaking the state. When changing the state
+beyond setting it as a result of processes be careful not to add or remove
+anything that is needed for your model.
+
+### Adding new fields to the global state
+
+If you are new to using ngcsimlib and looking for a way to add values to the
+global state, stop for a moment. Unless you know exactly what you are doing do
+not manually add values to the global state, instead see compartments, or
+components as those are the most common ways for adding fields to the global
+state.
+
+If you are actually trying to manually add values to the global state it is done
+through the use of the `add_key` method. This will create the appropriate key in
+the global state for at the given path and name, and it will be retrieved
+with `from_key` calls. This value however is not linked to a compartment and
+thus will be hard to get working properly in the compiled methods without some
+specific references.
+
+### Getting the current state
+
+To get the current state simply call `global_state_manager.state` this will give
+you a copy of the current state, meaning that any modifications made to it are
+not reflected in the global state.
+
+### Updating the global state
+
+To manually update the global state after modifying a local
+copy `global_state_manager.state = new_state` can be uased. This will update the
+state with the `.update` call for dictionaries meaning that a partial state will
+still update correctly.
+

docs/processes.md

@@ -0,0 +1,82 @@
+# Processes
+
+Processes in ngcsimlib are a way of defining a specific transition across a
+given model. They take in as many compilable method's across any number of
+components and produce a single top level method and a varying number of
+submethods needed to execute the entire chain of compilable methods in one step.
+This is done to interface nicely with just in time compilers and minimize the
+amount of read and write calls done across the chain of methods.
+
+## Building the chain
+
+Building the chain that a process will use is done in an iterative process. Once
+the process object is created steps are added using either `.then()` or `>>`.
+As an example
+
+```
+myProcess.then(myCompA.forward).then(myCompB.forward).then(myCompA.evolve).then(myCompB.evolve)
+```
+
+or
+
+```
+myProcess >> myCompA.forward >> myCompB.forward >> myCompA.evolve >> myCompB.evolve
+```
+
+In both cases this process will chain the four methods together into a single
+step, only updating the final state all steps are complete.
+
+## Types of processes
+
+There are two types of processes, the above example would be with what is
+referred to as a `MethodProcess`, these are used to chain together any
+compilable
+methods from any number of different components. The second type of process,
+called a `JointProcess`, is used to chain together entire processes.
+JointProcesses are especially useful if there are multiple method processes that
+need to be called but different orders of the processes are needed at different
+times.
+
+## Extras
+
+There are a few extra methods that come standard with each process type that can
+be useful for both regular operation and debugging.
+
+### Viewing the compiled method
+
+Behind the scenes a process is transforming and compiling down all the steps
+used to build it, these means that the exact code it is running to do its
+calculation is not what the user wrote. To allow for the end user to view and
+make sure that the two pieces of code; theirs and the compiled version, are
+equivalent every process has a `view_compiled_method()` call that can be used
+after the model is compiled. This call will return the code that it is running
+as a string. There will be some stark differences between the produced code and
+the code in the components used to build the steps. Please refer to the
+compiling page for a more indepth guide to comparing the outputs.
+
+### Needed Keywords
+
+As some methods will require external values such as `t` or `dt` for a given
+execution a process will also track all the keyword arguments that are needed to
+run their compiled process. To view which keywords a given process is expected
+calling `get_keywords()`. This is mostly used for debugging or a sanity check.
+
+### Packing keywords
+
+To add onto the needed keywords the process also provides an interface to
+produce the keywords needed to run in the form of two methods. The first method
+is `pack_keywords(...)`, this method packs together a single row of values
+needed to run a single execution of the process. The arguments are
+the `row_seed` which is a seed to be passed to all the keyword generators (only
+needed if generators are being used). The second set of arguments are keyword
+arguments that are either constant `dt=0.1` or
+generators `lambda row_seed: 0.1 * row_seed`. The second method for generating
+the keywords for a process is with `pack_rows(...)`. This method created many
+sets of keywords needed to run multiple iterations of the process. The arguments
+are slightly different, first it now has a `length` argument to indicate the
+number of rows being produced, second it has a `seed_generator` use to generate
+the seed of each row (for example to have only even
+seeds `seed_generator = lambda x: 2 * x`) if the generator
+is `None` `seed_generator = lamda x: x` is used. After that the same keyword
+arguments to define the needed parameters is used as in `pack_keywords`    
+