Merge pull request #58 from usethesource/adopt-tutor-docs

cloned Tutor documentation from deprecated rascal-tutor project to here

Author: jurgenvinju

courses/Tutor/API/API.md

@@ -0,0 +1,49 @@
+---
+title: API documentation
+---
+
+#### Synopsis
+
+The Tutor compiler reads Rascal source files and produces ((Concept)) markdown files 
+for each Rascal module.
+
+#### Description
+
+The compiler generates for every `Module.rsc` a markdown file with the following structure:
+
+``````
+---
+title: <moduleName>
+---
+
+// for each declaration (function, data, syntax, alias, import, extend)
+## <declarationName> {#fully-qualified-declaration-name}
+
+#### Synopsis
+
+...
+#### Description
+...
+
+// etc.
+``````
+
+So, for all standard ((Concept)) headers, like `Synopsis` and `Benefits` there is a place at every declaration.
+
+The content of the header paragraphs is either directly derived from source code (like function signature and data definitions), or it is taken from the following tag definitions on each declaration:
+* `@doc` may contain literally the headers of a ((Concept)), like `#### Synopsis`. This notation is deprecated in favor of the tags below.
+* `@synopsis` is a single line description of the definition.
+* `@description` is a multi-line explanantion of the definition.
+* `@benefits`, `@pitfalls`, `@examples`, `@types`, `@name` and `@function` each follow the intent of the standard ((Concept)) headers.
+
+The documentation written for features without an explicit synopsis is not rendered. In other words it is obligatory to use at least a `@synopsis` tag or start with a `#### Synopsis` header..
+
+#### Benefits
+
+* A (small) part of documentation writing is automated. The information about the name of a function, data or annotation declaration, or its signature is always consistent.
+* You can write examples of the usage of each definition using ((Listing)) markup that is run and checked at documentation compile-time.
+
+#### Pitfalls 
+
+* This approach requires that functions with the same name are grouped together in the source file. 
+* You need to run the tutor compiler before errors in the documentation tags are detected.

courses/Tutor/Architecture/Architecture.md

@@ -0,0 +1,62 @@
+---
+title: Architecture
+---
+
+#### Synopsis
+
+The global architecture of the Rascal Tutor
+
+#### Description
+
+The Rascal Tutor is a Markdown and Rascal source file pre-processor. As input it takes [Docusaurus Markdown](https://docusaurus.io/docs/markdown-features/) files organized in hierarchical folders, and Rascal modules organized in hierarchical packages. One root folder is called a "course". As output the pre-processor produces per course 
+a folder hierarchy again, where each folder has its own `index.md` file (generated or written). After the tutor
+compiler has generated such a consistent folder of interconnected markdown files, other downstream processors can turn them into (static) html websites, pdf files or otherwise. The standard way of processing is to use the [Docusaurus](https://docusaurus.io) static website generator.
+
+The important features of the pre-processor, "compiler", are:
+
+1. The compiler is configured via `util::Reflective::PathConfig`, where:
+   * each entry in the `srcs` list is a single _course_
+   * each entry in the `libs` list, be it a jar file or not, is searched for an `index.value` file to augment the current index. 
+1. Concept hierarchy - each folder `/X` has its own index file, called either `X/X.md` or `X/index.md`. Nested folders equal nested concepts.
+1. Indexing - to help in easy cross-linking between concepts:
+   * Each concept of course `A` stored in `X/Y/Z/Z.md` may be linked via `Z`, `Y-Z`, `X-Y-Z`, `A:Z`, `A:Y-Z`, `A:X-Y-Z`
+   * Each Rascal module of course `A` named `a::b::C` may be linked via `C`, `b-C`, `a::b::C`, `module:a::b::C`, `A:C`, `A:b-C`, `A:a::b::C`, `A:module:a::b::C`
+   * Each Rascal package of course `A` named `a::b` may be linked via `b`, `a::b`, `package:a::b`, `A:b`, `A:a::b`, `A:module:a::b`
+   * Each image `x.{png,svg,jpg,jpeg}` stored in `X/Y/Z` may be linked as though it were a concept file.
+   * The index is a `rel[str,str]` from the above described links to the `index.md` files in the generated hierarchy.
+   * The compiler reports missing links and ambiguous links as errors.
+   * A single `index.value` file is written in the output folder for future reference by depending projects.
+1. Code execution ensures lively demonstrations which are checked for correctness. 
+   * Code blocks marked `rascal-shell` are executed line-by-line on the REPL prompt. Each prompt starts with a fresh environment. All error and standard output is captured and printed back. The `Content` module that serves HTML or any other file is also inlined in the output Markdown file.
+   * Code blocks marked `rascal-shell,continue` are executed in the previously constructed environment (from top to bottom in the Markdown file) and behave the same otherwise.
+   * Code blocks marked `rascal-prepare` with or without `continue` behave the same as above, except that no output or input is printed back into the output file.
+1. Links written between `((` and `))` are resolved using the previously described index. 
+1. Table of contents are generated for the word TOC between three `(((` brackets `)))`; the content of each concept is the concept nested under it in the hierarchy.
+1. Each Rascal file is indexed to list the declarations it contains and information is extracted:
+   * data declarations with their `doc`, `synopsis`, `examples` etc. tags
+   * (overloaded) function declarations with their `doc`, `synopsis`, `examples` etc. tags
+   * alias declarations with their `doc`, `synopsis`, `examples` etc. tags
+1. Screenshots of interactive ((Library:module:Content) visualizations can be made by configuring the compiler to use selenium and chrome.
+
+Features on the TODO list are:
+1. Interactive Questions (have to be revived)
+
+
+To implement these features the following components are relevant:
+* `lang::rascal::tutor::Indexer` implements indexing and lookup
+* `lang::rascal::tutor::Compiler` implements linking, code execution, toc, and the generation of index files.
+* `lang::rascal::tutor::apidoc::ExtractInfo` parses Rascal code and generates an intermediate overview
+* `lang::rascal::tutor::apidoc::GenerateMarkdown` generates a single Markdown file for each Rascal module, and also reuses `Compiler` to process embedded markdown with code examples, links, etc.
+* `lang::rascal::tutor::repl::TutorCommandExecutor` encapsulates a Rascal REPL as a set of closures (`eval`, `reset`, and `prompt`) 
+   * for efficiency's sake a single executor is shared among all Markdown and Rascal files that are compiled in a single run. Between every file the executor is reset for a new environment, but previously loaded modules remain in the heap available for reuse. 
+
+#### Benefits
+
+* Index creation is modular: an index is created per course and these indices are combined at runtime.
+* Code execution is reasonably time efficient, although it may require quite some memory because eventually an entire course will be loaded.
+
+#### Pitfalls
+
+* Courses are compiled on a per-course basis. Support for incremental courses compilation is not yet available.
+* Badly balanced code block quotes are not detected by this pre-processor but will fail HTML compilation downstream.
+* Manual links are not checked at compile-time by this pre-processor but fill fail HTML compilation downstream.

courses/Tutor/Authoring/Authoring.md

@@ -0,0 +1,48 @@
+---
+title: Authoring
+---
+
+#### Synopsis
+
+Creating and writing a course for the Rascal Tutor.
+
+#### Syntax
+
+#### Types
+
+#### Function
+
+#### Description
+
+The life cycle of a course consists of the following steps:
+
+* A new course, say `MyCourse`, is created. This is achieved by:
+  * Creating a subdirectory named `MyCourse` in the `courses` directory of the current Rascal project.
+  * Creating a file `MyCourse/MyCourse.md`. This is the root concept of the new course.
+* The contents of the course are created by populating the course with subconcepts of the root concept.
+* A subconcept, say `CoolIdea` is created by:
+  * Creating a subdirectory `CoolIdea` of its parent concept.
+  * Creating a file `CoolIdea/CoolIdea.md` that describes the concept.
+  * Alternatively, a file `CoolIdea/index.md` works too.
+  * Renaming/moving/deleting concepts is done at the directory/file level.
+
+Concepts are represented as directories for the following reasons:
+
+* To represent subconcepts as subdirectories.
+* To contain all figures and other files that are included in the concept. In this way:
+  * A complete concept can be easily moved or renamed as a single unit.
+  * Name clashes between included files per concept are avoided.
+
+#### Examples
+
+#### Benefits
+
+* You can use your favourite Markdown editor and standard system commands to author a course.
+* The output of the Tutor compiler is Markdown/HTML that you can process further with any tool you like
+* Links to other concepts are all _relative_ to a root directory with all the courses. So `../../Course/A/index.md` would be a link generated from a link to `A` in `/Course/Z/index.md`. This makes it easy to include courses in a website at any subfolder.
+* The index that is generated for every project with several courses can be reused by other projects to generate consistent cross-referencing. For example a project `flybytes` which depends on the `rascal` project which contains the `Library` course, could use `Library:IO` to refer to the documentation about the `IO` module in the standard library of Rascal.
+
+#### Pitfalls
+
+* Special features like code execution and cross-referencing concepts are not implemented by generic Markdown editors, so you have to wait and see what happens until you compile the course.
+* All images are referenced from `/assets/` so images have to be installed there on any website you want to include the documentation in.

courses/Tutor/Compilation/Compilation.md

@@ -0,0 +1,122 @@
+---
+title: Tutor Compilation
+keywords:
+    - compilation
+    - compiler
+    - configuration
+    - maven
+    - deployment
+    - incremental
+    - dependency
+    - dependencies
+---
+
+After ((Authoring)) tutor files, they have to compiled down to plain docusaurus. The compiler implements features such as linking, screenshots, and executing code fragments.
+Most importantly, when there are linking errors (missing or ambiguous) or code execution errors, the compiler reports them such that they can be fixed before releasing the documentation.
+
+## Configuration
+
+The way to execute the rascal tutor compiler is via [rascal-maven-plugin](http://github.com/usethesource/rascal-maven-plugin). First it 
+has to be configured for your project.
+
+Do configure the tutor, add the plugin to the pom.xml like so:
+
+```xml
+<plugins>
+    <plugin>
+        <groupId>org.rascalmpl</groupId>
+        <artifactId>rascal-maven-plugin</artifactId>
+        <version>0.16.0</version>
+        <executions>
+            <execution>
+                <id>default-cli</id>
+                <phase>compile</phase>
+                <goals>
+                    <goal>tutor</goal>
+                </goals>
+                <configuration>
+                    <enableStandardLibrary>false</enableStandardLibrary> 
+                    <errorsAsWarnings>false</errorsAsWarnings>
+                    <isPackageCourse>true</isPackageCourse>
+                    <bin>${project.build.outputDirectory}</bin>
+                    <srcs>
+                        <src>${project.basedir}/doc/Course1</src>
+                        <src>${project.basedir}/doc/Course2</src>
+                        <src>${project.basedir}/src/main/rascal</src>
+                    </srcs>
+                    <ignores>
+                        <ignore>${project.basedir}/src/main/rascal/experimental</ignore>
+                    </ignores>
+                </configuration>
+                </execution>
+        </executions>
+    </plugin>
+</plugins>
+```
+
+1. `enableStandardLibrary` defines which library the links to standard library documentation refer to. Either the version of rascal that the rascal-maven-plugin depends on (`true`), or the version of rascal that the current project depends on (`false`). Also this defines which version of the library the code examples are executed against.
+2. `errorsAsWarnings` and `warningsAsError` define failure modes for the compiler. Maven will not fail if `errorsAsWarnings` is set to `true` even if there are errors.
+3. `isPackageCourse`, when set to `true` will nest all target documentation in `docs/Packages/projectName`, and rename `src/main/rascal` or `src` to `API`. All links will also be re-routed to these locations. Otherwise, when set to `false` all documentation ends up in `docs/Course1` and `docs/Course2`, etc.
+4. `bin` points to the place that defines the root of the resulting `jar` file. All generated files will endup in `bin/docs` and `bin/docs/assets`. This has to be the outputDirectory and if you forget to configure this, it will be set automatically anyway. The compiler also writes `bin/index.value` file after it is done, to store the entire linking index for future usage. This future use can be either the next run of the compiler, or a depending project downstream.
+5. `srcs` define root courses, some of which will be pure Concept hierarchies, and others could be the _root_ of Rascal source folders. Note that this also defines the search path for modules during code execution of `rascal-shell` blocks.
+6. `ignores` defines files or folders which are to be ignore. These could be folder names inside of Rascal source folders, or files of Rascal modules, or course concept folders, or course concept files. When a folder is ignored, this works transitively for the contents of that folder.
+
+## Dependencies
+
+If the current Maven project has dependencies then the `index.value` files in these jar files are located
+and imported in the current link index of the project. This allows you to refer to concepts in one of the 
+projects you depend on, provided that project was also using the `rascal-maven-plugin` to run the tutor compiler.
+
+The dependency feature depends on the assumption that the `/docs` folders in the jar of the current project and the jars of the
+project it depends on are extracted into the `docs` folder of a Docusaurus project later, and that everything
+under `/docs/assets` in the same jars is copied to `static/assets`. The `rascal-website` project
+achieves this using the `resources` plugin `copy` commands. If you are running your own docusaurus site,
+then this should be re-implemented.
+
+## Execution
+
+Simply run the maven `compile` or `package` or `install` command to trigger the rascal tutor compiler:
+
+```bash
+$ mvn install
+```
+
+However, to exclusively run the tutor you can try this:
+
+```bash
+$ mvn rascal:tutor
+```
+
+If there are screenshots in the documentation, then selenium must be configured with the location of `chrome` and `chromedriver`:
+
+```bash
+$ mvn rascal:tutor -Dwebdriver.chrome.driver=`which chromedriver` -Dwebdriver.chrome.browser=`which chrome`
+```
+
+Here we have used the bash feature to splice in the location of chromedrive and chrome using the `which` command. This requires both to
+be found on the system `PATH`. If this is not the case, provide the absolute path to the binaries of these two programs. Both on Mac
+and Windows the path to `Google Chrome`, for example, may contain many spaces and it is wise to wrap the path in double quotes.
+
+The tutor compiler is **incremental** in a certain way. Running the compile twice in a row, only the markdown files and Rascal modules that have been modified will be compiled again. The compiler
+reuses the output of the previous run, including error messages. All error messages and warnings, including those of unmodified files are
+always presented at the end. 
+
+Simple run the tutor twice to recompile only what has changed:
+
+```bash
+mvn rascal:tutor
+mvn rascal:tutor
+```
+
+:::warning
+When folders or files are renamed, the old output files remain in the `bin` folder as well as the old links to those in the link index file.
+This can lead to:
+* spurious link ambiguity; for example when `A/B.md` has moved to `C/B.md` both versions of `B` will be present in the link administration.
+* unreported missing links; for example when `A/B.md` has moved to A/C.md`, `B` will still be linked even though it is not present anymore.
+
+It is therefore always prudent to `mvn clean` such that the output directory including the `index.value` has been removed, when files or folder change name or location.
+:::
+
+## Deployment of documentation
+
+* When `mvn install` or `mvn package` is run with the correct configuration for the `bin` folder, then all markdown files and other assets (screenshots) end up in the `jar` file of the current project.

courses/Tutor/Concept/Benefits/Benefits.md

@@ -0,0 +1,32 @@
+---
+title: Benefits
+---
+
+#### Synopsis
+
+Briefly lists known advantages of the concept.
+
+#### Syntax
+
+```
+#### Benefits
+
+_MarkedText_
+```
+
+#### Types
+
+#### Function
+
+#### Description
+
+#### Examples
+
+#### Benefits
+
+* Benefits allow a description of the, subjectively, positive aspects of a concept which is clearly delineated. This allows other sections to remain more factual and neutral.
+* Benefits alert newcomers to possibilities they would not have thought of otherwise.
+
+#### Pitfalls
+
+* Benefits sections may grow into bragging lists.

courses/Tutor/Concept/Concept.md

@@ -0,0 +1,61 @@
+---
+title: Concept
+details:
+  - Title
+  - Keywords
+  - Synopsis
+  - Syntax
+  - Types
+  - Function
+  - Details
+  - Description
+  - Examples
+  - Benefits
+  - Pitfalls
+---
+
+#### Synopsis
+
+A _concept_ is the basic building block of a course. 
+
+#### Syntax
+
+* `module a/b/Concept` in `a/b/Concept.rsc`
+* Or a file `a/b/Concept/Concept.md`, `a/b/Concept/index.md`:
+   ``````
+   ---
+   title: Concept Title
+   keywords:
+      - keyword1
+      - keyword2
+   ---
+
+   #### Synopsis
+   
+   This is the synopsis.
+
+   #### Description
+
+   #### Types
+
+   #### Function
+
+   #### Examples
+
+   #### Benefits
+
+   #### Pitfalls
+   ``````
+
+All sections are optional, but not the title header. It is always recommended to have at least a Synopsis and some Examples. Empty sections are removed by the preprocessor.
+
+#### Description
+
+A concept describes a separate entity (idea, artefact, function, rule) that is relevant in the scope of the course in which it appears.
+It consists of the named sections above that are described separately. 
+Each section starts with a keyword that should appear at the begin of a line.
+
+Here is a brief summary of the sections of a concept:
+(((TOC)))
+
+