add some notes about architecture

Author: Arlen22

ARCHITECTURE.md

@@ -0,0 +1,127 @@
+# MWS Server Architecture Explained
+
+## Overview
+
+The system treats a public wiki as a named, database-backed resource addressed by a slug. Internally that wiki is backed by a recipe, but for most developer-facing purposes it is simplest to think in terms of a wiki assembled from three things: a template, a set of bags, and a plugin set.
+
+A template provides shared defaults such as HTML behavior, default bags, and plugin configuration. A wiki adds its own bag mappings, plugins, and access rules. At runtime, requests use the wiki's effective configuration rather than raw admin input.
+
+## Core Concepts
+
+A bag is a named tiddler store. Bags hold content and carry their own read and write permissions.
+
+A template is a reusable definition that can be shared across many wikis. It supplies default bag structure, plugin settings, and HTML behavior.
+
+A wiki is the public-facing site. It is addressed by slug and resolves to an effective set of bags and plugins.
+
+Plugins extend the client runtime. The server prepares and serves the plugin assets needed to bootstrap the wiki in the browser.
+
+Users, roles, and sessions determine who can see a wiki, who can edit it, and which bags they can read or write.
+
+## How A Wiki Is Assembled
+
+Each wiki is built from a template plus wiki-specific configuration. Together they determine which bags participate in the wiki, which of those bags are writable, which plugins are loaded, and what HTML shell the client receives.
+
+Writes are routed by title prefix. In practice, the most specific matching writable prefix determines where a title is saved. Readonly bags act as fallback content layers beneath the writable part of the wiki. This gives each wiki a predictable namespace model: a title is written to one resolved writable location and read from the first applicable source in the wiki's effective bag stack.
+
+Deleting follows the same model as writing. A delete applies to the resolved writable location for that title rather than removing the title from every bag that might contain it.
+
+## Runtime Surface
+
+The main runtime entry point is the wiki page itself:
+
+- `GET /wiki/:recipe_slug` returns the HTML bootstrap for the wiki.
+
+The recipe-scoped API provides the supporting runtime data:
+
+- `GET /recipe/:recipe_slug/status` returns the caller's view of the wiki's bag layout and writeability.
+- `GET /recipe/:recipe_slug/list.json` lists visible titles together with routing information.
+- `GET /recipe/:recipe_slug/store.js` returns the store payload used to bootstrap the client.
+- `GET /recipe/:recipe_slug/updates?since=` returns recipe-scoped changes since a known revision.
+- `PUT /recipe/:recipe_slug/batch/:op` performs batch list, read, save, and delete operations.
+
+The runtime surface is designed so that list, read, save, delete, updates, and store generation all describe the same wiki view.
+
+## Store And Client Bootstrap
+
+The client is bootstrapped from two pieces: HTML and store data.
+
+The HTML comes either from the default TiddlyWiki shell or from template-defined custom HTML. The store contains the resolved tiddlers for the wiki plus the metadata the client needs to understand the current recipe, bag ownership, host, and revision state.
+
+Depending on server and template configuration, plugin assets and store data may be delivered as external resources or injected into the HTML response. That delivery choice changes how the browser receives the data, but not what wiki content the client sees.
+
+## Administration
+
+The admin surface manages wikis, templates, bags, users, and roles. In this model, a wiki is the editable public resource, a template is the reusable shared definition, and a bag is the underlying storage unit.
+
+Saving a wiki or template changes the configuration that future runtime requests use. Template changes affect every wiki that depends on that template. Bag changes affect where titles are stored and resolved.
+
+Plugin entries are exposed through the admin load path so they can be selected and inspected, but this path does not treat plugins as editable database rows in the same way as wikis, templates, bags, users, and roles.
+
+User administration covers profile-like data and role membership. Password creation, login, reset, and password change belong to the session and password subsystem rather than to admin row saves.
+
+## Permissions
+
+Access is role-based.
+
+Recipe permissions determine whether a wiki is available to a user at all. Bag permissions determine whether the user can read from or write to the bags that make up that wiki. Template permissions govern template management rather than live wiki access.
+
+The runtime presents a wiki as a coherent whole rather than a partially filtered bag set. If a user does not have the required access to the bags that define a wiki, the wiki request is denied. A user can also have access to a wiki while still being unable to modify a specific title if that title resolves to a writable bag they cannot edit.
+
+## Sessions And Identity
+
+Sessions resolve the current user and that user's roles for every request. Those roles are then used consistently across wiki access checks, bag write checks, and admin operations.
+
+The public identifier for a wiki is its slug. Developers working on the runtime surface should treat that slug as the stable address of the wiki.
+
+## Repository Layout
+
+This is a monorepo divided into separate projects. The entry point is in `packages/mws/src/index.ts`. The entry point imports the `server`, `commander`, and `events` projects.
+
+The entire server uses Promises and async functions for pretty much everything. Synchronous file system calls should be avoided as much as possible.
+
+The `events` project is the foundation of MWS. It contains an `EventEmitter` instance that everything else subscribes to.
+
+Events are emitted asyncly. Event listeners are awaited with `Promise.all`, and rejections throw back to the event emit call. This is intentional because it is the heart of the entire server, not just a public event bus, and errors shouldn't be ignored. Errors that come from things like attempting to send SSE events to other clients, however, should not be thrown because they aren't relevent to the source of the event.
+
+There should be no singleton references other than the event emitter and event handlers should be as pure as possible, so that in theory it would be possible to run multiple completely separate MWS servers in the same process.
+
+The `commander` project handles the CLI parsing code. It exports a default function which MWS calls to execute the CLI. Commander emits several events during execution, which MWS hooks into to scaffold the rest of the server.
+
+It also exports the base class for commands to inherit from, and because it instantiates the commands, commands should not specify their own constructor. Additional instance properties may be added to commands in the `cli.execute.before` event.
+
+The `server` project handles all web related stuff, but in an MWS-agnostic way. It only requires the `events` project, so it could be used as the foundation for unrelated webservers. Its internal API is directly inspired by the TiddlyWiki server API, with routing and centralized handling of the request body.
+
+The `mws` project is the application layer of MWS and ties everything else together. It defines all the commands and web server routes and handles the database connection.
+
+## Server Events
+
+These are the hooks that run on startup. The listen command starts the server and returns, finishing the command run.
+
+```
+├ cli.register: 0.101ms
+├ cli.commander: 0.021ms
+├ Command: listen: --------
+├ ├ cli.execute.before: --------
+├ ├ ├ mws.cache.init.before: 0.012ms
+├ ├ ├ mws.cache.init.after: 0.016ms
+├ ├ ├ mws.adapter.init.before: 0.013ms
+├ ├ ├ mws.adapter.init.after: 0.007ms
+├ ├ ├ mws.config.init.before: 0.008ms
+├ ├ ├ mws.config.init.after: 0.013ms
+├ ├ cli.execute.before: 1.175s
+├ ├ listen.router.init: --------
+├ ├ ├ mws.router.init: 0.017ms
+├ ├ ├ mws.routes.important: 0.004ms
+├ ├ ├ mws.routes: 1.054ms
+├ ├ ├ mws.routes.fallback: 0.039ms
+├ ├ listen.router.init: 111.665ms
+├ ├ cli.execute.after: 0.008ms
+├ Command: listen: 1.296s
+```
+
+## Todo
+
+- server input validation: all the pieces are there, I just need to wire it up.
+- make commands easier to extend, probably just by making it easier to specify a list of commands via javascript. 
+- clean up my spaghetti code into nice straight rows of pasta.

CONTRIBUTING.md

@@ -2,96 +2,3 @@
 
 Contributors need to sign the TiddlyWiki CLA to, you know, contribute. \
 To sign the CLA, and understand what it means, [go here.](https://github.com/TiddlyWiki/TiddlyWiki5/blob/master/contributing.md#contributor-license-agreement)
-
-# MWS Server Architecture Explained
-
-This is a monorepo divided into separate projects. The entry point is in `packages/mws/src/index.ts`. The entry point imports the `server`, `commander`, and `events` projects. 
-
-The entire server uses Promises and async functions for pretty much everything. Synchronous file system calls should be avoided as much as possible. 
-
-The `events` project is the foundation of MWS. It contains an `EventEmitter` instance that everything else subscribes to. 
-
-Events are emitted asyncly. Event listeners are awaited with `Promise.all`, and rejections throw back to the event emit call. This is intentional because it is the heart of the entire server, not just a public event bus, and errors shouldn't be ignored. Errors that come from things like attempting to send SSE events to other clients, however, should not be thrown because they aren't relevent to the source of the event. 
-
-There should be no singleton references other than the event emitter and event handlers should be as pure as possible, so that in theory it would be possible to run multiple completely separate MWS servers in the same process.
-
-The `commander` project handles the CLI parsing code. It exports a default function which MWS calls to execute the CLI. Commander emits several events during execution, which MWS hooks into to scaffold the rest of the server. 
-
-It also exports the base class for commands to inherit from, and because it instantiates the commands, commands should not specify their own constructor. Additional instance properties may be added to commands in the `cli.execute.before` event. 
-
-The `server` project handles all web related stuff, but in an MWS-agnostic way. It only requires the `events` project, so it could be used as the foundation for unrelated webservers. Its internal API is directly inspired by the TiddlyWiki server API, with routing and centralized handling of the request body. 
-
-The `mws` project is the application layer of MWS and ties everything else together. It defines all the commands and web server routes and handles the database connection. 
-
-The `react-admin` package is the Admin UI. It's based on react, and is built automatically when the dev server is enabled. You can find the server route in `packages/mws/src/services/setupDevServer.ts`
-
-If this all sounds a bit confusing, here's all the events that are emitted on a normal startup. 
-
-- `zod.make`  - Emitted by `server` to extend the zod global.
-- `cli.register`  - Emitted by `commander` to register commands.
-- `cli.commander` - Emitted by `commander` after routes are registered, but before help is printed. 
-- `cli.execute.before` - Emitted by `commander` right before the command is executed. This allows adding properties to the command instance for use later in the application. 
-
-These next ones are attachment points for further extension of MWS. They are emitted in the main `cli.execute.before` listener that MWS attaches.  
-
-- `mws.cache.init.before`
-- `mws.cache.init.after`
-- `mws.adapter.init.before`
-- `mws.adapter.init.after`
-- `mws.init.before`
-- `mws.init.after`
-
-These are emitted by the listen command, either by `mws` or by `server`. 
-
-- `listen.router.init`
-- `mws.router.init`
-- `mws.routes.important`
-- `mws.routes`
-- `mws.routes.fallback`
-
-And finally the event emitted after the command ends. 
-
-- `cli.execute.after`
-
-Remember that this is the order the commands start executing in. Some of them are nested. For instance, if I log the event name when it is emitted and time it, this is the output.
-
-
-```
-zod.make
-zod.make: 0.123ms
-
-cli.register
-cli.register: 0.088ms
-
-cli.commander
-cli.commander: 0.019ms
-
-cli.execute.before
-  mws.cache.init.before
-  mws.cache.init.before: 0.014ms
-  mws.cache.init.after
-  mws.cache.init.after: 0.013ms
-  mws.adapter.init.before
-  mws.adapter.init.before: 0.007ms
-  mws.adapter.init.after
-  mws.adapter.init.after: 0.006ms
-  mws.init.before
-  mws.init.before: 0.008ms
-  mws.init.after
-  mws.init.after: 0.009ms
-cli.execute.before: 1.171s
-
-listen.router.init
-  mws.router.init
-  mws.router.init: 0.012ms
-  mws.routes.important
-  mws.routes.important: 0.006ms
-  mws.routes
-  mws.routes: 2.754ms
-  mws.routes.fallback
-  mws.routes.fallback: 0.054ms
-listen.router.init: 4.002ms
-
-cli.execute.after
-cli.execute.after: 0.015ms
-```

packages/commander/src/runCLI.ts

@@ -52,10 +52,11 @@ export async function runCLI() {
   // parse the CLI options
   const { params, options } = parseCLI(commander, cmd, process.argv.slice(3));
   const inst = new cmdDef.Command(params, options);
+  serverEvents.logBefore("Command: " + cmd);
   await serverEvents.emitAsync("cli.execute.before", cmd, params, options, inst);
   await inst.execute();
   await serverEvents.emitAsync("cli.execute.after", cmd, params, options, inst);
-
+  serverEvents.logAfter("Command: " + cmd);
 }
 
 

packages/events/src/index.ts

@@ -15,28 +15,61 @@ export interface ServerEventsMap {
 }
 // the node implementation handles once in the once handler, not in emit
 export class ServerEvents extends EventEmitter<ServerEventsMap> {
+  activeCount = 0;
+  activeName = "";
+  // events aren't supposed to be running on the first evaluation 
+  // so this stays true until the proper code changes it.
+  eventLogging = true;
+  logBefore(eventName: string) {
+    if (!this.eventLogging) return;
+    if (this.activeName)
+      console.log(
+        "├ ".repeat(this.activeCount)
+        + String(this.activeName)
+        + ":", "--------"
+      );
+    this.activeName = String(eventName);
+    this.activeCount++;
+    console.time("├ ".repeat(this.activeCount) + String(eventName));
+  }
+  logAfter(eventName: string) {
+    if (!this.eventLogging) return;
+    console.timeEnd("├ ".repeat(this.activeCount) + String(eventName));
+    this.activeCount--;
+    this.activeName = "";
+  }
+  logLine(text: string) {
+    if (!this.eventLogging) return;
+    console.log("├ ".repeat(this.activeCount + 1) + text)
+  }
   /** Use emitAsync instead */
   override emit!: never;
   /** Call all listeners in sequence sync'ly */
   emitSync<K>(
     eventName: keyof ServerEventsMap | K,
     ...args: K extends keyof ServerEventsMap ? ServerEventsMap[K] : never
   ) {
+    this.logBefore(String(eventName));
     this.listeners(eventName).map(e => e(...args));
+    this.logAfter(String(eventName));
   }
   /** Call all listeners via Promise.all. */
   async emitAsync<K>(
     eventName: keyof ServerEventsMap | K,
     ...args: K extends keyof ServerEventsMap ? ServerEventsMap[K] : never
   ) {
-    await Promise.all(this.listeners(eventName).map(e => e(...args)));
+    this.logBefore(String(eventName));
+    for (const e of this.listeners(eventName)) { await e(...args) };
+    this.logAfter(String(eventName));
   }
   /** Call all listeners sync'ly, await them, catch errors per listener and forward them to emitLogCatcher.  */
   async emitLog<K>(
     eventName: keyof ServerEventsMap | K,
     ...args: K extends keyof ServerEventsMap ? ServerEventsMap[K] : never
   ) {
+    this.logBefore(String(eventName));
     this.listeners(eventName).map(async e => { try { await e(...args) } catch (e) { this.emitLogCatcher?.(e); } })
+    this.logAfter(String(eventName));
   }
 
   emitLogCatcher?: (error: unknown) => void;

packages/mws/src/index.ts

@@ -1,15 +1,15 @@
-// import "./globals";
 // const originalconsole = console.log.bind(console);
-// console.log = (...args) => { originalconsole(new Error("console stack").stack); return originalconsole(...args) };
+// console.log = (...args) => {
+//   originalconsole(new Error("console stack").stack);
+//   return originalconsole(...args)
+// };
 import { install } from "source-map-support";
 install();
 import { serverEvents } from "@tiddlywiki/events";
 
-// these all use serverEvents
 import "@tiddlywiki/commander";
 import "@tiddlywiki/server";
 import "./startup";
-// import "./registerStartup";
 import "./new-commands";
 import "./new-managers";
 import "./zodAssert";
@@ -29,12 +29,26 @@ import { clientBuildDef } from "./startup";
 
 export * from "@tiddlywiki/server";
 export * from "@tiddlywiki/events";
-export { SessionManager, SessionManagerObject, AuthUser } from "./services/sessions";
-export { PasswordService } from "./services/PasswordService";
-export { WikiPluginCache, PluginDefinition, TiddlerHasher, defaultPreloadFunction } from "./services/cache";
+export {
+  SessionManager,
+  SessionManagerObject,
+  AuthUser
+} from "./services/sessions";
+export {
+  PasswordService
+} from "./services/PasswordService";
+export {
+  WikiPluginCache,
+  PluginDefinition,
+  TiddlerHasher,
+  defaultPreloadFunction,
+} from "./services/cache";
 export * from "./services/setupDevServer";
 export * from "./services/tiddlywiki";
-export { mountTW5Route, TW5Route } from "./services/tw-routes";
+export {
+  mountTW5Route,
+  TW5Route
+} from "./services/tw-routes";
 export * from "./new-commands";
 export * from "./new-managers";
 
@@ -64,8 +78,10 @@ export default async function runMWS(oldOptions?: any) {
   if (process.env.CLIENT_BUILD) {
     await runBuildOnce(clientBuildDef);
   } else {
+    serverEvents.eventLogging = !!process.env.VERBOSE;
     await startup();
     await runCLI();
+    serverEvents.eventLogging = false;
   }
 
 }

packages/mws/src/new-managers/NEW-DESIGN-V2.md

@@ -218,6 +218,7 @@ Permissions are role-based.
 ### Levels
 
 - Bag levels: `A_read`, `B_write`, `C_admin`
+- Template levels: `A_read`, `B_write`
 - Recipe levels: `A_read`, `B_write`
 
 The prefixed enum ordering is intentional. Descending lexical order yields the highest held privilege.