Remove retired earmark and unused phoenix_storybook dependencies (#4886)

* Migrate off retired earmark markdown dependency (#4878)

Replace the direct earmark dependency with mdex for rendering AI
assistant message content. earmark is retired and unmaintained;
mix hex.audit flags it.

mdex node structs cannot hold arbitrary HTML attributes, so per-element
Tailwind classes are now injected into the rendered HTML instead of via
AST manipulation. Raw HTML passthrough and the raw-content fallback are
preserved, and the language- prefix on code blocks is stripped to keep
earmark's previous output.

earmark remains a dev-only transitive dependency of phoenix_storybook
0.9.2 until that is upgraded to 1.2+ (which requires Phoenix 1.8).

* Remove unused phoenix_storybook dependency (#4846)

phoenix_storybook 0.9.2 carried three open advisories flagged by
mix deps.audit (critical RCE, atom-table DoS, PubSub topic injection).
It was dev-only with no production exposure, and storybook was dormant:
only three demo stories existed and nothing had changed since the 2023
prototype. Remove the package rather than upgrade to 1.1.0.

Drop the dep and its dev routes, delete the backend module, stories, and
assets, and strip the esbuild/tailwind/live-reload config. Clear the
storybook advisory IDs from the CI deps.audit ignore list, leaving only
the cowlib advisory (no patch available yet). Remove now-orphaned lock
entries via mix deps.unlock --unused.

* update changelog

* Keep MDEx's native language- class on code blocks

MDEx emits the standard `<code class="language-javascript">` for fenced
code blocks. We were rewriting it back to Earmark's old bare
`<code class="javascript">`, but nothing consumes that class (no syntax
highlighter is wired up, and the markdown CSS targets `pre code` by
element), so keep MDEx's conventional output instead.

* Enable GFM extensions for MDEx markdown rendering

The earmark→mdex migration left MDEx running as plain CommonMark, which
silently dropped the GFM features earmark rendered by default. Assistant
replies containing markdown tables showed as raw pipe text, strikethrough
and bare-URL autolinks stopped working.

Enable the table, strikethrough, autolink, and tasklist extensions to
restore the prior rendering, and add a regression test covering them.

Author: midigofrank

.circleci/config.yml

@@ -188,11 +188,8 @@ jobs:
           command: |
             # GHSA-g2wm-735q-3f56: cowlib cookie encoder CRLF injection (low);
             #   no patched release available yet.
-            # GHSA-55hg-8qxv-qj4p / GHSA-833p-95jq-929q / GHSA-mrhx-6pw9-q5fh:
-            #   phoenix_storybook advisories. Dev-only dep, not compiled into
-            #   prod and storybook routes are dev-only. Tracked for removal.
             sudo -u lightning mix deps.audit \
-              --ignore-advisory-ids GHSA-g2wm-735q-3f56,GHSA-55hg-8qxv-qj4p,GHSA-833p-95jq-929q,GHSA-mrhx-6pw9-q5fh \
+              --ignore-advisory-ids GHSA-g2wm-735q-3f56 \
               || echo "deps.audit" >> /tmp/lint_failed
       - run:
           name: "Check for retired Hex packages"

.formatter.exs

@@ -4,8 +4,7 @@
   inputs: [
     "*.{heex,ex,exs}",
     "priv/*/seeds.exs",
-    "{config,lib,test}/**/*.{heex,ex,exs}",
-    "storybook/**/*.exs"
+    "{config,lib,test}/**/*.{heex,ex,exs}"
   ],
   subdirectories: ["priv/*/migrations"],
   line_length: 81

CHANGELOG.md

@@ -25,6 +25,11 @@ and this project adheres to
 
 ### Changed
 
+- Migrated off the retired `earmark` markdown dependency in favour of `mdex`.
+  [#4878](https://github.com/OpenFn/lightning/issues/4878)
+- Removed the unused dev-only `phoenix_storybook` dependency, clearing its
+  advisories from the `mix deps.audit` ignore list.
+  [#4846](https://github.com/OpenFn/lightning/issues/4846)
 - Bump worker to 1.27.0
 
 ### Fixed

RUNNINGLOCAL.md

@@ -183,8 +183,8 @@ LOCAL_ADAPTORS=true mix phx.server
 ```
 
 Each path in `OPENFN_ADAPTORS_REPO` must contain a `packages` subdirectory.
-Paths that are missing or unreadable are logged and skipped, so the rest of
-the list still loads.
+Paths that are missing or unreadable are logged and skipped, so the rest of the
+list still loads.
 
 ### Problems with Apple Silicon
 

assets/css/storybook.css

@@ -17,7 +17,6 @@ ignore:
   - "lib/lightning/release.ex"
   - "lib/lightning/sentry_event_filter.ex"
   - "test/support/**"
-  - "storybook/**"
 
   # Phoenix declarative scaffolding — boot-only or pure DSL
   - "lib/lightning_web.ex"

assets/js/storybook.js

@@ -102,7 +102,6 @@ config :esbuild,
          --external:/fonts/*
          --external:/images/*
          js/app.js
-         js/storybook.js
          js/editor/Editor.tsx
          js/react/components/DataclipViewer.tsx
          js/react/components/CollectionPreviewViewer.tsx
@@ -145,13 +144,6 @@ config :tailwind,
       --output=priv/static/assets/app.css
     ),
     cd: Path.expand("..", __DIR__)
-  ],
-  storybook: [
-    args: ~w(
-      --input=assets/css/storybook.css
-      --output=priv/static/assets/storybook.css
-    ),
-    cd: Path.expand("..", __DIR__)
   ]
 
 # Configures Elixir's Logger

codecov.yml

@@ -33,8 +33,7 @@ config :lightning, LightningWeb.Endpoint,
     # Start the esbuild watcher by calling Esbuild.install_and_run(:default, args)
     esbuild:
       {Esbuild, :install_and_run, [:default, ~w(--sourcemap=inline --watch)]},
-    tailwind: {Tailwind, :install_and_run, [:default, ~w(--watch)]},
-    storybook_tailwind: {Tailwind, :install_and_run, [:storybook, ~w(--watch)]}
+    tailwind: {Tailwind, :install_and_run, [:default, ~w(--watch)]}
   ]
 
 config :lightning,
@@ -112,8 +111,7 @@ config :lightning, LightningWeb.Endpoint,
       ~r"priv/static/.*(js|css|png|jpeg|jpg|gif|svg)$",
       ~r"priv/gettext/.*(po)$",
       ~r"lib/lightning_web/(live|components|views)/.*(ex|heex)$",
-      ~r"lib/lightning_web/templates/.*(eex)$",
-      ~r"storybook/.*(exs)$"
+      ~r"lib/lightning_web/templates/.*(eex)$"
     ]
   ]
 

config/config.exs

@@ -4,7 +4,6 @@
     "lib/lightning/release.ex",
     "lib/lightning/sentry_event_filter.ex",
     "lib/mix/tasks/install_runtime.ex",
-    "test/support/*",
-    "storybook/*"
+    "test/support/*"
   ]
 }