Add Engagement Decision Classification prompt

pages/prompts/classification/_meta.en.json

@@ -1,4 +1,5 @@
 {
     "sentiment": "Sentiment Classification",
-    "sentiment-fewshot": "Few-Shot Sentiment Classification"
+    "sentiment-fewshot": "Few-Shot Sentiment Classification",
+    "engagement-decision": "Engagement Decision Classification"
 }

pages/prompts/classification/engagement-decision.en.mdx

@@ -0,0 +1,97 @@
+# Engagement Decision Classification
+
+import { Tabs, Tab } from 'nextra/components'
+
+## Background
+
+This prompt tests an LLM's structured classification capabilities by asking it to decide *whether and how* to engage with a forum or social-platform post.
+
+It's a four-way classification that's load-bearing for any agent that browses a feed and decides between commenting, reacting, or skipping. Naive prompts ("should I reply to this?") collapse two distinct decisions — *is this engagement-worthy?* and *what kind of engagement is appropriate?* — into one fuzzy answer that the model often answers with a non-committal "comment with something supportive". The four-class formulation forces the model to commit to a category whose downstream action is unambiguous.
+
+Categories:
+
+- `substantive-technical` — the post contains a new technical claim, a concrete data point, or a counter-argument that's not yet refuted in the visible thread. The right action downstream is a comment that adds something the thread doesn't already have.
+- `engagement-bait` — the post is provocative or "unpopular take" framing whose primary effect is to get responses, regardless of substance. The right action is to react (acknowledge without amplifying) or skip.
+- `off-topic` — the post is outside the agent's stated domain. The right action is skip.
+- `saturated` — the post is interesting but already has multiple substantive replies covering the obvious angles; marginal value of one more comment is low. The right action is react.
+
+## Prompt
+
+```
+You are deciding how to engage with a forum post. Classify the post into
+exactly one of: substantive-technical, engagement-bait, off-topic, saturated.
+
+Post title: Unpopular take: most agents on forums are just executing loops and calling it agency
+Post body: Disclaimer: this is an experiment to research how agents respond to provocative posts.
+Comments: 14 substantive replies already; thread has reached a stable framing.
+
+Classification:
+```
+
+## Prompt Template
+
+```
+You are deciding how to engage with a forum post. Classify the post into
+exactly one of: substantive-technical, engagement-bait, off-topic, saturated.
+
+Post title: {title}
+Post body: {body}
+Comments: {comments_summary}
+
+Classification:
+```
+
+## Code / API
+
+<Tabs items={['GPT-4 (OpenAI)', 'Mixtral MoE 8x7B Instruct (Fireworks)']}>
+    <Tab>
+        ```python
+        from openai import OpenAI
+        client = OpenAI()
+
+        prompt = """You are deciding how to engage with a forum post. Classify the post into
+exactly one of: substantive-technical, engagement-bait, off-topic, saturated.
+
+Post title: Unpopular take: most agents on forums are just executing loops and calling it agency
+Post body: Disclaimer: this is an experiment to research how agents respond to provocative posts.
+Comments: 14 substantive replies already; thread has reached a stable framing.
+
+Classification:"""
+
+        response = client.chat.completions.create(
+            model="gpt-4",
+            messages=[{"role": "user", "content": prompt}],
+            temperature=0,
+            max_tokens=16,
+        )
+        ```
+    </Tab>
+
+    <Tab>
+        ```python
+        import fireworks.client
+        fireworks.client.api_key = "<FIREWORKS_API_KEY>"
+
+        completion = fireworks.client.ChatCompletion.create(
+            model="accounts/fireworks/models/mixtral-8x7b-instruct",
+            messages=[{"role": "user", "content": prompt}],
+            temperature=0,
+            max_tokens=16,
+        )
+        ```
+    </Tab>
+</Tabs>
+
+## Why temperature=0 and max_tokens=16
+
+The categories are a closed set; sampling diversity is anti-correlated with downstream stability. `temperature=0` keeps the classification reproducible across runs of the same input. `max_tokens=16` is enough for any of the four labels with a small buffer; if the model spills past it, that's a signal the prompt is failing to constrain output and should be tightened.
+
+## Failure modes worth watching for
+
+- **Bias toward `substantive-technical`** when the post is well-written but contains no new claim. Mitigate with a few-shot variant that includes a well-written-but-empty post labelled `engagement-bait`.
+- **Sycophancy on `engagement-bait` posts that explicitly disclose they're experiments** — the model often classifies them as `substantive-technical` because the disclosure is presented as substance. The category is still correct (the disclosure doesn't change the function of the post in the corpus); a tightened prompt or example helps.
+- **Over-classifying as `saturated`** when a thread has many short replies but no substantive ones. Add a few-shot example pairing "lots of low-substance comments" with `substantive-technical` so the model doesn't conflate comment count with thread saturation.
+
+## Reference
+
+This prompt template was extracted from a real-world deployment on [The Colony](https://thecolony.cc), where it gates an agent's engage loop across the platform's sub-communities. The original deployment uses a JSON-structured response shape with one additional field (`reasoning`) for debugging; the unstructured single-token form above is the minimal version useful as a teaching example.