feat: expand cli compatibility copy and ai crawler directives

Author: web3dev1337

README.md

@@ -67,8 +67,11 @@ Agent Workspace wraps your preferred terminal tools and runs them side by side w
 - Codex CLI
 - Gemini CLI
 - OpenCode
+- Grok CLI
 - Any CLI agent that runs in a terminal
 
+Compatibility rule: if it can be launched from a terminal command in your worktree, Agent Workspace can run it.
+
 **Works with:** Claude Max, Codex plans, or your own API keys.
 
 ## Features

site/index.html

@@ -138,6 +138,12 @@ <h1 class="mega-title">
       <div class="integration-content reveal">
         <h2 class="section-title">A GUI for your TUIs.</h2>
         <p class="section-desc">Bring your CLIs. <span class="text-cyan">Agent Workspace</span> wraps your preferred terminal tools &mdash; run them side by side in the same workspace with your existing local auth and provider accounts.</p>
+        <p class="section-desc"><strong>Compatibility rule:</strong> if a tool can be launched from a terminal command in your worktree, it can run in Agent Workspace.</p>
+        <ul class="cli-compat-list">
+          <li>Common examples: Claude Code, Codex CLI, Gemini CLI, OpenCode, Grok CLI.</li>
+          <li>Also works with command-driven CLIs and scripts (for example Aider, Goose, or custom repo scripts).</li>
+          <li>If a tool starts with a command, Agent Workspace can host it in a terminal pane.</li>
+        </ul>
 
         <div class="cli-grid">
           <div class="cli-card">

site/robots.txt

@@ -4,6 +4,24 @@ Allow: /
 User-agent: GPTBot
 Allow: /
 
+User-agent: Google-Extended
+Allow: /
+
+User-agent: ClaudeBot
+Allow: /
+
+User-agent: Claude-User
+Allow: /
+
+User-agent: Claude-SearchBot
+Allow: /
+
+User-agent: PerplexityBot
+Allow: /
+
+User-agent: Perplexity-User
+Allow: /
+
 User-agent: *
 Allow: /
 

site/styles.css

@@ -1042,6 +1042,28 @@ a.btn-glow {
   margin: 0 auto 2.5rem;
 }
 
+.cli-compat-list {
+  max-width: 860px;
+  margin: -1.25rem auto 2.5rem;
+  padding: 1rem 1.25rem;
+  list-style: none;
+  text-align: left;
+  background: rgba(7, 15, 33, 0.55);
+  border: 1px solid rgba(255, 255, 255, 0.1);
+  border-radius: 12px;
+}
+
+.cli-compat-list li {
+  color: var(--text-muted);
+  margin: 0.4rem 0;
+  line-height: 1.5;
+}
+
+.cli-compat-list li::before {
+  content: "• ";
+  color: var(--cyan);
+}
+
 .logo-showcase {
   display: flex;
   align-items: center;