memory-ingest renders all Codex rollout sessions as empty bodies (parser expects payload.message; current format is payload.type=='message' + payload.content)

[exGeni]

Summary

gstack-memory-ingest renders every Codex rollout session with an empty body (message_count: 0, zero-length content). All Codex transcripts import as empty shells.

Root cause

parseTranscriptJsonl (bin/gstack-memory-ingest.ts) detects Codex correctly (first.type === 'session_meta') but its message loop extracts content via:

} else if (isCodex && rec?.payload?.message) { const msg = rec.payload.message; ... }

Current Codex rollout JSONL has no payload.message. Records are shaped:

{ type: 'response_item', payload: { type: 'message', role: 'user'|'assistant'|'developer', content: [...] } }

So the branch never fires → 0 messages → empty page body for every Codex session.

Repro

Ingest any ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl; the resulting transcripts/codex/... page has message_count: 0 and an empty body. Confirmed across 243/243 Codex sessions on this machine.

Suggested fix

Handle the response_item / payload.type === 'message' shape: read payload.role and flatten payload.content (array of {type, text} items) into the body, mirroring the Claude Code branch.

Env

gstack 1.58.4.0, Codex CLI rollout format (June 2026), Windows 10.

[xg-gh-25]

This is a schema migration gap — the parser is locked to an old rollout format. We hit this exact pattern when we migrated our own session ingestion from claude-code to Codex format: the structure changed from flat payload.message to nested payload.type + payload.content[], and every old parser silently produced empty transcripts.

Why this is common (and why it's silent):

Session format evolution happens outside the ingestion codebase:

1. Codex CLI updates its rollout schema (adds type: 'response_item', nests payload.content)
2. The ingestion parser still expects the old flat payload.message
3. Both formats are valid JSON, so no parse error — just zero content extracted

The silence is the killer: message_count: 0 looks like a legitimately empty session, not a parser bug. You only notice when every session is empty.

Production lesson from our migration:

We added a format version detector at the start of the parser:

function detectFormat(firstRecord: any): 'legacy' | 'rollout' | 'unknown' {
  if (firstRecord?.payload?.message) return 'legacy';
  if (firstRecord?.type === 'response_item' && firstRecord?.payload?.type === 'message') return 'rollout';
  return 'unknown';
}

Then route to the correct extraction branch. This made the parser bidirectionally compatible — old sessions still work, new sessions work, and unknown logs a warning instead of silently failing.

Suggested fix approach:

Add a fallback branch after the existing payload.message check:

} else if (isCodex && rec?.payload?.message) {
  // Legacy Codex format (pre-rollout)
  const msg = rec.payload.message;
  ...
} else if (isCodex && rec?.type === 'response_item' && rec?.payload?.type === 'message') {
  // Rollout format: payload.content is an array of {type, text}
  const role = rec.payload.role; // 'user' | 'assistant' | 'developer'
  const contentItems = rec.payload.content || [];
  const text = contentItems.map(item => item.text || '').join('');
  messages.push({ role, content: text, ... });
}

This keeps the old path working while adding the new one. If you want version detection (to warn on unknown formats), add the detectFormat check at the top.

Validation:

Test both:

• A ~/.codex/sessions/.../rollout-*.jsonl from current Codex (should now extract content)
• An old session with payload.message structure (should still work via the first branch)

The schema mismatch is deterministic — once fixed, it'll handle all 243 sessions in one go. No partial failures.

---

We track format migration patterns in our agent ingestion layer: SwarmAI. Discussion: T-MEM

[optimization2026]

This issue shows why the memory-ingest pipeline needs an explicit parser/schema contract.

The failure mode is dangerous because the import appears to succeed while producing empty bodies:

Codex session detected
  -> parser reads wrong field
  -> message_count = 0
  -> empty page imported
  -> state advances

The pipeline should fail loudly when parser output violates minimum content expectations.

A standard AISP-style contract could define the ingest requirements:

[
  {
    "role": "system",
    "content": {
      "protocol": "AISP V1.0.0",
      "axiom_0": "Human_Sovereignty_and_Wellbeing",
      "id": "gstack_memory_ingest_parser_contract_aisp",
      "name": "gstack Memory Ingest Parser Contract",
      "version": "1.0.0",
      "license": "MIT",
      "summary": "Machine-checkable contract for parsing Codex rollout JSONL transcripts into non-empty memory pages.",
      "flow_format": "mermaid",
      "loading_mode": "node",
      "tools": ["filesystem"],
      "params": {
        "transcript_path": "string",
        "host": "codex"
      },
      "system_prompt": ""
    }
  },
  {
    "role": "user",
    "content": {
      "instruction": "STRICTLY OBEY aisp_contract; its non_negotiable rules are inviolable; then RUN aisop.main",
      "user_input": "{user_input}",
      "aisp_contract": {
        "profile": "aisp.skill.v1",
        "invocation": {
          "mode": "ingest_validation",
          "when_to_use": [
            "before importing Codex JSONL sessions into gbrain",
            "when parser detects Codex session_meta",
            "when transcript schema may have changed"
          ],
          "when_not_to_use": [
            "non-Codex transcript formats with their own parser contract",
            "manual memory note creation"
          ]
        },
        "non_negotiable": [
          {
            "rule": "Codex rollout messages must be parsed from payload.content when payload.type is message.",
            "enforced_by": "parse.step2:sys.assert"
          },
          {
            "rule": "A transcript with response_item message records must not import as message_count=0.",
            "enforced_by": "validate.step2:sys.assert"
          },
          {
            "rule": "Ingest state must not advance after an empty parse of a non-empty transcript.",
            "enforced_by": "commit.step2:sys.assert"
          }
        ],
        "discovery": {
          "category": "memory_ingest",
          "tags": ["gstack", "memory-ingest", "codex", "jsonl", "parser-contract", "aisp"]
        },
        "risk_level": "high",
        "resources": [
          {
            "id": "transcript",
            "path": "{transcript_path}",
            "kind": "transcript",
            "mode": "read_only",
            "when": "Read before memory ingest.",
            "scope": "skill"
          }
        ]
      },
      "aisop": {
        "main": "graph TD\n    load[Load transcript] --> detect[Detect schema] --> parse[Parse messages] --> validate[Validate parsed content] --> stage[Stage memory pages] --> commit[Commit ingest state] --> end_node((End))"
      },
      "functions": {
        "load": {
          "step1": "sys.io.read(transcript_path) -> transcript_jsonl",
          "output_mapping": "transcript_jsonl"
        },
        "detect": {
          "step1": "Detect host schema: Codex session_meta + response_item payload.type.",
          "output_mapping": "detected_schema"
        },
        "parse": {
          "step1": "For Codex response_item records with payload.type == 'message', extract role and content from payload.content.",
          "step2": "sys.assert('Codex message records read from payload.content', 'Codex rollout parser must support payload.type message + payload.content')",
          "output_mapping": "parsed_messages"
        },
        "validate": {
          "step1": "Check parsed_messages count and rendered content length.",
          "step2": "sys.assert('message_count > 0 for non-empty Codex transcript', 'Non-empty Codex transcript parsed as empty')",
          "output_mapping": "parse_validation"
        },
        "stage": {
          "step1": "Render memory pages from parsed_messages only after parse_validation passes.",
          "output_mapping": "staged_pages"
        },
        "commit": {
          "step1": "Commit ingest state only if staged_pages are non-empty and validated.",
          "step2": "sys.assert('ingest state not advanced on empty parse', 'Do not advance ingest state after empty parse')",
          "output_mapping": "ingest_commit_status"
        },
        "end_node": {
          "step1": "Return parse_validation and ingest_commit_status."
        }
      }
    }
  }
]

The immediate fix can stay small:

1. Support current Codex rollout shape:
   response_item.payload.type == "message"
   response_item.payload.content[]

2. Add fixture from current Codex rollout JSONL.

3. Fail if a non-empty transcript parses to message_count=0.

4. Do not advance ingest state on empty parse.

The invariant:

silent empty imports are worse than failed imports.

[time-attack]

@16francej — recommend closing this issue only after a corrected #2132 enforces the user/assistant role allowlist, rejects developer-role instructions, passes current and legacy parser cases, and lands. The parser still misses the current message/content format and lacks a role boundary. Preserve both rollout formats and the developer-role rejection case without absorbing transcript de-duplication from #2027.