# Async Exec Duplicate Completion Investigation

## Scope

- Session: `agent:main:telegram:group:-1003774691294:topic:1`
- Symptom: the same async exec completion for session/run `keen-nexus` was recorded twice in LCM as user turns.
- Goal: identify whether this is most likely duplicate session injection or plain outbound delivery retry.

## Conclusion

Most likely this is **duplicate session injection**, not a pure outbound delivery retry.

The strongest gateway-side gap is in the **node exec completion path**:

1. A node-side exec finish emits `exec.finished` with the full `runId`.
2. Gateway `server-node-events` converts that into a system event and requests a heartbeat.
3. The heartbeat run injects the drained system event block into the agent prompt.
4. The embedded runner persists that prompt as a new user turn in the session transcript.

If the same `exec.finished` reaches the gateway twice for the same `runId` for any reason (replay, reconnect duplicate, upstream resend, duplicated producer), OpenClaw currently has **no idempotency check keyed by `runId`/`contextKey`** on this path. The second copy will become a second user message with the same content.

## Exact Code Path

### 1. Producer: node exec completion event

- `src/node-host/invoke.ts:340-360`
  - `sendExecFinishedEvent(...)` emits `node.event` with event `exec.finished`.
  - Payload includes `sessionKey` and full `runId`.

### 2. Gateway event ingestion

- `src/gateway/server-node-events.ts:574-640`
  - Handles `exec.finished`.
  - Builds text:
    - `Exec finished (node=..., id=<runId>, code ...)`
  - Enqueues it via:
    - `enqueueSystemEvent(text, { sessionKey, contextKey: runId ? \`exec:${runId}\` : "exec", trusted: false })`
  - Immediately requests a wake:
    - `requestHeartbeatNow(scopedHeartbeatWakeOptions(sessionKey, { reason: "exec-event" }))`

### 3. System event dedupe weakness

- `src/infra/system-events.ts:90-115`
  - `enqueueSystemEvent(...)` only suppresses **consecutive duplicate text**:
    - `if (entry.lastText === cleaned) return false`
  - It stores `contextKey`, but does **not** use `contextKey` for idempotency.
  - After drain, duplicate suppression resets.

This means a replayed `exec.finished` with the same `runId` can be accepted again later, even though the code already had a stable idempotency candidate (`exec:<runId>`).

### 4. Wake handling is not the primary duplicator

- `src/infra/heartbeat-wake.ts:79-117`
  - Wakes are coalesced by `(agentId, sessionKey)`.
  - Duplicate wake requests for the same target collapse to one pending wake entry.

This makes **duplicate wake handling alone** a weaker explanation than duplicate event ingestion.

### 5. Heartbeat consumes the event and turns it into prompt input

- `src/infra/heartbeat-runner.ts:535-574`
  - Preflight peeks pending system events and classifies exec-event runs.
- `src/auto-reply/reply/session-system-events.ts:86-90`
  - `drainFormattedSystemEvents(...)` drains the queue for the session.
- `src/auto-reply/reply/get-reply-run.ts:400-427`
  - The drained system event block is prepended into the agent prompt body.

### 6. Transcript injection point

- `src/agents/pi-embedded-runner/run/attempt.ts:2000-2017`
  - `activeSession.prompt(effectivePrompt)` submits the full prompt to the embedded PI session.
  - That is the point where the completion-derived prompt becomes a persisted user turn.

So once the same system event is rebuilt into the prompt twice, duplicate LCM user messages are expected.

## Why plain outbound delivery retry is less likely

There is a real outbound failure path in the heartbeat runner:

- `src/infra/heartbeat-runner.ts:1194-1242`
  - The reply is generated first.
  - Outbound delivery happens later via `deliverOutboundPayloads(...)`.
  - Failure there returns `{ status: "failed" }`.

However, for the same system event queue entry, this alone is **not sufficient** to explain the duplicate user turns:

- `src/auto-reply/reply/session-system-events.ts:86-90`
  - The system event queue is already drained before outbound delivery.

So a channel send retry by itself would not recreate the exact same queued event. It could explain missing/failed external delivery, but not by itself a second identical session user message.

## Secondary, lower-confidence possibility

There is a full-run retry loop in the agent runner:

- `src/auto-reply/reply/agent-runner-execution.ts:741-1473`
  - Certain transient failures can retry the whole run and resubmit the same `commandBody`.

That can duplicate a persisted user prompt **within the same reply execution** if the prompt was already appended before the retry condition triggered.

I rank this lower than duplicate `exec.finished` ingestion because:

- the observed gap was around 51 seconds, which looks more like a second wake/turn than an in-process retry;
- the report already mentions repeated message send failures, which points more toward a separate later turn than an immediate model/runtime retry.

## Root Cause Hypothesis

Highest-confidence hypothesis:

- The `keen-nexus` completion came through the **node exec event path**.
- The same `exec.finished` was delivered to `server-node-events` twice.
- Gateway accepted both because `enqueueSystemEvent(...)` does not dedupe by `contextKey` / `runId`.
- Each accepted event triggered a heartbeat and was injected as a user turn into the PI transcript.

## Proposed Tiny Surgical Fix

If a fix is wanted, the smallest high-value change is:

- make exec/system-event idempotency honor `contextKey` for a short horizon, at least for exact `(sessionKey, contextKey, text)` repeats;
- or add a dedicated dedupe in `server-node-events` for `exec.finished` keyed by `(sessionKey, runId, event kind)`.

That would directly block replayed `exec.finished` duplicates before they become session turns.