RAM Consumption Verification

[Created by Claude: ab7e3fa0-6c2e-49c0-a55f-21779f70b64a 2026-01-25]

Purpose

Verify that a process's self-reported RAM consumption is accurate by:

1. Measuring total system RAM with htop
2. Killing the target process(es)
3. Measuring total system RAM again
4. Comparing the difference

Prerequisites

• htop installed
• tmux installed
• Agent must know their suffix(5) (last 5 chars of Agent ID)

⚠️ CRITICAL: Always Clean Up

You MUST clean up the tmux session after the test completes.

The tmux session is ONLY for running htop temporarily. Failure to clean up leaves orphaned sessions.

Step-by-Step Procedure

Step 1: Create tmux session with htop

Use your suffix(5) to name the session:

tmux new-session -d -s ram-test-{suffix5} && \
tmux send-keys -t ram-test-{suffix5} 'htop' Enter

Example for agent with suffix 0b64a:

tmux new-session -d -s ram-test-0b64a && \
tmux send-keys -t ram-test-0b64a 'htop' Enter

Step 2: Wait and capture BEFORE measurement

sleep 2 && tmux capture-pane -t ram-test-{suffix5} -p | grep "^  Mem"

Record this value (e.g., 53.8G/128G)

Step 3: Kill target process(es)

Use pkill -9 for forceful termination:

pkill -9 -f "{process-pattern}"

Examples:

# Kill ingestors
pkill -9 -f "ingest-to-centralized-db"

# Kill a specific python script
pkill -9 -f "my_script.py"

# Kill by PID
kill -9 {pid1} {pid2}

Step 4: Wait and capture AFTER measurement

sleep 1 && tmux capture-pane -t ram-test-{suffix5} -p | grep "^  Mem"

Record this value (e.g., 43.2G/128G)

Step 5: Verify processes are dead

pgrep -f "{process-pattern}" || echo "Processes confirmed dead"

Step 6: Calculate and report

RAM Before: 53.8 GB
RAM After:  43.2 GB
Difference: 10.6 GB (this is the actual RAM used by killed processes)

Step 7: ⚠️ CLEANUP (MANDATORY)

Always run this step, even if earlier steps failed:

tmux kill-session -t ram-test-{suffix5}

Complete Example Script

For an agent with suffix e274f verifying ingestor RAM:

# Step 1: Create session
tmux new-session -d -s ram-test-e274f && \
tmux send-keys -t ram-test-e274f 'htop' Enter

# Step 2: Measure BEFORE
sleep 2 && tmux capture-pane -t ram-test-e274f -p | grep "^  Mem"
# Output: Mem[||||||||||||||||||||||||||||||||||||53.8G/128G]

# Step 3: Kill processes
pkill -9 -f "ingest-to-centralized-db"

# Step 4: Measure AFTER
sleep 1 && tmux capture-pane -t ram-test-e274f -p | grep "^  Mem"
# Output: Mem[||||||||||||||||||||||||||||||||||||43.2G/128G]

# Step 5: Verify dead
pgrep -f "ingest-to-centralized-db" || echo "Ingestors confirmed dead"

# Step 6: Report
echo "RAM dropped from 53.8GB to 43.2GB = 10.6GB used by ingestors"

# Step 7: CLEANUP
tmux kill-session -t ram-test-e274f

Interpreting Results

Why htop Instead of ps/top?

• ps aux RSS values can be misleading (shared memory, copy-on-write)
• htop shows real-time system-wide memory that the kernel reports
• Measuring before/after gives ground truth of actual memory freed

Troubleshooting

htop not showing in capture

# Increase sleep time
sleep 3 && tmux capture-pane -t ram-test-{suffix5} -p | grep "^  Mem"

Session already exists

# Kill existing and recreate
tmux kill-session -t ram-test-{suffix5} 2>/dev/null
tmux new-session -d -s ram-test-{suffix5}

Forgot to cleanup

# List all ram-test sessions
tmux ls | grep ram-test

# Kill all ram-test sessions
tmux ls | grep ram-test | cut -d: -f1 | xargs -I{} tmux kill-session -t {}