Conversation Metrics

View source on GitHub

CLI tool to retrieve cost and token usage metrics for OpenHands conversations. Supports both V0 and V1 APIs, automatically selecting the appropriate one based on the conversation version.

Features

Auto-detects conversation version (V0 vs V1) and uses the appropriate API
Graceful fallback chain for metric retrieval
Displays cost (USD), token counts, cache stats, and context window
JSON output for programmatic use
API call logging for debugging and development
Fixture-based testing - high coverage via recorded API responses
Zero dependencies - uses only Python standard library

How It Works

The tool uses a fallback chain to find metrics:

Check conversation version via /api/conversations/{id}
For V1 conversations:
- First try /api/v1/app-conversations?ids={id} which includes a metrics object
- If metrics are all zeros, fall back to /api/v1/conversation/{id}/events/search and extract metrics from ConversationStateUpdateEvent at value.stats.usage_to_metrics.agent
For V0 conversations (or if V1 fails): Use /api/conversations/{id}/events and find the latest event with llm_metrics
Last resort: Use /api/conversations/{id}/trajectory and scan for llm_metrics

Note: Some V1 conversations have metrics stored only in events (not in the app-conversations response). The fallback chain ensures these are still retrieved correctly.

Installation

No installation required - just make the script executable:

chmod +x oh-metrics

Requires Python 3.10+.

Run It

Set your API key

export OH_API_KEY="your-api-key"

Get metrics for a conversation

./oh-metrics <conversation_id>

Example output (V0 conversation):

────────────────────────────────────────────────────────────
Conversation: 7f3d57c4e5b2434d9ca78e5e27311137
Title: Hello and a Programming Joke
API Version: V0 via V0 (events)
────────────────────────────────────────────────────────────
💰 Total Cost: $0.083175 USD

📊 Token Usage:
   Prompt tokens:     12,865
   Completion tokens: 117
   Total tokens:      12,982

🗄️  Cache:
   Cache read:        0
   Cache write:       12,740

📐 Context window:    200,000
────────────────────────────────────────────────────────────

Example output (V1 conversation):

────────────────────────────────────────────────────────────
Conversation: 72d40619b8534f9b9de6c3f17a71072d
Title: 📝 V0 API for Conversation Costs & Tokens
API Version: V1 via V1 (events)
────────────────────────────────────────────────────────────
💰 Total Cost: $13.576675 USD

📊 Token Usage:
   Prompt tokens:     17,577,146
   Completion tokens: 63,985
   Total tokens:      17,641,131

🗄️  Cache:
   Cache read:        17,022,477
   Cache write:       553,973

🧠 Reasoning tokens:  1,169

📐 Context window:    200,000
────────────────────────────────────────────────────────────

JSON output

./oh-metrics <conversation_id> --json

{
  "conversation_id": "7f3d57c4e5b2434d9ca78e5e27311137",
  "title": "Hello and a Programming Joke",
  "api_version": "V0",
  "api_used": "V0 (events)",
  "metrics": {
    "accumulated_cost": 0.083175,
    "accumulated_token_usage": {
      "prompt_tokens": 12865,
      "completion_tokens": 117,
      "cache_read_tokens": 0,
      "cache_write_tokens": 12740,
      "context_window": 200000
    }
  }
}

Options

Option	Short	Description
`--api-key KEY`	`-k`	API key (defaults to `OH_API_KEY` env var)
`--base-url URL`	`-u`	Base URL (default: `https://app.all-hands.dev`)
`--json`	`-j`	Output in JSON format
`--log-api-calls`	`-l`	Log all API requests/responses to `.oh/api-logs/`
`--help`	`-h`	Show help message

Logging API Calls

For debugging or development, you can log all API requests and responses:

./oh-metrics <conversation_id> --log-api-calls

This creates timestamped files in .oh/api-logs/YYYYMMDD-HHMMSS/:

.oh/api-logs/20260227-145230/
├── 0001-request.json
├── 0001-response.json
├── 0002-request.json
└── 0002-response.json

Each request/response pair is numbered sequentially. The Authorization header is redacted from logged requests.

Metrics Explained

Field	Description
`accumulated_cost`	Total cost in USD
`prompt_tokens`	Tokens sent to the LLM
`completion_tokens`	Tokens generated by the LLM
`cache_read_tokens`	Tokens read from prompt cache
`cache_write_tokens`	Tokens written to prompt cache
`reasoning_tokens`	Tokens used for reasoning (if applicable)
`context_window`	Model’s context window size

Architecture

The library is organized into separate modules:

oh_api/
├── __init__.py    # Public API exports
├── client.py      # Base HTTP client with logging and fixture support
├── v0.py          # V0 API driver
├── v1.py          # V1 API driver
└── metrics.py     # High-level metrics retrieval with fallback chain

Using the Library Programmatically

from oh_api import APIClient, get_conversation_metrics

# Create a client
client = APIClient(
    base_url="https://app.all-hands.dev",
    api_key="your-api-key",
    log_api_calls=True,  # Optional: enable logging
)

# Get metrics
metrics = get_conversation_metrics(client, "conversation-id")
if metrics:
    print(f"Cost: ${metrics.accumulated_cost:.6f}")
    print(f"Tokens: {metrics.total_tokens:,}")

Using the V0/V1 Drivers Directly

from oh_api import APIClient
from oh_api.v0 import V0Driver
from oh_api.v1 import V1Driver

client = APIClient(base_url="...", api_key="...")

# V0 API
v0 = V0Driver(client)
conv = v0.get_conversation("conv-id")
events = v0.get_events("conv-id", limit=50)

# V1 API
v1 = V1Driver(client)
conv = v1.get_conversation("conv-id")
if conv and conv.metrics:
    print(f"Cost: ${conv.metrics.accumulated_cost}")

Testing

The test suite uses recorded API fixtures to achieve high coverage without making real API calls:

# Run tests
uv run pytest tests/ -v

# Run with coverage
uv run pytest tests/ --cov --cov-report=term-missing

Current coverage: 89% of library code.

Creating New Fixtures

Run the CLI with --log-api-calls to capture real API responses
Copy relevant response files to tests/fixtures/
Rename following the pattern: GET__api_path_q_param=value.json

APIs Used

For V1 Conversations

Endpoint	Purpose
`GET /api/conversations/{id}`	Get conversation info (to determine version)
`GET /api/v1/app-conversations?ids={id}`	Get conversation with metrics
`GET /api/v1/conversation/{id}/events/search`	Fallback: find metrics in `ConversationStateUpdateEvent`

For V0 Conversations

Endpoint	Purpose
`GET /api/conversations/{id}`	Get conversation info
`GET /api/conversations/{id}/events`	Get events with `llm_metrics`
`GET /api/conversations/{id}/trajectory`	Fallback for metrics

License

MIT License - see LICENSE for details.

Overview

Sandbox lifecycle

Conversation monitoring & reacting

Secrets & authentication

Plugins, skills & MCP

Custom agents & tools

Guardrails

View source on GitHub

Features

How It Works

Installation

Run It

Set your API key

Get metrics for a conversation

JSON output

Options

Logging API Calls

Metrics Explained

Architecture

Using the Library Programmatically

Using the V0/V1 Drivers Directly

Testing

Creating New Fixtures

APIs Used

For V1 Conversations

For V0 Conversations

License

View source on GitHub

​Features

​How It Works

​Installation

​Run It

​Set your API key

​Get metrics for a conversation

​JSON output

​Options

​Logging API Calls

​Metrics Explained

​Architecture

​Using the Library Programmatically

​Using the V0/V1 Drivers Directly

​Testing

​Creating New Fixtures

​APIs Used

​For V1 Conversations

​For V0 Conversations

​License

Features

How It Works

Installation

Run It

Set your API key

Get metrics for a conversation

JSON output

Options

Logging API Calls

Metrics Explained

Architecture

Using the Library Programmatically

Using the V0/V1 Drivers Directly

Testing

Creating New Fixtures

APIs Used

For V1 Conversations

For V0 Conversations

License