📚 Documentation
Quick Links
- Home - Main documentation page
- Quickstart Guide - Get started quickly with Arize Toolkit
- CLI Reference - Command-line interface documentation
Tools Documentation
| Tool Category | Description |
|---|---|
| Project/Model Tools | Access and manage projects/models, retrieve performance metrics and traces/inference volumes |
| Monitor Tools | Create, copy, and manage monitors and alerting |
| Custom Metrics Tools | Create and manage custom metrics |
| Language Model Tools | Work with prompts, annotations, evaluations, and LLM features |
| Space & Organization Tools | Navigate and manage spaces, organizations, and projects |
| Data Import Tools | Import data from cloud storage and databases |
| Trace Tools | Query traces and spans, discover columns, and export trace data |
| Dashboard Tools | Create and manage dashboards |
| Utility Tools | Client configuration and utility functions |
Extensions
| Extension | Description |
|---|---|
| Prompt Optimization | Automated prompt improvement based on historical performance |
Claude Code Plugin Marketplace
This repository is a Claude Code plugin marketplace with AI-assisted tools for the Arize platform.
# Add the marketplace
claude plugin marketplace add duncankmckinnon/arize_toolkit
# Install plugins
claude plugin install arize-toolkit@arize-toolkit
claude plugin install arize-toolkit-dev@arize-toolkit
| Plugin | Skills | Description |
|---|---|---|
| arize-toolkit | arize-toolkit-cli, arize-traces | Manage Arize resources via CLI, debug traces |
| arize-toolkit-dev | arize-graphql-analytics, new-query-workflow | Developer workflows for adding new functionality |
See the Plugin Marketplace docs for full details.
Arize Claude Code Tracing
For automatic tracing of Claude Code sessions to Arize AX or Phoenix, install the tracing plugin from the Arize Claude Code Plugin marketplace:
# Add the Arize plugin marketplace
claude plugin marketplace add Arize-ai/arize-claude-code-plugin
# Install the tracing plugin
claude plugin install claude-code-tracing@arize-claude-plugin
| Plugin | Skills | Description |
|---|---|---|
| claude-code-tracing | setup-claude-code-tracing | Automatic tracing of Claude Code sessions with OpenInference spans (9 lifecycle hooks) |
For Developers
- Development Guide - Information about extending the toolkit
- Integration Tests - Running integration tests
- Claude Code Plugins - Plugin development and skill authoring
Disclaimer
Although this package is used for development work with and within the Arize platform, it is not an Arize product. It is a open source project developed and maintained by an Arize Engineer. Feel free to add issues or reach out for help in the Arize community Slack channel.
Overview
Arize Toolkit is a set of tools packaged as a Python client that lets you easily interact with Arize AI APIs. Here's a quick overview of the main features in the current release:
- Access and manage models
- Retrieve performance metrics over a time period
- Retrieve inference volume over a time period
- Create, copy, and manage custom metrics
- Create, copy, and manage monitors and alerting
- Work with LLM features like prompts and annotations
- Import data from cloud storage (S3, GCS, Azure) and databases (BigQuery, Snowflake, Databricks)
- Create, update, and delete data import jobs with full lifecycle management
- Prompt Optimization Extension (optional): Automatically optimize prompts using meta-prompt techniques with feedback from evaluators
- Trace Inspection: List traces, retrieve spans with full attributes, discover available columns, and export to DataFrames or CSV
- Command-Line Interface (optional): Manage all Arize resources directly from the terminal with Rich table output and JSON mode
Installation
pip install arize_toolkit
Optional Dependencies
Command-Line Interface
To use the arize_toolkit CLI, install with the cli extras:
pip install arize_toolkit[cli]
Prompt Optimization Extension
For automated prompt optimization using meta-prompt techniques, install with the prompt_optimizer extras:
pip install arize_toolkit[prompt_optimizer]
Command-Line Interface
The CLI wraps all Client functionality so you can manage models, monitors, prompts, and more directly from the terminal.
Quick Start
# One-time setup — saves credentials to ~/.arize_toolkit/config.toml
arize_toolkit config init
# List models (Rich table output by default)
arize_toolkit models list
# Same thing, using the "projects" alias
arize_toolkit projects list
# JSON output, pipe to jq
arize_toolkit --json models list | jq '.[].name'
# Get monitor details for a specific model
arize_toolkit monitors get "accuracy-alert" --model "fraud-detection-v3"
# Create a performance monitor
arize_toolkit monitors create-performance "accuracy-alert" \
--model "fraud-detection-v3" \
--environment production \
--performance-metric accuracy \
--threshold 0.95
# Use a named profile for a different environment
arize_toolkit --profile staging spaces list
Global Options
| Flag | Description |
|---|---|
--version | Show the installed version |
--profile NAME | Use a named configuration profile |
--json | Output raw JSON instead of Rich tables |
--api-key KEY | Override the API key |
--org NAME | Override the organization |
--space NAME | Override the space |
--app-url URL | Override the Arize app URL |
Command Groups
| Group | Description |
|---|---|
config | Manage configuration profiles (init, list, show, use) |
spaces | List, create, and switch spaces or orgs |
orgs | List and create organizations |
users | Search users and manage space membership |
models / projects | List models, check volume, pull performance metrics |
monitors | Create, list, copy, and delete monitors |
prompts | Manage prompt templates and versions |
custom-metrics | Create and manage custom metrics |
evaluators | Manage LLM and code evaluators |
dashboards | Create and view dashboards |
traces | List traces, get spans, and discover available columns |
imports | Manage file and table import jobs |
See the full CLI documentation for detailed usage and examples.
Client Setup
The Client class is the entrypoint for interacting with the toolkit. It provides maintains the connection information for making requests to the Arize APIs, and offers a wide range of operations for interacting with models, monitors, dashboards, and more.
API Key
To create a client, you need to provide your Arize API key. Use this reference to get your API key from the Arize UI.
Organization and Space
You will also need to provide an organization name and space name. To give some context, models are scoped to a space, and the space is scoped to an organization. These can be found by navigating to the Arize UI and looking at the upper left dropdown from the landing page. You will first select an organization from the dropdown and then pick the space from the available spaces in that organization.
For the example below, the organization is Demo Org and the space is Test Space.
For On Prem deployments
For SaaS users, the default API endpoint is always going to be https://api.arize.com.
If you are using an on prem deployment of Arize, you will need to provide the api_url parameter.
This parameters should just be the base url of your Arize instance.