mcprobe
Verified Safeby Liquescent-Development
Overview
A conversational testing framework for MCP (Model Context Protocol) servers, validating that LLM agents can correctly answer real-world questions using synthetic users and LLM judges.
Installation
mcprobe run test-scenario.yaml --provider ollama --model llama3.2Environment Variables
- OPENAI_API_KEY
- OLLAMA_BASE_URL
- GOOGLE_API_KEY
- MCP_URL
- MCP_TOKEN
- GEMINI_MODEL
- AGENT_TEMPERATURE
Security Notes
The `mcprobe generate-scenarios` command executes arbitrary user-provided commands (e.g., `npx @example/weather-mcp`) via `subprocess.run` to connect to an MCP server. While intended for developers to launch their *own* trusted local servers for testing, this feature represents a significant security risk if the `server` argument were ever exposed to untrusted input. Otherwise, the project follows good security practices, such as using `SecretStr` for API keys in configurations and relying on environment variables.
Similar Servers
fastmcp
FastMCP is an ergonomic interface for the Model Context Protocol (MCP), providing a comprehensive framework for building and interacting with AI agents, tools, resources, and prompts across various transports and authentication methods.
mcp-interviewer
A Python CLI tool to evaluate Model Context Protocol (MCP) servers for agentic use-cases, by inspecting capabilities, running functional tests, and providing LLM-as-a-judge evaluations.
1xn-vmcp
An open-source platform for composing, customizing, and extending multiple Model Context Protocol (MCP) servers into a single logical, virtual MCP server, enabling fine-grained context engineering for AI workflows and agents.
mcp-advisor
Provides LLMs and humans with structured access to the Model Context Protocol (MCP) specification and documentation for understanding and compliance evaluation.