playwright-mcp-evals
Verified Safeby scalvert
Overview
A Playwright-based testing framework for evaluating and validating Model Context Protocol (MCP) servers, including tool calls, conformance, and LLM-driven scenarios.
Installation
npm testEnvironment Variables
- OPENAI_API_KEY
- ANTHROPIC_API_KEY
- MCP_ACCESS_TOKEN
- MCP_REFRESH_TOKEN
- MCP_TOKEN_TYPE
- MCP_TOKEN_EXPIRES_AT
- MCP_OAUTH_SERVER_URL
- MCP_AUTH_STATE_PATH
- MCP_OAUTH_CLIENT_ID
- MCP_OAUTH_CLIENT_SECRET
- MCP_OAUTH_SCOPES
- MCP_OAUTH_RESOURCE
- CI
- OAUTH_DEBUG
Security Notes
The framework itself is a test client. The stdio transport allows executing arbitrary local commands defined in `mcpConfig`, which is an expected feature for testing local servers, not a vulnerability, assuming trusted configuration inputs. LLM integration (OpenAI/Anthropic) requires API keys loaded from environment variables. OAuth authentication involves opening a browser and running a local HTTP callback server, which is a standard process. No direct `eval()` or obfuscation was found. The primary risk comes from configuring the `command` field in `mcpConfig` (for stdio transport) with an untrusted source, which could lead to arbitrary code execution outside the scope of the framework's direct operation.
Similar Servers
playwright-mcp
Provides a Model Context Protocol (MCP) server for LLMs to automate browser interactions using Playwright's accessibility tree, avoiding pixel-based vision models.
qa-use
Provides comprehensive browser automation and QA testing capabilities, integrating with a backend platform for automated tests, interactive debugging, and batch test execution.
fetcher-mcp
Fetching and processing web page content (HTML to Markdown) using a headless browser for AI-driven applications.
playwright-mcp-server
Provides a robust, token-aware Playwright browser automation server for Large Language Models (LLMs) and coding agents to interact with web content, supporting advanced features like anti-detection and multi-page management.