ui-controller-mcp
by phanijapps
Overview
Exposes desktop UI control tools over Server-Sent Events (SSE) for remote automation, augmented by AI vision and planning.
Installation
./start.shEnvironment Variables
- NGROK_AUTH_TOKEN
- PORT
- OLLAMA_VISION_MODEL
- OLLAMA_PLANNING_MODEL
- OLLAMA_BASE_URL
- NGROK_DOMAIN
Security Notes
The server exposes powerful desktop control tools (`launch_app`, `type_text`, `click`, `scroll`, `focus_window`, `screenshot`) that, if compromised, could lead to full system control or data exfiltration. A `SafetyGuard` is implemented to block specific dangerous commands (`rm -rf`, `shutdown`), but this list might not be exhaustive and could potentially be bypassed by a sophisticated attacker. The optional `ngrok` tunneling, while convenient, exposes the desktop control to the public internet, drastically increasing the attack surface. Users must be extremely cautious when exposing this server.
Similar Servers
Windows-MCP
This MCP server enables AI agents to directly interact with the Windows operating system, performing tasks such as file navigation, application control, UI interaction, and QA testing.
MCPControl
A Windows control server for the Model Context Protocol, enabling AI models to programmatically control system operations such as mouse, keyboard, window management, and screen capture.
Windows-MCP.Net
Enabling AI assistants to automate tasks and interact with the Windows desktop environment.
DelphiMCP
Enables AI-driven automation and control of Delphi VCL applications via the Model Context Protocol (MCP) and named pipes.