Codakey
Play
← Back to AbleMouse

How to Deploy & Use AbleMouse

AbleMouse Deployment and Usage Guide

1. Prerequisites

Operating Systems: Windows, macOS, iOS, iPadOS, Linux (Ubuntu), Android (specific editions vary by OS support) Python: Version 3.7 or higher Hardware (varies by edition):

  • AbleMouse DIY Edition: ESP32 board, basic electronic components (wires, resistors), 3D-printable case (optional).
  • AbleMouse AI Edition: A webcam (built-in or external).
  • AbleMouse Beyond Switch Edition: No special hardware beyond a standard PC.

Python Libraries (Core dependencies from source files):

  • opencv-python (cv2) – For computer vision in AI Edition.
  • mediapipe – For facial landmark detection in AI Edition.
  • pyautogui – For system-level mouse and keyboard control.
  • pygame – For audio playback in Beyond Switch Edition.
  • tkinter – For graphical menus (usually comes with Python).
  • numpy – For numerical operations.

Optional/Additional Tools:

  • For AI Edition with advanced features: A local LLM server (e.g., Ollama) running on http://localhost:8080 for the talk2cam.py script.
  • For Windows-specific tools: AutoHotKey (for MouseCommander).
  • For DIY Edition: Arduino IDE or similar for ESP32 programming, soldering tools.

2. Installation

  1. Clone the Repository:

    git clone https://github.com/aradzhabov/AbleMouse.git
cd AbleMouse

  2. Set up a Python Virtual Environment (Recommended):

    python -m venv venv
# On Windows:
venv\Scripts\activate
# On macOS/Linux:
source venv/bin/activate

  3. Install Python Dependencies: The project does not have a central requirements.txt. Install packages based on the edition you want to use.

    • For AI Edition: 
      pip install opencv-python mediapipe pyautogui numpy
    • For Beyond Switch Edition: 
      pip install pygame pyautogui
    • For R&D Tools (e.g., talk2cam.py): 
      pip install opencv-python requests pillow

3. Configuration

Configuration is primarily done via editing Python source files. There are no central .env files.

For AbleMouse AI Edition (able_mouse_ai_edition.py):

  • camera = 0: Change to 1 or 2 if you need to use an external webcam.
  • bln_cam_mouse_control = False: Set to True to enable cursor control via head movement.
  • USE_MENU_SYSTEM = False: Set to True if you want this AI Edition to connect to and control the Beyond Switch Edition menu server.
  • MENU_HOST & MENU_PORT: Configure if USE_MENU_SYSTEM is True.
  • Adjust time thresholds (LEFT_EYE_CLOSE_TIME_THRESHOLD, MOUTH_OPEN_RIGHT_CLICK_TIME_THRESHOLD, etc.) to match the user's physical capabilities.

For AbleMouse Beyond Switch Edition (app_config.py & menu_config.py):

  • SERVER_CONFIG: Modify host and port if needed.
  • UI_CONFIG: Adjust transparency, highlight_interval, etc.
  • AUDIO_CONFIG: Ensure audio file paths are correct. Place your .mp3 files in the audio/ directory.
  • MENU_CONFIG (in menu_config.py): This is the core navigation structure. Customize the menu entries (like FOOD, REQUESTS) and their hierarchy to suit the specific user's needs.

For Camera Llama Analyzer (talk2cam.py):

  • self.server_url = "http://localhost:8080": Point this to your local LLM API endpoint.
  • self.current_instruction: Change the default prompt for the vision model.

4. Build & Run

Choose your edition based on the Product Compatibility Table.

AbleMouse AI Edition (Head/Face Control)

  1. Ensure your webcam is connected and accessible.
  2. Run the main script: 
    cd src/ablemouse_ai_edition
python able_mouse_ai_edition.py
  3. A camera feed window will open. Position your face within the frame.
    • Cursor Control: Move your head to move the cursor.
    • Mode Toggle: Close your left eye for LEFT_EYE_CLOSE_TIME_THRESHOLD seconds to enable/disable camera mouse control.
    • Right Click: Open your mouth for MOUTH_OPEN_RIGHT_CLICK_TIME_THRESHOLD seconds.
    • Left Click: Close your right eye for RIGHT_EYE_RIGHT_CLICK_TIME_THRESHOLD seconds.
    • Menu Selection: Open your mouth briefly (MOUTH_OPEN_MENU_SELECTION_THRESHOLD) when integrated with the Beyond Switch menu.

AbleMouse Beyond Switch Edition (Single-Switch Menu System)

  1. This edition provides an on-screen menu navigable with a single switch (like a mouth click).
  2. Run the server: 
    cd src/ablemouse_beyond_switch_edition
python main.py
  3. A semi-transparent menu window will appear, always on top. It automatically starts a local server on port 12345.
  4. Navigation: The menu highlights items sequentially. Press the Spacebar to select the currently highlighted item. Use the "↩ Back" option to go back.
  5. This menu can be controlled remotely by other AbleMouse editions (like AI Edition) or any client that sends simple commands (like "select") to localhost:12345.

AbleMouse DIY Edition (Tongue/Touch Pedal)

  1. This is a hardware project. Follow the detailed build guide in AbleMouseDIY.md.
  2. It involves flashing an ESP32 board with provided firmware and constructing physical pedals or sensors.
  3. Once built, it acts as a Bluetooth or USB HID device, controlling the cursor directly.

MouseCommander (Windows Gesture Tool)

  1. Requires AutoHotKey installed.
  2. Run the scripts from src/windows/auto_hot_key/ to add mouse gesture shortcuts.

5. Deployment

AbleMouse is designed to run locally on a user's personal computer. "Deployment" here means setting it up to start automatically and run reliably.

  • Windows: Add a shortcut to the desired Python script (e.g., main.py for Beyond Switch) in the Startup folder (shell:startup). Run the script as Administrator if you encounter permissions issues with pyautogui.
  • macOS/Linux: Create a systemd service (Linux) or a Launch Agent (macOS) to run the Python script at login. Ensure the virtual environment is activated in the startup script.
  • Cross-Platform Consideration: The Beyond Switch Edition's menu server (main.py) is a good candidate to run at startup, as it provides a consistent interface that other assistive tools can connect to.

Not recommended for cloud deployment, as the software requires direct access to local hardware (HID, camera) and low-latency control of the GUI.

6. Troubleshooting

  • "No module named 'cv2' / 'mediapipe' / 'pyautogui'"

    • Solution: Ensure your virtual environment is activated and you have installed the required packages for your chosen edition using pip. See Installation section.

  • Webcam not found (camera = 0 doesn't work)

    • Solution (AI Edition): Try changing camera = 0 to camera = 1 in able_mouse_ai_edition.py. On macOS, ensure camera permissions are granted to your terminal or IDE.
    • Solution (General): Test your camera with a simple OpenCV script or a standard app like Zoom first.

  • pyautogui fails to control mouse/keyboard (especially on macOS)

    • Solution: You must grant Accessibility or Input Monitoring permissions to your terminal (e.g., Terminal.app, iTerm2) or the app running the script (e.g., VS Code) in System Preferences > Security & Privacy > Privacy.

  • Beyond Switch menu is unresponsive or doesn't highlight

    • Solution: Ensure the application window is in focus. Check that no other application is intercepting the Spacebar key. Verify the highlight_interval in UI_CONFIG is not set too high.

  • Audio feedback not playing in Beyond Switch Edition

    • Solution: Check that the AUDIO_DIR path in app_config.py is correct relative to where main.py is run. Ensure the .mp3 files exist in that directory. Check your system's audio output.

  • AI Edition cursor control is too sensitive/jumpy

    • Solution: Enable bln_do_not_move_cursor_if_xy_move_within_threshold and adjust SKIP_X_THRESHOLD and SKIP_Y_THRESHOLD values upward. Enable bln_nose_center for more stable tracking.

  • Connection refused when AI Edition tries to connect to menu server

    • Solution: Ensure the Beyond Switch Edition (main.py) is running first and that USE_MENU_SYSTEM = True and MENU_HOST/MENU_PORT in the AI Edition script match the server configuration.
← Project overviewView alternatives to AbleMouse →