exo-explore/exo

exo is an innovative open-source project with the primary goal of enabling you to run your own AI cluster at home. Here are its main features:

1. Device Integration

• Unifies your existing devices (iPhone, iPad, Android, Mac, NVIDIA GPU, Raspberry Pi, etc.) into a powerful GPU cluster.

2. Core Features:

• Broad Model Support: Supports a variety of models, including LLaMA (MLX and tinygrad), Mistral, LlaVA, Qwen, and Deepseek.
• Dynamic Model Partitioning: Automatically optimizes model allocation based on the current network topology and available device resources, allowing you to run models larger than a single device can handle.
• Automatic Device Discovery: Automatically discovers other devices on the network, eliminating the need for manual configuration.
• ChatGPT Compatible API: Provides a ChatGPT-compatible API interface, allowing you to run models on your own hardware with just a one-line code change.

3. Device Equality:

• Employs a P2P architecture instead of a master-slave architecture.
• Any device connected anywhere on the network can be used to run models.

4. Hardware Requirements:

• The only requirement is that the total memory of all devices is sufficient to hold the entire model.
• For example, running llama 3.1 8B (fp16) requires 16GB of total memory, which can be achieved with the following configurations:
  • 2 x 8GB M3 MacBook Air
  • 1 x 16GB NVIDIA RTX 4070 Ti Laptop
  • 2 x Raspberry Pi with 4GB RAM + 1 x 8GB Mac Mini

5. Installation Requirements:

• Requires Python >= 3.12.0
• If using an NVIDIA GPU on Linux, you also need:
  • NVIDIA Driver
  • CUDA Toolkit
  • CUDNN

The innovation of this project lies in its ability to allow ordinary users to build AI computing clusters using existing devices at home, greatly reducing the hardware threshold for running large AI models. It supports heterogeneous device collaboration, and even adding weaker devices can improve the overall throughput of the cluster.

For all detailed information, please refer to the official website (https://github.com/exo-explore/exo)