ollama_proxy_server/README.md

Ollama Proxy Server

Ollama Proxy Server is a lightweight, secure proxy server designed to add a security layer to one or multiple Ollama servers. It routes incoming requests to the backend server with the lowest load, minimizing server strain and improving responsiveness. Built with Python, this project is ideal for managing distributed Ollama instances with authentication and logging capabilities.

Author: ParisNeo

License: Apache 2.0

Repository: https://github.com/ParisNeo/ollama_proxy_server

Features

• Load Balancing: Routes requests to the Ollama server with the fewest ongoing requests.
• Security: Implements bearer token authentication using a user:key format.
• Asynchronous Logging: Logs access and errors to a CSV file without blocking request handling.
• Connection Pooling: Uses persistent HTTP connections for faster backend communication.
• Streaming Support: Properly forwards streaming responses from Ollama servers.
• Command-Line Tools: Includes utilities to run the server and manage users.
• Cross-Platform: Runs on any OS supporting Python 3.11.

Project Structure

ollama_proxy_server/
  |- add_user.py               # Script to add users to the authorized list
  |- main.py                   # Main proxy server script
  example.authorized_users.txt # Example authorized users file
  example.config.ini           # Example configuration file
  .gitignore                    # Git ignore file
  Dockerfile                    # Docker configuration
  LICENSE                       # Apache 2.0 license text
  requirements.txt              # Runtime dependencies
  requirements_dev.txt          # Development dependencies
  setup.py                      # Setup script for installation
  README.md                     # This file

Installation

Prerequisites

• Python 3.11 or higher
• Git (optional, for cloning the repository)

Option 1: Install from PyPI (Not Yet Published)

Once published, install using pip:

pip install ollama_proxy_server

Option 2: Install from Source

Clone the repository:

git clone https://github.com/ParisNeo/ollama_proxy_server.git
cd ollama_proxy_server

Install dependencies:

pip install -r requirements.txt

Install the package:

pip install .

Option 3: Use Docker

Build the Docker image:

docker build -t ollama_proxy_server .

Run the container:

docker run -p 8080:8080 -v $(pwd)/config.ini:/app/config.ini -v $(pwd)/authorized_users.txt:/app/authorized_users.txt ollama_proxy_server

Test that it works:

curl localhost:8080 -H "Authorization: Bearer user1:0XAXAXAQX5A1F"

Configuration

config.ini

Copy config.ini.example to config.ini and edit it:

[server0]
url = http://localhost:11434

# Add more servers as needed
# [server1]
# url = http://another-server:11434

• url: The URL of an Ollama backend server.

authorized_users.txt

Copy authorized_users.txt.example to authorized_users.txt and edit it:

user:key
another_user:another_key

Usage

Running the Server

python main.py --config config.ini --users_list authorized_users.txt

Managing Users

Use the add_user.py script to add new users.

python add_user.py <username> <key>

Alternatively, you can use the newly created ops command:

sudo ops add_user username:password

Setup as a Service

Using setup_service.sh

The repository includes a script called setup_service.sh to set up Ollama Proxy Server as a systemd service. This allows it to run in the background and start on boot.

1. Download the Repository:

   git clone https://github.com/ParisNeo/ollama_proxy_server.git
   cd ollama_proxy_server
   

2. Make setup_service.sh Executable:

   chmod +x setup_service.sh
   

3. Run the Script with sudo Privileges:

   sudo ./setup_service.sh /path/to/working/directory
   

   Replace /path/to/working/directory with the path where you want to set up your proxy server.

4. Follow Prompts:

   • You will be prompted to provide a port number (default is 11534) and log path.
   • You'll also add users and their passwords which will populate /etc/ops/authorized_users.txt.

5. Start the Service:

   sudo systemctl start ollama-proxy-server
   

6. Enable the Service to Start on Boot:

   sudo systemctl enable ollama-proxy-server
   

7. Check the Status of the Service:

   sudo journalctl -u ollama-proxy-server -f
   

Managing Users with ops Command

After setting up the service, you can add more users using the new ops command:

sudo ops add_user username:password

Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository.
2. Create a feature branch (git checkout -b feature/your-feature).
3. Commit your changes (git commit -am 'Add your feature').
4. Push to the branch (git push origin feature/your-feature).
5. Open a Pull Request.

See CONTRIBUTING.md for more details (to be added).

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

Acknowledgments

Built by ParisNeo.

Thanks to the open-source community for tools like requests and ascii_colors.

See you soon!