OnLLM 🧠

OnLLM is the platform to run Large Language Model (LLM) or SLM models using OnnxRuntime directly on low-end devices like low power computers, mobile phones etc. It is cross-platform using Kivy & open-source. Currently we have the light-weight app to run AI chatbot on Android , Windows , Linux or other OS using Python directly.

The name is derived from: On from Onnx + LLM . Our approach is always local & offline first (you will need internet to download the model files for the first time). Arificial Intelligence (AI) to be accesible to everyone.

Features

✅️ An inbuild chatbot to chat with the AI model(s).
✅️ A local offline RAG to query on our selected document (pdf/docx) directly in the chatbot.
✅️ No trackers, no Ads, No data collection, completely Private.

You can buy me a coffee via this link or tap on below image. Thank you 🙏.

📽️ Demo

You can play below Video or click this Youtube Link to see the demo. Please let me know in the comments, how do you feel about this App.

🧑‍💻 Quickstart Guide

📱 Download & Run the Android App

Get the app from Google PlayStore

You can check the Releases and downlaod the latest version of the android app (APK file) on your phone if you do not want to get it from PlayStore.

In the initial screen, you need to click Start .
Once you enter the Chat screen, you can select the required model from drop-down. There is another drop-down menu to select the max lenght of the AI response , if you choose small value, your response may be incomplete due to the number of words.
Now the app will sync the available models from backend if Internet is available and will update the dropdown menu. Once you click on new model, it will prompt you to download. You may get different models on phone & desktop (due to performance limitations).
Now you can use your pdf or docx document file to query on the document. BUT, due to Android (11+) file restrictions, it is really difficult to use .pdf or .docx without full file manager access. SO, you need to keep & rename your .pdf with .pdf.jpg in either Downloads or Pictures (sub-folder will work under) folder.
> Example: mydoc.pdf should be renamed as mydoc.pdf.jpg
You can delete the downloaded models from Settings from the top-right menu.
You can check for update, check documentation / demo & contact us from the top-right menu.
When you select a model from drop-down in the chat window, please wait for the internal session to be started. It may take few seconds depending on the model size, you can check the below table .

Requirements

Minimum Android 9

💻 Download & Run the Windows or Linux App

You can check the Releases and downlaod the latest version of the application on your computer. If you are on Windows , download the OnLLM_X.X.X.exe file & double click to run it. If you are on Linux , download OnLLM_Linux_XXX file and run it.

Notes:

Some Antivirus softwares might give you detection alert for the exe (happens for PyInstaller apps), you need to add an exception in that case.
On Linux you may need to change file permission to execute it.
```
chmod +x OnLLM_Linux_XXX
./OnLLM_Linux_XXX
```

🐍 Run with Python

Clone the repo

git clone https://github.com/daslearning-org/OnLLM.git

Run the application

cd OnLLM/app/
pip install -r requirements.txt # virtual environment is recommended
python main.py

🔗 LLM Models used in this project

LLM (QA) models:

The models are being taken from different sources & below are the models available as on today. Size is the compressed size (approax). Few models which are marked with warning (⚠️) symbol may be very slow or unresponsive on Android or low power devices.

Model	Size	Android	Desktop
SmolLM2-135M	95MB	✅️	✅️
SmolLM2-360M	241MB	✅️	✅️
Gemma3-1B	645MB	⚠️	✅️

Embed models (Document RAG):

It follows the same priciples as above.

Model	Size	Android	Desktop
all-MiniLM-L6-V2	85MB	✅️	✅️

🖧 Our Scematic Architecture

To be added...

🦾 Build your own App

The Kivy project has a great tool named Buildozer which can make mobile apps for Android & iOS

📱 Build Android App

A Linux environment is recommended for the app development. If you are on Windows, you may use WSL or any Virtual Machine . As of now the buildozer tool works on Python version 3.11 at maximum. I am going to use Python 3.11

Install Rust & Add that in path

curl https://sh.rustup.rs -sSf | sh # then follow the prompt
export PATH="$HOME/.cargo/bin:$PATH"
# optional
rustup default stable

Install other dependencies & build APK

# add the python repository
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update

# install all dependencies.
sudo apt install -y ant autoconf automake ccache cmake g++ gcc libbz2-dev libffi-dev libltdl-dev libtool libssl-dev lbzip2 make ninja-build openjdk-17-jdk patch patchelf pkg-config protobuf-compiler python3.11 python3.11-venv python3.11-dev

# optionally we can default to python 3.11
sudo ln -sf /usr/bin/python3.11 /usr/bin/python3
sudo ln -sf /usr/bin/python3.11 /usr/bin/python
sudo ln -sf /usr/bin/python3.11-config /usr/bin/python3-config

# optionally you may check the java installation with below commands
java -version
javac -version

# install python modules
git clone https://github.com/daslearning-org/OnLLM.git
cd OnLLM/app/
python3.11 -m venv .env # create python virtual environment
source .env/bin/activate
pip install -r req_android.txt

# build the android apk
buildozer android debug # this may take a good amount of time for the first time & will generate the apk in the bin directory

🖳 Build Computer Application (Windows / Linux / MacOS)

A Python virtual environment is recommended and will use PyInstaller to package it.

# install pyinstaller
git clone https://github.com/daslearning-org/OnLLM.git
cd OnLLM/app/
python3.11 -m venv .env # create python virtual environment
source .env/bin/activate
pip install -r requirements.txt
pip install pyinstaller

# then update the spec file as needed
# then build your app which will be native to the OS i.e. Linux or Windows or MAC
pyinstaller desktopApp.spec