> For the complete documentation index, see [llms.txt](https://voxrad.gitbook.io/voxrad/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://voxrad.gitbook.io/voxrad/running-models-locally.md).

# Running Models Locally

There are various ways to run models locally and create OpenAI compatible endpoints. Some of the example open-source libraries are:

* [Inference](https://github.com/xorbitsai/inference?tab=readme-ov-file)
* [LocalAI](https://github.com/mudler/LocalAI)
* [LMStudio](https://lmstudio.ai/docs/local-server)
* [Ollama](https://ollama.com/blog/openai-compatibility)
* [Jan](https://jan.ai/docs/local-api)
* [OpenLLM](https://github.com/bentoml/OpenLLM?tab=readme-ov-file#start-an-llm-server)
* [GPT4ALL](https://github.com/nomic-ai/gpt4all/tree/cef74c2be20f5b697055d5b8b506861c7b997fab/gpt4all-api)
* [Faster Whisper Server](https://github.com/fedirz/faster-whisper-server)

If you are planning to run the models on consumer grade personal computers, you may want to use models in GGUF format. Read this discussion on [conversion](https://github.com/ggerganov/llama.cpp/discussions/2948). 

## Remotely Hosted Models

You can also input OpenAI compatible Base URL and API key of any remotely hosted service, however this is not recommended for sensitive data. For example:

{% hint style="warning" %}
This is not recommended for clinical use.
{% endhint %}

* OpenAI: `https://api.openai.com/v1/`
* Gemini: `https://my-openai-gemini-henna.vercel.app/v1` 

       (Note: Cannot fetch models, defaults to Gemini-1.5-pro)

* Groq: `https://api.groq.com/openai/v1`