Running Models Locally

There are various ways to run models locally and create OpenAI compatible endpoints. Some of the example open-source libraries are:

If you are planning to run the models on consumer grade personal computers, you may want to use models in GGUF format. Read this discussion on conversion.

Remotely Hosted Models

You can also input OpenAI compatible Base URL and API key of any remotely hosted service, however this is not recommended for sensitive data. For example:

This is not recommended for clinical use.

OpenAI: https://api.openai.com/v1/
Gemini: https://my-openai-gemini-henna.vercel.app/v1

(Note: Cannot fetch models, defaults to Gemini-1.5-pro)

Groq: https://api.groq.com/openai/v1

PreviousTroubleshooting NextGenerating a Report

Last updated 13 days ago

Running Models Locally

There are various ways to run models locally and create OpenAI compatible endpoints. Some of the example open-source libraries are:

If you are planning to run the models on consumer grade personal computers, you may want to use models in GGUF format. Read this discussion on conversion.

Remotely Hosted Models

You can also input OpenAI compatible Base URL and API key of any remotely hosted service, however this is not recommended for sensitive data. For example:

This is not recommended for clinical use.

OpenAI: https://api.openai.com/v1/
Gemini: https://my-openai-gemini-henna.vercel.app/v1

(Note: Cannot fetch models, defaults to Gemini-1.5-pro)

Groq: https://api.groq.com/openai/v1

PreviousTroubleshooting NextGenerating a Report

Last updated 13 days ago