Other models#
LLM supports OpenAI models by default. You can install plugins to add support for other models. You can also add additional OpenAI-API-compatible models using a configuration file.
Installing and using a local model#
LLM plugins can provide local models that run on your machine.
To install llm-gpt4all, providing 17 models from the GPT4All project, run this:
llm install llm-gpt4all
Run llm models
to see the expanded list of available models.
To run a prompt through one of the models from GPT4All specify it using -m/--model
:
llm -m orca-mini-3b-gguf2-q4_0 'What is the capital of France?'
The model will be downloaded and cached the first time you use it.
Check the plugin directory for the latest list of available plugins for other models.
Adding more OpenAI models#
OpenAI occasionally release new models with new names. LLM aims to ship new releases to support these, but you can also configure them directly, by adding them to a extra-openai-models.yaml
configuration file.
Run this command to find the directory in which this file should be created:
dirname "$(llm logs path)"
On my Mac laptop I get this:
~/Library/Application Support/io.datasette.llm
Create a file in that directory called extra-openai-models.yaml
.
Let’s say OpenAI have just released the gpt-3.5-turbo-0613
model and you want to use it, despite LLM not yet shipping support. You could configure that by adding this to the file:
- model_id: gpt-3.5-turbo-0613
aliases: ["0613"]
The model_id
is the identifier that will be recorded in the LLM logs. You can use this to specify the model, or you can optionally include a list of aliases for that model.
If the model is a completion model (such as gpt-3.5-turbo-instruct
) add completion: true
to the configuration.
With this configuration in place, the following command should run a prompt against the new model:
llm -m 0613 'What is the capital of France?'
Run llm models
to confirm that the new model is now available:
llm models
Example output:
OpenAI Chat: gpt-3.5-turbo (aliases: 3.5, chatgpt)
OpenAI Chat: gpt-3.5-turbo-16k (aliases: chatgpt-16k, 3.5-16k)
OpenAI Chat: gpt-4 (aliases: 4, gpt4)
OpenAI Chat: gpt-4-32k (aliases: 4-32k)
OpenAI Chat: gpt-3.5-turbo-0613 (aliases: 0613)
Running llm logs -n 1
should confirm that the prompt and response has been correctly logged to the database.
OpenAI-compatible models#
Projects such as LocalAI offer a REST API that imitates the OpenAI API but can be used to run other models, including models that can be installed on your own machine. These can be added using the same configuration mechanism.
The model_id
is the name LLM will use for the model. The model_name
is the name which needs to be passed to the API - this might differ from the model_id
, especially if the model_id
could potentially clash with other installed models.
The api_base
key can be used to point the OpenAI client library at a different API endpoint.
To add the orca-mini-3b
model hosted by a local installation of LocalAI, add this to your extra-openai-models.yaml
file:
- model_id: orca-openai-compat
model_name: orca-mini-3b.ggmlv3
api_base: "http://localhost:8080"
If the api_base
is set, the existing configured openai
API key will not be sent by default.
You can set api_key_name
to the name of a key stored using the API key management feature.
Add completion: true
if the model is a completion model that uses a /completion
as opposed to a /completion/chat
endpoint.
Having configured the model like this, run llm models
to check that it installed correctly. You can then run prompts against it like so:
llm -m orca-openai-compat 'What is the capital of France?'
And confirm they were logged correctly with:
llm logs -n 1
Extra HTTP headers#
Some providers such as openrouter.ai may require the setting of additional HTTP headers. You can set those using the headers:
key like this:
- model_id: claude
model_name: anthropic/claude-2
api_base: "https://openrouter.ai/api/v1"
api_key_name: openrouter
headers:
HTTP-Referer: "https://llm.datasette.io/"
X-Title: LLM