Installing and using a local model#
LLM plugins can provide local models that run on your machine.
llm install llm-gpt4all
llm models to see the expanded list of available models.
To run a prompt through one of the models from GPT4All specify it using
llm -m orca-mini-3b-gguf2-q4_0 'What is the capital of France?'
The model will be downloaded and cached the first time you use it.
Check the plugin directory for the latest list of available plugins for other models.
Adding more OpenAI models#
OpenAI occasionally release new models with new names. LLM aims to ship new releases to support these, but you can also configure them directly, by adding them to a
extra-openai-models.yaml configuration file.
Run this command to find the directory in which this file should be created:
dirname "$(llm logs path)"
On my Mac laptop I get this:
Create a file in that directory called
Let’s say OpenAI have just released the
gpt-3.5-turbo-0613 model and you want to use it, despite LLM not yet shipping support. You could configure that by adding this to the file:
- model_id: gpt-3.5-turbo-0613 aliases: ["0613"]
model_id is the identifier that will be recorded in the LLM logs. You can use this to specify the model, or you can optionally include a list of aliases for that model.
If the model is a completion model (such as
completion: true to the configuration.
With this configuration in place, the following command should run a prompt against the new model:
llm -m 0613 'What is the capital of France?'
llm models to confirm that the new model is now available:
OpenAI Chat: gpt-3.5-turbo (aliases: 3.5, chatgpt) OpenAI Chat: gpt-3.5-turbo-16k (aliases: chatgpt-16k, 3.5-16k) OpenAI Chat: gpt-4 (aliases: 4, gpt4) OpenAI Chat: gpt-4-32k (aliases: 4-32k) OpenAI Chat: gpt-3.5-turbo-0613 (aliases: 0613)
llm logs -n 1 should confirm that the prompt and response has been correctly logged to the database.
Projects such as LocalAI offer a REST API that imitates the OpenAI API but can be used to run other models, including models that can be installed on your own machine. These can be added using the same configuration mechanism.
model_id is the name LLM will use for the model. The
model_name is the name which needs to be passed to the API - this might differ from the
model_id, especially if the
model_id could potentially clash with other installed models.
api_base key can be used to point the OpenAI client library at a different API endpoint.
To add the
orca-mini-3b model hosted by a local installation of LocalAI, add this to your
- model_id: orca-openai-compat model_name: orca-mini-3b.ggmlv3 api_base: "http://localhost:8080"
api_base is set, the existing configured
openai API key will not be sent by default.
You can set
api_key_name to the name of a key stored using the API key management feature.
completion: true if the model is a completion model that uses a
/completion as opposed to a
Having configured the model like this, run
llm models to check that it installed correctly. You can then run prompts against it like so:
llm -m orca-openai-compat 'What is the capital of France?'
And confirm they were logged correctly with:
llm logs -n 1
Extra HTTP headers#
Some providers such as openrouter.ai may require the setting of additional HTTP headers. You can set those using the
headers: key like this:
- model_id: claude model_name: anthropic/claude-2 api_base: "https://openrouter.ai/api/v1" api_key_name: openrouter headers: HTTP-Referer: "https://llm.datasette.io/" X-Title: LLM