Other models

LLM supports OpenAI models by default. You can install plugins to add support for other models. You can also add additional OpenAI-API-compatible models using a configuration file.

Installing and using a local model

LLM plugins can provide local models that run on your machine.

To install llm-gpt4all, providing 17 models from the GPT4All project, run this:

llm install llm-gpt4all

Run llm models to see the expanded list of available models.

To run a prompt through one of the models from GPT4All specify it using -m/--model:

llm -m orca-mini-3b-gguf2-q4_0 'What is the capital of France?'

The model will be downloaded and cached the first time you use it.

Check the plugin directory for the latest list of available plugins for other models.

OpenAI-compatible models

Projects such as LocalAI offer a REST API that imitates the OpenAI API but can be used to run other models, including models that can be installed on your own machine. These can be added using the same configuration mechanism.

The model_id is the name LLM will use for the model. The model_name is the name which needs to be passed to the API - this might differ from the model_id, especially if the model_id could potentially clash with other installed models.

The api_base key can be used to point the OpenAI client library at a different API endpoint.

To add the orca-mini-3b model hosted by a local installation of LocalAI, add this to your extra-openai-models.yaml file:

- model_id: orca-openai-compat
  model_name: orca-mini-3b.ggmlv3
  api_base: "http://localhost:8080"

If the api_base is set, the existing configured openai API key will not be sent by default.

You can set api_key_name to the name of a key stored using the API key management feature.

Add completion: true if the model is a completion model that uses a /completion as opposed to a /completion/chat endpoint.

If a model does not support streaming, add can_stream: false to disable the streaming option.

If a model supports structured output via JSON schemas, you can add supports_schema: true to support this feature.

If a model is a vision model, you can add vision: true to support this feature and use image attachments.

If a model is an audio model, you can add audio: true to support this feature and use audio attachments.

Having configured the model like this, run llm models to check that it installed correctly. You can then run prompts against it like so:

llm -m orca-openai-compat 'What is the capital of France?'

And confirm they were logged correctly with:

llm logs -n 1

Extra HTTP headers

Some providers such as openrouter.ai may require the setting of additional HTTP headers. You can set those using the headers: key like this:

- model_id: claude
  model_name: anthropic/claude-2
  api_base: "https://openrouter.ai/api/v1"
  api_key_name: openrouter
  headers:
    HTTP-Referer: "https://llm.datasette.io/"
    X-Title: LLM