Plugin directory

The following plugins are available for LLM. Here’s how to install them.

Local models

These plugins all help you run LLMs directly on your own computer:

• llm-gguf uses llama.cpp to run models published in the GGUF format.

• llm-mlx (Mac only) uses Apple’s MLX framework to provide extremely high performance access to a large number of local models.

• llm-ollama adds support for local models run using Ollama.

• llm-llamafile adds support for local models that are running locally using llamafile.

• llm-mlc can run local models released by the MLC project, including models that can take advantage of the GPU on Apple Silicon M1/M2 devices.

• llm-gpt4all adds support for various models released by the GPT4All project that are optimized to run locally on your own machine. These models include versions of Vicuna, Orca, Falcon and MPT - here’s a full list of models.

• llm-mpt30b adds support for the MPT-30B local model.

Remote APIs

These plugins can be used to interact with remotely hosted models via their API:

• llm-mistral adds support for Mistral AI’s language and embedding models.

• llm-gemini adds support for Google’s Gemini models.

• llm-anthropic supports Anthropic’s Claude 3 family, 3.5 Sonnet and beyond.

• llm-command-r supports Cohere’s Command R and Command R Plus API models.

• llm-reka supports the Reka family of models via their API.

• llm-perplexity by Alexandru Geana supports the Perplexity Labs API models, including llama-3-sonar-large-32k-online which can search for things online and llama-3-70b-instruct.

• llm-groq by Moritz Angermann provides access to fast models hosted by Groq.

• llm-grok by Benedikt Hiepler providing access to Grok model using the xAI API Grok.

• llm-anyscale-endpoints supports models hosted on the Anyscale Endpoints platform, including Llama 2 70B.

• llm-replicate adds support for remote models hosted on Replicate, including Llama 2 from Meta AI.

• llm-fireworks supports models hosted by Fireworks AI.

• llm-openrouter provides access to models hosted on OpenRouter.

• llm-cohere by Alistair Shepherd provides cohere-generate and cohere-summarize API models, powered by Cohere.

• llm-bedrock adds support for Nova by Amazon via Amazon Bedrock.

• llm-bedrock-anthropic by Sean Blakey adds support for Claude and Claude Instant by Anthropic via Amazon Bedrock.

• llm-bedrock-meta by Fabian Labat adds support for Llama 2 and Llama 3 by Meta via Amazon Bedrock.

• llm-together adds support for the Together AI extensive family of hosted openly licensed models.

• llm-deepseek adds support for the DeepSeek’s DeepSeek-Chat and DeepSeek-Coder models.

• llm-lambda-labs provides access to models hosted by Lambda Labs, including the Nous Hermes 3 series.

• llm-venice provides access to uncensored models hosted by privacy-focused Venice AI, including Llama 3.1 405B.

If an API model host provides an OpenAI-compatible API you can also configure LLM to talk to it without needing an extra plugin.

Tools

The following plugins add new tools that can be used by models:

• llm-tools-simpleeval implements simple expression support for things like mathematics.

• llm-tools-quickjs provides access to a sandboxed QuickJS JavaScript interpreter, allowing LLMs to run JavaScript code. The environment persists between calls so the model can set variables and build functions and reuse them later on.

• llm-tools-sqlite can run read-only SQL queries against local SQLite databases.

• llm-tools-datasette can run SQL queries against a remote Datasette instance.

• llm-tools-exa by Dan Turkel can perform web searches and question-answering using exa.ai.

• llm-tools-rag by Dan Turkel can perform searches over your LLM embedding collections for simple RAG.

Fragments and template loaders

LLM 0.24 introduced support for plugins that define -f prefix:value or -t prefix:value custom loaders for fragments and templates.

• llm-video-frames uses ffmpeg to turn a video into a sequence of JPEG frames suitable for feeding into a vision model that doesn’t support video inputs: llm -f video-frames:video.mp4 'describe the key scenes in this video'.

• llm-templates-github supports loading templates shared on GitHub, e.g. llm -t gh:simonw/pelican-svg.

• llm-templates-fabric provides access to the Fabric collection of prompts: cat setup.py | llm -t fabric:explain_code.

• llm-fragments-github can load entire GitHub repositories in a single operation: llm -f github:simonw/files-to-prompt 'explain this code'. It can also fetch issue threads as Markdown using llm -f issue:https://github.com/simonw/llm-fragments-github/issues/3.

• llm-hacker-news imports conversations from Hacker News as fragments: llm -f hn:43615912 'summary with illustrative direct quotes'.

• llm-fragments-pypi loads PyPI packages’ description and metadata as fragments: llm -f pypi:ruff "What flake8 plugins does ruff re-implement?".

• llm-fragments-pdf by Dan Turkel converts PDFs to markdown with PyMuPDF4LLM to use as fragments: llm -f pdf:something.pdf "what's this about?".

• llm-fragments-site-text by Dan Turkel converts websites to markdown with Trafilatura to use as fragments: llm -f site:https://example.com "summarize this".

• llm-fragments-reader runs a URL theough the Jina Reader API: llm -f 'reader:https://simonwillison.net/tags/jina/' summary.

Embedding models

Embedding models are models that can be used to generate and store embedding vectors for text.

• llm-sentence-transformers adds support for embeddings using the sentence-transformers library, which provides access to a wide range of embedding models.

• llm-clip provides the CLIP model, which can be used to embed images and text in the same vector space, enabling text search against images. See Build an image search engine with llm-clip for more on this plugin.

• llm-embed-jina provides Jina AI’s 8K text embedding models.

• llm-embed-onnx provides seven embedding models that can be executed using the ONNX model framework.

Extra commands

• llm-cmd accepts a prompt for a shell command, runs that prompt and populates the result in your shell so you can review it, edit it and then hit <enter> to execute or ctrl+c to cancel.

• llm-cmd-comp provides a key binding for your shell that will launch a chat to build the command. When ready, hit <enter> and it will go right back into your shell command line, so you can run it.

• llm-python adds a llm python command for running a Python interpreter in the same virtual environment as LLM. This is useful for debugging, and also provides a convenient way to interact with the LLM Python API if you installed LLM using Homebrew or pipx.

• llm-cluster adds a llm cluster command for calculating clusters for a collection of embeddings. Calculated clusters can then be passed to a Large Language Model to generate a summary description.

• llm-jq lets you pipe in JSON data and a prompt describing a jq program, then executes the generated program against the JSON.

Just for fun

• llm-markov adds a simple model that generates output using a Markov chain. This example is used in the tutorial Writing a plugin to support a new model.