Writing plugins to add new embedding models#
Read the plugin tutorial for details on how to develop and package a plugin.
This page shows an example plugin that implements and registers a new embedding model.
There are two components to an embedding model plugin:
An implementation of the
register_embedding_models()hook, which takes aregistercallback function and calls it to register the new model with the LLM plugin system.A class that extends the
llm.EmbeddingModelabstract base class.The only required method on this class is
embed_batch(texts), which takes an iterable of strings and returns an iterator over lists of floating point numbers.
The following example uses the sentence-transformers package to provide access to the MiniLM-L6 embedding model.
- class llm.EmbeddingModel#
- embed(item: str | bytes | Attachment, *, key: str | None = None) list[float]#
Embed a single text string, binary blob or attachment, returning floats
- abstract embed_batch(items: Iterable[str | bytes | Attachment], *, key: str | None = None) Iterator[list[float]]#
Embed a batch of strings, blobs or attachments, returning float vectors
- embed_multi(items: Iterable[str | bytes | Attachment], batch_size: int | None = None, *, key: str | None = None) Iterator[list[float]]#
Embed multiple items in batches according to the model batch_size
import llm
from sentence_transformers import SentenceTransformer
@llm.hookimpl
def register_embedding_models(register):
model_id = "sentence-transformers/all-MiniLM-L6-v2"
register(SentenceTransformerModel(model_id, model_id), aliases=("all-MiniLM-L6-v2",))
class SentenceTransformerModel(llm.EmbeddingModel):
def __init__(self, model_id, model_name):
self.model_id = model_id
self.model_name = model_name
self._model = None
def embed_batch(self, texts):
if self._model is None:
self._model = SentenceTransformer(self.model_name)
results = self._model.encode(texts)
return (list(map(float, result)) for result in results)
Embedding models that call a hosted API should set needs_key and optionally key_env_var, then accept a keyword-only key= argument in embed_batch(). LLM resolves explicit keys, stored key aliases and environment variables before calling this method:
class HostedEmbeddingModel(llm.EmbeddingModel):
model_id = "hosted-embedding-model"
needs_key = "hosted"
key_env_var = "HOSTED_API_KEY"
def embed_batch(self, texts, *, key=None):
# key is the resolved API key
results = hosted_client.embed(texts, api_key=key)
return (list(map(float, result)) for result in results)
The key= value can come from llm embed --key, llm embed-multi --key or the Python embed(..., key=) and embed_multi(..., key=) methods. For compatibility, LLM also supports older plugins whose embed_batch() method does not accept key= by temporarily exposing the resolved value as self.key.
Once installed, the model provided by this plugin can be used with the llm embed command like this:
cat file.txt | llm embed -m sentence-transformers/all-MiniLM-L6-v2
Or via its registered alias like this:
cat file.txt | llm embed -m all-MiniLM-L6-v2
llm-sentence-transformers is a complete example of a plugin that provides an embedding model.
Execute Jina embeddings with a CLI using llm-embed-jina talks through a similar process to add support for the Jina embeddings models.
Embedding binary content#
If your model can embed binary content, use the supports_binary property to indicate that:
class ClipEmbeddingModel(llm.EmbeddingModel):
model_id = "clip"
supports_binary = True
supports_text= True
supports_text defaults to True and so is not necessary here. You can set it to False if your model only supports binary data.
If your model accepts binary, your .embed_batch() model may be called with a list of Python bytestrings. These may be mixed with regular strings if the model accepts both types of input.
llm-clip is an example of a model that can embed both binary and text content.
Embedding typed attachments#
Binary strings do not include a MIME type or information about where they came from. Models that need that information can declare the attachment MIME types they support using attachment_types:
class MultimodalEmbeddingModel(llm.EmbeddingModel):
model_id = "multimodal"
attachment_types = frozenset({"image/png", "image/jpeg", "audio/wav"})
def embed_batch(self, items, *, key=None):
for item in items:
if isinstance(item, llm.Attachment):
mimetype = item.resolve_type()
content = item.content_bytes()
# Decode content according to mimetype, then embed it
else:
# Embed text strings as usual
...
An embedding model with a non-empty attachment_types collection receives matching inputs as llm.Attachment objects. Attachments provide type, path, url, and content attributes, along with resolve_type() and content_bytes() methods.
The CLI accepts typed attachments using -a/--attachment, which detects the MIME type, or --at/--attachment-type PATH MIME, which sets it explicitly.
For backwards compatibility, a model with supports_binary = True and no declared attachment_types receives the attachmentâs content as bytes. This allows existing binary embedding plugins to work with the typed attachment CLI without modification.