Component Boundaries and Interactions#

User Code
  └─→ LlamaCPP.__init__
      ├─→ Validators (field_validator, model_validator):
      │   ├─→ _validate_model_path_exists: Check local file exists
      │   ├─→ _check_model_source: Ensure one source is set
      │   ├─→ _check_formatters: Ensure formatters are provided
      │   └─→ _prepare_kwargs: Merge n_ctx, verbose, n_gpu_layers into model_kwargs
      ├─→ Store: model_path, model_url, temperature, max_new_tokens, etc.
      ├─→ Create Metadata: is_chat_model=True, is_function_calling_model=False
      └─→ model_post_init:
          ├─→ _resolve_model_path: Download if needed, return local Path
          └─→ _load_model: Load Llama with caching

_resolve_model_path()
  ├─→ If model_path is set:
  │   └─→ Return Path(model_path)
  ├─→ If hf_model_id is set:
  │   ├─→ Create cache directory
  │   ├─→ Call _fetch_model_file_hf(repo_id, filename, cache_dir)
  │   │   └─→ hf_hub_download → Path
  │   └─→ Update self.model_path, return Path
  └─→ Else (model_url or default):
      ├─→ Compute cache path from URL filename
      ├─→ If not cached:
      │   └─→ Call _fetch_model_file(url, path)
      │       └─→ requests.get(stream=True) with progress bar
      └─→ Update self.model_path, return Path

_load_model(model_path)
  ├─→ Compute cache_key = (str(path), json(model_kwargs))
  ├─→ With _MODEL_CACHE_LOCK: check cache
  │   └─→ result = _MODEL_CACHE.get(cache_key)
  ├─→ If cache miss:
  │   ├─→ loaded = Llama(model_path, **model_kwargs)
  │   └─→ With _MODEL_CACHE_LOCK: store if still missing
  └─→ Return result

User → LlamaCPP.complete(prompt, formatted, stream, **kwargs)
  ├─→ If not formatted:
  │   └─→ prompt = self.completion_to_prompt(prompt)
  ├─→ If stream:
  │   └─→ _stream_complete(prompt, **kwargs)
  │       ├─→ _guard_context(prompt) → count tokens, raise if too long
  │       ├─→ Build call_kwargs from generate_kwargs + defaults
  │       ├─→ Acquire _model_lock
  │       └─→ Yield CompletionResponse per chunk
  └─→ Else:
      └─→ _complete(prompt, **kwargs)
          ├─→ _guard_context(prompt)
          ├─→ Build call_kwargs
          ├─→ Acquire _model_lock
          ├─→ self._model(prompt=prompt, **call_kwargs)
          └─→ Return CompletionResponse(text=choices[0]["text"])

User → LlamaCPP.chat(messages, stream, **kwargs)
  └─→ CompletionToChatMixin.chat(messages, stream, **kwargs)
      ├─→ formatted = self.messages_to_prompt(messages)
      ├─→ result = self.complete(formatted, formatted=True, stream, **kwargs)
      └─→ If stream:
      │   └─→ Convert each CompletionResponse chunk → ChatResponse chunk
      └─→ Else:
          └─→ Wrap CompletionResponse → ChatResponse

User → LlamaCPP.acomplete(prompt, formatted, stream, **kwargs)
  ├─→ If stream:
  │   ├─→ chunks = await asyncio.to_thread(lambda: list(self.complete(..., stream=True)))
  │   └─→ Return async generator yielding chunks
  └─→ Else:
      └─→ await asyncio.to_thread(self.complete, prompt, formatted, stream=False, **kwargs)

User → TextCompletionLLM(output_parser=parser, prompt=prompt, llm=LlamaCPP(...))
  └─→ TextCompletionLLM stores LlamaCPP instance

User → text_llm(key="value")
  ├─→ Check llm.metadata.is_chat_model → True
  ├─→ Format prompt with variables → List[Message]
  ├─→ LlamaCPP.chat(messages) → ChatResponse
  ├─→ Extract response.message.content
  ├─→ PydanticParser.parse(content) → Model instance
  └─→ Return Model instance

Initialization:
  - model_path: str (set during init or model_post_init)
  - model_url: Optional[str] (immutable after init)
  - hf_model_id: Optional[str] (immutable after init)
  - temperature: float (immutable after init)
  - max_new_tokens: int (immutable after init)
  - context_window: int (immutable after init)
  - n_gpu_layers: int (immutable after init)
  - stop: list[str] (immutable after init)
  - generate_kwargs: dict (immutable after init)
  - model_kwargs: dict (immutable after init)
  - _model: Llama (loaded during model_post_init)
  - _model_lock: threading.Lock (per-instance)

Runtime:
  - _model: Llama instance (loaded, never changes)
  - _model_lock: Acquired during each inference call

_MODEL_CACHE: WeakValueDictionary
  - Keys: (model_path_str, model_kwargs_json)
  - Values: Llama instances (weak references)
  - Entries auto-removed when all LlamaCPP refs are GC'd

_MODEL_CACHE_LOCK: threading.Lock
  - Protects read/write to _MODEL_CACHE

Input:
  - prompt: str
  - formatted: bool
  - stream: bool
  - **kwargs: Additional generation options

Processing:
  - formatted_prompt: str (after completion_to_prompt if needed)
  - call_kwargs: dict (merged generate_kwargs + defaults + overrides)
  - raw_response: dict from Llama backend

Output:
  - CompletionResponse or CompletionResponseGen

Validators:
  ├─→ _validate_model_path_exists: ValueError if path doesn't exist
  ├─→ _check_model_source: ValueError if no source specified
  ├─→ _check_model_source: ValueError if hf_model_id without hf_filename
  └─→ _check_formatters: ValueError if formatters missing

_resolve_model_path:
  ├─→ _fetch_model_file: ValueError (Content-Length < 1 MB)
  ├─→ _fetch_model_file: requests.ConnectionError
  ├─→ _fetch_model_file: RuntimeError (download succeeded but file missing)
  ├─→ _fetch_model_file_hf: ImportError (huggingface-hub not installed)
  └─→ _fetch_model_file_hf: Network/auth errors from hf_hub_download

_guard_context:
  └─→ ValueError if prompt tokens exceed context_window

_complete / _stream_complete:
  └─→ RuntimeError from llama-cpp-python backend