Llama (ChatGPT@Home) 🦙

A 2 files on prem LLM. - Intro to Large Language Models / Llama In My Living Room 🎶

# Build ⮺

Integrates with any harness like OpenFox

$ git clone --depth 1  https://github.com/ggerganov/llama.cpp
$ cd llama.cpp

# compile with a proper option (see below), then install:
$ sudo "$(command -v cmake)" --install build
$ sudo ldconfig

nix cmake does not search /usr. cmake --system-information reports CMAKE_SYSTEM_PREFIX_PATH "/home/yves/.nix-profile;/usr/local", so find_package(Vulkan) takes the nix headers but never finds the distro loader: Could NOT find Vulkan (missing: Vulkan_LIBRARY). Tell it where the loader lives. Durable form: home.sessionVariables.CMAKE_LIBRARY_PATH.

$ export CMAKE_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu   # distro loader, nix headers stay

$ cmake -B build -DGGML_VULKAN=1
$ cmake --build build --config Release

# Install ⮺

Pitfall: sudo cannot see cmake

sudo cmake --install build fails with cmake: command not found. cmake exists only in ~/.nix-profile/bin (there is no /usr/bin/cmake), and sudo resets PATH to its secure_path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin. sudo -E does not help, secure_path wins over a preserved PATH.

Pick one:

$ sudo "$(command -v cmake)" --install build     # most surgical, absolute path only
$ sudo env "PATH=$PATH" cmake --install build    # hand your whole PATH to the root shell

Then refresh the runtime linker cache, else the installed binaries cannot resolve libggml*:

$ sudo ldconfig

⚠️ Summary of compilation fix

Use sudo ldconfig, not bare ldconfig. Bare resolves to ~/.nix-profile/bin/ldconfig first and dies with Can't open cache file /nix/store/...-glibc-2.42-84/etc/ld.so.cache. sudo resolves through secure_path to the real /usr/sbin/ldconfig, which reads /etc/ld.so.conf.d/{libc,usr_local_lib}.conf.

see

LD_LIBRARY_PATH & ldconfig

# Summary of change

What	Change	Why
`CMAKE_LIBRARY_PATH`	add `/usr/lib/x86_64-linux-gnu`	nix cmake never searches `/usr` -> `Vulkan_LIBRARY-NOTFOUND`
`hm-session-vars.sh` sourcing	guarded `source` added to `~/.profile` + `~/.bashrc`	home-manager only writes the file, nothing sources it
`CMAKE_INSTALL_LIBDIR`	`lib64` -> `lib`	`/usr/local/lib64` does not exist and is not in `/etc/ld.so.conf`
`CMAKE_INSTALL_RPATH`	left unset	`file(RPATH_CHANGE)` aborts on `libggml-base.so`, which has no build-tree RUNPATH
install command	`sudo cmake` -> `sudo "$(command -v cmake)"`	`sudo` drops `~/.nix-profile/bin`
`sudo ldconfig`	run once after install	`/usr/local/lib` only reaches the loader through the ld.so cache

Result: loader from the distro, headers from nix 1.4.357, no nix path in RUNPATH.

# Check

$ /usr/local/bin/llama-cli --list-devices
Available devices:
  Vulkan0: Intel(R) UHD Graphics (CML GT2) (11796 MiB, 5185 MiB free)
  Vulkan1: NVIDIA GeForce MX350 (2294 MiB, 2231 MiB free)

# Running Model 🤖

# Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller

A custom hybrid quantization of the Qwen3.8-27B base model, specifically designed to fit Multi-Token Prediction (MTP) and long context into a strict 16GB VRAM hardware budget (like consumer RTX 4080 / RTX 5080 cards).

--ctx-size 100000 does not fit on 16 GB NVIDIA — it aborts at startup while loading the mmproj. see where-the-vram-goes

But --ctx-size 65000 does

$ llama-server \
  --model ~/.lmstudio/models/jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf \
  --mmproj ~/.lmstudio/models/bartowski/Qwen3.8-27B-GGUF/mmproj-Qwen3.8-27B-f16.gguf \
  --ctx-size 65000 \
  --n-gpu-layers 999 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --parallel 1 \
  --reasoning-preserve \
  --host 0.0.0.0 --port 8081 \
  --alias "Qwen3.8-27B-i1"

Detailed analysis

# Where the VRAM goes

ggml_vulkan: Device memory allocation of size 927587264 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
GGML_ASSERT(buffer) failed

927587264 is the vision projector allocation. By the time it runs, weights + SSM state + KV cache have already taken the card.

Why 100k fits on the RX 7800 XT but not here

The NVIDIA driver holds back a fixed 461 MiB Reserved pool at init. See cuda. Which are at play

Item	Size
model weights (65 layers, full offload)	12.61 GiB
SSM state (49 linear-attention layers, `q4_0`)	0.58 GiB — fixed, independent of `--ctx-size`
KV cache `q4_0` @ 100096 ctx	1.72 GiB
subtotal	~15.4 GiB
mmproj (vision encoder)	0.86 GiB
free VRAM	15.47 GiB

KV math — read from the GGUF header (qwen35 architecture):

hybrid attention, full_attention_interval = 4 → only 16 of 65 layers keep a KV cache
key_length = value_length = 256 — not the 128 you’d assume from head_count = 24; this doubles the cache
head_count_kv = 4 → per layer per token: 2 × 4 × 256 × 0.5625 B (q4_0 = 4.5 bits) = 1152 B
× 16 layers = 18 KiB/token → × 100096 ≈ 1.72 GiB

The SSM state comes from llama_memory_recurrent (598.50 MiB, logged identically at every --ctx-size tested). It is architectural, so switching Vulkan → CUDA cannot recover it — see cuda.

Change	Frees	Cost
`--ctx-size 65536`	~0.59 GiB	best quality-per-cost; recommended — projector stays on GPU
`--no-mmproj-offload`	1.11 GiB	projector runs on CPU; vision still works, but image encode is 7.5× slower
drop `--n-gpu-layers 999`	varies	lets `common_fit_params` auto-fit instead of aborting
`--spec-draft-n-max 2`	small	lower speculation hit rate

At 65000 the projector on GPU leaves only 56 MiB free (measured); with --no-mmproj-offload it leaves 1189 MiB.

So the report that “100k context won’t fit on NVIDIA/CUDA but does on AMD + Vulkan” has a concrete cause - due to driver reservation and a bit less vram on the card ( 16303 MiB fo RTX 5070 Ti vs 16384 MiB for RX 7800 XT). Using CUDA instead of Vulkan on the RTX 5070 Ti won’t change the outcome.

# Early Run

This date from 2022 early test with ROCm on a RX 580 - 8Gb

# Download Model ⮺

Already quantized/converted model

# Running Chat ⮺

$ ./main -m ./models/7B/llama-2-7b.Q4_K_M.gguf  --repeat_penalty 1.0 -ngl 100 --color -i -r "User:" -f prompts/chat-with-bob.txt

User:Where is Bratislava ?
Bob: Bratislava is the capital of Slovakia, which is a country in the European Union.
User:When was it founded ?
Bob: Bratislava was founded in 1536, by King Ferdinand I. # Beware of LLM answer...

see

Offloading 0 layers to GPU #1956 - use -ngl 100 to force using VRAM
Computer Hardware Required to Run LLaMA AI Model Locally (GPU, CPU, RAM, SSD)

# UI ? ⮺

gpt-llama.cpp - Replace OpenAi’s GPT APIs with llama.cpp’s supported models locally
LlamaGPT - self-hosted, offline, ChatGPT-like chatbot, powered by Llama 2. 100% private, with no data leaving your device.

# Llama vs ChatGPT ⮺

ChatGPT vs Llama2 - A Detailed Statistical Comparison
How To Run Llama 2 on Anything
- Run a Chatgpt-like Chatbot on a Single GPU with ROCm
- Hardware requirements for Llama 2 #425 - llama-2-13b-chat.ggmlv3.q4_0.bin, llama-2-13b-chat.ggmlv3.q8_0.bin and llama-2-70b-chat.ggmlv3.q4_0.bin from TheBloke. - huggingface-cli download TheBloke/Llama-2-13B-chat-GGUF llama-2-13b-chat.Q4_K_M.gguf
  - ./main -ngl 32 -m /home/yves/.cache/huggingface/hub/models--TheBloke--Llama-2-13B-chat-GGUF/snapshots/4458acc949de0a9914c3eab623904d4fe999050a/llama-2-13b-chat.Q4_K_M.gguf --color --temp 0.7 --repeat_penalty 1.1 -n -1 -i -ins
- Cheapest hardware to run Llama 2 70B - Anything with 64GB of memory will run a quantized 70B model. What else you need depends on what is acceptable speed for you.

# see also

ruby_llm / HN - A delightful Ruby way to work with AI. No configuration madness, no complex callbacks, no handler hell – just beautiful, expressive Ruby code.
NanoChat – The best ChatGPT that $100 can buy

# Microgpt ⮺ / HN

A single file of 200 lines of pure Python with no dependencies that trains and inferences a GPT. This file contains the full algorithmic content of what is needed: dataset of documents, tokenizer, autograd engine, a GPT-2-like neural network architecture, the Adam optimizer, training loop, and inference loop. Everything else is just efficiency.

Written on December 3, 2023, Last update on March 11, 2026

LLM at_home

#4577