Running Model locally - Cursor@Home

see also

• 4-node AMD cluster - runing 512GB Model locally / needs \(\)
  • 4-node Minisforum MS-S1 Max Strix Halo cluster
• tokenspeed / HN

# Models ⮺

• Deepseek V4.1 Flash vs Qwen 3.8 Flash, GLM 5.3 Flash, Gemini 3.8 Flash

see also

• Model benchmarks
• Top Model by task
• The Best Local Agentic Coding Workflow (Complete Guide)

# Security

Agentic backdoors via pretraining poisoning - Of course this is authored by people at Anthropic. Continuing their narrative that open sourced models are dangerous and that people should trust the closed sourced providers.

• Your Open Source Model Could Have a Hidden Time-Release Backdoor - demonstrate that harness context can be used to trigger a different answer in controlled condition.

# Qwen 3.8

• I gave Qwen 3.8 27B a reverse-engineering job and it finished in 30 minutes

# 16 Go

# Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller ⮺

• Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller - A custom hybrid quantization of the Qwen3.8-27B base model, specifically designed to fit Multi-Token Prediction (MTP) and long context into a strict 16GB VRAM hardware budget (like consumer RTX 4080 / RTX 5080 cards).
• mmproj-Qwen3.8-27B-f16.gguf - vision to text
• Le quant Q4_XS_Smaller : un 27B dense sur 16 Go de VRAM
• Le KV Cache en Q4 : 100 000 tokens de contexte, avec vision et MTP activés
• 100k tokens de contexte avec un KV Cache en Q4 - en theorie => 65k sur Nvdia

$ llama-server \
  --model Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf \
  --mmproj mmproj-Qwen3.8-27B-f16.gguf \               # vision support
  --ctx-size 100000 \
  --n-gpu-layers 999 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --parallel 1 \
  --reasoning-preserve \
  --host 0.0.0.0 --port 8080 \
  --alias "Qwen3.8-27B-i1"

# see also

• Qwen 3.8 27B sur 16 Go de VRAM : il a résolu le bug que DeepSeek V4 Flash a raté

• Qwen3.8 Flash: Over 7x Faster First Token - llama.cpp vs SGLang vs FreeToken. Benchmarked! (96GB)

• Qwen 3.8 27B GSQ RCO tested - 16GB Local LLM setup - from ISTA DAS Lab Austria,
  • Qwen 3.8 27B Quantizations Q1 - Q8 compared
  • Qwen 3.8 27B Reasoning Levels Tested - Not What I Expected
  • Qwen 3.8 27B Cold Fusion tested - 16GB Local LLM setup

# Qwen3.6 ⮺

This release delivers substantial upgrades, particularly in

• Agentic Coding: the model now handles frontend workflows and repository-level reasoning with greater fluency and precision.

• Thinking Preservation: we’ve introduced a new option to retain reasoning context from historical messages, streamlining iterative development and reducing overhead.

see also

• Qwen 3.6 Locally: Can I use it for my daily tasks?
  • 27B - better on coding task (but dense)
  • 35B - faster
  • Prompt Vault - a curated collection of structured coding prompts and challenges designed for testing Large Language Models (LLMs)
• QWEN 3.6 27B sur 16GB VRAM : La meilleure configuration
  • RTX 5080 and RTX 3090 Setup: 80 Tok/s on Qwen 3.6 27B Q8

# Qwen3.5 ⮺

Qwen3.5 is Alibaba’s new model family, including Qwen3.5-35B-A3B, 27B, 122B-A10B and 397B-A17B and the new Small series: Qwen3.5-0.8B, 2B, 4B and 9B. The multimodal hybrid reasoning LLMs deliver the strongest performances for their sizes. They support 256K context across 201 languages, have thinking + non-thinking, and excel in agentic coding, vision, chat, and long-context tasks. The 35B and 27B models work on a 22GB Mac / RAM device.

See all GGUFs here.

• How to run Qwen 3.5 locally

# Qwen3

# qwen3-14b-claude-4.5-opus-high-reasoning-distill

# Qwen3-Coder-30B-A3B-Instruct-GGUF

This is the standard instruction-tuned code model.
Typical strengths:

• generating code from prompts
• explaining code
• quick coding tasks
• IDE autocomplete / chat coding

It’s also easier to run locally because the total model size is smaller (30B).

• use Qwen3 Coder 30B A3B Instruct. This model delivers strong coding performance and reliable tool use.
  • Which local models actually work with Cline? AMD tested them all

RTX 5070 Ti

• Running 3.5 9B on my ASUS 5070ti 16G with lm studio

# Qwen3-Coder-Next ⮺

Much larger model: ~80B parameters total / ~3B active per token

• very sparse Mixture-of-Experts
• optimized for fast repeated calls in agent loops

Designed for coding agents that run loops:

• read repo
• plan changes
• edit files
• run tests
• debug
• repeat

It was trained with environment interaction and executable coding tasks, so it learns from test results and feedback loops.
Typical strengths:

• repo-scale refactors
• debugging from logs
• multi-step tool use
• IDE agents (Cline, Claude Code, etc.)

# DeepSeek v4.1 Flash

• I Run Full DeepSeek v4.1 Flash on a 12GB GPU

# Quantisation

# Q4_K_M ⮺

Q4_K_M is a compressed 4-bit version of an AI model that balances small size with good accuracy.

Q4 - means each model weight uses 4 bits instead of 16 bits, reducing memory usage by about 4×.

• smaller files
• faster inference
• lower RAM/VRAM requirements

K stands for K-block quantisation, an improved quantisation method used in GGML/GGUF models.
Instead of compressing each weight independently, it compresses blocks of weights together, which:

• preserves more information
• improves accuracy compared to older Q4 formats

M a variant of the quantisation scheme (S (small) / M (Medium) / L (Large) precision)

# GSQ-RCO

A particular quantization method/format intended to preserve quality at very low bit rates

# Model Parameters

# A3B ⮺

Some newer models use a Mixture-of-Experts (MoE) architecture.
A3B → Active 3B parameters during inference.

Instead of using all parameters every time, the model:

• Contains a large total parameter count (e.g., 35B).
• Activates only a subset of them per token using a router that selects a few “experts.”

Example

# -mtp

-mtp build (about 0.35 GB larger) that carries the model’s Multi-Token Prediction head for speculative decoding in llama.cpp. The weights are otherwise identical, so quality is unchanged.

you’re adding machinery that can make decoding faster.

And the MTP speedup only happens when your inference engine actually supports/activates MTP. This repo specifically targets llama.cpp’s draft-mtp speculative decoding.

# instruct ⮺

This refers to a model that is specifically trained or fine-tuned to follow instructions from users in a helpful, safe, and coherent way.

# DFlash ⮺

DFlash is a novel speculative decoding method that utilizes a lightweight block diffusion model for drafting. It enables efficient, high-quality parallel drafting that pushes the limits of inference speed.

# Uncensored

• Uncensored AI Costs $5 to Make. Nobody Can Stop It.
  • Heretic: Fully automatic censorship removal for language models - a tool that removes censorship (aka “safety alignment”) from transformer-based language models without expensive post-training. It combines an advanced implementation of directional ablation, also known as “abliteration” (Arditi et al. 2024, Lai 2025 (1, 2)), with a TPE-based parameter optimizer powered by Optuna.

# GGUF

• What’s in a GGUF, besides the weights - and what’s still missing? / HN