Llama-swap
De Le Wiki de Lug
Autres actions
Installation
Debian
# apt update && apt upgrade # apt install -y nodejs npm golang-go # cd /opt # git clone https://github.com/mostlygeek/llama-swap.git # cd llama-swap # make clean all
- On crée le service SystemD :
# vi /etc/systemd/system/llama-swap.service
[Unit] Description=llama-swap After=network.target [Service] Type=simple ExecStart=/opt/llama-swap/build/llama-swap-linux-amd64 --config /etc/llama-swap/config.yaml --listen 0.0.0.0:8080 Restart=on-failure RestartSec=3 [Install] WantedBy=multi-user.target
systemctl daemon-reload systemctl enable --now llama-swap
Configuration
- Si nécessaire, on crée le dossier :
# mkdir /etc/llama-swap
- On édite le fichier de configuration :
# vi /etc/llama-swap/config.yaml
- Exemple de configuration de base :
apiKeys: - "CLE_API_1" - "CLE_API_2" logLevel: info healthCheckTimeout: 500 globalTTL: 0 unloadTimeout: 10 logToStdout: both
Exemples modèle unique
Qwen3.8-27B Context 144384 Kvarn6/Kvarn6 (2x 12 Go de VRAM) Spécialiste codeur lent
...
models:
"Qwen3.8-27B":
name: "Qwen3.8-27B"
capabilities:
in:
- text
- image
out:
- text
tools: true
context: 144384
cmd: |
/opt/beellama.cpp/build/bin/llama-server
-m /opt/models/Qwen3.8-27B/UD-Q4_K_XL/Qwen3.8-27B-UD-Q4_K_XL.gguf
--mmproj /opt/models/Qwen3.8-27B/UD-Q4_K_XL/mmproj-BF16.gguf
--no-mmproj-offload
--image-min-tokens 1024
-ngl 999
-sm tensor
-ts 1/1
-t 4
-tb 4
-b 1024
-ub 1024
-c 144384
-np 1
-fa on
--load-mode none
-ctk kvarn6
-ctv kvarn6
--kv-tail-tokens 1024
--spec-type ngram-mod,ngram-map-k4v,draft-mtp
--spec-ngram-mod-n-match 24
--spec-ngram-mod-n-min 12
--spec-ngram-mod-n-max 48
--spec-ngram-map-k4v-size-n 12
--spec-ngram-map-k4v-size-m 48
--spec-ngram-map-k4v-min-hits 1
--spec-draft-n-max 3
--spec-draft-n-min 0
--spec-draft-p-min 0.75
--reasoning on
--reasoning-preserve
--jinja
--chat-template-file /opt/models/Qwen-Fixed-Chat-Templates/chat_template.jinja
--chat-template-kwargs '{"preserve_thinking": true}'
--reasoning-format deepseek
--reasoning-budget 16192
--reasoning-budget-message "D'accord, assez réfléchi, plus d'attente. Passons à l'action."
--slot-save-path /kv_cache/
--temp 1.0
--top-k 20
--top-p 0.95
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
--host 127.0.0.1
--port ${PORT}
gemma-4-31B-it-qat Context 32768 pour traduction (2x 12 Go de VRAM) Spécialiste traduction
...
models:
"traducteur":
name: "Gemma4-31B-QAT"
capabilities:
in:
- text
out:
- text
tools: true
context: 32768
cmd: |
env LLAMA_GRAPH_REUSE_DISABLE=1 /opt/llama.cpp/build/bin/llama-server
-m /opt/models/gemma-4-31B-it-qat/UD-Q4_K_XL/gemma-4-31B-it-qat-UD-Q4_K_XL.gguf
--host 127.0.0.1
--port ${PORT}
-ngl 999
-sm tensor
--tensor-split 1,1
-c 32768
-ctk q8_0
-ctv q8_0
-np 1
-b 1024
-ub 1024
-fa on
--spec-type ngram-mod,ngram-map-k4v,draft-mtp
--spec-ngram-mod-n-match 24
--spec-ngram-mod-n-min 12
--spec-ngram-mod-n-max 48
--spec-ngram-map-k4v-size-n 12
--spec-ngram-map-k4v-size-m 48
--spec-ngram-map-k4v-min-hits 1
--spec-draft-model /opt/models/gemma-4-31B-it-qat/UD-Q4_K_XL/MTP/mtp-gemma-4-31B-it-Q8_0.gguf
--spec-draft-n-max 4
--spec-draft-n-min 0
--spec-draft-p-min 0.75
--slot-save-path /kv_cache/
--temp 1.0
--top-p 0.95
--top-k 64
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
gemma-4-26B-A4B-it-qat (2x 12 Go VRAM) Généraliste rapide
# hf download unsloth/gemma-4-26B-A4B-it-qat-GGUF --include "*Q4*.gguf" --include "*mtp*Q4_0*.gguf" --include "*mmproj-BF16*.gguf" --local-dir /opt/models/gemma-4-26B-A4B-it-qat/UD-Q4_K_XL
...
models:
"Gemma4-26B":
name: "Gemma4-26B-A4B-QAT"
capabilities:
in:
- text
out:
- text
tools: true
context: 196608
cmd: |
env LLAMA_GRAPH_REUSE_DISABLE=1 /opt/llama.cpp/build/bin/llama-server
-m /opt/models/gemma-4-26B-A4B-it-qat/UD-Q4_K_XL/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf
--host 127.0.0.1
--port ${PORT}
-ngl 999
-sm tensor
--tensor-split 1,1
-c 196608
-ctk q8_0
-ctv q8_0
-np 1
-b 1024
-ub 1024
-fa on
--reasoning on
--slot-save-path /kv_cache/
--temp 1.0
--top-p 0.95
--top-k 64
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
--spec-type ngram-mod,ngram-map-k4v,draft-mtp
--spec-draft-model /opt/models/gemma-4-26B-A4B-it-qat/UD-Q4_K_XL/MTP/mtp-gemma-4-26B-A4B-it-Q4_0.gguf
--spec-draft-n-max 4
--spec-draft-n-min 0
--spec-draft-p-min 0.75
--spec-ngram-mod-n-match 24
--spec-ngram-mod-n-min 12
--spec-ngram-mod-n-max 48
--spec-ngram-map-k4v-size-n 12
--spec-ngram-map-k4v-size-m 48
--spec-ngram-map-k4v-min-hits 1
--mmproj /opt/models/gemma-4-26B-A4B-it-qat/UD-Q4_K_XL/mmproj-BF16.gguf
--no-mmproj-offload
gemma-4-31B-it (2x 12 Go VRAM) UD-IQ3_XXS Généraliste dense
# hf download unsloth/gemma-4-31B-it-GGUF --include "*UD-IQ3_XXS*.gguf" --include "*mtp-gemma-4-31B-it-Q8_0*.gguf" --include "*mmproj-BF16*.gguf" --local-dir /opt/models/gemma-4-31B-it-GGUF/UD-IQ3_XXS
...
models:
"Gemma4-31B":
name: "gemma-4-31B-it"
capabilities:
in:
- text
out:
- text
tools: true
context: 98304
cmd: |
env LLAMA_GRAPH_REUSE_DISABLE=1 /opt/llama.cpp/build/bin/llama-server
-m /opt/models/gemma-4-31B-it-GGUF/UD-IQ3_XXS/gemma-4-31B-it-UD-IQ3_XXS.gguf
--host 127.0.0.1
--port ${PORT}
-ngl 999
-sm tensor
--tensor-split 1,1
-c 98304
-ctk q8_0
-ctv q8_0
-np 1
-b 1024
-ub 1024
-fa on
--reasoning on
--slot-save-path /kv_cache/
--temp 1.0
--top-p 0.95
--top-k 64
--min-p 0.0
--presence-penalty 0.0
--repeat-penalty 1.0
--spec-type ngram-mod,ngram-map-k4v,draft-mtp
--spec-draft-model /opt/models/gemma-4-31B-it-GGUF/UD-IQ3_XXS/MTP/mtp-gemma-4-31B-it-Q8_0.gguf
--spec-draft-n-max 4
--spec-draft-n-min 0
--spec-draft-p-min 0.75
--spec-ngram-mod-n-match 24
--spec-ngram-mod-n-min 12
--spec-ngram-mod-n-max 48
--spec-ngram-map-k4v-size-n 12
--spec-ngram-map-k4v-size-m 48
--spec-ngram-map-k4v-min-hits 1
--mmproj /opt/models/gemma-4-31B-it-GGUF/UD-IQ3_XXS/mmproj-BF16.gguf
--no-mmproj-offload