Skip to content

Commit fa966b9

Browse files
0cc4mggerganov
authored andcommitted
Vulkan k-quant mmq and ggml-backend offload functionality (llama/6155)
* Fix Vulkan no kv offload incoherence * Add k-quant mul mat mat shaders * Rework working buffer allocation, reduces vram use noticeably Clean up cpu assist code, replaced with ggml-backend offload function * Default to all dedicated GPUs * Add fallback for integrated GPUs if no dedicated GPUs are found * Add debug info which device is allocating memory * Fix Intel dequant issue Fix validation issue * Fix Vulkan GGML_OP_GET_ROWS implementation * Clean up merge artifacts * Remove Vulkan warning
1 parent b83a9fc commit fa966b9

File tree

3 files changed

+328
-353
lines changed

3 files changed

+328
-353
lines changed

0 commit comments

Comments
 (0)