jeans/ollama
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

lower default NUM_PARALLEL to 2

Browse Source 
this is in part to "pay" for #10452, which doubled the default context length. The combination isn't fully neutral though, because even though the old 4x2k limit and the new 2x4k limit are memory equivalent, the 1x fallback is larger with 4k
This commit is contained in:
Devon Rifkin 2025-04-29 02:03:51 -07:00 committed by GitHub
parent 6ec71d8fb6
commit 67335dede2
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
server/sched.go
View File

@ -58,7 +58,7 @@ var defaultModelsPerGPU = 3
// Default automatic value for parallel setting // Default automatic value for parallel setting
// Model will still need to fit in VRAM. If this setting won't fit // Model will still need to fit in VRAM. If this setting won't fit
// we'll back off down to 1 to try to get it to fit // we'll back off down to 1 to try to get it to fit
var defaultParallel = 4 var defaultParallel = 2
var ErrMaxQueue = errors.New("server busy, please try again. maximum pending requests exceeded") var ErrMaxQueue = errors.New("server busy, please try again. maximum pending requests exceeded")