feat: vLLM serving numind/NuExtract3 on full GPU, ollama disabled
- Ollama scaled to 0 — vLLM gets entire 16GB GPU - NuExtract3 (9B) running at 80% GPU utilization - Both OLLAMA_BASE_URL and VLLM_BASE_URL point to vLLM service - Fixed gremlin-1 missing default route (no internet access) - Removed GPU resource limit (time-slicing handles allocation) - Removed init container (direct download works with internet fix)
This commit is contained in:
@@ -16,6 +16,7 @@ spec:
|
||||
app: vllm
|
||||
spec:
|
||||
runtimeClassName: nvidia
|
||||
enableServiceLinks: false
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: gremlin-1
|
||||
containers:
|
||||
@@ -32,7 +33,7 @@ spec:
|
||||
- "--port"
|
||||
- "8000"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.45"
|
||||
- "0.80"
|
||||
- "--max-model-len"
|
||||
- "8192"
|
||||
- "--max-num-seqs"
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: vllm
|
||||
name: nuextract
|
||||
namespace: vllm-service
|
||||
spec:
|
||||
ports:
|
||||
@@ -16,7 +16,7 @@ spec:
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: vllm-external
|
||||
name: nuextract-external
|
||||
namespace: vllm-service
|
||||
spec:
|
||||
ports:
|
||||
|
||||
@@ -37,7 +37,7 @@ metadata:
|
||||
data:
|
||||
default.conf: |
|
||||
upstream vllm_backend {
|
||||
server vllm.vllm-service.svc.cluster.local:8000;
|
||||
server nuextract.vllm-service.svc.cluster.local:8000;
|
||||
}
|
||||
|
||||
server {
|
||||
|
||||
Reference in New Issue
Block a user