Compare commits

..
3 Commits
Author SHA1 Message Date
Celes Renata 98764dd3f3 fix: add Harbor registry login to Woodpecker build steps
ci/woodpecker/push/woodpecker Pipeline failed
Build and Push / lint-and-test (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.adapters.broker_adapter name:broker-adapter]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.aggregation.worker name:aggregation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.extractor.worker name:extractor]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.ingestion.worker name:ingestion]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.lake_publisher.worker name:lake-publisher]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.parser.worker name:parser]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.recommendation.worker name:recommendation]) (push) Has been cancelled
Build and Push / build-services (map[cmd:python -m services.scheduler.app name:scheduler]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.api.app:app --host 0.0.0.0 --port 8000 name:query-api]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.risk.app:app --host 0.0.0.0 --port 8000 name:risk]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.symbol_registry.app:app --host 0.0.0.0 --port 8000 name:symbol-registry]) (push) Has been cancelled
Build and Push / build-services (map[cmd:uvicorn services.trading.app:app --host 0.0.0.0 --port 8000 name:trading-engine]) (push) Has been cancelled
Build and Push / build-dashboard (push) Has been cancelled
Build and Push / build-superset (push) Has been cancelled
Build and Push / integration-test (push) Has been cancelled
2026-04-19 07:56:55 +00:00
Celes Renata 911e42996b fix: use HARBOR_USERNAME secret in CI, add idempotent Harbor API setup to deploy script
- GitHub Actions: login with secrets.HARBOR_USERNAME + HARBOR_PASSWORD
- deploy.sh step 7: creates stonks-oracle project, robot account, tag retention
- All API calls are idempotent (safe to re-run)
2026-04-19 07:45:58 +00:00
Celes Renata 5be3ce2db9 feat: migrate CI/CD from GHCR to local Harbor registry
- Makefile: GHCR -> registry.celestium.life/stonks-oracle
- GitHub Actions: login to Harbor, use HARBOR_PASSWORD secret
- infra/k8s/*.yaml: all image refs -> registry.celestium.life
- inttest pipeline: remove GHCR pull secret (local registry, no auth)
- Steering docs: update registry/git endpoints
2026-04-19 07:34:28 +00:00
18 changed files with 305 additions and 48 deletions
+11 -13
View File
@@ -7,8 +7,8 @@ on:
branches: [main] branches: [main]
env: env:
REGISTRY: ghcr.io REGISTRY: registry.celestium.life
IMAGE_BASE: ghcr.io/${{ github.repository_owner }}/stonks-oracle IMAGE_BASE: registry.celestium.life/stonks-oracle
jobs: jobs:
lint-and-test: lint-and-test:
@@ -83,12 +83,12 @@ jobs:
steps: steps:
- uses: actions/checkout@v5 - uses: actions/checkout@v5
- name: Log in to GHCR - name: Log in to Harbor
uses: docker/login-action@v4 uses: docker/login-action@v4
with: with:
registry: ${{ env.REGISTRY }} registry: ${{ env.REGISTRY }}
username: ${{ github.actor }} username: ${{ secrets.HARBOR_USERNAME }}
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.HARBOR_PASSWORD }}
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 uses: docker/setup-buildx-action@v4
@@ -117,12 +117,12 @@ jobs:
steps: steps:
- uses: actions/checkout@v5 - uses: actions/checkout@v5
- name: Log in to GHCR - name: Log in to Harbor
uses: docker/login-action@v4 uses: docker/login-action@v4
with: with:
registry: ${{ env.REGISTRY }} registry: ${{ env.REGISTRY }}
username: ${{ github.actor }} username: ${{ secrets.HARBOR_USERNAME }}
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.HARBOR_PASSWORD }}
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 uses: docker/setup-buildx-action@v4
@@ -149,12 +149,12 @@ jobs:
steps: steps:
- uses: actions/checkout@v5 - uses: actions/checkout@v5
- name: Log in to GHCR - name: Log in to Harbor
uses: docker/login-action@v4 uses: docker/login-action@v4
with: with:
registry: ${{ env.REGISTRY }} registry: ${{ env.REGISTRY }}
username: ${{ github.actor }} username: ${{ secrets.HARBOR_USERNAME }}
password: ${{ secrets.GITHUB_TOKEN }} password: ${{ secrets.HARBOR_PASSWORD }}
- name: Set up Docker Buildx - name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4 uses: docker/setup-buildx-action@v4
@@ -217,8 +217,6 @@ jobs:
kubectl cluster-info || echo "WARNING: kubectl cannot reach cluster API" kubectl cluster-info || echo "WARNING: kubectl cannot reach cluster API"
- name: Run integration tests - name: Run integration tests
env:
GHCR_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: | run: |
bash infra/inttest/run_pipeline.sh \ bash infra/inttest/run_pipeline.sh \
--image-tag ${{ github.sha }} \ --image-tag ${{ github.sha }} \
+2 -3
View File
@@ -13,7 +13,7 @@ The namespace is NOT managed by Helm — it's created by `runmefirst.sh` with He
- Services defined in `values.yaml` under `services:` — the deployments template iterates over them - Services defined in `values.yaml` under `services:` — the deployments template iterates over them
- Adding a new service: add entry to `values.yaml`, add network policy if it needs ingress, add ingress if it needs external access - Adding a new service: add entry to `values.yaml`, add network policy if it needs ingress, add ingress if it needs external access
- Dashboard uses nginx-unprivileged on port 8080 (not 80) - Dashboard uses nginx-unprivileged on port 8080 (not 80)
- Superset uses custom image `ghcr.io/celesrenata/stonks-oracle/superset:latest` with trino + psycopg2 drivers - Superset uses custom image `registry.celestium.life/stonks-oracle/superset:latest` with trino + psycopg2 drivers
## TLS ## TLS
- Internal services: use `ca-issuer` ClusterIssuer (local CA) - Internal services: use `ca-issuer` ClusterIssuer (local CA)
@@ -44,9 +44,8 @@ The namespace is NOT managed by Helm — it's created by `runmefirst.sh` with He
- Ollama: `ollama.ollama-service.svc.cluster.local:11434` - Ollama: `ollama.ollama-service.svc.cluster.local:11434`
## Images ## Images
- All images from `ghcr.io/celesrenata/stonks-oracle/<service>:latest` - All images from `registry.celestium.life/stonks-oracle/<service>:latest`
- Use `imagePullPolicy: Always` - Use `imagePullPolicy: Always`
- Use `imagePullSecrets` referencing `ghcr-credentials`
## Labels ## Labels
- `app.kubernetes.io/part-of: stonks-oracle` - `app.kubernetes.io/part-of: stonks-oracle`
+5 -3
View File
@@ -29,18 +29,20 @@ Three-layer signal aggregation engine:
- Trading Engine: `https://stonks-trading.celestium.life` - Trading Engine: `https://stonks-trading.celestium.life`
- Superset: `https://stonks-dash.celestium.life` - Superset: `https://stonks-dash.celestium.life`
- Trino: `https://stonks-trino.celestium.life` - Trino: `https://stonks-trino.celestium.life`
- Gitea: `https://git.celestium.life`
- Harbor Registry: `https://registry.celestium.life`
## Infrastructure ## Infrastructure
- Kubernetes cluster: 4x NixOS nodes (gremlin-1 through gremlin-4), reachable via `kubectl`, `virtctl`, `ssh root@gremlin-{1,2,3,4}` - Kubernetes cluster: 4x NixOS nodes (gremlin-1 through gremlin-4), reachable via `kubectl`, `virtctl`, `ssh root@gremlin-{1,2,3,4}`
- NixOS configs stored at `/etc/nixos` on gremlin-1, git-pushed to other hosts - NixOS configs stored at `/etc/nixos` on gremlin-1, git-pushed to other hosts
- Ingress: Traefik, domain `*.celestium.life` - Ingress: Traefik, domain `*.celestium.life`
- Cert-Manager: `ca-issuer` (local CA) for internal services - Cert-Manager: `ca-issuer` (local CA) for internal services
- Container registry: `ghcr.io/celesrenata/stonks-oracle` - Container registry: `registry.celestium.life/stonks-oracle`
## CI/CD ## CI/CD
- GitHub Actions workflow at `.github/workflows/build.yml` - GitHub Actions workflow at `.github/workflows/build.yml`
- Push to `main` triggers: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to GHCR - Push to `main` triggers: lint → pytest → frontend vitest → build all service images + dashboard + superset → push to Harbor
- Images tagged as `ghcr.io/celesrenata/stonks-oracle/<service>:<sha>` and `:latest` - Images tagged as `registry.celestium.life/stonks-oracle/<service>:<sha>` and `:latest`
- Dashboard image: `frontend/Dockerfile` (multi-stage: node:24 → nginx-unprivileged on port 8080) - Dashboard image: `frontend/Dockerfile` (multi-stage: node:24 → nginx-unprivileged on port 8080)
- Superset image: `docker/Dockerfile.superset` (apache/superset + trino + psycopg2) - Superset image: `docker/Dockerfile.superset` (apache/superset + trino + psycopg2)
- Python service images: `docker/Dockerfile` with `SERVICE_CMD` build arg - Python service images: `docker/Dockerfile` with `SERVICE_CMD` build arg
+70
View File
@@ -39,6 +39,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -68,6 +73,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -97,6 +107,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -126,6 +141,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -155,6 +175,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -184,6 +209,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -213,6 +243,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -242,6 +277,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -271,6 +311,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -300,6 +345,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -329,6 +379,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -358,6 +413,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -387,6 +447,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
@@ -415,6 +480,11 @@ steps:
custom_dns: 192.168.42.1 custom_dns: 192.168.42.1
no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life" no_proxy: "10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,.local,.celestium.life"
logins: logins:
- registry: https://registry.celestium.life
username:
from_secret: harbor_username
password:
from_secret: harbor_password
- registry: https://index.docker.io/v1/ - registry: https://index.docker.io/v1/
username: username:
from_secret: docker_username from_secret: docker_username
+11 -11
View File
@@ -1,6 +1,6 @@
REPO_OWNER := celesrenata REPO_OWNER := celesrenata
REPO_NAME := stonks-oracle REPO_NAME := stonks-oracle
GHCR := ghcr.io/$(REPO_OWNER)/$(REPO_NAME) REGISTRY := registry.celestium.life/stonks-oracle
SHA := $(shell git rev-parse --short HEAD 2>/dev/null || echo "dev") SHA := $(shell git rev-parse --short HEAD 2>/dev/null || echo "dev")
SERVICES := scheduler symbol-registry ingestion parser extractor aggregation recommendation risk broker-adapter lake-publisher query-api SERVICES := scheduler symbol-registry ingestion parser extractor aggregation recommendation risk broker-adapter lake-publisher query-api
@@ -12,7 +12,7 @@ help:
@echo " lint - Run ruff linter" @echo " lint - Run ruff linter"
@echo " test - Run pytest" @echo " test - Run pytest"
@echo " build - Build all service images locally" @echo " build - Build all service images locally"
@echo " push - Push all images to GHCR" @echo " push - Push all images to registry"
@echo " deploy - Apply K8s manifests" @echo " deploy - Apply K8s manifests"
@echo " clean - Remove local images" @echo " clean - Remove local images"
@@ -40,26 +40,26 @@ build:
echo "Building $$svc ($$cmd)..."; \ echo "Building $$svc ($$cmd)..."; \
docker build \ docker build \
--build-arg "SERVICE_CMD=$$cmd" \ --build-arg "SERVICE_CMD=$$cmd" \
-t $(GHCR)/$$svc:$(SHA) \ -t $(REGISTRY)/$$svc:$(SHA) \
-t $(GHCR)/$$svc:latest \ -t $(REGISTRY)/$$svc:latest \
-f docker/Dockerfile . || exit 1; \ -f docker/Dockerfile . || exit 1; \
done done
@echo "Building dashboard..." @echo "Building dashboard..."
docker build \ docker build \
-t $(GHCR)/dashboard:$(SHA) \ -t $(REGISTRY)/dashboard:$(SHA) \
-t $(GHCR)/dashboard:latest \ -t $(REGISTRY)/dashboard:latest \
-f frontend/Dockerfile frontend/ || exit 1 -f frontend/Dockerfile frontend/ || exit 1
@echo "Building superset..." @echo "Building superset..."
docker build \ docker build \
-t $(GHCR)/superset:$(SHA) \ -t $(REGISTRY)/superset:$(SHA) \
-t $(GHCR)/superset:latest \ -t $(REGISTRY)/superset:latest \
-f docker/Dockerfile.superset docker/ || exit 1 -f docker/Dockerfile.superset docker/ || exit 1
push: push:
@for svc in $(SERVICES); do \ @for svc in $(SERVICES); do \
echo "Pushing $$svc..."; \ echo "Pushing $$svc..."; \
docker push $(GHCR)/$$svc:$(SHA); \ docker push $(REGISTRY)/$$svc:$(SHA); \
docker push $(GHCR)/$$svc:latest; \ docker push $(REGISTRY)/$$svc:latest; \
done done
deploy: deploy:
@@ -70,5 +70,5 @@ deploy:
clean: clean:
@for svc in $(SERVICES); do \ @for svc in $(SERVICES); do \
docker rmi $(GHCR)/$$svc:$(SHA) $(GHCR)/$$svc:latest 2>/dev/null || true; \ docker rmi $(REGISTRY)/$$svc:$(SHA) $(REGISTRY)/$$svc:latest 2>/dev/null || true; \
done done
+4 -7
View File
@@ -236,15 +236,12 @@ if ! kubectl create namespace "$NAMESPACE"; then
fi fi
# ── Create GHCR image pull secret (if token available) ─────────────────────── # ── Create GHCR image pull secret (if token available) ───────────────────────
# NOTE: Images now served from Harbor at registry.celestium.life (no auth needed for pulls)
# This block is kept for backward compatibility but is no longer required
if [ -n "${GHCR_TOKEN:-}" ]; then if [ -n "${GHCR_TOKEN:-}" ]; then
log "Creating ghcr-credentials secret ..." log "GHCR_TOKEN set but images are on local Harbor — skipping GHCR secret"
kubectl create secret docker-registry ghcr-credentials \
--docker-server=ghcr.io \
--docker-username=celesrenata \
--docker-password="$GHCR_TOKEN" \
-n "$NAMESPACE" || true
else else
log "GHCR_TOKEN not set — skipping image pull secret (images must be pullable without auth)" log "Images served from registry.celestium.life (no pull secret needed)"
fi fi
# ── Create Docker Hub pull secret (avoid rate limits) ──────────────────────── # ── Create Docker Hub pull secret (avoid rate limits) ────────────────────────
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: aggregation-worker - name: aggregation-worker
image: ghcr.io/celesrenata/stonks-oracle/aggregation:latest image: registry.celestium.life/stonks-oracle/aggregation:latest
imagePullPolicy: Always imagePullPolicy: Always
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: broker-adapter - name: broker-adapter
image: ghcr.io/celesrenata/stonks-oracle/broker-adapter:latest image: registry.celestium.life/stonks-oracle/broker-adapter:latest
imagePullPolicy: Always imagePullPolicy: Always
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: extractor-worker - name: extractor-worker
image: ghcr.io/celesrenata/stonks-oracle/extractor:latest image: registry.celestium.life/stonks-oracle/extractor:latest
imagePullPolicy: Always imagePullPolicy: Always
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: ingestion-worker - name: ingestion-worker
image: ghcr.io/celesrenata/stonks-oracle/ingestion:latest image: registry.celestium.life/stonks-oracle/ingestion:latest
imagePullPolicy: Always imagePullPolicy: Always
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: lake-publisher - name: lake-publisher
image: ghcr.io/celesrenata/stonks-oracle/lake-publisher:latest image: registry.celestium.life/stonks-oracle/lake-publisher:latest
imagePullPolicy: Always imagePullPolicy: Always
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: parser-worker - name: parser-worker
image: ghcr.io/celesrenata/stonks-oracle/parser:latest image: registry.celestium.life/stonks-oracle/parser:latest
imagePullPolicy: Always imagePullPolicy: Always
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: query-api - name: query-api
image: ghcr.io/celesrenata/stonks-oracle/query-api:latest image: registry.celestium.life/stonks-oracle/query-api:latest
imagePullPolicy: Always imagePullPolicy: Always
ports: ports:
- containerPort: 8000 - containerPort: 8000
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: recommendation-worker - name: recommendation-worker
image: ghcr.io/celesrenata/stonks-oracle/recommendation:latest image: registry.celestium.life/stonks-oracle/recommendation:latest
imagePullPolicy: Always imagePullPolicy: Always
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: risk-engine - name: risk-engine
image: ghcr.io/celesrenata/stonks-oracle/risk:latest image: registry.celestium.life/stonks-oracle/risk:latest
imagePullPolicy: Always imagePullPolicy: Always
ports: ports:
- containerPort: 8000 - containerPort: 8000
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: scheduler - name: scheduler
image: ghcr.io/celesrenata/stonks-oracle/scheduler:latest image: registry.celestium.life/stonks-oracle/scheduler:latest
imagePullPolicy: Always imagePullPolicy: Always
securityContext: securityContext:
allowPrivilegeEscalation: false allowPrivilegeEscalation: false
+1 -1
View File
@@ -28,7 +28,7 @@ spec:
type: RuntimeDefault type: RuntimeDefault
containers: containers:
- name: symbol-registry-api - name: symbol-registry-api
image: ghcr.io/celesrenata/stonks-oracle/symbol-registry:latest image: registry.celestium.life/stonks-oracle/symbol-registry:latest
imagePullPolicy: Always imagePullPolicy: Always
ports: ports:
- containerPort: 8000 - containerPort: 8000
+191
View File
@@ -0,0 +1,191 @@
#!/usr/bin/env bash
set -euo pipefail
# deploy.sh — Deploy Harbor container registry to Kubernetes
# Run from gremlin-1: bash ~/sources/kube/harbor/deploy.sh
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
echo "=== Harbor Container Registry Deploy ==="
echo ""
# -------------------------------------------------------
# 1. Create namespace
# -------------------------------------------------------
echo "--- Step 1: Creating namespace ---"
kubectl create namespace harbor-service --dry-run=client -o yaml | kubectl apply -f -
echo " ✓ harbor-service namespace ready"
echo ""
# -------------------------------------------------------
# 2. Proxy CA cert (for Squid SSL bump)
# -------------------------------------------------------
echo "--- Step 2: Proxy CA cert ---"
CA_CERT_PATH="/tmp/harbor-home.crt"
if curl -sf http://192.168.42.1/home.crt -o "$CA_CERT_PATH" 2>/dev/null; then
if ! kubectl get configmap proxy-ca-cert -n harbor-service > /dev/null 2>&1; then
kubectl create configmap proxy-ca-cert --from-file=ca.crt="$CA_CERT_PATH" -n harbor-service
echo " ✓ proxy-ca-cert created"
else
echo " ✓ proxy-ca-cert already exists"
fi
else
echo " ⚠ Could not fetch CA cert (non-fatal)"
fi
echo ""
# -------------------------------------------------------
# 3. Create NFS directories (requires root for mount)
# -------------------------------------------------------
echo "--- Step 3: Ensuring NFS directories ---"
if [ "$(id -u)" -eq 0 ]; then
mkdir -p /tmp/harbor-nfs-init
mount -t nfs 192.168.42.8:/volume1/Kubernetes/harbor /tmp/harbor-nfs-init 2>/dev/null || true
mkdir -p /tmp/harbor-nfs-init/data/registry
mkdir -p /tmp/harbor-nfs-init/data/redis
mkdir -p /tmp/harbor-nfs-init/data/jobservice
mkdir -p /tmp/harbor-nfs-init/data/trivy
umount /tmp/harbor-nfs-init 2>/dev/null || true
rmdir /tmp/harbor-nfs-init 2>/dev/null || true
echo " ✓ NFS directories ready"
else
echo " ⚠ Not root — skipping NFS dir creation (dirs may already exist)"
fi
echo ""
# -------------------------------------------------------
# 4. Apply PVs and PVCs
# -------------------------------------------------------
echo "--- Step 4: Applying PVs and PVCs ---"
kubectl apply -f "$SCRIPT_DIR/../pvs/harbor-pv.yaml"
kubectl apply -f "$SCRIPT_DIR/pvcs.yaml"
echo " ✓ PVs and PVCs applied"
echo ""
# -------------------------------------------------------
# 5. Remove old plain Docker Registry ingress
# -------------------------------------------------------
echo "--- Step 5: Checking old registry ingress ---"
if kubectl get ingress registry-ingress -n git-server > /dev/null 2>&1; then
echo " Removing old registry ingress from git-server namespace..."
kubectl delete ingress registry-ingress -n git-server
echo " ✓ Old registry ingress removed"
else
echo " ✓ No old registry ingress found"
fi
echo ""
# -------------------------------------------------------
# 6. Install/upgrade Harbor via Helm
# -------------------------------------------------------
echo "--- Step 6: Installing Harbor ---"
helm repo add harbor https://helm.goharbor.io 2>/dev/null || true
helm repo update harbor 2>/dev/null || true
HARBOR_EXISTS=$(helm list -n harbor-service -q 2>/dev/null | grep -c harbor || true)
if [ "${HARBOR_EXISTS:-0}" -gt 0 ]; then
echo " Harbor already installed — upgrading..."
else
echo " Fresh Harbor install..."
fi
helm upgrade --install harbor harbor/harbor \
--namespace harbor-service \
--values "$SCRIPT_DIR/values.yaml" \
--timeout 10m \
--wait
echo ""
echo " Waiting for Harbor core to be ready..."
kubectl wait --for=condition=ready pod -l app=harbor,component=core -n harbor-service --timeout=180s || true
echo ""
# -------------------------------------------------------
# 7. Configure Harbor via API (idempotent)
# -------------------------------------------------------
echo "--- Step 7: Configuring Harbor project and robot account ---"
HARBOR_API="https://registry.celestium.life/api/v2.0"
HARBOR_AUTH="admin:St0nks0racl3!"
# Create stonks-oracle project if it doesn't exist
if curl -sf -u "$HARBOR_AUTH" "$HARBOR_API/projects?name=stonks-oracle" | python3 -c "import sys,json; sys.exit(0 if json.load(sys.stdin) else 1)" 2>/dev/null; then
echo " ✓ Project stonks-oracle already exists"
else
curl -sf -X POST -u "$HARBOR_AUTH" \
-H "Content-Type: application/json" \
"$HARBOR_API/projects" \
-d '{"project_name":"stonks-oracle","public":true,"metadata":{"auto_scan":"true","severity":"high"},"storage_limit":-1}'
echo " ✓ Project stonks-oracle created (public, auto-scan enabled)"
fi
# Create CI robot account if it doesn't exist
EXISTING_ROBOT=$(curl -sf -u "$HARBOR_AUTH" "$HARBOR_API/robots" 2>/dev/null | python3 -c "
import sys, json
robots = json.load(sys.stdin)
for r in robots:
if 'ci-push' in r.get('name', ''):
print(r['name'])
break
" 2>/dev/null || true)
if [ -n "$EXISTING_ROBOT" ]; then
echo " ✓ Robot account already exists: $EXISTING_ROBOT"
else
ROBOT_RESP=$(curl -sf -X POST -u "$HARBOR_AUTH" \
-H "Content-Type: application/json" \
"$HARBOR_API/robots" \
-d '{
"name":"ci-push",
"description":"CI/CD pipeline push account",
"duration":-1,
"level":"project",
"permissions":[{
"namespace":"stonks-oracle",
"kind":"project",
"access":[
{"resource":"repository","action":"push"},
{"resource":"repository","action":"pull"},
{"resource":"tag","action":"create"},
{"resource":"tag","action":"list"},
{"resource":"artifact","action":"read"}
]
}]
}')
ROBOT_NAME=$(echo "$ROBOT_RESP" | python3 -c "import sys,json; print(json.load(sys.stdin)['name'])")
ROBOT_SECRET=$(echo "$ROBOT_RESP" | python3 -c "import sys,json; print(json.load(sys.stdin)['secret'])")
echo " ✓ Robot account created"
echo " Username: $ROBOT_NAME"
echo " Secret: $ROBOT_SECRET"
echo " ⚠ Save these — add as HARBOR_USERNAME and HARBOR_PASSWORD in your CI secrets"
fi
# Set tag retention policy (idempotent — check if one exists first)
PROJECT_ID=$(curl -sf -u "$HARBOR_AUTH" "$HARBOR_API/projects?name=stonks-oracle" | python3 -c "import sys,json; print(json.load(sys.stdin)[0]['project_id'])" 2>/dev/null || true)
if [ -n "$PROJECT_ID" ]; then
EXISTING_RETENTION=$(curl -sf -u "$HARBOR_AUTH" "$HARBOR_API/retentions" 2>/dev/null || true)
if echo "$EXISTING_RETENTION" | python3 -c "import sys,json; d=json.load(sys.stdin); sys.exit(0 if isinstance(d,list) and len(d)>0 else 1)" 2>/dev/null; then
echo " ✓ Tag retention policy already exists"
else
curl -sf -X POST -u "$HARBOR_AUTH" \
-H "Content-Type: application/json" \
"$HARBOR_API/retentions" \
-d "{
\"algorithm\":\"or\",
\"scope\":{\"level\":\"project\",\"ref\":$PROJECT_ID},
\"trigger\":{\"kind\":\"Schedule\",\"settings\":{\"cron\":\"0 0 0 * * *\"}},
\"rules\":[
{\"disabled\":false,\"action\":\"retain\",\"scope_selectors\":{\"repository\":[{\"kind\":\"doublestar\",\"decoration\":\"repoMatches\",\"pattern\":\"**\"}]},\"tag_selectors\":[{\"kind\":\"doublestar\",\"decoration\":\"matches\",\"pattern\":\"latest\"}],\"params\":{}},
{\"disabled\":false,\"action\":\"retain\",\"scope_selectors\":{\"repository\":[{\"kind\":\"doublestar\",\"decoration\":\"repoMatches\",\"pattern\":\"**\"}]},\"tag_selectors\":[{\"kind\":\"doublestar\",\"decoration\":\"matches\",\"pattern\":\"**\"}],\"params\":{\"latestPushedK\":10}}
]
}" 2>/dev/null && echo " ✓ Tag retention policy created (keep latest + last 10 tags, daily)" || echo " ⚠ Could not create retention policy (non-fatal)"
fi
fi
echo ""
echo "=== Harbor Deploy Complete ==="
echo ""
echo " URL: https://registry.celestium.life"
echo " Login: admin / St0nks0racl3!"
echo ""
echo " Pods:"
kubectl get pods -n harbor-service