admin/stonks-oracle
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5f6d23888aae7b6c354f68f49cb2173597436c10
stonks-oracle/.kiro/specs/local-cicd-pipeline/tasks.md
T

9.7 KiB
Raw Blame History

Implementation Plan: Local CI/CD Pipeline

Overview

Migrate the Stonks Oracle CI/CD pipeline from GitHub-dependent infrastructure (ARC + GHCR) to a fully local pipeline using Gitea, Woodpecker CI, and the local Docker registry. Update ArgoCD and Kargo to source from local infrastructure. All pipeline scripts and manifests live in ~/sources/kube/pipelines/ on gremlin-1.

Tasks

  • 1. Tear down ARC infrastructure

    • 1.1 Remove pipelines/arc/ directory (values.yaml, runner-scaleset.yaml, runner-rbac.yaml)
      • Requirements: 7.1, 7.2, 7.4
    • 1.2 Remove pipelines/pvs/arc-pv.yaml — ARC NFS PV no longer needed
      • Requirements: 7.5
    • 1.3 Update pipelines/runmefirst.sh — add ARC teardown section at the beginning: uninstall arc-runner-set and arc Helm releases, delete arc/runner-rbac.yaml ClusterRoleBinding, delete pipeline-arc-pv PV, delete arc-system namespace. Use || true and --ignore-not-found for idempotency.
      • Requirements: 7.1, 7.2, 7.3, 7.4, 7.5, 11.7

  • 2. Create Woodpecker CI NFS PersistentVolume

    • 2.1 Create pipelines/pvs/woodpecker-pv.yaml — NFS PV for Woodpecker server data (5Gi, nfs://192.168.42.8:/volume1/Kubernetes/pipelines/woodpecker, persistentVolumeReclaimPolicy: Retain, label app: pipeline-woodpecker)
      • Requirements: 24.1, 24.4

  • 3. Create Gitea configuration script

    • 3.1 Create pipelines/gitea/setup.sh — Shell script that automates Gitea initial setup via REST API: complete install wizard or create admin user, create OAuth2 application for Woodpecker CI (callback URL https://stonks-ci.celestium.life/authorize), create stonks-oracle repository. Script outputs OAuth2 client_id and client_secret for use by Woodpecker Helm install. All operations idempotent (check-before-create pattern).
      • Requirements: 1.1, 1.2, 1.5, 2.3

  • 4. Create Woodpecker CI manifests

    • 4.1 Create pipelines/woodpecker/values.yaml — Helm values for woodpecker/woodpecker chart in woodpecker namespace. Server: Gitea OAuth2 config (WOODPECKER_GITEA=true, WOODPECKER_GITEA_URL, WOODPECKER_GITEA_CLIENT, WOODPECKER_GITEA_SECRET), WOODPECKER_HOST=https://stonks-ci.celestium.life, WOODPECKER_ADMIN=admin, Traefik ingress at stonks-ci.celestium.life with TLS via ca-issuer, NFS-backed persistent volume (5Gi). Agent: 2 replicas, Kubernetes backend (WOODPECKER_BACKEND=kubernetes), WOODPECKER_BACKEND_K8S_NAMESPACE=woodpecker, WOODPECKER_BACKEND_K8S_VOLUME_SIZE=10G.
      • Requirements: 2.1, 2.2, 2.3, 2.5, 11.1, 11.5, 28.1
    • 4.2 Create pipelines/woodpecker/network-policy.yaml — NetworkPolicy in woodpecker namespace allowing Traefik ingress (from kube-system namespace) to Woodpecker server on HTTP port
      • Requirements: 28.5
    • 4.3 Create pipelines/woodpecker/agent-rbac.yaml — ClusterRoleBinding granting the Woodpecker agent service account cluster-admin for integration test steps that create ephemeral namespaces. Mirrors the existing ARC runner RBAC pattern.
      • Requirements: 26.1, 26.2, 26.3

  • 5. Checkpoint — Verify Woodpecker and Gitea manifests

    • Ensure all YAML manifests are syntactically valid. Verify Woodpecker Helm values include correct Gitea OAuth2 configuration. Verify NFS PV paths are distinct from ArgoCD and Kargo PVs. Ask the user if questions arise.

  • 6. Update ArgoCD manifests for Gitea

    • 6.1 Update pipelines/argocd/repo-secret.yaml — Change repository URL from https://github.com/celesrenata/stonks-oracle.git to the Gitea repository URL (http://gitea-http.git-server.svc.cluster.local:3000/admin/stonks-oracle.git). Update credentials to use Gitea admin username/password instead of GitHub token.
      • Requirements: 25.1, 9.4
    • 6.2 Update pipelines/argocd/apps/stonks-beta.yaml — Change spec.source.repoURL from GitHub to Gitea repository URL
      • Requirements: 9.1, 25.3
    • 6.3 Update pipelines/argocd/apps/stonks-paper.yaml — Change spec.source.repoURL from GitHub to Gitea repository URL
      • Requirements: 9.2, 25.3
    • 6.4 Update pipelines/argocd/apps/stonks-live.yaml — Change spec.source.repoURL from GitHub to Gitea repository URL
      • Requirements: 9.3, 25.3

  • 7. Update Kargo Warehouse for local registry

    • 7.1 Update pipelines/kargo/warehouse.yaml — Change spec.subscriptions[0].image.repoURL from ghcr.io/celesrenata/stonks-oracle/query-api to registry.celestium.life/stonks-oracle/query-api
      • Requirements: 8.1, 8.2

  • 8. Update Helm chart for local registry

    • 8.1 Update infra/helm/stonks-oracle/values.yaml — Change image.registry from ghcr.io/celesrenata/stonks-oracle to registry.celestium.life/stonks-oracle. Remove imagePullSecrets list (remove ghcr-credentials entry). Remove ghcrAuth section entirely.
      • Requirements: 10.1, 10.2, 10.4

  • 9. Checkpoint — Verify ArgoCD, Kargo, and Helm chart updates

    • Ensure all updated YAML is syntactically valid. Verify no references to ghcr.io or github.com remain in ArgoCD apps, Kargo warehouse, or Helm values. Verify values-beta.yaml and values-paper.yaml don't need changes (they inherit image.registry from base). Ask the user if questions arise.

  • 10. Create Woodpecker pipeline file

    • 10.1 Create .woodpecker.yml in the repository root — Define all CI pipeline steps in Woodpecker's native YAML format: lint-python (ruff check), test-python (pytest), test-frontend (vitest), build steps for all 12 Python services using woodpeckerci/plugin-docker-buildx pushing to registry.celestium.life/stonks-oracle/<service>:<sha> and :latest, build-dashboard, build-superset, integration-test (run_pipeline.sh), and mirror-github. Use when conditions to restrict build/push/mirror steps to main branch push events. Use depends_on for step ordering. Use from_secret for registry and GitHub credentials.
      • Requirements: 14.1, 14.2, 14.3, 14.4, 14.5, 27.1, 27.2, 27.3, 27.4, 27.5, 3.1, 3.2, 3.3, 3.4, 3.5, 4.1, 4.2, 4.3, 4.4, 4.5, 5.1, 5.2, 5.3, 5.4, 5.5, 6.1, 6.2, 6.3, 6.4, 6.5

  • 11. Update install script (runmefirst.sh)

    • 11.1 Rewrite pipelines/runmefirst.sh — New install order: (1) ARC teardown, (2) create namespaces (woodpecker, argocd, kargo, stonks-beta, stonks-paper), (3) apply NFS PVs (argocd, kargo, woodpecker), (4) run Gitea setup script (gitea/setup.sh), (5) add Woodpecker Helm repo and install Woodpecker with Gitea OAuth2 credentials injected, (6) apply Woodpecker network policy and agent RBAC, (7) install ArgoCD via Helm with updated values, apply repo secret and Applications, (8) install Kargo via Helm, apply project/warehouse/stages. Use set -euo pipefail, idempotent operations throughout.
      • Requirements: 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7, 1.1, 1.2, 1.4, 1.5

  • 12. Update teardown script (runmelast.sh)

    • 12.1 Rewrite pipelines/runmelast.sh — New teardown order: (1) remove Kargo resources + Helm release, (2) remove ArgoCD resources + Helm release, (3) uninstall Woodpecker Helm release, delete Woodpecker agent RBAC and network policy, (4) delete namespaces (woodpecker, argocd, kargo). Preserve: NFS PVs, NFS data, stonks-oracle namespace, stonks-beta, stonks-paper, git-server namespace (Gitea + registry). Use --ignore-not-found and || true for idempotency.
      • Requirements: 12.1, 12.2, 12.3, 12.4, 12.5, 13.1, 13.2, 13.3

  • 13. Checkpoint — Verify all scripts and pipeline file

    • Ensure runmefirst.sh install order matches design (ARC teardown → Gitea → Woodpecker → ArgoCD → Kargo). Ensure runmelast.sh teardown order is reverse (Kargo → ArgoCD → Woodpecker). Verify .woodpecker.yml covers all 12 services + dashboard + superset + integration test + GitHub mirror. Verify no references to ARC, arc-system, or GHCR remain in any pipeline scripts. Ask the user if questions arise.

  • 14. Final review

    • Review all created and modified files for completeness. Verify the full pipeline flow: Gitea push → Woodpecker webhook → lint/test → build/push to local registry → integration test → GitHub mirror → Kargo detects new tag → beta auto-promote → paper/live with market-hours gates. Ensure all 28 requirements are addressed. Ask the user if questions arise.

Notes

  • Pipeline infrastructure scripts (~/sources/kube/pipelines/) are on gremlin-1, separate from the stonks-oracle repo
  • The .woodpecker.yml pipeline file and Helm chart changes are in the stonks-oracle repo
  • No property-based tests — this feature is entirely IaC (shell scripts, YAML manifests, Helm values)
  • Gitea is already deployed in git-server namespace but unconfigured (still on install page)
  • Local registry is already deployed and healthy at registry.celestium.life
  • ArgoCD and Kargo are already deployed — only config updates needed (repo URL, image source)
  • ARC is currently deployed and needs to be torn down before Woodpecker install
  • The .github/workflows/build.yml remains in the repo for reference but won't be used by Woodpecker
  • Woodpecker agents use the Kubernetes backend — each pipeline step runs as a standalone Pod
  • Image builds use woodpeckerci/plugin-docker-buildx with privileged mode
  • GitHub mirror is a post-CI step that pushes via SSH key — failure does not block promotion
  • Break-glass, audit trail, and Kargo Dashboard features are provided by Kargo out of the box (Requirements 19, 20, 21, 22, 23)
  • Market-hours AnalysisTemplate is unchanged from the existing pipeline
  • Kargo stages, project, and project-config are unchanged
  • values-beta.yaml and values-paper.yaml inherit image.registry from base values.yaml — no changes needed
  • k3s nodes may need containerd mirror config to trust the local registry for image pulls
Reference in New Issue View Git Blame Copy Permalink