Scaling GitOps Repositories Across Teams and Projects

Use per-project GitOps repositories with a shared platform repo for cross-cutting concerns to balance ownership and scalability.

JR

3 minute read

Use per-project GitOps repositories with a shared platform repo for cross-cutting concerns to balance ownership and scalability.

Scaling GitOps repositories requires balancing autonomy, maintainability, and operational overhead. Monorepos create coordination bottlenecks as teams grow, while per-app repos introduce fragmentation. A hybrid approach—per-project repositories paired with a centralized platform repo—provides the best path forward for most organizations.

Diagnosis: When Repositories Become a Problem

  • Monorepo pitfalls: Tight coupling between projects leads to merge conflicts, noisy CI pipelines, and broad blast radius for misconfigurations.
  • Per-app repo pitfalls: Duplication of boilerplate, inconsistent policy enforcement, and increased operational toil managing dozens of repositories.
  • Key metric: Repository churn rate (commits/day) and team size correlate strongly with operational friction.

Actionable Workflow

  1. Define ownership boundaries

    • Map teams to projects (1:1 or 1:N).
    • Identify shared components (e.g., base configurations, RBAC policies) for the platform repo.
  2. Create per-project repositories

    • Structure: project-name repo containing app-specific manifests, overlays, and values.
    • Example: team-a-service, team-b-platform.
  3. Set up a platform repository

    • Contains:
      • Cluster-wide policies (e.g., PodSecurityPolicies, resource quotas).
      • Shared Helm charts or Kustomize bases.
      • Operator manifests (e.g., Prometheus, Istio).
  4. Configure repository synchronization

    • Use Argo CD or Flux to sync both repository types.
    • Example Argo CD Application for platform repo:
      apiVersion: argoproj.io/v1alpha1  
      kind: Application  
      metadata:  
        name: platform-config  
      spec:  
        project: default  
        source:  
          repoURL: https://github.com/org/platform-repo  
          path: ./cluster-config  
          targetRevision: main  
        destination:  
          server: https://kubernetes.default.svc  
          namespace: argocd  
      
  5. Automate policy enforcement

    • Use OPA Gatekeeper or Kyverno in the platform repo to validate changes across all repositories.
  6. Monitor repository health

    • Track repository size, commit frequency, and sync errors via Prometheus alerts.

Policy Example: CODEOWNERS Enforcement

# platform-repo/CODEOWNERS  
# Platform team owns cluster-wide configurations  
src/cluster-config/* @platform-team  

# Team-specific ownership in per-project repos  
# team-a-service/CODEOWNERS  
src/* @team-a  

Tooling

  • Argo CD: Supports multi-repo setups with clear separation of concerns.
  • Flux: GitOps operator with flexible repository synchronization.
  • Tekton: Automate repository creation and policy injection.
  • OLM (Operator Lifecycle Manager): Manage operators via platform repo.

Tradeoffs

  • Operational overhead: Managing 10+ repositories requires robust CI/CD pipelines and access controls.
  • Learning curve: Teams must understand repository boundaries and cross-cutting dependencies.
  • Sync latency: Platform repo changes may take time to propagate to all projects.

Troubleshooting Common Issues

  • Sync failures:

    • Check Argo CD Application status:
      argo app get platform-config  
      
    • Verify repository URL and targetRevision match expectations.
  • Permission errors:

    • Ensure service account in the GitOps tool has access to all repositories.
    • For OpenShift, validate the gitops-service-account has proper roles.
  • Drift between repositories:

    • Use kubectl get syncwaves (if using SyncWave) or Argo CD’s diff tool to identify inconsistencies.
    • Reconcile via argocd sync --repo <repo-name>.
  • Repository bloat:

    • Split oversized repositories when commit frequency exceeds 50/day or size >5GB.
    • Archive unused projects instead of deleting to preserve history.

This approach minimizes contention while maintaining flexibility. Start with per-project repositories and a platform repo, then adjust based on team feedback and operational metrics.

Source thread: How should I scale my GitOps repositories when deploying multiple projects?

comments powered by Disqus