feat: Add CI/CD secrets configuration and documentation
- Add comprehensive CI/CD secrets section to .env.example - Create detailed CI-CD-SECRETS.md guide with: - Step-by-step secret setup instructions - Container registry authentication examples - PyPI token configuration - GitHub integration setup - Security best practices - Troubleshooting guide - Workflow-specific requirements Includes support for: - Docker Hub, GitLab, GitHub container registries - Automated PyPI publishing - Slack/Discord notifications - Kubernetes deployment secrets - Security scanning integration Updated .env template with placeholder values and detailed comments.
This commit is contained in:
William Valentin
274
docs/CI-CD-SECRETS.md
Normal file
274
docs/CI-CD-SECRETS.md
Normal file
@@ -0,0 +1,274 @@
|
||||
# CI/CD Secrets Setup Guide
|
||||
|
||||
This guide explains how to configure the required secrets for UnitForge's Gitea CI/CD workflows.
|
||||
|
||||
## Overview
|
||||
|
||||
UnitForge's CI/CD pipelines require several secrets to function properly:
|
||||
- **Container Registry** credentials for pushing multi-arch images
|
||||
- **PyPI** token for automated package publishing
|
||||
- **GitHub** token for release management
|
||||
- **Optional** notification and deployment secrets
|
||||
|
||||
## Required Secrets
|
||||
|
||||
### 1. Container Registry Authentication
|
||||
|
||||
**Required for:** All build workflows (`build-container.yml`, `release.yml`, `nightly.yml`)
|
||||
|
||||
```bash
|
||||
CONTAINER_REGISTRY_USERNAME=your-registry-username
|
||||
CONTAINER_REGISTRY_PASSWORD=your-registry-password-or-token
|
||||
```
|
||||
|
||||
#### Docker Hub Example:
|
||||
```bash
|
||||
CONTAINER_REGISTRY_USERNAME=dockerhubusername
|
||||
CONTAINER_REGISTRY_PASSWORD=dckr_pat_xxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
```
|
||||
|
||||
#### GitLab Container Registry Example:
|
||||
```bash
|
||||
CONTAINER_REGISTRY_USERNAME=gitlab-ci-token
|
||||
CONTAINER_REGISTRY_PASSWORD=glpat-xxxxxxxxxxxxxxxxxxxx
|
||||
```
|
||||
|
||||
#### GitHub Container Registry Example:
|
||||
```bash
|
||||
CONTAINER_REGISTRY_USERNAME=github-username
|
||||
CONTAINER_REGISTRY_PASSWORD=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
```
|
||||
|
||||
### 2. PyPI Publishing Token
|
||||
|
||||
**Required for:** Release workflow (`release.yml`) - stable releases only
|
||||
|
||||
```bash
|
||||
PYPI_API_TOKEN=pypi-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
```
|
||||
|
||||
#### How to get PyPI token:
|
||||
1. Go to [PyPI Account Settings](https://pypi.org/manage/account/token/)
|
||||
2. Click "Add API token"
|
||||
3. Set scope to "Entire account" or specific project
|
||||
4. Copy the token (starts with `pypi-`)
|
||||
|
||||
### 3. GitHub Token (Optional)
|
||||
|
||||
**Required for:** GitHub releases and integration
|
||||
|
||||
```bash
|
||||
GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
```
|
||||
|
||||
#### How to get GitHub token:
|
||||
1. Go to GitHub Settings > Developer settings > Personal access tokens
|
||||
2. Click "Generate new token (classic)"
|
||||
3. Select scopes: `repo`, `write:packages`
|
||||
4. Copy the token (starts with `ghp_`)
|
||||
|
||||
## Setting Up Secrets in Gitea
|
||||
|
||||
### Repository Secrets
|
||||
|
||||
1. Navigate to your repository in Gitea
|
||||
2. Go to **Settings** > **Secrets and Variables** > **Actions**
|
||||
3. Click **New repository secret**
|
||||
4. Add each required secret:
|
||||
|
||||
```
|
||||
Name: CONTAINER_REGISTRY_USERNAME
|
||||
Value: your-registry-username
|
||||
|
||||
Name: CONTAINER_REGISTRY_PASSWORD
|
||||
Value: your-registry-password
|
||||
|
||||
Name: PYPI_API_TOKEN
|
||||
Value: pypi-your-token-here
|
||||
|
||||
Name: GITHUB_TOKEN
|
||||
Value: ghp_your-token-here
|
||||
```
|
||||
|
||||
### Organization Secrets (Optional)
|
||||
|
||||
For multiple repositories, set up organization-level secrets:
|
||||
|
||||
1. Go to your Gitea organization
|
||||
2. Navigate to **Settings** > **Secrets and Variables** > **Actions**
|
||||
3. Add secrets that will be shared across repositories
|
||||
|
||||
## Optional Secrets
|
||||
|
||||
### Notification Services
|
||||
|
||||
```bash
|
||||
# Slack notifications
|
||||
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
|
||||
|
||||
# Discord notifications
|
||||
DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/000000000000000000/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
||||
|
||||
# Email notifications
|
||||
EMAIL_NOTIFICATION_ADDRESS=ci-notifications@yourdomain.com
|
||||
```
|
||||
|
||||
### Deployment Secrets
|
||||
|
||||
```bash
|
||||
# SSH key for server deployments (base64 encoded)
|
||||
DEPLOYMENT_SSH_KEY=LS0tLS1CRUdJTi...
|
||||
|
||||
# Kubernetes service account token
|
||||
KUBERNETES_TOKEN=eyJhbGciOiJSUzI1NiIsImtpZCI6Ii...
|
||||
|
||||
# Production API key
|
||||
PRODUCTION_API_KEY=prod-api-key-xxxxxxxxxxxx
|
||||
```
|
||||
|
||||
### Security Scanning
|
||||
|
||||
```bash
|
||||
# Enhanced security scanning (Snyk, etc.)
|
||||
SECURITY_SCAN_TOKEN=your-security-service-token
|
||||
```
|
||||
|
||||
## Environment Configuration
|
||||
|
||||
Update your `.env` file with registry and configuration details:
|
||||
|
||||
```bash
|
||||
# Container Registry Configuration
|
||||
CONTAINER_REGISTRY_URL=https://your-registry.example.com/your-namespace/unitforge
|
||||
CONTAINER_TAG=latest
|
||||
|
||||
# Build Configuration
|
||||
DOCKER_PLATFORMS=linux/amd64,linux/arm64
|
||||
BUILDX_CACHE_FROM=type=gha
|
||||
BUILDX_CACHE_TO=type=gha,mode=max
|
||||
```
|
||||
|
||||
## Workflow-Specific Requirements
|
||||
|
||||
### Pull Request Workflow (`pr-check.yml`)
|
||||
- **No secrets required** - Only builds and tests, no pushing
|
||||
|
||||
### Build Container Workflow (`build-container.yml`)
|
||||
- `CONTAINER_REGISTRY_USERNAME` ✅ Required
|
||||
- `CONTAINER_REGISTRY_PASSWORD` ✅ Required
|
||||
|
||||
### Release Workflow (`release.yml`)
|
||||
- `CONTAINER_REGISTRY_USERNAME` ✅ Required
|
||||
- `CONTAINER_REGISTRY_PASSWORD` ✅ Required
|
||||
- `PYPI_API_TOKEN` ✅ Required (stable releases)
|
||||
- `GITHUB_TOKEN` ⚠️ Optional (for GitHub releases)
|
||||
|
||||
### Nightly Workflow (`nightly.yml`)
|
||||
- `CONTAINER_REGISTRY_USERNAME` ✅ Required
|
||||
- `CONTAINER_REGISTRY_PASSWORD` ✅ Required
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Secret Rotation
|
||||
- Rotate secrets regularly (quarterly recommended)
|
||||
- Use tokens with limited scope and expiration
|
||||
- Monitor token usage in registry/service dashboards
|
||||
|
||||
### 2. Principle of Least Privilege
|
||||
- Grant minimum required permissions
|
||||
- Use service-specific tokens where possible
|
||||
- Avoid using personal access tokens in production
|
||||
|
||||
### 3. Token Security
|
||||
- Never commit secrets to version control
|
||||
- Use environment-specific secrets
|
||||
- Enable audit logging where available
|
||||
|
||||
### 4. Backup and Recovery
|
||||
- Document secret sources and renewal procedures
|
||||
- Maintain secure backup of critical tokens
|
||||
- Test secret rotation procedures
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Authentication Failed:**
|
||||
```
|
||||
Error: unauthorized: authentication required
|
||||
```
|
||||
- Verify `CONTAINER_REGISTRY_USERNAME` and `CONTAINER_REGISTRY_PASSWORD`
|
||||
- Check token permissions and expiration
|
||||
- Ensure registry URL matches token scope
|
||||
|
||||
**PyPI Publishing Failed:**
|
||||
```
|
||||
Error: 403 Forbidden
|
||||
```
|
||||
- Verify `PYPI_API_TOKEN` format (should start with `pypi-`)
|
||||
- Check token scope (project vs account-wide)
|
||||
- Ensure package name matches PyPI project
|
||||
|
||||
**GitHub Release Failed:**
|
||||
```
|
||||
Error: Resource not accessible by integration
|
||||
```
|
||||
- Verify `GITHUB_TOKEN` permissions include `repo` scope
|
||||
- Check if token is expired
|
||||
- Ensure repository access is granted
|
||||
|
||||
### Testing Secrets
|
||||
|
||||
Test your secrets locally:
|
||||
|
||||
```bash
|
||||
# Test container registry login
|
||||
echo "$CONTAINER_REGISTRY_PASSWORD" | docker login "$CONTAINER_REGISTRY_URL" -u "$CONTAINER_REGISTRY_USERNAME" --password-stdin
|
||||
|
||||
# Test PyPI token (dry run)
|
||||
twine check dist/* --repository testpypi
|
||||
|
||||
# Test GitHub API access
|
||||
curl -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/user
|
||||
```
|
||||
|
||||
## Monitoring and Alerts
|
||||
|
||||
### Registry Usage
|
||||
- Monitor container registry storage and bandwidth
|
||||
- Set up alerts for failed pushes
|
||||
- Track image pull statistics
|
||||
|
||||
### Build Metrics
|
||||
- Monitor workflow success rates
|
||||
- Track build duration trends
|
||||
- Alert on consecutive failures
|
||||
|
||||
### Security Monitoring
|
||||
- Enable security scanning alerts
|
||||
- Monitor vulnerability trends
|
||||
- Set up notifications for critical issues
|
||||
|
||||
## Support
|
||||
|
||||
For additional help:
|
||||
- Check [Gitea Actions Documentation](https://docs.gitea.io/en-us/usage/actions/)
|
||||
- Review workflow logs in Gitea Actions UI
|
||||
- Open an issue in the UnitForge repository
|
||||
- Consult your container registry documentation
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Secret | Required For | Format | Purpose |
|
||||
|--------|-------------|--------|---------|
|
||||
| `CONTAINER_REGISTRY_USERNAME` | All builds | Username | Registry authentication |
|
||||
| `CONTAINER_REGISTRY_PASSWORD` | All builds | Token/Password | Registry authentication |
|
||||
| `PYPI_API_TOKEN` | Releases | `pypi-...` | Package publishing |
|
||||
| `GITHUB_TOKEN` | Releases | `ghp_...` | GitHub integration |
|
||||
| `SLACK_WEBHOOK_URL` | Notifications | HTTPS URL | Slack alerts |
|
||||
| `DEPLOYMENT_SSH_KEY` | Deployments | Base64 SSH key | Server access |
|
||||
|
||||
---
|
||||
|
||||
*Last updated: December 2024*
|
||||
*For the latest information, see the [CI/CD Workflows README](.gitea/workflows/README.md)*
|
||||
58
docs/ci-cd/local-testing.md
Normal file
58
docs/ci-cd/local-testing.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Local CI/CD Testing
|
||||
|
||||
This guide helps you test CI/CD workflows locally before pushing to the repository.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Docker with Buildx support
|
||||
- uv package manager
|
||||
- Python 3.8+
|
||||
|
||||
## Local Testing Commands
|
||||
|
||||
```bash
|
||||
# Test local build
|
||||
make docker-build
|
||||
|
||||
# Test multi-arch build
|
||||
make docker-buildx-local
|
||||
|
||||
# Test full development workflow
|
||||
make dev
|
||||
|
||||
# Run health checks
|
||||
./scripts/health_check.sh
|
||||
```
|
||||
|
||||
## Workflow Testing
|
||||
|
||||
Use `act` to test GitHub/Gitea workflows locally:
|
||||
|
||||
```bash
|
||||
# Install act
|
||||
curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash
|
||||
|
||||
# Test PR workflow
|
||||
act pull_request -s CONTAINER_REGISTRY_USERNAME=test -s CONTAINER_REGISTRY_PASSWORD=test
|
||||
|
||||
# Test release workflow
|
||||
act push -e tests/fixtures/release-event.json
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Build Issues
|
||||
- Ensure all vendor assets are committed
|
||||
- Check Docker daemon is running
|
||||
- Verify buildx is properly configured
|
||||
|
||||
### Registry Issues
|
||||
- Check .env file configuration
|
||||
- Verify registry credentials
|
||||
- Test registry connectivity
|
||||
|
||||
### Performance Issues
|
||||
- Use build cache: `--cache-from type=gha`
|
||||
- Optimize Docker layers
|
||||
- Use multi-stage builds
|
||||
```
|
||||
Reference in New Issue
Block a user