3 Dev Forgejo Deployment Pipeline
Forgejo Actions edited this page 2026-10-13 10:07:04 +00:00

Forgejo Deployment Pipeline

Memories uses Forgejo Actions to build Docker Compose service images and deploy the local self-hosted stack. The workflow lives in .forgejo/workflows/build-deploy.yml.

Runner

The deployment workflow targets a dedicated Forgejo runner label:

memories-deploy

This runner is intentionally separate from the general docker/DIND runner. It has access to the host Docker daemon so it can build images and restart the same Docker Compose stack that the operator runs manually.

Current local runner setup on the deployment host:

container: memories-deploy-runner
image: data.forgejo.org/forgejo/runner:11
runner name: memories-deploy
label: memories-deploy:docker://docker:28-cli
scope: repo-specific for chris/memories
config directory: /home/chris/git/memories-deploy-runner
workspace: /home/chris/git/memories-deploy-runner/workspace
host Docker socket: /var/run/docker.sock
host source mount: /home/chris/git:/home/chris/git

The runner job containers use network: host. Workflow checkouts therefore use the public Forgejo URL (https://git.wiwie.dk/...) rather than the internal Compose hostname gitea, which is not resolvable from host networking.

Important runner config details:

  • Keep container.docker_host: unix:///var/run/docker.sock.
  • Do not also mount /var/run/docker.sock through container.options; the runner already mounts it for the job container when docker_host is set.
  • Keep /home/chris/git mounted into job containers so host Docker can access the checked-out Compose build context.
  • If the Docker socket group changes, restart the runner with the matching --group-add <docker-socket-gid>.

Useful runner checks:

docker ps --filter 'name=memories-deploy-runner'
docker logs --tail 80 memories-deploy-runner
stat -c '%g' /var/run/docker.sock

Workflow Triggers

The workflow runs on:

  • every branch push
  • manual workflow_dispatch
  • nightly schedule (02:17 UTC)

Branch pushes run scoped tests for CI feedback. Deployments and Docker image builds are gated to main only. Manual and nightly runs execute the full test suite without building or deploying images.

Change Detection And Test Scopes

The changes job runs scripts/ci/detect-scopes.sh, computes changed paths from the pushed commit range, and emits booleans for each deployable service:

  • api
  • web
  • worker_media
  • worker_ml

Service-specific changes build only that service image on main. Broad operational or shared files build all images, including:

  • .forgejo/workflows/build-deploy.yml
  • deploy runtime configuration such as deploy/compose*.yaml and deploy/example.env
  • root Gradle settings/build files
  • gradle/*
  • app version touchpoints such as apps/web/package.json

The same script also emits test-scope booleans. Branch pushes use these scoped checks instead of running the full suite every time:

  • People API changes compile the API and run People API tests.
  • General API changes run API tests.
  • Web changes run npm run check and Playwright tests.
  • CLI changes run CLI build/tests.
  • ML worker changes run ML unit tests.
  • Docs/spec-only changes run no runtime tests.

Risky or broad paths fall back to broad smoke checks on branch pushes, not the full suite. These paths include workflow/script changes, Compose/runtime deploy configuration, root Gradle settings, database migrations, API auth/security/sharing/worker-protocol paths, and unknown paths. Deploy markdown runbooks are treated as docs-only. The fallback checks compile JVM services, run the web type/build check, run CLI build/tests, and run ML unit tests. Full JVM and Playwright suites are reserved for nightly/manual runs.

App version touchpoints such as apps/web/package.json mark all service images as affected for main deployment, but run web-scoped checks on branch pushes so normal runtime PRs do not lose the benefit of scoped CI solely because they bump the release version.

On main, the workflow runs fast smoke checks for changed service areas before building and deploying affected images. The full suite is intentionally reserved for nightly/manual runs so main deploys stay fast while the slower confidence gate still runs regularly.

Build Jobs

Build jobs are separate so Forgejo shows per-service status:

  • Build API image
  • Build web image
  • Build media worker image
  • Build ML worker image

Each build uses Docker Compose from deploy/ with the production environment file path supplied by the workflow:

docker compose --env-file "$MEMORIES_DEPLOY_ENV_FILE" -f compose.yaml build <service>

MEMORIES_DEPLOY_ENV_FILE currently points to:

/home/chris/git/memories-deploy-runner/.ci-secrets/deploy.env

This is a stable host path shared by both deploy runners (memories-deploy, memories-deploy-runner-2). Runner workspaces are ephemeral per run, so the env file must not live inside a checkout.

The Compose file still defaults to local .env when the variable is not set, so manual local commands keep working.

Android CI Image

The android job runs from a dedicated image instead of downloading the Android SDK and Node.js on every run:

  • Dockerfile: scripts/ci/android-image/Dockerfile
  • Base gradle:8.13-jdk21 plus Node.js 20 and the Android SDK (cmdline-tools, platform 37, build-tools 37, licences accepted) baked in.
  • Registry: git.wiwie.dk/chris/memories/android-ci (:<version> and :latest).

detect-scopes.sh emits android_ci_image=true only when scripts/ci/android-image/* changes; the build-android-ci-image job then rebuilds and pushes the image on main. The debug keystore is deliberately NOT part of the image (ISSUE-121) — the job provisions it at run time from ANDROID_DEBUG_KEYSTORE_BASE64.

Forgejo pulls the job container image before any step runs, so the android job authenticates that pull through container.credentials (PACKAGE_PUSH_TOKEN); a docker login step could never do it. Caveat: act_runner reuses a locally present image instead of pulling, so a stale host-side :latest would shadow a rebuilt image for up to 7 days (until the cleanup prune drops it) — build-android-ci-image therefore docker rmis all local tags of the image after a successful push to force the next job to pull (best-effort, with a warning on failure). To force a rebuild without touching the Dockerfile, change any file under scripts/ci/android-image/ — every path in that directory sets android_ci_image=true.

Version And Image Tags

The pipeline derives the application version from apps/web/package.json. Keep that version aligned with the release-policy touchpoints before merge.

Tag rules:

  • main: MEMORIES_APP_VERSION=<version>
  • branch builds: MEMORIES_APP_VERSION=<version>-<branch>-<shortsha>

Branch tags are still computed for inspectability, but branch pushes no longer build service images by default. main builds use the release version tag.

The API container also receives MEMORIES_APP_VERSION as an environment override from Compose so runtime version metadata matches the deployed image tag instead of an older value in deploy/.env.

Memories service images built by Compose carry Docker labels:

  • dev.memories.project=memories
  • dev.memories.service=<compose-service-name>

These labels let the cleanup job prune only unused Memories images created by this pipeline instead of deleting unrelated self-hosted service images.

Deploy Jobs

Deploy jobs run only on main and only for services that were rebuilt.

Deployment order:

  1. Deploy API starts or verifies memories-postgres, then recreates memories-api and waits for health.
  2. Deploy web, Deploy media worker, and Deploy ML worker depend on the API deploy and recreate their own containers with --no-deps --no-build.

The workflow uses Compose like this for service deployment:

docker compose --env-file "$MEMORIES_DEPLOY_ENV_FILE" -f compose.yaml up -d \
  --force-recreate --no-deps --no-build --wait --wait-timeout 300 <service>

Expected Status Patterns

On a branch push:

  • Detect changed services succeeds.
  • Run scoped PR tests runs the selected scoped checks, or broad smoke fallback checks for risky paths.
  • Build and deploy jobs are skipped.

On a main push:

  • Detect changed services succeeds.
  • Run main smoke checks runs fast checks for affected service areas.
  • Affected build jobs run after smoke checks.
  • Matching deploy jobs run.
  • Unaffected service deploy jobs are skipped.

On a nightly or manual run:

  • Run full test suite runs JVM, web, CLI, and ML tests.
  • Build and deploy jobs are skipped.

Docker Cleanup

The Clean Docker build artifacts job runs after successful affected builds and deploys. It is intentionally conservative:

  • It prints docker system df before and after cleanup.
  • It prunes unused Memories images older than seven days using the dev.memories.project=memories label.
  • It prunes BuildKit cache older than 72 hours while keeping up to 8GB of cache.
  • It does not run docker system prune.
  • It does not prune volumes.
  • It cannot remove images used by running containers.

Current old Memories images that predate the labels are not removed by the workflow automatically. Operators may clean those manually after verifying which release tags they want to keep.

Troubleshooting

If a job fails before the first step, inspect runner logs and Forgejo action logs. Local action logs are stored compressed by Forgejo, for example:

/data/gitea/actions_log/chris/memories/<prefix>/<task-id>.log.zst

Common failures seen during setup:

  • Duplicate mount point: /var/run/docker.sock: remove the explicit socket mount from runner container.options; keep container.docker_host only.
  • Could not resolve host: gitea: job container is using host networking; use the public Forgejo URL for checkout.
  • Build succeeds but deploy does not change the running container tag: confirm the deploy job exported MEMORIES_APP_VERSION=<release_version> and that Compose is using the expected env file.

Check the deployed stack after a main run:

docker ps --filter 'name=memories-' --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'