Pushbox: CI

froog Pushbox runs your CI two ways:

GitHub Actions runners: runs-on: froog

Each job gets a fresh sandbox, deleted when the job ends. CI is billed for job time only: $0.0006 per vCPU per minute (2 vCPU $0.0012, 4 vCPU $0.0024, 8 vCPU $0.0048, 16 vCPU $0.0096 a minute), per second, from when GitHub starts the job until it completes. Booting the VM, setting up the runner, saving a cache and a kept /debug VM aren't billed. Connect GitHub with the repositories, then change runs-on:

jobs:
  test:
    runs-on: froog          # was: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test

The same sizes by vCPU count: froog-2vcpu, froog-4vcpu, froog-8vcpu, froog-16vcpu.

Add options after the size with /, for example runs-on: froog-large/from=ci-base/debug. from=NAME starts the job from a saved snapshot with your toolchain in place. debug keeps the sandbox 30 minutes after a failed job, so you can froog ci ssh into it. Every job's log on GitHub shows the VM it runs on and the command to look inside while it runs (froog ssh ci-…, or froog ci ssh ci_…). cache=NAME keeps a warm disk between jobs: the job starts from the saved snapshot cache-NAME when there is one (otherwise from from= or the base image), and when a job of a push, a schedule or a manual run on the repository's default branch succeeds, its disk is saved back as cache-NAME. Pull request jobs read the cache and never write it. Pull requests from forks don't read it either, since their authors get none of the repository's secrets. The disk is kept, so toolchains, ~/.cargo, node_modules and Docker layers are there next time. Keep build outputs inside the checkout with clean: false on actions/checkout (it deletes untracked files by default). Before saving, froog removes what later jobs mustn't read: logins in ~/.docker/config.json, ~/.git-credentials, ~/.config/gh, ~/.npmrc, ~/.pypirc, ~/.netrc, ~/.aws, ~/.config/gcloud, ~/.azure, ~/.kube, cargo and gem credentials and SSH private keys (root's too), shell histories, the runner's _temp (step scripts with secrets filled in, env and output files) and logs, the token actions/checkout leaves in .git/config, and /tmp. An executable ~/.froog/cache-scrub runs first, for anything else your jobs write. If it or the cleanup fails, nothing is saved. Don't use /cache on jobs that use deployment secrets. A cache is a saved snapshot: it counts toward your saved snapshots, and froog snapshots --rm cache-NAME starts it over. A label with an unknown size or option never runs: froog refuses the job and says why.

How it works: GitHub sends froog a workflow_job event. froog creates a sandbox named ci-<repo>-<job id> (label froog.ci=true), asks GitHub for a just-in-time runner configuration, and the sandbox runs the GitHub Actions runner with it. The runner is ephemeral: it takes one job and exits. When GitHub reports the job completed, froog deletes the sandbox. Any CI sandbox still there 6 hours 15 minutes after it was created is deleted (GitHub's longest job is 6 hours). The job runs in ~/actions-runner/_work on a full VM with Docker, so services:, container: and docker build work.

• Limits and billing. CI sandboxes count against your sandboxes-at-once cap and start limits. Their job time comes out of your plan's usage like sandbox time, and shows in froog usage and the dashboard as CI jobs. At a cap or a start limit a job waits, and froog tries again every 15 seconds. A job that still waits after 24 hours fails.
• Refused jobs show the reason: an unknown label, a public repository, a from= snapshot that doesn't exist, a sandbox that can't be created. A refused job of a public repository cancels its workflow run when the GitHub App has Actions: Read and write (otherwise the run stays queued until GitHub gives up).
• Which jobs a runner takes. GitHub registers a just-in-time runner on the whole repository, so it can take any queued job with the same labels. In a private repository that's fine: when two runs are queued at once, a job may run on the VM started for the other one, and that job's VM takes this one. In a public repository froog's runner has a job-started hook that fails any job that isn't from the run it was started for, before any step runs, so it never takes a job froog refused. The token that registers the runner (Administration, on that one repository) stays on froog's side. The sandbox gets only the runner configuration.
• When something goes wrong. A VM that doesn't start, or a runner that exits before taking its job, gets one more try on a new VM. After that the job fails with the reason (froog ci job). GitHub doesn't resend a webhook that failed, so froog asks it to (for the last day's events). A job still starting or running after 10 minutes is looked up on GitHub, so a lost event never leaves a VM running. A job still queued on GitHub whose VM went to another job of the repository gets a new VM, and runners GitHub still lists for VMs that are gone are removed from the repository every 10 minutes (only the ones froog named). The runner is GitHub's latest release, which GitHub requires.
• Public repositories don't run on froog by default: anyone can open a pull request, and its jobs would run on your bill. froog ci public on --users "me,a-teammate" allows them for runs started by those GitHub users (no --users: anyone, then turn on "Require approval for all outside collaborators" in the repository's Actions settings). froog ci public off turns them off again.
• Secrets. A job's sandbox starts without your environment: no variables, secret files, GitHub token or keys. Use GitHub Actions secrets.

$ froog ci                                   # setup, 7-day stats, the last jobs
$ froog ci --repo acme/api --state failed    # --state all|active|failed, --limit
$ froog ci job ci_…                          # state, reason, wait, run time, sandbox
$ froog ci logs ci_…                         # the job's log from GitHub
$ froog ci metrics ci_…                      # CPU and memory over time, steps with their times
$ froog ci tests ci_…                        # test results (--all: passed ones too)
$ froog ci ssh ci_…                          # a shell on the job's VM while it runs
$ froog ci search 'level:error timeout'      # every finished job's log, line by line
$ froog ci stats --days 14                   # per repository and job: runs, failures, p50/p90, cost, flaky tests
$ froog ci rightsize                         # jobs that would fit another size better
$ froog ci caches                            # the /cache=NAME disks; --rm NAME starts one over

In the dashboard. Pushbox lists your workflow runs, 25 to a page, newest first: each run's commit title, run number, who started it, its branch, when and how long, with filters for workflow (in the sidebar), repository, status and branch, and a search over titles, job names, commits and people. A run opens as a summary: what triggered it, its status, duration and cost, a box for each of its jobs on froog, annotations (the ##[error], ##[warning] and ##[notice] lines of each job's log, refusals, out-of-memory kills and failed tests) and usage per job. A job opens as its log, a collapsible section per step (the failed or running one open), with its CPU and memory and its tests. Insights, Log search, Monitors, Caches and Settings each have a page of their own under Pushbox. Runs, jobs and pages have their own links (#ci/run/RUN_ID, #ci/job/ci_…).

API: GET /ci/runs?workflow=&repo=&branch=&state=&q=&limit=&offset= ({runs, total, workflows, repos, branches}: a page of workflow runs, each with GitHub's title, event, run_number, path and actor and its jobs; 25 by default, up to 100), GET /ci/runs/{run_id}, GET /ci?repo=&state=&limit= ({configured, connected, label, public_repos, public_users, stats, jobs}), GET /ci/jobs/{id}, GET /ci/jobs/{id}/logs (text, the last 1 MiB, from GitHub, or the runner's output while GitHub has none), PUT /ci/settings {public_repos, public_users}, GET /ci/jobs/{id}/metrics ({resources, points, steps}), GET /ci/jobs/{id}/tests ({summary, tests}, failed first), GET /ci/analytics?days=&repo= ({repos, jobs, flaky_tests}), GET /ci/logs/search?q=&days=&limit=&order=newest|oldest ({matches: [{job_id, repo, job_name, branch, line, step, at, text}]}), GET / POST /ci/monitors ({name, kind: failure|duration, repo, branch, workflow, job, log_pattern, duration_secs, consecutive, cooldown_mins, email}, adding and removing need account.admin), DELETE /ci/monitors/{id}. Webhooks can send ci.job.* events. Job states: queued (waiting for a sandbox), starting, running, completed (with GitHub's conclusion), refused and failed, with reason. Reading needs account.read, the settings account.admin. Jobs are kept 30 days.

actions/cache, and what's installed

Your caches stay beside the VMs. actions/cache and the actions that use it (setup-node, setup-python, setup-go, setup-java, setup-dotnet, ruby/setup-ruby, Swatinem/rust-cache, astral-sh/setup-uv, Gradle's) save to and restore from froog's own cache service instead of GitHub's, with no change to your workflow. Like GitHub's cache, a job writes its own branch's cache and reads its own branch's first, then the default branch's, so a pull request can't change what main restores. Pull requests from forks, and accounts under zero data retention, use GitHub's cache as before, and so does a workflow that sets its own NODE_OPTIONS. Up to 10 GB per account is kept, the least recently used going first, and entries unused for 7 days are removed. If froog's cache can't answer, actions/cache carries on as on a miss. Other actions, such as upload-artifact, still talk to GitHub.

What's installed. The image has what GitHub's ubuntu-latest workflows usually expect: Docker with Buildx and Compose, Node 24 with pnpm, yarn and bun, Python with uv, Go, Rust, Java 21 with Maven, Ruby, PHP, .NET 8 and 10, the AWS CLI, kubectl, helm, gh, Chrome, shellcheck and the PostgreSQL and MySQL clients. actions/setup-python (3.10 to 3.13), setup-node (20, 22, 24) and setup-go (the last three releases) find those versions already in /opt/hostedtoolcache instead of downloading them, as on GitHub's runners. cat /etc/froog-image lists everything with versions. A /cache=NAME disk keeps what it was saved with: start one over (froog ci caches --rm NAME) to pick up a newer image.

Insights: metrics, tests and the right size

froog keeps more about each job than GitHub does, for 30 days:

• CPU and memory. The VM is sampled every 30 seconds while the job runs: average and peak CPU, peak memory, and the kernel's out-of-memory kills. froog ci metrics ci_… draws them with the job's steps and how long each took.
• Tests. When a job ends, froog reads the JUnit XML files it wrote in its workspace (changed during the job, up to 900 KB): totals, and each failed test with its message. Most test runners write JUnit with a flag: pytest --junitxml=report.xml, jest --reporters=jest-junit, vitest --reporter=junit --outputFile=report.xml, go test -v 2>&1 | go-junit-report > report.xml or gotestsum --junitfile report.xml, cargo nextest run --profile ci (with [profile.ci.junit]), Maven Surefire and Gradle by default. Not under zero data retention.
• Flaky tests. A test that failed and passed on the same commit (a re-run, or two jobs of a matrix) is listed as flaky, with how often.
• Log search. Each finished job's log is kept (the last 4 MiB), so you can search every job at once: froog ci search 'branch:main level:error timeout', or the dashboard's Log search. Words must all be in the line (any case), "a phrase" must appear as is, -word excludes, and repo:, branch:, workflow:, job:, conclusion:, step: and level:error / level:warning narrow it down. --oldest lists the oldest job first, to find where a failure first appeared. Not under zero data retention.
• Monitors. An alert when matching jobs fail or run long: by email to the account, and as the ci.monitor.triggered webhook event ({"id", "type", "created_at", "monitor", "ci_job", "reason"}) for Slack or your own tools. Match by repository, branch, workflow and job name (any part of the name). A failure monitor can wait for several failures in a row (a success starts over) and look for a pattern in the failed job's log (OutOfMemory, SIGKILL). A duration monitor fires once per job while it runs, when it passes the limit. After an alert, a monitor waits its cooldown (default 60 minutes) before the next. Up to 50 per account. froog ci monitors add "main breaks" --repo acme/api --branch main, froog ci monitors add slow --duration 20m, froog ci monitors add oom --log-pattern OutOfMemory --consecutive 2, froog ci monitors, froog ci monitors rm cim_….
• Analytics and size advice. froog ci stats and the dashboard add up each repository and job: runs, failure rate, p50 and p90 run time and queue wait, minutes and cost, CPU and memory. After three runs at a size, a job gets advice when another size fits better: one size up when it ran out of memory or used 90% of it, or when its CPUs were busy 80% of the time (faster); one size down when it never used more than 45% of the CPU and 40% of the memory (half the price). froog ci rightsize lists them with the label to use. Each job's row also shows its slowest steps (p50 and p90 over its runs) and, with /cache=, how often it started from the cache instead of cold.

Moving your workflows

froog ci migrate, in your repository, shows the runs-on lines it would change in .github/workflows; --write saves them. ubuntu-latest, ubuntu-24.04 and ubuntu-22.04 become froog (--size small|large|xlarge for another size, --cache NAME to add a warm disk). Blacksmith's blacksmith-<N>vcpu-ubuntu-* and Depot's depot-ubuntu-*-<N> become the froog size with the same vCPUs. Windows, macOS and Arm jobs stay where they are, and runs-on expressions (${{ matrix.os }}) are pointed out to change by hand.

$ froog ci migrate                   # show the changes
$ froog ci migrate --write --cache deps
$ git commit -am "CI on froog" && git push

To run your local changes on the same warm disk as CI, from your laptop or a coding agent: froog run --from cache-NAME -- make test.

The CLI from any CI

froog run creates a sandbox, pushes the checkout to it, runs the command, prints the cost, deletes the sandbox and exits with the command's code, so the CI job fails when the tests fail. Create a key with the ci preset and store it as a secret:

$ froog api-key create github-ci --preset ci --ttl 90d
$ gh secret set FROOG_API_KEY        # paste the froog_… key it printed once

# .github/workflows/froog-tests.yml
name: Tests on froog
on: [push, pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      FROOG_API_KEY: ${{ secrets.FROOG_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - run: curl -fsSL https://froogcorp.com/install | sh
      - run: froog run --size large -- make test        # or --from ci-base, -e KEY=VALUE

Install your toolchain once and keep it as a saved snapshot (froog snapshot box --name ci-base), then froog run --from ci-base starts with it. The same two lines work in GitLab CI, CircleCI or Buildkite with FROOG_API_KEY set.