駒形電産

TL;DR

Scale-to-zero can make review apps, staging environments, demos, and low-traffic internal tools much cheaper to keep around. I extended the existing draft in kamal-proxy#197 and tested it with a real Rails application on a standard 2 GB Ubuntu VPS.

• The Rails container used roughly 200–315 MiB while running and 0 B while stopped.
• The next request woke Rails and received a valid response in about 3.15 seconds.
• Large POST bodies, 20 concurrent wake requests, proxy restarts, failures, and deploys while sleeping were tested successfully.
• The exact experimental kamal-proxy and Kamal revisions are available for others to try. They are not part of an official release.
• The main open design question is whether kamal-proxy should access the Docker socket directly or delegate container lifecycle operations to another service.

The rest of this article explains the motivation, test results, problems found on real Docker, and how to reproduce the experiment.

Why I wanted scale-to-zero

Many teams run applications that need to remain available but receive traffic only occasionally: review apps, staging environments, internal tools, demos, and customer-specific applications.

Keeping every Rails container running all the time is expensive. Each one can sit there using a few hundred megabytes of memory even when nobody has accessed it for days. Kubernetes or a PaaS can solve parts of this problem, but they may add much more cost and operational complexity than these small applications justify.

For this kind of workload, a cold-start delay of a few seconds is often a reasonable tradeoff. What I wanted was the old Passenger-style behavior: stop an application after it has been idle for a while, then start it again when the next request arrives.

I have the same problem in a more specific setting. I run a programming school called FJORD BOOT CAMP. Our students build Rails applications as graduation projects, and I would like to keep those applications online after they graduate. They are useful as portfolios and as a record of what each student made, but most receive almost no traffic. Paying for a separate PaaS instance for every old project does not make much sense.

Kamal already makes it easy to run many applications on one server, and kamal-proxy already knows which container belongs to each host. So scale-to-zero in kamal-proxy felt like a natural experiment.

Starting from an existing draft

I found basecamp/kamal-proxy#197, a draft implementation by @martijnenco, and its corresponding Kamal change, basecamp/kamal#1800. The basic design was already there:

• stop a write container after an idle timeout
• start it on the next request
• hold the request until the health check succeeds
• persist the sleeping state in the existing kamal-proxy state file

I used that work as the starting point instead of creating another unrelated implementation. I then fixed the concurrency and lifecycle problems I found, added tests, and tried it against real Docker and a real Rails application.

This work is still experimental and is not part of an official Kamal or kamal-proxy release. The biggest architectural question is still open: should kamal-proxy have direct access to the Docker socket, or should container start/stop operations be delegated to another service?

The direct socket approach is wonderfully simple, but mounting /var/run/docker.sock gives the proxy effectively host-level control over Docker. That is a serious security tradeoff, not a harmless configuration detail.

What happened on a real VPS

I tested it on a fairly ordinary 2 GB Ubuntu 24.04 VPS with no swap. The server happens to be hosted by Sakura Internet in Japan, but nothing in the test depended on that provider. It used Docker Engine 29.1.3 and Docker API 1.52.

The first real test immediately found something the unit tests had missed. The original tiny Docker client used API v1.41 in its URLs, but this Docker daemon required at least v1.44. The daemon rejected every start and stop request. I changed the client to query Docker's unversioned /version endpoint once and use the API version reported by the daemon.

I also verified that kamal-proxy could continue running as its non-root user. Kamal mounts the Docker socket only when idle mode is enabled and adds the socket's numeric group ID to the proxy container. This matters because the Docker group ID on the host is not guaranteed to match a group baked into the image.

The low-level validation covered more than a happy-path GET request:

• a 65,536-byte chunked POST body survived a cold wake unchanged
• 20 concurrent GET and POST requests caused exactly one Docker start
• a wake timeout returned 503 and a later request could retry successfully
• a Docker start failure returned 503 and did not permanently break the service
• restarting kamal-proxy while the application was sleeping preserved the state
• deploying a new target while the old target was sleeping did not wake the stale container
• in-flight requests, WebSockets, and streaming responses prevent idle shutdown until they finish

Then I deployed a real Rails application called Calltodo. It uses Rails, Puma, SQLite, and Solid Queue. Here are the useful numbers from the final runs:

I also ran three independent Calltodo services, each with its own container and SQLite volume. Together they used about 606 MiB while idle-but-running. After scale-to-zero, all three application containers reported 0 B. Waking only one service left the other two stopped. Waking all three at the same time completed in roughly 3.00–3.23 seconds, with one Docker start per container.

On this particular 2 GB server, three simultaneously active instances look like a sensible initial limit. The useful part is that many more applications can be installed as long as most of them are asleep. The real capacity limit becomes simultaneous wake-ups rather than the total number of deployed projects.

Of course, stopping a web container also stops anything running inside it. An application with an always-on job worker, scheduled work, WebSockets that must stay connected, or another background requirement needs a different arrangement or must leave idle mode disabled. Scale-to-zero is explicitly opt-in.

Trying the experimental forks

I published the exact code used for the experiment in two branches:

• komagata/kamal-proxy at b6290ce
• komagata/kamal at 5064ede

There is deliberately no upstream PR for this version yet. These branches are available for testing while the upstream design is still being discussed.

First, build the experimental kamal-proxy image and push it to a registry your VPS can access:

git clone https://github.com/komagata/kamal-proxy.git
cd kamal-proxy
git checkout b6290ce039d4565474955bb6ca3918c12324016a

docker build -t YOUR_DOCKERHUB_USER/kamal-proxy:scale-to-zero-b6290ce .
docker push YOUR_DOCKERHUB_USER/kamal-proxy:scale-to-zero-b6290ce

In the Rails application's Gemfile, use the matching experimental Kamal revision:

gem "kamal",
  git: "https://github.com/komagata/kamal.git",
  ref: "5064ede15fa7d4bed0c0917e23f61fe6f53564cf"

Run bundle update kamal, then configure the proxy in config/deploy.yml:

proxy:
  host: app.example.com
  ssl: true
  idle:
    timeout: 300
    wake_timeout: 30
  run:
    repository: YOUR_DOCKERHUB_USER/kamal-proxy
    version: scale-to-zero-b6290ce

timeout and wake_timeout are seconds. Enabling proxy.idle makes this Kamal branch mount /var/run/docker.sock, add its numeric group ID to the proxy container, and pass the idle settings to kamal-proxy. If rootless Docker or a nonstandard socket is in use, set proxy.run.docker_socket explicitly.

After updating the proxy image, deploy normally:

bundle exec kamal proxy reboot
bundle exec kamal deploy

Start with a disposable environment, a generous wake timeout, and an application whose health check is reliable. Do not put this straight into an important production server. To disable the feature, remove proxy.idle, switch back to an official Kamal and kamal-proxy release, reboot the proxy, and manually start any application container that is currently sleeping before the switch.

The design and upstream direction are being discussed in basecamp/kamal-proxy Discussion #222. Test results from other environments—especially review apps—would be much more useful than another parallel implementation.

For my use case, the result is already pretty exciting: a Rails application that consumed 200–315 MiB while nobody was using it now consumes nothing, then comes back in about three seconds. That is exactly the tradeoff I wanted.