Skip to content

Pause and resume the environment

Eureka 1.0: pause and eureka admin resume --start are implemented. Neither has been proven on a live environment. To resize control or login, use the cloud console, and only when nothing is running. Those steps are in doc/resize-control-login.md. The rest of this page is the pause UI.

In Eureka 1.0 the scheduler is the control node. Admin > Nodes > Pause environment… in the environment bar turns off the whole environment: running jobs are handled, the queue is held, users are told, the login node stops, and the control node stops itself last. Nothing runs and no VM is billed while it is paused. Disks, the login domain and its certificate are kept.

Nodes shows an environment bar above three cards. The bar reads Environment: Running, with jobs running and queued, and Pause environment…. Login node shows the domain, IP, certificate expiry, how many people are signed in, and Restart (reboots the login VM and disconnects signed-in users). Scheduler is the control node and offers Restart. Compute lists running VMs, grouped by flow. A Running jobs list under the cards shows jobs that are actually running (name, user, submitted, and state), so the count matches the pause dialog. Compute has no pause of its own. Pause environment… waits for those jobs or kills them.

Google Cloud

Nodes page with the Environment: Running bar, Pause environment button, and the Login node, Scheduler, and Compute cards

Pause when jobs finish waits for running jobs to complete, holds the queue, then stops login and control. You can cancel while it waits. Pause now and kill marks running jobs Killed: environment paused, runs their cleanup, then stops login and control.

  1. Open Admin > Nodes and choose Pause environment… in the environment bar.
  2. If jobs are running, pick one:
    • Pause when jobs finish (default, recommended). No new jobs start. The Scheduler card shows Pausing after N jobs with Cancel pause. The environment bar hides Pause environment…, and Restart on the Login and Scheduler cards stays off until it finishes or you cancel.
    • Pause now and kill N jobs. Running jobs are marked Killed: environment paused, and their normal cleanup runs so no cloud resources are left behind. Stopping a machine that never got an address, or that is still connecting, tears it down at once instead of waiting out SSH retries. With no jobs running, it’s a simple confirm.
  3. Also pause the login node is on by default. Leave it on. While the environment is paused there is no projectEureka UI. Resume with eureka admin resume --start from a machine that has the cloud credentials and the resume hint, or start the control instance from the cloud console. The login node and everything else come back with it.
  4. Notify users (on by default) shows the system notice banner and writes the text to the login’s motd and wall.
  5. Keep queued jobs held after resume is off by default. Turn it on and the queued jobs stay held after resume. Release them one at a time or all at once from the job queue on the Nodes page (eureka admin jobs release).

Queued jobs show Held (paused) to admins and Held: environment paused to their owners. On the project Jobs page the same jobs show Held: environment paused, or Held (kept after resume) when you kept them held.

Google Cloud

Pause environment dialog with Pause when jobs finish selected, the option to also pause the login node, Notify users, and the resume instructions

Google Cloud

The same two pause choices appear on Google Cloud.

The order is always: jobs (wait, or kill plus cleanup), hold the queue, notify, stop the login, stop the control. The login and control are stopped through the platform’s own managed group or VM API (an AFW Adaptive Script on control), never by deleting anything:

  • GCP: the login managed instance group’s stopInstances keeps the instance in the MIG as stopped, so autohealing doesn’t recreate it. Control runs instances.stop on itself.
  • AWS: the login Auto Scaling group gets suspendProcesses (so the ASG doesn’t replace a stopped instance), then EC2 StopInstances. On resume, StartInstances runs, then resumeProcesses. Control runs StopInstances on itself.
  • Kubernetes: the KubeVirt VMs are set to runStrategy: Halted. On resume they go back to Always.

The control stop is sent after the portal has answered and the state is saved as paused, so control always stops last.

While the environment is paused, there is no projectEureka UI. The portal does not answer, so users see a connection error rather than a paused page. When the pause reaches stopping, Eureka writes /var/lib/eureka/resume-hint.json (cloud, env, control name, id, project, zone, region, namespace; no secrets) on the machine that ran eureka admin pause or eureka admin pause --status, and on control and the login node. Run --status from the workstation you will resume from while control is still up.

eureka admin resume prints the console command below. eureka admin resume --start reads that hint file, which works while control (AFW) is down, and starts the control VM with credentials already on that machine:

  • GCP: application-default credentials. The caller needs compute.instances.get and compute.instances.start on the control instance.
  • AWS: the default credential chain. The caller needs ec2:DescribeInstances and ec2:StartInstances on the control instance. The login node has no instance profile, so run --start from an admin workstation that already has those credentials and the hint file.
  • Kubernetes: KUBECONFIG or ~/.kube/config. The command patches the control VirtualMachine runStrategy to Always.

The command then waits until AFW answers. If the permission is missing it exits 1, names the permission, prints the console command, and does not start the VM. If the VM is already running it does nothing. This start path has not been proven on a live environment.

The cloud console still works: Compute Engine on Google Cloud, EC2 on AWS, or the control KubeVirt VM on Kubernetes (for example virtctl start). Control then starts the login node, and the rest come back automatically.

The control instance name is shown in the pause dialog and banner as “To resume, start the control instance name in your cloud console”. That’s usually <env>-control.

Google Cloud

In Compute Engine > VM instances, open <env>-control and choose Start / Resume. Or run:

Terminal window
gcloud compute instances start <env>-control --zone <zone> --project <project>

When control boots, it resumes on its own. Control:

  1. starts the login node through its managed group;
  2. releases held jobs (unless you chose to keep them held);
  3. clears the banner.

The login node re-links its existing domain to its new public address. On AWS the public DNS changes when the instance stops and starts, so the scheduler keeps the same login record (by instance id) and does not treat the new address as a new node. A new node would lose the domain binding, and the next start would hand out a new domain. The login reuses its certificate when that certificate is valid for more than a day, so the same URL works. It asks for a new domain only when the domain service no longer knows the instance.

On Kubernetes the login keeps its load-balancer address and its domain is not managed that way, so the re-link step is skipped.

Google Cloud

While the environment is paused, there is no projectEureka UI. Run eureka admin resume --start from a machine with Google Cloud credentials, or start the control instance in Compute Engine, for example gcloud compute instances start <env>-control --zone <zone>. Login starts on its own after control boots.

Allow a few minutes after the control starts.

Restart on the Scheduler card reboots the control VM. Running jobs keep running, and omnisched adopts them again after the reboot. Restart on the Login card stops and starts the login node through its managed group, which disconnects signed-in users; its domain is re-linked the same way. Restart is unavailable while a pause is pending. Pause environment… is on the environment bar above the cards.

The System notice card on Nodes is the banner every signed-in user sees. Text is the message (up to 500 characters). Lead time is minutes until a planned event, or 0 for none. Set notice publishes it. Clear removes it.

Pause uses the same banner when Notify users is on. You can also set a notice when you are not pausing.

eureka admin notice set TEXT [--lead MINUTES] [--motd] [--wall] and eureka admin notice clear [--motd] match the card. --motd and --wall also write the login host’s message of the day or send wall, and need root on that host. MCP: system_notice_get, system_notice_set, system_notice_clear.

Google Cloud

Nodes page after Pause when jobs finish is requested, with the pause banner, Scheduler showing Pausing after 1 job, and the System notice text filled in
Portal CLI MCP tool
Pause environment eureka admin pause [--kill-now] [--keep-login] [--no-notify] [--message TEXT] [--keep-held] admin_env_pause
Status eureka admin pause --status admin_env_pause_status
Cancel pause eureka admin pause --cancel admin_env_pause_cancel
Restart eureka admin restart [control|login] admin_env_restart
Keep held after resume eureka admin resume --keep-held / --release-on-resume admin_env_resume_options
Start control from the hint file eureka admin resume --start (local command; control is down, so there is no MCP tool)
Release / Release all eureka admin jobs release PATH… | --all admin_jobs_release

eureka admin resume prints the console steps above and the --start command. eureka admin pause --status prints a stop error line with the text of a failed control stop. eureka admin resume --start is not proven live.