Skip to content

Backing up an environment

This page lists what a system admin needs to back up so a projectEureka environment can be rebuilt. projectEureka 1.0 does not ship a one-command backup or restore. The reliable approach is to snapshot the disks that hold this data, keep a copy of the launch configuration off the environment, and use the cloud’s or NFS server’s own backups for buckets and external storage.

The file you launched with (~/.eureka/eureka.yaml, or the path you passed with --eureka-config or EUREKA_CONFIG) is not stored only on the environment. Keep a copy somewhere else. You need it to launch a replacement with the same storage layout, instance defaults, and other launch settings. After launch, a copy is also written to /etc/eureka/eureka.yaml on the control node.

The control node holds the environment’s records: projects, users, launchers, settings, credentials, and the scheduler’s change feed. Snapshot or archive these together so they stay consistent.

Eureka objects are stored in the AFW LMDB database at /afw/lmdb. That includes projects, launchers, jobs, admin and project credentials, sign-in settings (the AuthConfig object, including the Google client secret when one is set), and related records. There is no separate SQL database for this data.

The AFW journal, the scheduler’s change feed, is the directory /afw/eureka/journal.

Authorization policies live under /afw/auth.

Back these up together. A store without its journal, or a journal from a different store, is not a consistent restore.

User accounts and password hashes are in OpenLDAP on the control node. The data directory and TLS certificates depend on the machine architecture:

  • On x86_64, the database is /var/symas/openldap-data. The server configuration is /opt/symas/etc/openldap/slapd.conf. The LDAP TLS certificate and key are /opt/symas/etc/openldap/cert.pem and /opt/symas/etc/openldap/key.pem.
  • On aarch64, the database is /var/lib/ldap. The TLS certificate and key are /etc/openldap/certs/cert.pem and /etc/openldap/certs/key.pem.

In both cases, the LDAP manager bind password is /root/eureka-ldap-bind. Restore that file with the database. If it is missing, a later start can generate a new password that will not open the restored database.

Each node has a personality at /etc/eureka-personality and a secret at /var/eurekakey. The control node also keeps a verifier for each live node under /var/eurekakeys.

If you snapshot and restore the control and login disks together, keep these files as part of that snapshot. If the control host was compromised, treat every verifier as compromised and do not restore verifier files from that host. See JWT security.

  • /etc/eureka/, including eureka.yaml and instance-type-defaults.json. Other files may also appear here (for example a desktop-gate secret). Back up the whole directory.
  • /var/lib/eureka/storage-admin, which holds Switch state for external filesystems. Include it if a Switch might be in progress.

/afw/models and much of /afw/config are generated when the control node starts from the product image. Whether a rebuild needs those files restored, or can recreate them, has not been verified as a standalone restore; a full control-disk snapshot includes them.

Include /vault (the standard /vault/home, /vault/projects, and /vault/containers trees) only when those trees are still on the default login-VM NFS. In that case the data lives on the login disk at those paths, and the login VM exports them. See Storage for production.

When /vault is on external storage (Filestore, EFS, or another NFS you pointed the standard trees at), that storage’s own backups cover it. Do not treat a copy of /vault on the login VM as the backup of that data; the login node’s copy is hidden after a Switch.

Sign-in settings (which sources are on, the Google client, the InCommon allow list, and each user’s linked identities) are stored on the control node in AFW, as described above. The login node also has files that were generated from those settings and certificates that must stay the same if you have already registered the environment with a campus IdP:

  • /opt/simplesamlphp/cert (including incommon.key and incommon.crt, created once and reused)
  • /opt/simplesamlphp/config and /opt/simplesamlphp/metadata
  • /etc/httpd/mellon (Mellon key, certificate, and IdP metadata)
  • /var/data/simplesamlphp (including simplesamlphp.sqlite)
  • /etc/eureka/sign-in-sources.json
  • /etc/ood/config/ood_portal.yml

The public HTTPS certificate, when Let’s Encrypt issued it, is under /etc/letsencrypt. Apache is configured to use /etc/pki/tls/certs/localhost.crt and /etc/pki/tls/private/localhost.key, which are linked to that lineage when a Let’s Encrypt certificate is installed. A paused login keeps its domain, and an existing valid certificate is reused.

Managed bucket contents live in the cloud account (Cloud Storage or S3), not on the control or login disk. Back those buckets up with the cloud’s own tools. projectEureka’s records of the buckets (names, which project they belong to, flow-bucket settings) are in the AFW store on the control node.

External buckets are not created by projectEureka. Use that bucket’s own backups. Deleting the projectEureka entry does not delete the cloud bucket.

On Kubernetes there are no managed buckets; only homes, project files, and any external buckets you add.

  1. A copy of the launch configuration, kept off the environment.
  2. A snapshot of the control node disk (this covers the AFW store, journal, authorization policies, LDAP, the bind password, JWT verifiers, and /etc/eureka).
  3. A snapshot of the login node disk if /vault is still on login-VM NFS, and in any case if you need the existing Let’s Encrypt lineage or SimpleSAMLphp and Mellon certificates.
  4. The cloud’s backups or versioning for managed project buckets.
  5. The NFS service’s own backups when home, projects, or containers are on Filestore, EFS, or another external NFS.

Take the control snapshot while the environment is idle if you can, so the AFW store, journal, and LDAP are not mid-update. How to quiesce AFW and LDAP for a perfectly consistent filesystem copy has not been separately verified; a disk snapshot of a paused or idle control node is the intended method.

There is no supported one-command restore. These are the approaches that follow from where the data lives:

  • Same virtual machines, restored disks. Restore the control snapshot (and the login snapshot if you took one). Attach the same external NFS and the same buckets. Start the control node, then the login node, as you would after a pause. See Pause and resume the environment.
  • New virtual machines, same storage. Launch from the saved launch configuration so the new environment mounts the same external NFS. Restoring the AFW store, journal, and LDAP piece by piece onto a freshly launched control node has not been verified. If you do not restore the control disk as a whole, treat that as a procedure to verify before you rely on it. JWT secrets on a new set of nodes are issued when those nodes start; do not copy verifiers from a compromised control host.
  • Login-VM /vault. If you are restoring login-hosted data, restore /vault/home, /vault/projects, and /vault/containers onto the login disk before clients mount them. If those trees were moved to external NFS, restore them with that storage’s tools instead.

After a restore, confirm that you can sign in, that projects and buckets appear, and that a Terminal can see the expected home and project directories.

Storage for production · External filesystems and clusters · Sign-in · JWT security · Pause and resume the environment · Environment overview