Backing up an environment
This page lists what a system admin needs to back up so a projectEureka environment can be rebuilt. projectEureka 1.0 does not ship a one-command backup or restore. The reliable approach is to snapshot the disks that hold this data, keep a copy of the launch configuration off the environment, and use the cloud’s or NFS server’s own backups for buckets and external storage.
Keep the launch configuration
Section titled “Keep the launch configuration”The file you launched with (~/.eureka/eureka.yaml, or the path you passed with --eureka-config or EUREKA_CONFIG) is not stored only on the environment. Keep a copy somewhere else. You need it to launch a replacement with the same storage layout, instance defaults, and other launch settings. After launch, a copy is also written to /etc/eureka/eureka.yaml on the control node.
Control node
Section titled “Control node”The control node holds the environment’s records: projects, users, launchers, settings, credentials, and the scheduler’s change feed. Snapshot or archive these together so they stay consistent.
AFW object store and journal
Section titled “AFW object store and journal”Eureka objects are stored in the AFW LMDB database at /afw/lmdb. That includes projects, launchers, jobs, admin and project credentials, sign-in settings (the AuthConfig object, including the Google client secret when one is set), and related records. There is no separate SQL database for this data.
The AFW journal, the scheduler’s change feed, is the directory /afw/eureka/journal.
Authorization policies live under /afw/auth.
Back these up together. A store without its journal, or a journal from a different store, is not a consistent restore.
Users and passwords (LDAP)
Section titled “Users and passwords (LDAP)”User accounts and password hashes are in OpenLDAP on the control node. The data directory and TLS certificates depend on the machine architecture:
- On x86_64, the database is
/var/symas/openldap-data. The server configuration is/opt/symas/etc/openldap/slapd.conf. The LDAP TLS certificate and key are/opt/symas/etc/openldap/cert.pemand/opt/symas/etc/openldap/key.pem. - On aarch64, the database is
/var/lib/ldap. The TLS certificate and key are/etc/openldap/certs/cert.pemand/etc/openldap/certs/key.pem.
In both cases, the LDAP manager bind password is /root/eureka-ldap-bind. Restore that file with the database. If it is missing, a later start can generate a new password that will not open the restored database.
JWT keys
Section titled “JWT keys”Each node has a personality at /etc/eureka-personality and a secret at /var/eurekakey. The control node also keeps a verifier for each live node under /var/eurekakeys.
If you snapshot and restore the control and login disks together, keep these files as part of that snapshot. If the control host was compromised, treat every verifier as compromised and do not restore verifier files from that host. See JWT security.
Other control files
Section titled “Other control files”/etc/eureka/, includingeureka.yamlandinstance-type-defaults.json. Other files may also appear here (for example a desktop-gate secret). Back up the whole directory./var/lib/eureka/storage-admin, which holds Switch state for external filesystems. Include it if a Switch might be in progress.
/afw/models and much of /afw/config are generated when the control node starts from the product image. Whether a rebuild needs those files restored, or can recreate them, has not been verified as a standalone restore; a full control-disk snapshot includes them.
Login node
Section titled “Login node”/vault, only on login-VM NFS
Section titled “/vault, only on login-VM NFS”Include /vault (the standard /vault/home, /vault/projects, and /vault/containers trees) only when those trees are still on the default login-VM NFS. In that case the data lives on the login disk at those paths, and the login VM exports them. See Storage for production.
When /vault is on external storage (Filestore, EFS, or another NFS you pointed the standard trees at), that storage’s own backups cover it. Do not treat a copy of /vault on the login VM as the backup of that data; the login node’s copy is hidden after a Switch.
Sign-in and the portal certificate
Section titled “Sign-in and the portal certificate”Sign-in settings (which sources are on, the Google client, the InCommon allow list, and each user’s linked identities) are stored on the control node in AFW, as described above. The login node also has files that were generated from those settings and certificates that must stay the same if you have already registered the environment with a campus IdP:
/opt/simplesamlphp/cert(includingincommon.keyandincommon.crt, created once and reused)/opt/simplesamlphp/configand/opt/simplesamlphp/metadata/etc/httpd/mellon(Mellon key, certificate, and IdP metadata)/var/data/simplesamlphp(includingsimplesamlphp.sqlite)/etc/eureka/sign-in-sources.json/etc/ood/config/ood_portal.yml
The public HTTPS certificate, when Let’s Encrypt issued it, is under /etc/letsencrypt. Apache is configured to use /etc/pki/tls/certs/localhost.crt and /etc/pki/tls/private/localhost.key, which are linked to that lineage when a Let’s Encrypt certificate is installed. A paused login keeps its domain, and an existing valid certificate is reused.
Project buckets and other cloud data
Section titled “Project buckets and other cloud data”Managed bucket contents live in the cloud account (Cloud Storage or S3), not on the control or login disk. Back those buckets up with the cloud’s own tools. projectEureka’s records of the buckets (names, which project they belong to, flow-bucket settings) are in the AFW store on the control node.
External buckets are not created by projectEureka. Use that bucket’s own backups. Deleting the projectEureka entry does not delete the cloud bucket.
On Kubernetes there are no managed buckets; only homes, project files, and any external buckets you add.
A practical backup set
Section titled “A practical backup set”- A copy of the launch configuration, kept off the environment.
- A snapshot of the control node disk (this covers the AFW store, journal, authorization policies, LDAP, the bind password, JWT verifiers, and
/etc/eureka). - A snapshot of the login node disk if
/vaultis still on login-VM NFS, and in any case if you need the existing Let’s Encrypt lineage or SimpleSAMLphp and Mellon certificates. - The cloud’s backups or versioning for managed project buckets.
- The NFS service’s own backups when home, projects, or containers are on Filestore, EFS, or another external NFS.
Take the control snapshot while the environment is idle if you can, so the AFW store, journal, and LDAP are not mid-update. How to quiesce AFW and LDAP for a perfectly consistent filesystem copy has not been separately verified; a disk snapshot of a paused or idle control node is the intended method.
Restore or rebuild
Section titled “Restore or rebuild”There is no supported one-command restore. These are the approaches that follow from where the data lives:
- Same virtual machines, restored disks. Restore the control snapshot (and the login snapshot if you took one). Attach the same external NFS and the same buckets. Start the control node, then the login node, as you would after a pause. See Pause and resume the environment.
- New virtual machines, same storage. Launch from the saved launch configuration so the new environment mounts the same external NFS. Restoring the AFW store, journal, and LDAP piece by piece onto a freshly launched control node has not been verified. If you do not restore the control disk as a whole, treat that as a procedure to verify before you rely on it. JWT secrets on a new set of nodes are issued when those nodes start; do not copy verifiers from a compromised control host.
- Login-VM
/vault. If you are restoring login-hosted data, restore/vault/home,/vault/projects, and/vault/containersonto the login disk before clients mount them. If those trees were moved to external NFS, restore them with that storage’s tools instead.
After a restore, confirm that you can sign in, that projects and buckets appear, and that a Terminal can see the expected home and project directories.
Related
Section titled “Related”Storage for production · External filesystems and clusters · Sign-in · JWT security · Pause and resume the environment · Environment overview