Server

The server is hosted by Infomaniak, via a partnership. Configuration (e.g. port forwarding) is done from their website (here, credentials are on the Vaultwarden) and on a sheet at the Desk.

Infrastructure

Infrastructure configuration is on the Clic Infra repository. Each service is hosted as a Docker stack. This makes it easy to add new services without worrying about dependencies, architecture, etc. Updating the server when the configuration has been modified is done using the [Ansible] tool (https://www.ansible.com/).

Important note: the reverse-proxy (Caddy) is not in a Docker container, but directly installed on the machine. See explanation below.

Add/Remove/Modify a service

Modifications to the infrastructure must first be made on the infra repository, then applied to the server using Ansible (see below).

Each service is configured via a docker-compose.yaml file (doc), in a folder unique to the service at the repository root. Services generally never communicate with each other other other than via their exposed APIs (e.g. the website accesses Directus via its public url, not via a dedicated virtual network). If a service requires a database, it is instantiated in the same stack as the service and is exclusively dedicated to it.

If the service needs to be accessible via the Internet, it must expose one or more ports, which will then be exposed to the outside world by the reverse-proxy.

Secret management

Services may require private (or server-specific) information, such as an admin password, or a mailbox password. These values are stored in a secrets.yaml file (in /home/debian), and loaded by Ansible during deployment. They must be referenced in the playbook/deploy.yaml playbook in the verification task, and passed as environment variables to the stack deployment task. They can then be retrieved from each stack's configuration using the syntax ${MY_SECRET}.

Deployment

Deployment is carried out using the playbook playbook/deploy.yaml, which will update all stacks. /!\ Warning /!: stacks must be modifiable with a simple docker stack deploy; don't use config, as these are immutable. The stack would have to be removed and then redeployed, which is sub-optimal. Instead, configure services via their environment variables.

Backups

Backups are managed by BorgBackup, enabling encrypted, incremental backups. The ~~[Borgmatic]~~Borgmatic tool, configured by Playbook playbook/deploy.yaml, periodically generates and uploads backups. Its configuration (also in the Playbook) must be updated whenever a service is added/removed/modified.

To manually create a backup, you can run sudo borgmatic (ideally in a detached screen screen sudo borgmatic, since it takes several hours to complete). You can inspect the backup repository using the borg CLI tool.

Installation

In the event of a complete server reinstallation, here are the steps to follow:

Use the init playbook: create an inventory on your computer (see the docs), then ansible-playbook -i <inventory> <path/to/init.yaml>.
SSH into the server.
Set up the /var/secrets.yaml file (see the example in the infra repo). Move the database dumps to the directories set in this file. If you do not have access to the Directus tokens, use a stub.
Generate an SSH key for the default user an one for root and save both in the deployment keys of the infra repository.
Launch a swarm with Playbook deploy.yaml: ansible-pull -U git@github.com:clicepfl/clic-infra.git -e @/var/secrets.yaml playbooks/deploy.yaml.
Load the databases content, using docker exec -it <container-name> sh then either psql -U <user> <database> < /docker-entrypoint-initdb.d/init.sql or mysql --user <user> -p < /docker-entrypoint-initdb.d/<script-name>.sql.
Stop all the services (docker swarm leave --force).
Copy/Move the file content in the volumes (stored under /var/lib/docker/volumes/<volume-name>/data), after having remove the content placed by the first run of each service. Skip database volumes (already initialized).
If needed, regenerate token for Directus and set them in the secrets file.
Restart all the services using the playbook deploy.yaml.
Launch caddy: ansible-pull -U git@github.com:clicepfl/clic-infra.git playbooks/caddy.yaml
Launch the webhook: ansible-pull -U git@github.com:clicepfl/clic-infra.git playbooks/webhook.yaml

Everything should be set now !