#Podman als Zwischenstufe von #Docker zu #Kubernetes.
Ich hatte allerdings gehofft, er geht etwas auf das Pod-Konzept ein. Aber egal, war trotzdem wieder sehr lehrreich.
Danke #GLT
#GLT2024 #Graz #ccc #Chaos #systemd #Container #GNU/Linux #PoweredByRSS
The planning took a lot longer than the actual move time, and the downtime for my main blog was really only around 10 minutes. But this has inspired me to also document some of the steps I took for my own future reference. Also, many videos and guides I watched, often only dealt with a single aspect of the migration, such as the Docker volumes backup and restore.
I’m not going to make this a detailed step by step guide as everyone’s situation is different, but hopefully this conceptual overview, and some detail with links, will help many others. I’m not sure yet if I’m going to do an explainer video, but maybe this post will help me make up my mind.
My hosting environment is a hosted VPS service running Ubuntu Linux, with Docker and Portainer, which host various web services, each in their own Docker container and related Docker volumes for persistent storage. One of those Docker containers is a Nginx Proxy Manager reverse proxy which routes incoming requests to the correct web services. There is also an OpenVPN container that allows me to authenticate and drop into the LAN environment to do maintenance. The DNS (for resolving the URL for each service) is handled through a free CloudFlare service. That DNS service points to the correct main public IP address for the server. The theory is, as soon as the service has been backed up and restored (and tested) to the new server, I update the IP address in the DNS to point to the new server, and any visitors get routed immediately to the new server without knowing the difference.
The high-level steps one needs to perform are:
Check some details on your existing server environment such as:
Perform Portainer backup inside Portainer – this saves all the Stacks with their configs to recreate the container images on the new server. It does not backup Docker networks or volumes.
Before I started the actual backups, I first got the new VPS setup with:
install Go (required for Docker-Backup) and installed Docker-Backup on both existing server and new server
Stop containers on existing server before backing up – to ensure all data is written to files and databases can be safely copied. I started with stand-alone services first which were not using the shared database, so I could test them being copied and restored first on the new server. So, my OpenVPN and NginxPM, Wordle, and Glances all worked without the shared database.
Run Docker-Backup on existing server for first one or two container IDs from inside /root/docker-backup with the command ./docker-backup backup --tar <containerID>. Running as root saved those compressed files to /root/docker-backup as a .tar file. This application includes all the volumes for that container as well as user permissions.
Log into terminal on the NEW server, go to /root/docker-backup (because we want to pull across the backed-up file into the same location on the new server). Copy the backed up file from this directory with the command scp root@191.101.59.143:/root/docker-backup/backedupfile.tar backedfile.tar . This will bring across that backed up file into the new server with that same name in /rot/docker-backup. Note for remote scm copy the existing server needs to allow password login, so you may need to reenable that for this to work.
Run Docker-Backup with the restore option to recreate the container’s volumes with all the persistent data and user file permissions, from the backed-up file. The share database will also just be a container volume being restored. You can browse /var/lib/docker/volumes to check the persistent data has been restored. I noted two issues though, and one was sometimes a volume name like glances would not be recreated with that name, but instead a long number of digits (I have no idea why) so I’d verify by its contents what it should be, and then rename it back to glances (or whatever). Also often got the error: Error response from daemon: No such image: kylemanna/openvpn:latest. I solved this in my case by just running docker pull <image-name> to pull the image, before running the restore again which then worked.
In theory now you could just spin up that container and it should work by finding its persistent data in its volumes. For some containers this did not work, and what fixed it, was going into Portainer and opening the Stack for that service, and just forcing a Stack Update (in /Stacks/Edit). In that way I also ensured the container and volumes were all properly connected. Don’t worry, it won’t overwrite any of the volume data you’ve restored.
The above is why I also temporarily opened ports 9000 and 81, so that I could run Portainer and NginxPM from my web browser with just the public IP of the server (easy, and I lock it down as soon as I finished setting up).
That in essence was it, I did this one by one, testing each one. When it came to my blog and the shared database, I just did that batch all together with the database restored first, so that the others could all come online and connect to the database.
My last steps after everything was working and tested, was to close ports 9000 and 81 on the new server, and to shut down the old server and notify the provider to cancel the account.
So, like I said the actual WordPress blog was only down around 10 minutes or so while I backed up and restored that batch with the database. I did my 11GB of photos in Piwigo last as that took longest to copy across and restore. I did ditch my Immich photos setup as I realised, I’d complicated things by having that connect directly into the Piwigo photos volume. That resulted in the Immich backup, including the 11GB of photos, and worse, restoring them in an odd location which was no longer linked to Piwigo. But no harm done, as it was just a test install and all my photos were still safely in, and attached to, Piwigo.
The Docker-Backup install guide was pretty long across more than one page, so I could summarise my steps here as its instructions at https://github.com/muesli/docker-backup and to install Go at https://go.dev/doc/install/source using:
wget -c https://go.dev/dl/go1.22.1.linux-amd64.tar.gz
rm -rf /usr/local/go # In case it exists already
tar -C /usr/local -xzf go1.22.1.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
go version # To check its installed
Installing Docker-Backup itself:
git clone https://github.com/muesli/docker-backup.git
cd docker-backup
go build
./docker-backup # Should show help info
In hindsight maybe I would try just the straight backup of the volume data using the docker command docker run --rm --volumes-from dbstore -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /dbdata and its corresponding restore on the other side.
It all went through fine although I also realised that my Nextcloud setup was quite out of date. I’ve been running it for many years and some of the configuration setup has actually changed. I decided to actually wipe the container as well as all volume data as I’m the only user, and the data is anyway synced from my desktop PC. The new install is way faster, and as soon as I connected the Nextcloud desktop sync again, all my documents and photos were just resynced to the server.
All-in-all about a full day’s research and testing bits of this to see how I’d do it, and I used an afternoon the following day to do the actual migration. It was lots of fun and now at least I know how to switch hosting providers again if I need, or want, to.
#Blog, #docker, #hosting, #migration, #technology
Nun gut - der Schritt von #docker zu #podman ist getan und Dank #podman-compose läuft unsere #Nextcloud quasi 1:1 seit 8 Monaten stabil.
Es wird Zeit, den zweiten Schritt zu tun: Den Compose-Rotz in einen #Pod zu migrieren, um dann rootless Port 80 und 443 anzubieten und einfache #systemd - Integration in einem Rutsch zu erhalten.
Hab' ja noch keinen Schimmer, wie ich vorgehen soll...
Convert any container into a website (well, most containers…)
Auf meinem für einen ganzen Schwung von Aufgaben dienenden Raspberry Pi laufen Home Assistant, Whoogle, Watchtower und Portainer in Docker-Containern. Kurz- bis mittelfristig werden wohl noch weitere auf dieser Art bereitgestellte Dienste hinzukommen.
Möchte ich da eigentlich mittlerweile unbedingt zu podman wechseln? Oder sollte ich vorher genau recherchieren, in welchen Fällen podman aktuell doch noch nicht so ein passgenaues Drop-In-Replacement darstellt?
Apparently the amd64 emulation used when building a #Docker image on arm64 hardware is only so good. When installing #Ruby with rvm it decides to install the arm64 version anyway.
$ uname -a
Linux 628f228dda6f 4.14.22 #1 SMP Thu Oct 27 08:01:25 UTC 2022 Build-38 x86_64 GNU/Linux
$ file /home/diaspora/.rvm/rubies/ruby-2.7.8/bin/ruby
/home/diaspora/.rvm/rubies/ruby-2.7.8/bin/ruby: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=e3ea67d64604f36deef9f6eb95cc99078337d287, with debug_info, not stripped
So the new Diaspora Docker image will be up in a few minutes while I build it on some amd64 hardware I have lying around.
Friendica actually has its own federating protocol, but it supports others too, such as Activity Pub (Fediverse), Twitter, Tumblr, and more. So it becomes a single client app that you can use as a hub to various other social networks (including Mastodon as part of the Fediverse).
Friendica’s functionality is not just about posting a status with an image, as it has calendaring, photo albums, forums, and more. I did a video about it at https://www.youtube.com/watch?v=nS6oAy7ibqc.
In many ways it is similar to Hubzilla (although Hubzilla does even more including its nomadic profiles) but one big differentiator is that Friendica can be easily installed and run with Docker. This makes it really easy to host as part of a virtual private server, or anywhere Docker can operate.
Friendica is a great social network to host for family and organisations, especially with its photo albums and calendaring. With its ActivityPub protocol, it still easily integrates with the entire Fediverse of social networks as well, so you are never isolated.
See https://github.com/friendica/docker
#Blog, #docker, #Friendica, #opensource, #socialnetworks, #technology
Docker is a popular containerization platform that allows you to easily package, distribute, and run applications in lightweight, portable containers. In light of this, one of the essential tasks in Docker container management is keeping container images up-to-date with the latest software patches, security fixes, and feature updates.
In this article, Linuxiac introduces you to an exciting tool called “Watchtower,” which automates updating Docker container images. They explore what Watchtower is, how it works, and how it can simplify your Docker workflow.
There is a school of thought though that says you should not just allow auto-updates to take place because things can go wrong. For example, over the last 3 or so years I’ve been using Docker, there was once or twice that I had to manually run a command that did an update on the external database (which meant had to roll back the image update, run the update on the database, and then re-install the latest image). So, there is an option for Watchtower to just notify you of any image updates, and to not install them automatically (the option I use).
See https://linuxiac.com/watchtower-automatically-update-docker-container-images/
#Blog, #docker, #technology, #Watchtower
Source: https://blog.alexellis.io/docker-is-deleting-open-source-images/
Yesterday, Docker sent an email to all Docker Hub users explaining that anyone who has created an "organisation" will have their account deleted including all images, if they do not upgrade to a paid team plan.
#floss #foss #warning #problem #economy #capitalism #danger #software #security #news
It took way too long, but my #ChatOps quest continues. Today I finished adding #Matrix support to my #Dovecot Sieve scripts, so that things like #DKIM and MTA-STS TLS reports could go to a matrix chat channel rather than sitting in a mail folder. Basically I now have an email 👉️ matrix bridge.
It took a fair amount of fussing about, mostly because (a) it takes matrix commander (a Matrix CLI) a long time to post a message to Synapse (read: more than ten seconds), (b) Dovecot’s documentation for altering settings for external scripts is byzantine, and (c) rather than cramming matrix commander into my already bloated Dovecot #Docker image I wanted to use the Docker image they provide, and I needed to work out a way to invoke matrix-commander from another container on the same pod.
But now it’s done, and I have another vector for #admin alerts that I can coalesce into a single place for easy review.
I worked quite hard to solve this problem, and I’m happy with its (eventual) simplicity. When you have (for example) two Kubernetes containers in a pod (or two processes that can share a named pipe) and you need to run a process on one of them from the other one, I have just the tool for you. It’s basically ssh without all the pesky networking, using named pipes instead of TCP streams.
Wait what? 👀
jaywink@lentil:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.8G 0 3.8G 0% /dev
tmpfs 777M 1.4M 776M 1% /run
/dev/sda1 38G 31G 4.7G 87% /
tmpfs 3.8G 0 3.8G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
/dev/sda15 253M 1.1M 252M 1% /boot/efi
tmpfs 777M 0 777M 0% /run/user/1000
jaywink@lentil:~$ docker system prune
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all dangling build cache
Are you sure you want to continue? [y/N] y
Deleted Containers:
50a6dc932118113ad8a2f221ea9f89f8285dfd6a8b1b9ddb0455c6e64b9b7eb2
Total reclaimed space: 706.3GB
Well, at least this got rid of the unexplained "no space left on device" error, where there was actual space available.
Ugh. The tool file stuff in the latest #Diaspora release has broken my #docker build. It'll take some time to sort out.
I can understand the desire to use toml to generate the diaspora.yaml file, but the unintended consequence is that it won't work well in an image-based ephemeral environment, preferring a single mode of deployment instead.
I'll see what I can do.