Upgrading to Amazon Linux 2023
by Sebastien Mirolo on Thu, 4 Jan 2024
Over time we ran production workloads on Fedora, CentOS7, Amazon Linux 2,
and now Amazon Linux 2023 (which means we are back on Fedora I guess).
This post is about the quirks we encountered migrating from Amazon Linux 2
to Amazon Linux 2023.
Picking an Amazon Linux 2023 AMI
The reference document from AWS Comparing Amazon Linux 2 and Amazon Linux 2023 is worth reading first. Then let's pick an AMI to get started with:
$ aws ec2 describe-images \
--owners amazon \
--filters "Name=name,Values=al2023-ami-*" "Name=state,Values=available" \
--query "reverse(sort_by(Images, &CreationDate))" | jq '.[].Description'
"Amazon Linux 2023 AMI 2023.2.20230920.1 x86_64 Minimal HVM kernel-6.1"
"Amazon Linux 2023 AMI 2023.2.20230920.1 arm64 Minimal HVM kernel-6.1"
"Amazon Linux 2023 AMI 2023.2.20230920.1 x86_64 HVM kernel-6.1"
"Amazon Linux 2023 AMI 2023.2.20230920.1 arm64 HVM kernel-6.1"
"Amazon Linux AMI 2023.0.20230912 arm64 ECS HVM EBS"
"Amazon Linux AMI 2023.0.20230912 x86_64 ECS HVM EBS"
"Amazon Linux AMI 2023.0.20230912 x86_64 ECS HVM EBS"
No EPEL repo
The most inconvenient of AL2023 is the lack of support for the EPEL repo. That means, we cannot easily install common tools such as fail2ban, syslog-ng, or openssh-ldap.
authconfig is replaced by authselect
Basically when you run authconfig, it runs a "compatibility tool" with a warning message.
$ authconfig --update --enablesssd --enablesssdauth Running authconfig compatibility tool. The purpose of this tool is to enable authentication against\ chosen services with authselect and minimum configuration. It does not\ provide all capabilities of authconfig. IMPORTANT: authconfig is replaced by authselect, please update your scripts. See Fedora 28 Change Page: https://fedoraproject.org/wiki/Changes/AuthselectAsDefault See man authselect-migration(7) to help you with migration to authselect
The warning message seems to imply the command is still working, though will
be deprecated soon. None-the-less things stopped working because of it properly.
Running authselect directly fixes all permission issues.
$ authselect select sssd --force
IMDSv1 vs IMDSv2
We used to suffix logs by the instanceId before uploading them to storage.
$ curl -v http://instance-data/latest/meta-data/instance-id HTTP/1.1 401 Unauthorized
That doesn't work anymore. We ended up getting a token, then calling the API using that token, IMDSv2-style:
$ TOKEN=`curl --silent -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` curl --silent -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id
Interpretation of '\' by systemctl
The systemd services were configured with variables defined in a /etc/sysconfig environment config file that was referenced by EnvironmentFile in the service unit. Previously we had to double-backslash (i.e. '\\') the backslash in the /etc/sysconfig file to make everything work. That's no longer the case.
# AmazonLinux2 $ systemctl --version systemd 219 +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN # Amazon Linux 2023 $ systemctl --version systemd 252 (252.16-1.amzn2023.0.1) +PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP -GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 -BZIP2 -LZ4 +XZ -ZLIB -ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
No cron by default
AL2023 does not have anacron installed by default (fortunately
you can still install it). Instead AL2023 relies on
systemd timers
to run commands like logrotate.
$ systemctl list-timers NEXT LEFT LAST PASSED UNIT ACTIVATES Wed 2024-07-17 00:00:00 UTC 6h left Tue 2024-07-16 00:00:03 UTC 17h ago logrotate.timer logrotate.service
Upgrading packages
AL2023 introduces Deterministic upgrades. In practice, this means you are often not upgrading packages individually as updates become available, but upgrading a whole set of packages at once. It has the benefit to be able to map all package versions from a single AL2023 version number.
The AL2023 smart-restart package will be convenient to restart systemd services after they have been updated on a system update.
$ dnf install smart-restart $ dnf update --releasever=2023.3.20231218
Et Voila!
More to read
If you are looking for related posts, Upgrading to Java11 on Amazon Linux 2 and Running an Amazon Linux2 virtual machine on your own hardware are good reads.
More technical posts are also available on the DjaoDjin blog. For fellow entrepreneurs, business lessons learned running a subscription hosting platform are also available.
by Sebastien Mirolo on Thu, 4 Jan 2024
