Documentation
Automatic security updates
Contents
Overview
TurnKey automatically installs the latest security updates over the network:
- The first time you boot a new appliance deployment (you can choose to skip this)
- Every night, around 4 AM.
Usually automatically updating software is considered to be a risky practice since updates may occasionally break existing functionality (e.g., changes to file formats, software interfaces, or expected behavior).
Debian mitigate this risk by carefully backporting security fixes so that security updates change as little as possible, minimizing the likelyhood that things will break.
In practice we've found it is very rare for a security update to break something, so we believe it is beneficial to configure software appliances to auto-update security fixes by default. Advanced users can always disable this mechanism and apply security fixes manually if they want.
Installing security updates on demand
In a root shell, run the following command:
turnkey-install-security-updates
If that doesn't work you may be running an older version of TurnKey. Try this instead:
/usr/sbin/cron-apt
Caution: This isn't 100% full-proof. Make sure we can reach you.
Unfortunately, we can't fix everything automatically so it's still very important that we be able to contact you when necessary. Make sure you're subscribed to TurnKey's low-traffic announcements newsletter. (Requires registration for a free website user account, "Security and News Announcements" subcriptipon checkbox should be ticked. If already registered, check the "subscriptions" tab of your user account info).
Otherwise you may not know that a problem requires your attention until it's too late. Sure, thanks to automatic security updates we usually don't need to bother you regarding security issues, but there are occasional exceptions...
- Not everything can be updated automatically: automatic security updates only work for supported software that is maintained using the package management system. Not all software is installed through the package management system. Not all software installed through the package management is supported. See the limitations section below for details.
- Some bugs can break automatic updates: even though security updates change as little as possible and are exceptionally well tested, mistakes can still happen. Usually these can be caught and fixed with another automatic update, but manual intervention is still required for bugs that break the auto-updates mechanism or one of its dependencies (e.g., Ubuntu broke cron).
How it works
Users who wish to tweak the auto-update mechanism may find it helpful to understand how it is set up.
1) A cron job is configured to run cron-apt daily.
# cat /etc/cron.d/cron-apt # # Regular cron jobs for the cron-apt package # # Every night at 4 o'clock. 0 4 * * * root test -x /usr/sbin/cron-apt && /usr/sbin/cron-apt
2) cron-apt is configured to only update from the security sources list.
# cat security.sources.list deb http://archive.turnkeylinux.org/debian buster-security main deb http://security.debian.org/ buster/updates main deb http://security.debian.org/ buster/updates contrib # deb http://security.debian.org/ buster/updates non-free
3) cron-apt is configured to install the updates automatically:
$ cat /etc/cron-apt/action.d/5-install
autoclean -y
dist-upgrade -y -o APT::Get::Show-Upgraded=true \
-o Dir::Etc::sourcelist=/etc/apt/sources.list.d/security.sources.list \
-o Dir::Etc::sourceparts=nonexistent \
-o DPkg::Options::=--force-confdef \
-o DPkg::Options::=--force-confold
4) cron-apt logs to /var/log/cron-apt/log
Issue resolution options
There is a slim chance that a security update may have a new dependency that is not located in one of the security repos configured in /etc/apt/sources.d/security.sources.list. Because auto security updates will only ever install packages from the security repositories, if a security update requires a new dependency that is not in a security repo (e.g. in "main") it will unfulfillable. The default auto sec update config will then cause the package with an available security update to be uninstalled. I'm aware of that occurring once in TurnKey's history (Nov 2018 - v15.x). In that particular incident MariaDB was auto-uninstalled, crashing applications that depend on the very popular and common database that is included in ~70% of the TurnKey library.
While the TurnKey server itself will continue to run, obviously a broken website is far from ideal! The upside of this default behavior is that an application crash will ensure that your server remains secure and you're aware of the issue very quickly. Although your web site might be broken, it will not be exploitable because of an insecure package. You can manually reinstall the removed package and all will be well again.
However, if your TurnKey server is "mission critical" an outage that can only be resolved by manual intervention might be a real problem. In this scenario, the security trade-off of not installing the update might be worth it. I.e. not installing a security update and not crashing your web app. This would particularly be the case if the server is not running in a hostile network - e.g. within a LAN and not publicly accessible.
This potential update issue could be worked around by disabling auto security updates altogether. However, there is a middle ground. TurnKey provides a config option to install security updates, but avoid this potential issue via an alternate strategy. The alternate strategy will never remove any packages. If a security update can't install, then it will leave the (unpatched and potentially insecure) package as is. This alternate strategy can be selected via TurnKey Confconsole:
Confconsole >> System settings >> Secupdate behavior
Whichever strategy you use, you will need to be aware of the inherent risk trade-offs. And either way, be sure to make sure we can reach you so we can keep you in the loop around any manual steps required to resolve the issue.
Limitations
The latest TurnKey Linux release is generally based on the latest version of Debian stable (the exception is a small lag when a new Debian stable is released). The Debian Security Team provides backported security fixes for all packages in Debian 'stable' as required, which TurnKey systems are configured to automatically install (as per this page). The Debian Security team also continue to provide security updates for Debian 'oldstable' for a year after the release of a new 'stable'. After this initial year, security maintenance is handed over to the LTS (long term support) team. The LTS team will continue to provide security updates to most packages for the life of the LTS release (4 years after the Security team hand maintence over - i.e. 5 years from the following 'stable' release).
However, Debian's security coverage does not apply to packages that do not originate from Debian.
- Trusted third party repositories: ideally the Debian package repositories would cover 100% of our software needs. Unfortunately in practice there's a lot of good software out there that Debian does not package. In these cases, where possible TurnKey will install software directly from trusted third party repositories.
Note that any packages that does not originate from Debian/TurnKey repositories is documented on the appliance product page, and also in the product's source code.
- TurnKey Linux custom packages: TurnKey contains a few custom packages which are updated directly by the Core developers from the project's cryptographically signed package repository. Security updates will be provided via the TurnKey security repo and auto installed by the default automated security update mechanism.
- Software installed from source code: unfortunately, many of the most popular open source web applications (e.g., Joomla) are not packaged by Debian. This means that they have to be installed and maintained by hand directly from upstream source code and no automatic security updates can be provided through the package management system.
Fortunately, most web applications run with reduced privileges and are developed in high-level programming languages that are less susceptible to many of the most serious low-level security vulnerabilities. Also in the appliance model, each application is confined to its own virtual machine. This limits the potential damage somewhat but vigilance is still recommended, especially for high-risk usage scenarios.
When a TurnKey appliance includes software installed from upstream source code, this is usually the first thing documented on the appliance page. Many (most?) also have the update/upgrade process documented. As much as possible we try to install applications as per upstream recommendations, so the upstream upgrade docs will generally apply. If you encoutner any problems upgrading, please feel free to report it to us and we will provide support. We will always provide "best effort" support, but please be aware that upgrading particularly old software can sometimes be problematic.
You can use the "apt policy" command to determine a package's origin. Note that you should generally run "apt update" prior to ensure that your local package index is up to date with the repository servers. E.g.:
# apt policy openssh-server openssh-server: Installed: 1:9.2p1-2+deb12u3 Candidate: 1:9.2p1-2+deb12u3 Version table: *** 1:9.2p1-2+deb12u3 500 500 http://security.debian.org/debian-security bookworm-security/main amd64 Packages 500 http://deb.debian.org/debian bookworm/main amd64 Packages 100 /var/lib/dpkg/status
So in the case of openssh-server, we have the most recent version installed and are receiving updates automatically from the security.debian.org repository.