How do we verify apt packages using GPG keys?

If you download/install packages from the Debian archive(s) using apt-get(8), I believe what you want is done for you automatically. See:

https://unix.stackexchange.com/questions/63054/apt-get-update-and-package-authentication

Here's the list of keys for my Debian 10 (Buster) system:

stevewi@dave:/etc/apt/trusted.gpg.d$ ll
total 60
-rw-r--r-- 1 root root 8132 Apr 23  2019 debian-archive-buster-automatic.gpg
-rw-r--r-- 1 root root 8141 Apr 23  2019 debian-archive-buster-security-automatic.gpg
-rw-r--r-- 1 root root 2332 Apr 23  2019 debian-archive-buster-stable.gpg
-rw-r--r-- 1 root root 5106 Apr 23  2019 debian-archive-jessie-automatic.gpg
-rw-r--r-- 1 root root 5115 Apr 23  2019 debian-archive-jessie-security-automatic.gpg
-rw-r--r-- 1 root root 2763 Apr 23  2019 debian-archive-jessie-stable.gpg
-rw-r--r-- 1 root root 7443 Apr 23  2019 debian-archive-stretch-automatic.gpg
-rw-r--r-- 1 root root 7452 Apr 23  2019 debian-archive-stretch-security-automatic.gpg
-rw-r--r-- 1 root root 2263 Apr 23  2019 debian-archive-stretch-stable.gpg

On Ubuntu, there is the concept of a PPA (personal package archive). If you want to use one of these, you have to install the GPG key for the PPA repository into apts key collection so that packages installed from the PPA can be authenticated.

https://itsfoss.com/ppa-guide/

Note that this probably works for Debian too…but YMMV.

-- sw

I am also using Debian 10, so this is hopeful. However, according to the article, the presence of the gpg keys does not imply that any checks are being done.

In my /etc/dpkg/dpkg.cfg, at least, I see no-debsig being used, which might imply that this feature is disabled. So I'm still worried that package contents are not being verified.

I'm a bit lost, really. I am not finding a lot of helpful documentation, unfortunately. I can't even find any .deb files on my system, even after installing a package. Do the files get cleaned up? Who knows. I'm hoping I don't have to look at source code to figure out what's going on. :-)

I do have the .gpg files mentioned by stevewi are on my system too, but I don't know what they are used for (maybe they are just for package metadata verification?).

If nobody else has set this up, how do I find out more information? Is there a way to talk to the Debian people involved?

You write:

I can't even find any .deb files on my system, even after installing a package. Do the files get cleaned up? Who knows. I'm hoping I don't have to look at source code to figure out what's going on. :-)

I have a few in /var/cache/apt/archives… If you run sudo apt-get clean, cached .deb files are removed (to reclaim disc space). I routinely do this… The man(1) page for dpkg(1) lists a number of files/directories it uses. Those may contain useful information.

Regarding:

I do have the .gpg files mentioned by stevewi are on my system too, but I don't know what they are used for (maybe they are just for package metadata verification?).

These may help:

From Ubuntu: https://help.ubuntu.com/community/SecureApt

From Debian: https://wiki.debian.org/SecureApt

-- sw

Ah! Thank you for the help! Now I'm getting somewhere. Certainly, from a quick skim of the Debian SecureApt page, it seems that everything is properly signed and verified. It's hard to tell how up-to-date that page is, of course, and I will read it more carefully later.

Some doubts remain, but I will see if I can somehow confirm through testing that verification is happening. Especially now that I found some .deb files.

My /var/cache/apt/archives dir was empty, but I eventually found that this was because I'm using apt instead of apt-get. I guess this is to be expected, as a more "user-friendly" front-end to dpkg, apt may be doing a bit more cleanup (you probably wouldn't need to "apt-get clean" periodically).

I will try to carve out some time to poke and prod at these files. I'm still being niggled by the words on the web-page I mentioned, the no-debsig in /etc/dpkg/dpkg.cfg, and a suggestion from our friends at the CIS that the output from "apt-key list" (I get "E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation") might indicate that my server isn't properly configured.

@Metallo writes:

Certainly, from a quick skim of the Debian SecureApt page, it seems that everything is properly signed and verified.

I couldn't imagine this process without some form of security/authentication.

It's hard to tell how up-to-date that page is, of course, and I will read it more carefully later.

At the bottom of the page is the following:

SecureApt (last modified 2020-11-09 03:27:14)

…so pretty recent.

FWIW, I use apt(1) too (mostly because I don't have to remember a lot of arcane stuff). The files I found in /var/cache/apt/archives were all very old. I've removed them.

-- sw

I've put in a few more hours into this, and here's what I have to share. First of all, at the top of wiki.debian.org/SecureApt, in the basic concepts section, it says that apt-key can be used to show the security keys in the key-ring. Running apt-key list, gives the GnuPG* not installed error. So I ran, apt install gnupg2, and then it worked, showing me the public keys corresponding to the files we have already talked about in etc/apt/trusted.gpg.d.

OK. So far so good. The next section, "Secure apt groundwork: checksums", talks about the contents of a Debian archive and the checksums in it. To take a look at this, I started a fresh Linode, then ran apt download pslist to get pslist_1.4.0-2_all.deb, which I hope is an example of what they are calling a Debian archive.

After apt install binutils (needed for ar), I extracted (ar x pslist_1.3.0-2_all.deb) the contents of this archive to get control.tar.xz (meta-data), data.tar.xz (package-contents), and debian-binary (Debian Archive Format Version). I then used tar to extract and examine the contents of the embedded .xz archives. In the package contents, there is a perl script at usr/bin/pslist. I edited that to output a message showing it had been changed.

I then reassembled the modified data.tar.xz file (tar cJf data.tar.xz usr) and then the Debian archive (ar r pslist_1.4.0-2_all.deb debian-binary control.tar.xz data.tar.xz) and used apt to install the newly reassembled package (apt install ./pslist_1.4.0-2_all.deb).

Unfortunately, apt did not complain at all. So, locally at least, I was able to change the contents of a .deb archive, and install it without the system detecting anything amiss. Running pslist includes the print statement I inserted, confirming corruption of the package.

Of course, this doesn't mean that when downloading and installing from the original source, some mechanism isn't doing the required checks. But it certainly is not as confidence inspiring as if it had stopped what I did.

I also noticed that the control.tar.xz archive includes an md5sums file that has m5d checksums for some, but not all of the executables in this package. I don't get the impression that the md5sums file is being forced to account for all package contents. If it is part of the apt security model, it might be unreliable.

And another ding to my confidence is finding that even wikipedia (that reliable source of information on linux minutiae) says that package signing probably isn't enabled.

At this point, I have to be wondering if I should adopt some pragmatic stance relative to apt security. For my purposes (ie hosting a personal web app), it's not worth the time I'm putting into this, so far. Not using apt at all is suddenly an option to be considered.

An update on this, in case anyone else stumbles on this question. There is a description of how apt package verification works in section 7.5.1 of the Securing Debian Manual. This description is a bit out of date, but it gets the broad strokes right.

The very short version of my conclusion is that, assuming that your system, the GPG key for Debian, and the SHA256 hash algorithm have not been compromised, then you can assume that apt will download and install only files that the Debian Package people have approved for distribution. The flaws in this scenario are not worth worrying about for most people.

Given that the above system is already ensuring that you're most likely installing valid packages, there is no need to use the secure apt mechanism that I was initially asking about, and that's described in section 7.5.2 of the Securing Debian Manual, and in the web-page I initially referenced. Essentially, this is a mostly redundant mechanism, and most people don't need to enable it.

The slightly longer version, if anyone wants details, is that:

a) apt only does package verification during downloads. And it simply tells you that what has been downloaded doesn't match what was expected. If you install it manually after that (eg like I did with my corrupted pslist package) it will not complain.

b) I believe that any md5 signatures in the package files (eg the md5 signatures I found in control.tar.xz in the pslist package) have nothing to do with this process. So the incomplete coverage does not matter. Instead an SHA256 hash for the entire package is being used. And that hash is found in a Packages file that apt downloads from whatever mirror you are using.

c) Here's the verification process that I have inferred from the manual:

• The local keys /etc/apt/trusted.gpg.d/*.gpg are used to verify that the InRelease file that is downloaded from https://[mirrorsite]/debian/dists/buster/ is trustworthy.
• That InRelease file contains sha256 signatures that can be used to verify the contents of various files, including Packages.gz that would be downloaded from https://[mirrorsite]/debian/dists/buster/main/binary-amd64/).
• In that Packages file, there are sha256 signatures for individual packages that apt would install, and those could be used to verify all the files that were downloaded by an apt install command before installation.