Than you for the thoughtful reply.
Yes the apt upgrade submit ends there with no further error messages. When moving that source list it proceeds and asks me if i want to install the upgrades.
Please help me to understand the MITM scenario. So you're saying an
attacker could be able to redirect all sources except one? Is that even technically likely? Has that ever happened?
I appreciate you explaining the risk, as I'm trying to understand why.
Please understand that this also prevents unattended upgrades from
proceeding, which aside from a real indication of MITM attack seems like unnecessarily putting many systems at a security risk. (Also, shouldn't
https be used by default in the main sources list that includes security
repos? Doesn't https prevent MITM?)
Re space checking: I understand estimates are inaccurate. But it seems
better to fail on the side of caution in this instance. To be clear, I'm
not expecting to make waves for so many other users, this could be rolled
out with a flag option like (--check-space). I do think opt out should be
the default though.
I know there are separate partitions, but doesn't the estimation look at
say /boot when a kernel is queued for install? If it knows where to install packages, it can check those exact locations for free space. (But still
imho better to fail by default)
I would take the conservative side of the estimate, as I've said the
uninstall process can be very complicated, especially when inside a VM with
no other space available. Usually means manually deleting files that upsets dpkg and then having to reconfigure all that first before making apt happy again.
I think 1.5GB is reasonable. Doesn't the filesystem not have to be
uncompressed to install? Wouldn't it already be uncompressed during apt
use?
I understand about rolling out in a way to see what possible problems arise before implementing defaults.
Thank you
On Mon, Oct 10, 2022, 6:56 AM David Kalnischkies <
[email protected]>
wrote:
On Sun, Oct 09, 2022 at 01:21:45PM -0700, xaq xaq wrote:
To simplify a frequently seen halt of `apt upgrade` I have this
example.. I
had to remove keepsolidinc.list
from /etc/apt/sources.list.d. Then apt update proceeds. Here is the
error I
get with that source list in place:
*Hit:1 http://archive.ubuntu.com/ubuntu
<http://archive.ubuntu.com/ubuntu> - InRelease
Hit:2 http://security.ubuntu.com/ubuntu
<http://security.ubuntu.com/ubuntu> --security InRelease
Hit:3 http://archive.ubuntu.com/ubuntu
<http://archive.ubuntu.com/ubuntu> --updates InRelease
Hit:4 http://archive.ubuntu.com/ubuntu
<http://archive.ubuntu.com/ubuntu> --backports InRelease
Ign:5 http://apt.keepsolid.com/ubuntu
<http://apt.keepsolid.com/ubuntu> - InRelease
Err:6 http://apt.keepsolid.com/ubuntu
<http://apt.keepsolid.com/ubuntu> - Release
404 Not Found [IP: 144.217.71.199 80]
Reading package lists... Done
E: The repository 'http://apt.keepsolid.com/ubuntu <http://apt.keepsolid.com/ubuntu> - Release' does not have a Release
file.
N: Updating from such a repository can't be done securely, and is
therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user
configuration details.*
What do you mean with "halt of `apt upgrade`"? You have shown us output
of 'apt update'? And that output suggests that the other archives were updated (just that they were already current: cache "Hit").
I suppose if the old data still on disk for keepsolid is indicating
newer versions of packages than you have installed an upgrade would try
to download them potentially running also into 404 errors (assuming this
repo is entirely offline). That would indeed "block" upgrades of other packages not from that repo, but what is apt supposed to do instead…
It is asked to upgrade all packages and can't: Partial upgrades are technically supported, but hardly tested in practice as it is
a combinatorial explosion if you tried. So in theory you upgrade from
a perfectly supported system with updates available to a system with
unknown support status and still more updates available. That doesn't
look like a good default move.
You can tell apt to carry on with --fix-missing if that is what you
desire, but really, you should either remove that repository (and the packages coming from it) for good or work with its provider on fixing it/making it available again. After all, --fix-missing lets you carry
on in a pinch, but it remains that the repository fails to serve updates
to you, which in the worst case means public known security issues remain exploitable on your system.
(This could be part of an attack. It usually isn't, but it is
indistinguishable from a MITM blocking access to a repository so
that you can not get the security updates the attacker wants to
exploit. There are better ways, hence its unlikely, but it is just
a less good way as it shows errors grabbing attention that something
is wrong. If apt wouldn't, this would be really easy to use.)
For the size estimation, I understand it is an estimation, and there
could
be guesswork, but I would posit that if the estimate is even roughly correct, a user most likely will not want to proceed, because the consequences greatly outweigh the benefits of proceeding.
As I tried to explain already apt downloads itself at the expense of 1.5
MB. It will take up installed 4.5 MB. It hardly changes size from
version to version, so more space freed/used after the upgrade amounts
to more or less 0 MB. Running and using apt requires additional data
which easily takes up 200 MB. Lets assume every package is similar to
apt (they aren't, you e.g. have things like texlive which comes in a
couple hundred MBs of download already). A typical upgrade can contain
more than 1000 packages, so which value between 0 and 200 GB are we
gonna use as a rough estimate for that upgrade? And again, which
partitions are we gonna talk about if you have multiple, like e.g.
a separate /boot ?
APT in this scenario picks 1.5 GB (the size it will need to download all
the packages) on the partition housing its download location:
apt's free space check for the download is done against free blocks
available for everyone. In a default configuration root has an additional
5% of a partition as free space available for their exclusive use.
If your idea is that lets say apt should not continue if you don't have
100 MB free on /boot and 5 GB on / (should be enough, right?) that is
fine for your system and pretty easy to implement with e.g. a dpkg::pre-invoke hook (no need for the heavier dpkg::pre-install-pkgs
hook). That wont work in the general case though… there are systems
which don't even have 5 GBs of disk space to begin with.
(Did you think apts 1.5 GB is a reasonable low bound? For some people
that is already way too high! Like if your filesystem supports
compression)
That script you could package up and make available for everyone who
needs it. Probably with config knobs to specific sizes. Probably with
more iterations as people come up with pretty interesting partition
schemes if you give them the option to… perhaps indeed moving to a pre-install-pkgs hook later on to make better guesses at what a given
upgrade might need (install of two packages, probably fine… upgrading
2000 package ~ now we are talking!) …
If a few months down the line we discover that there is indeed a setup
which works for – lets say – 95% of Debian users we can always set it
to prio:standard and install it by default or integrate it into apt
so (basically) everyone gets it everywhere, but until then it is more beneficial for you and us if we are not burdening ourselves with
making it work for everyone from day 1.
Best regards
David Kalnischkies
<div dir="auto"><div>Than you for the thoughtful reply.</div><div dir="auto"><br></div><div dir="auto">Yes the apt upgrade submit ends there with no further error messages. When moving that source list it proceeds and asks me if i want to install the
upgrades.</div><div dir="auto"><br></div><div dir="auto">Please help me to understand the MITM scenario. So you're saying an attacker could be able to redirect all sources except one? Is that even technically likely? Has that ever happened? </div><
div dir="auto"><br></div><div dir="auto">I appreciate you explaining the risk, as I'm trying to understand why. Please understand that this also prevents unattended upgrades from proceeding, which aside from a real indication of MITM attack seems
like unnecessarily putting many systems at a security risk. (Also, shouldn't https be used by default in the main sources list that includes security repos? Doesn't https prevent MITM?)</div><div dir="auto"><br></div><div dir="auto"><br></div><
div dir="auto">Re space checking: I understand estimates are inaccurate. But it seems better to fail on the side of caution in this instance. To be clear, I'm not expecting to make waves for so many other users, this could be rolled out with a flag
option like (--check-space). I do think opt out should be the default though.</div><div dir="auto"><br></div><div dir="auto">I know there are separate partitions, but doesn't the estimation look at say /boot when a kernel is queued for install? If it
knows where to install packages, it can check those exact locations for free space. (But still imho better to fail by default)</div><div dir="auto"><br></div><div dir="auto">I would take the conservative side of the estimate, as I've said the
uninstall process can be very complicated, especially when inside a VM with no other space available. Usually means manually deleting files that upsets dpkg and then having to reconfigure all that first before making apt happy again.</div><div dir="auto">
<br></div><div dir="auto">I think 1.5GB is reasonable. Doesn't the filesystem not have to be uncompressed to install? Wouldn't it already be uncompressed during apt use? </div><div dir="auto"><br></div><div dir="auto">I understand about rolling
out in a way to see what possible problems arise before implementing defaults.</div><div dir="auto"><br></div><div dir="auto"><br>Thank you<br><br><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Mon, Oct 10, 2022, 6:56 AM
David Kalnischkies <<a href="mailto:
[email protected]">
[email protected]</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Sun, Oct 09, 2022 at 01:21:45PM -0700,
xaq xaq wrote:<br>
> To simplify a frequently seen halt of `apt upgrade` I have this example.. I<br>
> had to remove keepsolidinc.list<br>
> from /etc/apt/sources.list.d. Then apt update proceeds. Here is the error I<br>
> get with that source list in place:<br>
> <br>
> *Hit:1 <a href="
http://archive.ubuntu.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://archive.ubuntu.com/ubuntu</a><br>
> <<a href="
http://archive.ubuntu.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://archive.ubuntu.com/ubuntu</a>> - InRelease<br>
> Hit:2 <a href="
http://security.ubuntu.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://security.ubuntu.com/ubuntu</a><br>
> <<a href="
http://security.ubuntu.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://security.ubuntu.com/ubuntu</a>> --security InRelease<br>
> Hit:3 <a href="
http://archive.ubuntu.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://archive.ubuntu.com/ubuntu</a><br>
> <<a href="
http://archive.ubuntu.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://archive.ubuntu.com/ubuntu</a>> --updates InRelease<br>
> Hit:4 <a href="
http://archive.ubuntu.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://archive.ubuntu.com/ubuntu</a><br>
> <<a href="
http://archive.ubuntu.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://archive.ubuntu.com/ubuntu</a>> --backports InRelease<br>
> Ign:5 <a href="
http://apt.keepsolid.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://apt.keepsolid.com/ubuntu</a><br>
> <<a href="
http://apt.keepsolid.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://apt.keepsolid.com/ubuntu</a>> - InRelease<br>
> Err:6 <a href="
http://apt.keepsolid.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://apt.keepsolid.com/ubuntu</a><br>
> <<a href="
http://apt.keepsolid.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://apt.keepsolid.com/ubuntu</a>> - Release<br>
> 404 Not Found [IP: 144.217.71.199 80]<br>
> Reading package lists... Done<br>
> E: The repository '<a href="
http://apt.keepsolid.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://apt.keepsolid.com/ubuntu</a><br>
> <<a href="
http://apt.keepsolid.com/ubuntu" rel="noreferrer noreferrer" target="_blank">
http://apt.keepsolid.com/ubuntu</a>> - Release' does not have a Release<br>
> file.<br>
> N: Updating from such a repository can't be done securely, and is<br> > therefore disabled by default.<br>
> N: See apt-secure(8) manpage for repository creation and user<br>
> configuration details.*<br>
What do you mean with "halt of `apt upgrade`"? You have shown us output<br>
of 'apt update'? And that output suggests that the other archives were<br>
updated (just that they were already current: cache "Hit").<br>
I suppose if the old data still on disk for keepsolid is indicating<br>
newer versions of packages than you have installed an upgrade would try<br>
to download them potentially running also into 404 errors (assuming this<br> repo is entirely offline). That would indeed "block" upgrades of other<br>
packages not from that repo, but what is apt supposed to do instead…<br>
It is asked to upgrade all packages and can't: Partial upgrades are<br> technically supported, but hardly tested in practice as it is<br>
a combinatorial explosion if you tried. So in theory you upgrade from<br>
a perfectly supported system with updates available to a system with<br> unknown support status and still more updates available. That doesn't<br> look like a good default move.<br>
You can tell apt to carry on with --fix-missing if that is what you<br>
desire, but really, you should either remove that repository (and the<br> packages coming from it) for good or work with its provider on fixing<br> it/making it available again. After all, --fix-missing lets you carry<br>
on in a pinch, but it remains that the repository fails to serve updates<br>
to you, which in the worst case means public known security issues remain<br> exploitable on your system.<br>
(This could be part of an attack. It usually isn't, but it is<br> indistinguishable from a MITM blocking access to a repository so<br>
that you can not get the security updates the attacker wants to<br> exploit. There are better ways, hence its unlikely, but it is just<br>
a less good way as it shows errors grabbing attention that something<br>
is wrong. If apt wouldn't, this would be really easy to use.)<br>
> For the size estimation, I understand it is an estimation, and there could<br>
> be guesswork, but I would posit that if the estimate is even roughly<br> > correct, a user most likely will not want to proceed, because the<br>
> consequences greatly outweigh the benefits of proceeding.<br>
As I tried to explain already apt downloads itself at the expense of 1.5<br> MB. It will take up installed 4.5 MB. It hardly changes size from<br>
version to version, so more space freed/used after the upgrade amounts<br>
to more or less 0 MB. Running and using apt requires additional data<br>
which easily takes up 200 MB. Lets assume every package is similar to<br>
apt (they aren't, you e.g. have things like texlive which comes in a<br> couple hundred MBs of download already). A typical upgrade can contain<br>
more than 1000 packages, so which value between 0 and 200 GB are we<br>
gonna use as a rough estimate for that upgrade? And again, which<br>
partitions are we gonna talk about if you have multiple, like e.g.<br>
a separate /boot ?<br>
APT in this scenario picks 1.5 GB (the size it will need to download all<br> the packages) on the partition housing its download location:<br>
apt's free space check for the download is done against free blocks<br> available for everyone. In a default configuration root has an additional<br> 5% of a partition as free space available for their exclusive use.<br>
If your idea is that lets say apt should not continue if you don't have<br> 100 MB free on /boot and 5 GB on / (should be enough, right?) that is<br>
fine for your system and pretty easy to implement with e.g. a<br> dpkg::pre-invoke hook (no need for the heavier dpkg::pre-install-pkgs<br> hook). That wont work in the general case though… there are systems<br>
which don't even have 5 GBs of disk space to begin with.<br>
(Did you think apts 1.5 GB is a reasonable low bound? For some people<br> that is already way too high! Like if your filesystem supports<br> compression)<br>
That script you could package up and make available for everyone who<br>
needs it. Probably with config knobs to specific sizes. Probably with<br>
more iterations as people come up with pretty interesting partition<br>
schemes if you give them the option to… perhaps indeed moving to a<br> pre-install-pkgs hook later on to make better guesses at what a given<br> upgrade might need (install of two packages, probably fine… upgrading<br> 2000 package ~ now we are talking!) …<br>
If a few months down the line we discover that there is indeed a setup<br> which works for – lets say – 95% of Debian users we can always set it<br> to prio:standard and install it by default or integrate it into apt<br>
so (basically) everyone gets it everywhere, but until then it is more<br> beneficial for you and us if we are not burdening ourselves with<br>
making it work for everyone from day 1.<br>
Best regards<br>
David Kalnischkies<br>
</blockquote></div></div></div>
--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)