Hello,

these days, I found a package in Debian (four-digit popcon count) that
in an upgrade happily removed some some changes I had made to a
configuration file of that package, in /etc/.

My immediate reaction was to consider this a gross violation of the
Debian Policy (10.7.3 "Behaviour"). Upon closer inspection however I
found there are markers in that file that define a "managed section", an
area where the upgrade process wants to do things - local modification
ought to be placed outside of that, they will not be harmed, then. FWIW,
this functionality was implemented upstream.

So I'm a bit undecided here how to proceed:

Either we understand the policy literally - then the maintainer will see
an RC bug that will require some work to fix.

Or we adopt a pragmatic approach since an administrator still can modify
that file without losing these changes, although not in every place. It
ought to be possible to revert any of the lines in the managed section
if they are undesired. The administrator will however have to respect
the markers, though. To be honest, I failed to see them, which alone
might be reason to prefer the first, strict approach.

Thoughts?


-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEWXMI+726A12MfJXdxCxY61kUkv0FAmPr19oACgkQxCxY61kU kv27og//XapPbpEJrmgU3ahf6qv3RtNHpNhl//g39X0JAhuwnugoDRGXfoPI5fPg 2bANmoO834I7Uc7lqgKBazFIwKTnE7CLA/PT/MxdTTq39RK77FLDW8/t9Lz6Pimt T3jGL+qNA+DXo7j8vqXUpKu5RUhsLpuTlyB012FeogXfABsvhyxuyHOh5BM9sxFe Jw4zaXmjMPe805aOC5oyI595pLDCFaeLr/TtDgFetxuyAKNIEONrlYpG1qctRNWd lePOQhIfE/TrGdrfAs7Bz6+n1P+PaqKmyszE3tbwJDbk7ADYndK0NDHI9RvtPEal +6/5jNw2pBvAGWSUyyUo+7iCfl9XvyyyqJnoWZNUr2xJbnVocXJAHQQskGpbgf+k xkr8jOf0r4BuQoocjqNZq7DOgXplTLxGA4DnE9ECzxo1lO0ziEwkZTNYH9jVr8Ym zbYDDPlpnnWrOcsgFkOYOilwlViygoFBMv64hGpwVjsNYCLIvlXSgJ5nUHwuZAGL Ti0zbAbSfEm4MlJJkQ+WfcP5vSPxHD0Sn0VlzZszVAwYbkQHLViVe2jx7RkTl/N7 AgZMYdagqIn7PniwnBSQdqtbS9fh3oVrIwC1p5BH1opxxzqP1bapCttBqhYSIBBb /ewsLuLYt2qn5GRA1/AHCDBWjjvalRo22omMosLz4++qbITxIz4=
=jK3J
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Christoph Biedl <[email protected]> writes:

these days, I found a package in Debian (four-digit popcon count) that
in an upgrade happily removed some some changes I had made to a
configuration file of that package, in /etc/.

My immediate reaction was to consider this a gross violation of the
Debian Policy (10.7.3 "Behaviour"). Upon closer inspection however I
found there are markers in that file that define a "managed section", an
area where the upgrade process wants to do things - local modification
ought to be placed outside of that, they will not be harmed, then. FWIW,
this functionality was implemented upstream.

Policy is fairly explicit that you can't do that, but I suspect we do this
in other places as well because some very natural patterns for debconf prompting (load default answers from a file, ask some questions, write out
a file containing the answers) tend to cause the same thing if someone
puts other unexpected settings in the same file.

I think the right answer (which as is often the case involves a lot more
work) is to break the configuration file into separate parts, one of which
is a true configuration file in the Policy definition and the other of
which is the settings that are needed by the upstream software but that
aren't a configuration file in the Debian sense (and thus aren't user-modifiable), put the latter in /usr, and convince the program to load
them both in some way.

I'm guessing that we're working around upstream limitations here, but if upstream's configuration file syntax supports some sort of include
directive, that's ideal, and if it doesn't, maybe there's some way to add
that?

If the right fix isn't available, I would be tempted to convert the whole configuration handling to ucf or something similar that has built-in functionality to try to handle cases like this.

--
Russ Allbery ([email protected]) <https://www.eyrie.org/~eagle/>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, 14 Feb 2023 11:23:55 -0800, Russ Allbery <[email protected]>
wrote:

I think the right answer (which as is often the case involves a lot more >work) is to break the configuration file into separate parts, one of which
is a true configuration file in the Policy definition and the other of
which is the settings that are needed by the upstream software but that >aren't a configuration file in the Debian sense (and thus aren't >user-modifiable), put the latter in /usr, and convince the program to load >them both in some way.

I must say that I really like the idea of having "managed sections",
allowing half-conffiles which is much easier to handle for non-complex
cases where configuration is just a few lines and which would appear
overly complex if we'd split that into conffile and non-conffile.

The "split it" approach is something that comes naturally to someone
who has been heavily socialized in the Debian Universe because we
handle conffiles on a file level. It feels unnatural and clumsy for
someone who is not familiar with the deep historic reasons for us to
do it like that.

The issue is, however, a lot more complicated than one would might
think, imagine a structured configuration file like a systemd unit or
an icinga or bind or ISC DHCP config file which would need multiple
"managed sections", and the special case of a setting moving from
managed to non-managed in the package or at the local admin's
discretion.

<---rant--->
On the other hand, putting non-changing configuration in /usr reminds
me of the systemd way to handle things, with a complicated combination
of "if an identically named file is in /etc, it overrides the
package-delivered configuration entirely without the local admin
noticing that there was an upstream change" and "put configuration
snippets in a magic place in /etc and it will augment the
package-delivered configuration without the local admin notiting that
the upstram changed in a way that is incompatible with the local augumentation". This way to handle things was of course invented in
the Red Hat world because rpm's conffile handling is so vastly
inferior to what we have available for 25 years now, and sadly we have
taken this over 1:1 into Debian, not making use of your vastly
superior methods here in favor of being compatible with Red Hat's
worse solution.
</---rant--->

If the right fix isn't available, I would be tempted to convert the whole >configuration handling to ucf or something similar that has built-in >functionality to try to handle cases like this.

ucf is actually what dpkg should have been, I'd recommend it
wholeheartedly, but we need more automation for that.

Debian's way to handle conffiles is the best the Linux world has. That
doesn't mean that we don't have room to improve. And sadly, we haven't
moved a single inch here in 15 years.

Greetings
Marc
--
-------------------------------------- !! No courtesy copies, please !! ----- Marc Haber | " Questions are the | Mailadresse im Header Mannheim, Germany | Beginning of Wisdom " |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, 14 Feb 2023 at 19:24, Russ Allbery <[email protected]> wrote:

Christoph Biedl <[email protected]> writes:

these days, I found a package in Debian (four-digit popcon count) that
in an upgrade happily removed some some changes I had made to a configuration file of that package, in /etc/.

My immediate reaction was to consider this a gross violation of the
Debian Policy (10.7.3 "Behaviour"). Upon closer inspection however I
found there are markers in that file that define a "managed section", an area where the upgrade process wants to do things - local modification ought to be placed outside of that, they will not be harmed, then. FWIW, this functionality was implemented upstream.

Policy is fairly explicit that you can't do that, but I suspect we do this
in other places as well because some very natural patterns for debconf prompting (load default answers from a file, ask some questions, write out
a file containing the answers) tend to cause the same thing if someone
puts other unexpected settings in the same file.

I think the right answer (which as is often the case involves a lot more work) is to break the configuration file into separate parts, one of which
is a true configuration file in the Policy definition and the other of
which is the settings that are needed by the upstream software but that aren't a configuration file in the Debian sense (and thus aren't user-modifiable), put the latter in /usr, and convince the program to load them both in some way.

I'm guessing that we're working around upstream limitations here, but if upstream's configuration file syntax supports some sort of include
directive, that's ideal, and if it doesn't, maybe there's some way to add that?

If the right fix isn't available, I would be tempted to convert the whole configuration handling to ucf or something similar that has built-in functionality to try to handle cases like this.

If any code changes to handle configuration are done, the absolute
best thing is to just switch upstream to use libeconf:

https://github.com/openSUSE/libeconf
https://tracker.debian.org/pkg/libeconf

This automatically does the right, modern and expected thing and works
in the same way across all distributions, removing the need to carry
any downstream technical debt.

Kind regards,
Luca Boccassi

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Marc Haber <[email protected]> writes:

The issue is, however, a lot more complicated than one would might
think, imagine a structured configuration file like a systemd unit or an icinga or bind or ISC DHCP config file which would need multiple
"managed sections", and the special case of a setting moving from
managed to non-managed in the package or at the local admin's
discretion.

I had that sort of configuration file in mind when I was talking about
include, but I guess it depends on what sort of settings one expects to be managed. I can think of a few possibilities, all of which include handles
okay (complicated configuration that people can selectively override but
where we don't want to treat the whole thing as a configuration file, a
small Debian-generated group of settings that should be included in a
larger conffile, or a bunch of completely static configuration that no one should ever change although that last case is weird).

Yes, nested structure is a bit complicated but in the examples you list
surely there isn't very much configuration that should be inaccessible to
the local admin? Since that's what's implied by having it *not* be in a managed section.

<---rant--->
On the other hand, putting non-changing configuration in /usr reminds
me of the systemd way to handle things, with a complicated combination
of "if an identically named file is in /etc, it overrides the package-delivered configuration entirely without the local admin
noticing that there was an upstream change" and "put configuration
snippets in a magic place in /etc and it will augment the
package-delivered configuration without the local admin notiting that
the upstram changed in a way that is incompatible with the local augumentation". This way to handle things was of course invented in
the Red Hat world because rpm's conffile handling is so vastly
inferior to what we have available for 25 years now, and sadly we have
taken this over 1:1 into Debian, not making use of your vastly
superior methods here in favor of being compatible with Red Hat's
worse solution.
</---rant--->

Well, I adore this way of configuring things and think it's way better
than how Debian has been doing it and I haven't used Red Hat since the
late 1990s, so *shrug*. :)

But the point wasn't to advocate for that approach specifically, only that
if you *do* have configuration that the user is not allowed to change
because the package is going to override it, it needs to not be in /etc, because if it's in /etc it's going to be really confusing. I don't care
where you put it, but /usr is the logical spot?

I think your argument is that maybe you shouldn't have a bunch of
configuration the user can't change, and I agree!

--
Russ Allbery ([email protected]) <https://www.eyrie.org/~eagle/>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Wed, Feb 15, 2023 at 07:50:10AM -0800, Russ Allbery wrote:

Well, I adore this way of configuring things and think it's way better
than how Debian has been doing it and I haven't used Red Hat since the
late 1990s, so *shrug*. :)

But the point wasn't to advocate for that approach specifically, only that
if you *do* have configuration that the user is not allowed to change
because the package is going to override it, it needs to not be in /etc, because if it's in /etc it's going to be really confusing. I don't care where you put it, but /usr is the logical spot?

I think your argument is that maybe you shouldn't have a bunch of configuration the user can't change, and I agree!

I think, making a parallel, that there has been since forever a bunch of configuration the user couldn't change, namely the default configuration
values embeded in code, and since before the existance of RedHat itself
it was in /usr

The way I see it, now it is sometimes being made explicit by storing it
in human-readable config files instead of code, which is great because I
can go and see what the defaults are and override in /etc only what I
need to override. A well documented default configuration in /usr is a
better interface for me than default values hidden in documentation that
risk going out of sync with code.

Along this parallel, one useful point I'd make an effort to extract from
Marc's rant is that changes of configuration in /usr need to be treated
like changes in default values for configuration in code: that is, as
interface breaking changes, which users need to be aware of, and need to
be convered in upgrading documentation as would any other behavioural
change that may break existing configured systems.


Enrico

--
GPG key: 4096R/634F4BD1E7AD5568 2009-05-08 Enrico Zini <[email protected]>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Marc Haber wrote...

The "split it" approach is something that comes naturally to someone
who has been heavily socialized in the Debian Universe because we
handle conffiles on a file level. It feels unnatural and clumsy for
someone who is not familiar with the deep historic reasons for us to
do it like that.

Yeah, "socialized in the Debian Universe" applies to me - no doubt a
split (or "conf.d") is the right thing to do in a package-based
distribution, for packages that provide a service for other package.
Perhaps logrotate is a good example for that.

As you mentioned, there are limits - when the configuration file format
is complex and and cannot be easily combined from various sources. And a
few other situations as well.


Anyway, my initial question was of a different kind: Is the situation
I've found worth a grave bug report? I just don't want to put some
pressure on the maintainer if there is some rough consensus such
"managed section" markers are good enough.

Fixing this then is none of by business. Still, having some ucf instead
of just writing the modified file should be enough.

Christoph

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEWXMI+726A12MfJXdxCxY61kUkv0FAmPtOy4ACgkQxCxY61kU kv0zgw/9E1G9knhD/CGloHGpM4Ova5viToFY/YbgX9/sUX1qVlkfCKpsroGEK8F1 B/NHiC+BMMA+YGWsT3fXODTcN9DrfVepMiUAk0eke38LMxrMtj3zFDHms1IMPYd9 2gR4ExkFvaeCrLUXSXNOX9p2JQsvfEP7iQcVHLlcLdXWWpevMWd5Q1/aZH7lACVu iBLfifZkAemukM5eDatMh+LWbu68A4mfQe/1aYUYSLQQGwucFRYVvOLi4J9yFZ5N zSdImLuO7Kc5JHFh5Ph8OOocm5K7wGLrnqp73e8gub07nGWrD6uBU8HwY6abPsoo a6StzVp2ZhPBj/Ssv8izO3ugQTam+2l4EVuMkkrEAFQ45+3sb5RIEm4dyDsrrSzY lA08+NMHhTrbFkTs1vbE9MhGHbJ1oEm8iHnhGkqCN9T2BQ66vqLY6BpvR5yLMzgr urykL2zuMq/EiMDRIs4nMaRZQYoUXqKWaXz0RsdZJiuIaF7ynMg2Kns8IMQaGExS 8FYfjjC2jieKzZWF5WmaKm4sIc0LsuzjKwBDpS+lLb3yX1wVPRG5JIn6Fnp6mX+C JcO0Uvz+ge0+xgCfDgMsMfjUkZk5UKrgmeMMEBh38Kwql83EY0RjLbNNUI7SDrNM sVdT3GZ3F0fn7ibu6C8wAj+GMplYftQYcAnKe3MUm0xyV2/dgNc=
=jhRm
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Marc Haber wrote:

On Tue, 14 Feb 2023 11:23:55 -0800, Russ Allbery <[email protected]> wrote:

I think the right answer (which as is often the case involves a lot more >work) is to break the configuration file into separate parts, one of which >is a true configuration file in the Policy definition and the other of >which is the settings that are needed by the upstream software but that >aren't a configuration file in the Debian sense (and thus aren't >user-modifiable), put the latter in /usr, and convince the program to load >them both in some way.

I must say that I really like the idea of having "managed sections",
allowing half-conffiles which is much easier to handle for non-complex
cases where configuration is just a few lines and which would appear
overly complex if we'd split that into conffile and non-conffile.

Half-conffiles like that prove *really* difficult to work with in other
ways, however.

A half-conffile causes many problems when trying to track /etc in git,
for instance.

And a half-conffile (or a single full-conffile with no .d directory) is
more difficult to work with when the admin wants to make programmatic
changes of their own. For instance, if you want to package up your configuration in a .deb package to install on multiple systems, you can
do that more easily if the software you're configuring handles .d files
or similar, rather than *only* supporting one big configuration file
that you have to edit.

For that and various other reasons, I'm very happy with the trend
towards "you can override package configuration in /usr with local configuration in /etc by creating a file of the same name".

But that said, I absolutely think Debian can improve further on that
design and tooling. For instance, I'd love to see ucf-style support for noticing "You're currently overriding /etc/foo.d/thing.conf, and
a new version of the package just shipped a new /usr/share/foo.d/conf,
would you like to review the changes and figure out if you want to
incorporate any?".

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)