We have two separate issues here:

a/ /tmp-on-tmpfs
b/ time based clean-up of /tmp and /var/tmp

I think it makes sense to discuss/handle those separately.

Agreed.

I also don't see any issue with a/, at worst people will be annoyed
with it for some reason and can then change it back.

Regarding b/: ...

The tmpfiles rule tmp.conf as shipped by systemd upstream ...
Files that are older then 10 days or 30 days are automatically cleaned up.

This seems like a rather dangerous thing to spring on people.

First of all, time can be pretty fluid on user machines.

Someone has a laptop, they do some work and generate big files in
/var/tmp/, maybe they're editing videos or processing medical images
or downloading and pre-processing large datasets for machine learning.
Now they close their laptop and go on vacation for six weeks. When
they come back they open it and ... shazam all the files they were
working on are deleted.

Another scenario. I download a big source tarball, firefox say. I want
to make a few small changes and compile a custom version. I keep it in /var/tmp/ for obvious reasons. (The patches are stored elsewhere of
course.) Now I download a new patch and apply it and recompile. And
get an error, because *some* of the files in the unpacked tree were
too old and got auto-deleted.

Imagine suspending a computer in the middle of a big build, and when I un-suspend it a few weeks later the build fails because some files are
now too old and get deleted. Is this really desirable?

To me, the purpose of /var/tmp/ when I have my "user" hat on is: a
place to put files I don't want backed up, particularly large ones,
and which if I run out of disk space is a place to look for stuff to
delete. it's not "a place to put files that the system might
capriciously delete just because."

I do understand the imperative to keep the filesystem clean and to do
routine maintenance and upkeep without user intervention. But changing something that used to be "stable storage unless your disk crashes"
into "files placed here can mysteriously disappear" seems like a major potentially-breaking change that can disrupt common work flows.

I can see popping up a warning: "disk space on the filesystem
containing /var/tmp is getting low and /var/tmp contains xxxGb of
files, hit OKAY to delete the xxxGb of oldest files there" or
something like that. But doing it silently is scary.

One compromise would be to enable the /var/tmp/ reaper by default only
on new installations. People can always turn it on if they want. Also,
when files are deleted, I think the user should get warnings all over
their desktop. I'd also suggest that the reaper by default never
delete files from /var/tmp if there is a lot of space available, or if
/var/tmp isn't consuming much space.

--Barak Pearlmutter

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Barak A. Pearlmutter, le lun. 06 mai 2024 11:15:35 +0100, a ecrit:

To me, the purpose of /var/tmp/ when I have my "user" hat on is: a
place to put files I don't want backed up, particularly large ones,
and which if I run out of disk space is a place to look for stuff to
delete. it's not "a place to put files that the system might
capriciously delete just because."

Definite +1 on this.

My /var/tmp has various "I want to get rid of this at some point"
temporary stuff, and I would be quite annoyed if it would disappear unexpectedly.

One compromise would be to enable the /var/tmp/ reaper by default only
on new installations.

At the very least, yes.

Samuel

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 6 May 2024 at 11:33, Barak A. Pearlmutter <[email protected]> wrote:

We have two separate issues here:

a/ /tmp-on-tmpfs
b/ time based clean-up of /tmp and /var/tmp

I think it makes sense to discuss/handle those separately.

Agreed.

I also don't see any issue with a/, at worst people will be annoyed
with it for some reason and can then change it back.

Regarding b/: ...

The tmpfiles rule tmp.conf as shipped by systemd upstream ...
Files that are older then 10 days or 30 days are automatically cleaned up.

This seems like a rather dangerous thing to spring on people.

First of all, time can be pretty fluid on user machines.

Then upon reading the release notes, on such a machine, one can simply do:

touch /etc/tmpfiles.d/tmp.conf

And they get no automated cleanups. This stuff is designed to be
trivially overridable, both by end-users and image builders. What I am
looking for is, what packages need bugs/MRs filed to deal with this
change, if any.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Then upon reading the release notes, on such a machine, one can simply do:

touch /etc/tmpfiles.d/tmp.conf

And they get no automated cleanups.

This also disables on-boot cleaning of /tmp/.

The root issue here is that deleting not-read-in-a-while but-maybe-stat'ed-recently-by-make-that-doesn't-count files from
/var/tmp/ by default, particularly when the system didn't used to,
violates the principle of least surprise. It will cause problems for
end users. And disabling it *properly* requires serious sysadmin
knowledge and is easy to get wrong.

There's an old debugging story: Maclisp on a DEC-10 was glitchy some
days and not others. Someone marked the crashes on a calendar, and
they correlated with the phase of the moon! Turns out the program
printed the lunar phase ("crescent", "waning gibbous", etc) in its
header, just for fun, and when the string was long it wrote past the
end of a buffer, etc.

The story is still told fifty years later because it's so funny, it's
a total violation of levels. People debugging in a windowless basement
should not need to know the phase of the moon!

Users want their systems to run reliably and reproducibly. Getting
different behaviour depending on how long it's been since some of the
files you're using have been *accessed* adds oodles of extra state
people have to keep track of in their mental model of what some
program needs to run. Sure, it can be administratively disabled on one particular system; but then you run the same program or perform the
same work-flow on another system, and it fails. Something fails
reliably after running for over 30 days ... unless you keep an eye on
it.

As someone who regularly deals with large datasets, and keeps them in
the "approved" don't-back-these-up location /var/tmp/, this just seems
crazy to me. When I tell a grad student to install Debian on their
laptop, do I really need to spend time explaining to them to be sure
to turn off the "delete your old files for no reason" option which is
enabled by default? When they're using someone else's cluster, should
I remind them to run a cron job "find /var/tmp/foo -exec touch ..." so
the 25 Tb of data we pushed onto scratch storage on that cluster
doesn't partly disappear? Our priority should be our users. Our users
are not well served by having their files deleted. They put them there
for a reason!

--Barak.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 6 May 2024 at 13:42, Barak A. Pearlmutter <[email protected]> wrote:

Then upon reading the release notes, on such a machine, one can simply do:

touch /etc/tmpfiles.d/tmp.conf

And they get no automated cleanups.

This also disables on-boot cleaning of /tmp/.

Yes, as it's going to be a tmpfs, so it is no longer needed. Trivial
to maintain though if one wants to do so, though.

The root issue here is that deleting not-read-in-a-while but-maybe-stat'ed-recently-by-make-that-doesn't-count files from
/var/tmp/ by default, particularly when the system didn't used to,
violates the principle of least surprise.

Which is what release notes are for, if everything was always the same
we wouldn't spend time to put those together.

There's an old debugging story

While personal anecdotes and stories can be interesting and amusing in
many circumstances, I am not really looking for those at this very
moment. What I am looking for right now is packages or internal
infrastructure that need
an update to cope with these two changes before I upload them, so if
you know of any please do let me know and I'll happily look into it
and at least file a bug, if not a MR. Thanks.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 06 May 2024 at 13:41:58 +0100, Barak A. Pearlmutter wrote:

As someone who regularly deals with large datasets, and keeps them in
the "approved" don't-back-these-up location /var/tmp

Independent of whether we make the change Luca is suggesting or not,
I don't think /var/tmp is a good recommendation for users storing
temporary data manually, because safe use of "obvious" filenames in a world-writeable location like /var/tmp requires you to be aware of the possibility of symlink attacks, and either ensure that your kernel has a mitigation for them (as Debian's does with default runtime configuration),
or be careful to avoid them, or be OK with any human user account and
any system uid on the system being able to achieve at least denial of
service and maybe privilege escalation to your account.

/tmp and /var/tmp are fine places for *programs* to store temporary data -
for example Flatpak uses a subdirectory in /var/tmp for half-downloaded
apps and runtimes - but only if they're using mkstemp(), mkdtemp()
or equivalent to generate an unpredictable location and allocate it
without race conditions, which is the only way these directories are
safe to use in portable software. This is not particularly user-friendly
to do manually.

For files you don't want backed up, I'd personally recommend storing
them somewhere in your home directory, perhaps ~/tmp, with a CACHEDIR.TAG (http://justsolve.archiveteam.org/wiki/Cache_Directory_Tagging_Standard),
and using backup software that understands that convention (for example
borg, restic, tar).

smcv

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 6 May 2024 at 15:31, Barak A. Pearlmutter <[email protected]> wrote:

What I am looking for right now is packages or internal
infrastructure that need
an update to cope with these two changes before I upload them, so if
you know of any please do let me know and I'll happily look into it
and at least file a bug, if not a MR. Thanks.

Okay.

git and other version control systems should be modified to warn if
the work tree or git dir are in /var/tmp and therefore exposed to
unexpected deletion of files, and possibly modified to refuse to clone
there without an explicit --force-unstable-storage option.

Does it do something similar when cloning into /tmp/ or any other
tmpfs right now?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

What I am looking for right now is packages or internal
infrastructure that need
an update to cope with these two changes before I upload them, so if
you know of any please do let me know and I'll happily look into it
and at least file a bug, if not a MR. Thanks.

Okay.

git and other version control systems should be modified to warn if
the work tree or git dir are in /var/tmp and therefore exposed to
unexpected deletion of files, and possibly modified to refuse to clone
there without an explicit --force-unstable-storage option.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

If it clones into /tmp the *entire* tree will either be reaped (upon
reboot) or not.

But having just some files deleted from a git dir or git working dir
is much more dangerous, because various git commands can treat files
deleted from the working tree as deliberate changes to be committed,
and files deleted from the git dir can render it unusable.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 6 May 2024 at 16:03, Barak A. Pearlmutter <[email protected]> wrote:

If it clones into /tmp the *entire* tree will either be reaped (upon
reboot) or not.

But having just some files deleted from a git dir or git working dir
is much more dangerous, because various git commands can treat files
deleted from the working tree as deliberate changes to be committed,
and files deleted from the git dir can render it unusable.

It's not really that different, it's a variation of the same problem.
I take it that git doesn't warn when cloning on tmpfs, then, so that's
not an issue.
tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
and have done so for ~15 years. Does git currently parse all installed tmpfiles.d snippets and warn if cloning on an arbitrary directory
covered by one of them?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

tmpfiles.d snippets can be defined to cleanup on a timer _anything_,

It's a question of what the *default* behaviour should be.

For whatever reason, a lot of people who process large data use
/var/tmp/FOO/ as a place to store information that should not be
backed up, but also should not just disappear.

Maybe they should use ~/FOO/ but put a CACHEDIR.TAG file with the
right magic contents there. Or touch ~/FOO/.nobackup and use duplicity --exclude-if-present .nobackup. Or ~/.cache/FOO/, although that's a
bit counterintuitive because it's "hidden" and not really meant for
this purpose.

There's a reason people don't want these files below ~/, which is that
some backup mechanisms, like btrfs snapshots or LVMs snapshots, may
snapshot files there regardless; and often /home/ is on more
stable/expensive storage (raid k for k≥1) while /var/tmp/ is on a
cheaper substrate.

Given the era of large data we're living in, this is a ubiquitous
problem. Since we don't give users a standard solution, they've rolled
their own. If we think that solution is wrong, we should (a) make an
actual official suggested location for this sort of material,
/var/scratch/ or /var/scratch/${USER}/ with a link there from
~/Scratch or something, and (b) transition them to it, ideally without unexpectedly deleting their files and messing up their work in the
process.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 6 May 2024 at 16:51, Barak A. Pearlmutter <[email protected]> wrote:

tmpfiles.d snippets can be defined to cleanup on a timer _anything_,

It's a question of what the *default* behaviour should be.

No, it is not, at least not for the strawman you conjured. So I gather
that git doesn't warn when cloning on other directories that might be
cleaned up - great, that's a non-issue too.
If you want it to do so, then the appropriate way is for you to file
an RFE upstream and ask them to parse the installed tmpfiles.d on a
system before cloning. This is not that hard to do, and the format is
stable, and if they ask about that I'll happily back you up on this
stability guarantee.
But what git does or does not allow is then completely unrelated to
anything happening here.

For whatever reason, a lot of people who process large data use /var/tmp/FOO/ as a place to store information that should not be backed up, but also should not just disappear.

Then such people, assuming they actually exist, can configure their
custom systems accordingly upon reading the release notes before
upgrading, as they would do anyway if installing on CentOS or any
other major OS. Hence, not an issue either.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, May 06, 2024 at 04:50:50PM +0100, Barak A. Pearlmutter wrote:

tmpfiles.d snippets can be defined to cleanup on a timer _anything_,

It's a question of what the *default* behaviour should be.

For whatever reason, a lot of people who process large data use
/var/tmp/FOO/ as a place to store information that should not be
backed up, but also should not just disappear.

To be honest I'm greatly surprised by this idea, and by the suggestion
that a lot of people do this, to me this is very similar by that half-joke about people storing useful files in the Recycle Bin.
But I also thought that "/tmp is on tmpfs, /var/tmp isn't, and that's
their main difference" are our defaults for years, when they apparently
aren't (now I understand why people sometimes suggest using /dev/shm or
/run for this or that to increase performance).

--
WBR, wRAR

-----BEGIN PGP SIGNATURE-----

iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmY5D/8tFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh oNcP/2f+UEv/NyWKFTE4BnVurIeiYzmOFC+GMelKApgTKYZZkydn5WKcXGo1F9dx HtnIBawCUN4Kn45vY/4Iz1C4DE2bAbX9uh7Mu/FCmT14ulUG3KKHfiskBtSQrP7s TeX6TFEk1+m7gVd6hrTD+c47JqYxv0cN79qGBXU+Ryhe47E0uuE/TRGIodK/JPap Dzu0vl4R8PvIyTHZi8oa3ASUdCexnLtAFGBqAwc+Vrbe6nPEbWADzOdJaR1qjTma j9159tzYxjrUFXG8hxrd+KkYG/5629zXlnGS0G6P+wbLusJBtL9hOnkNMO2Fu0IY 8qzIp4Dh16/0vB0EZA0oTMdvXkBw3WB0Q6Ml5E68f2U/0dZvpbU9PLp4bP8ZmVlb 7pW0mRKmD+yG/d0gLgMWoQU+OyUdwScD2XP5QkT2ezSl2pflmM+Bjy14OMWi+sni zbJECnsL6nzXhi+xFM71tKFrMhjDRERpfSSFKXHYkb1EtNPnTtU0i31X009U5CR5 6NfY07OXu7XgzLPaquuLJiuuAlSvrLcdsxXbSko5fEPKfy63fUUwtWPxh25VXpWr T13TDvLN1D4i4uy86fLHdZqG5nld7WxMklZu3PXGT/gaEbY+4kwxQcWGzd5FXgr5 LF4FtSQ9WIGmOLZ2B+qTOHcVttVSKDG4vzIul9RBy/j2bBTS
=UULi
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

Quoting Luca Boccassi (2024-05-06 15:20:08)

While personal anecdotes and stories can be interesting and amusing in many circumstances, I am not really looking for those at this very moment. What I am looking for right now is packages or internal infrastructure that need an update to cope with these two changes before I upload them, so if you know of any please do let me know and I'll happily look into it and at least file a bug, if not a MR. Thanks.

mmdebstrap uses $TMPDIR (because that's what Perl's File::Temp uses) and falls back to /tmp for storing the chroot directories it creates. The modification times on files in that chroot directory are potentially much older than 10 days as they match the timestamps on the packages installed in the chroot. Do I understand it correctly that with the proposed change and an unfortunate timing it can happen that files older than 10 days get removed from a temporary chroot directory in /tmp? If yes, how should applications guard against that from happening?

Thanks!

cheers, josch
--==============A23974101798848545=MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Description: signature
Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEElFhU6KL81LF4wVq58sulx4+9g+EFAmY5ONEACgkQ8sulx4+9 g+FSfhAAkDh6PpsO2xyowRzyvlwci+xqamT2zYOIWPcolhH2W7A3JVJPEIX9ObUG ki7Xs9/bKuo3pcWOApVGCkK5JEemDmZAruomksiUfOFZqH3uRUw0mVxgc8ASTOUi IRR7DpTBDD8DkcMQVEKaAkTzwcd7iszegaVyrtCqgDlZAqNf6+YX0QHUGYD2QCdo uNfVJoF/8DhB7yDOFLgl+EzaCw3lfk0S4tmk9Y+Y7rNDX9okrR0fgygS9qseYHnm NBWw6lzx/+yzFKyb8/ZtykAJWtL0E9cp1+bXKFOIagvfgoixrxcb7RE7zwwgSHbG 7nCePmIeq+zQ4oGa9qR7JdOVSbRuscCo7UhHAh9hDwiv4/MdQ5fx8AwwrZkXMQrH CtX5ViHy0lARdX0ibZN8QJsK/ePSpDesNpQh1hZe1G15giF8SktGjcZcRcvaRR/J x1/KHmH1B3yyeIUIrGeG4vqH4FRyHneRk0YZ/1lw9jMN+nReSf0Fs7yHtsbctb6h GbETdeJpRafsYvaOD1Q9xWw/iJrvEhZikMdIkzZPGQ+jPPvQtsJJpy6km5dgQEu3 kShM5O+a50x9NuwZfKqTVgFhz2rJawyoWcjLEG7JUviIrDatO5pSzUjUfne1Sfzl vwSplNS3Df2G5gVAZhmeBtrPWaex7ixN/86iicpPK4jkrjKb7yM=
=g35m
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 6 May 2024 at 22:27, Simon McVittie <[email protected]> wrote:

On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues wrote:

If [files can be deleted automatically while mmdebstrap is using them],
how should applications guard against that from
happening?

As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
flock(2) lock on its chroot's root directory, systemd-tmpfiles should
fail to take out its own lock on the directory during cleanup, and
respond to that by treating the directory as "in use" and skipping it.

That also works, but only as long as mmdebootstrap is actually
running, and as far as I understand it is not a long-running service,
not sure if it works for this use case

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues wrote:

If [files can be deleted automatically while mmdebstrap is using them],
how should applications guard against that from
happening?

As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
flock(2) lock on its chroot's root directory, systemd-tmpfiles should
fail to take out its own lock on the directory during cleanup, and
respond to that by treating the directory as "in use" and skipping it.

smcv

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 6 May 2024 at 21:08, Johannes Schauer Marin Rodrigues <[email protected]> wrote:

Hi,

Quoting Luca Boccassi (2024-05-06 15:20:08)

While personal anecdotes and stories can be interesting and amusing in many circumstances, I am not really looking for those at this very moment. What I
am looking for right now is packages or internal infrastructure that need an
update to cope with these two changes before I upload them, so if you know of
any please do let me know and I'll happily look into it and at least file a bug, if not a MR. Thanks.

mmdebstrap uses $TMPDIR (because that's what Perl's File::Temp uses) and falls
back to /tmp for storing the chroot directories it creates. The modification times on files in that chroot directory are potentially much older than 10 days
as they match the timestamps on the packages installed in the chroot. Do I understand it correctly that with the proposed change and an unfortunate timing
it can happen that files older than 10 days get removed from a temporary chroot
directory in /tmp? If yes, how should applications guard against that from happening?

More specific settings win, so your application can ship its own
tmpfiles.d snippet that defines a longer (or infinite) cleanup time
for directories under /tmp or /var/tmp.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 6 May 2024 at 23:00, Johannes Schauer Marin Rodrigues <[email protected]> wrote:

Quoting Luca Boccassi (2024-05-06 23:28:59)

On Mon, 6 May 2024 at 22:27, Simon McVittie <[email protected]> wrote:

On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues wrote:

If [files can be deleted automatically while mmdebstrap is using them], how should applications guard against that from
happening?

As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive flock(2) lock on its chroot's root directory, systemd-tmpfiles should fail to take out its own lock on the directory during cleanup, and respond to that by treating the directory as "in use" and skipping it.

That also works, but only as long as mmdebootstrap is actually
running, and as far as I understand it is not a long-running service,
not sure if it works for this use case

I guess those users who want to place their chroots in /tmp would have to disable the cleanup timer on their systems.

For the mmdebstrap user who just wants to create some tarballs, I think the flock solution would be sufficient, thank you.

More specific settings win, so your application can ship its own tmpfiles.d snippet that defines a longer (or infinite) cleanup time for directories under /tmp or /var/tmp.

How would that look like? Say, my application creates temporary directories with the pattern /tmp/mmdebstrap.XXXXXXXX -- how would the recommended line which disables cleanup in /usr/lib/tmpfiles.d/mmdebstrap.conf look like?

To be more specific, as per documentation:

https://www.freedesktop.org/software/systemd/man/latest/tmpfiles.d.html

'x' lines can be used to override cleanup rules, and support globbing,
so something like:

x /tmp/mmdebstrap.*

And just to confirm (I read this elsewhere in this thread): if my /etc/fstab has an entry for /tmp (with a tmpfs) does this automatically mean that no cleanup will happen or do i still have to put something into /etc to disable the periodic cleanup?

That's something different, fstab is about whether /tmp is a tmpfs or
not, cleanups still happen regardless of the filesystem type.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Quoting Luca Boccassi (2024-05-06 23:28:59)

On Mon, 6 May 2024 at 22:27, Simon McVittie <[email protected]> wrote:

On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues wrote:

If [files can be deleted automatically while mmdebstrap is using them], how should applications guard against that from
happening?

As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive flock(2) lock on its chroot's root directory, systemd-tmpfiles should
fail to take out its own lock on the directory during cleanup, and
respond to that by treating the directory as "in use" and skipping it.

That also works, but only as long as mmdebootstrap is actually
running, and as far as I understand it is not a long-running service,
not sure if it works for this use case

I guess those users who want to place their chroots in /tmp would have to disable the cleanup timer on their systems.

For the mmdebstrap user who just wants to create some tarballs, I think the flock solution would be sufficient, thank you.

More specific settings win, so your application can ship its own tmpfiles.d snippet that defines a longer (or infinite) cleanup time for directories under /tmp or /var/tmp.

How would that look like? Say, my application creates temporary directories with the pattern /tmp/mmdebstrap.XXXXXXXX -- how would the recommended line which disables cleanup in /usr/lib/tmpfiles.d/mmdebstrap.conf look like?

And just to confirm (I read this elsewhere in this thread): if my /etc/fstab has an entry for /tmp (with a tmpfs) does this automatically mean that no cleanup will happen or do i still have to put something into /etc to disable the periodic cleanup?

Thanks!

cheers, josch
--==============A52964082893687403=MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Description: signature
Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEElFhU6KL81LF4wVq58sulx4+9g+EFAmY5Uw0ACgkQ8sulx4+9 g+HpnQ/9F6yd3pyo38ADFRHHm8yP3oy7nJoNlFYpM2y//bAto/cbcwzrc3tDezJ6 /P1BdxHriYx+JAYeQ+Zrb+Kwzu4bwdu+GHteakx3h0TbI279QwRkWOqoKIesyQ+7 sekqH82DvjrXvh6ceUTmDcJbPlzUQIZNesk7QIjbVt2zScJ41S2s5cMO1rH5Pzgk Z4N2BqtWMnLkvF3ynw2/+xaBezJuV6rC5Yv6wLfJxNtKBn4jvUiubAQHzfeJAKwM icrAiYS4KaCgcL/Tc0TcZKkc0jyrbYViQqk8lJ+VP1/d3GNpiAVghc4mGKj7VMC6 DLtDH2YTKedR+HnGXq0s7u2CS6krpDT/U/IKaBAAJ3BBJ7HqOvtyPPU5lu+9Fhl4 4AVhE+mi69qzSaZ5hhfIRz8QbRzJhNs+8caFBBcdTxoDnJgaSMg3QYObIALTX1Sd AGt8p+QIvDHOkkIUozm0K0D0h+J86j+tOj6bE4JpUI02r0YlhXd3rI9jH92yGkM7 sD0Z0R5+9sJCkVDiFYFnrK2YrDSVK6RJYXdCsVK6gKP4hyqpQbt1aCYvKMZGYBJW 5BHGFS9djf56EF/GLJJT7isC/AI3xpa+bMYYTG08FLYImKVT68erx4LdWPlkt7bE 3hq4D9Q1wt5KKj2Wpea5zj9ZyJ37cqJcY7POJUInsMVatjWXR1o=
=AbFT
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

Quoting Luca Boccassi (2024-05-07 00:09:51)

To be more specific, as per documentation:

https://www.freedesktop.org/software/systemd/man/latest/tmpfiles.d.html

'x' lines can be used to override cleanup rules, and support globbing,
so something like:

x /tmp/mmdebstrap.*

thank you for being patient with me. I saw the man page but I also tried using codesearch to look for other packages doing the same thing already and was unable to find one. This made me doubt whether I had understood this correctly. For example, tmpfiles.d(5) makes it look like there are several minuses required after the path but apparently those are optional. I've never written such files before, so your input is useful for me, thank you!

And just to confirm (I read this elsewhere in this thread): if my /etc/fstab has an entry for /tmp (with a tmpfs) does this automatically mean that no cleanup will happen or do i still have to put something into /etc to disable the periodic cleanup?

That's something different, fstab is about whether /tmp is a tmpfs or not, cleanups still happen regardless of the filesystem type.

I shall put the following into my local /etc/tmpfiles.d/josch.conf to disable the cleanup of /tmp completely, then:

x /tmp

Thanks!

cheers, josch
--==============�85683799973894544=MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Description: signature
Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEElFhU6KL81LF4wVq58sulx4+9g+EFAmY5Xc0ACgkQ8sulx4+9 g+ElhhAAnJ+mwA/u/WBCuIvJjbl/pe2y9fUbC/1viyVSqclEbaTBrYsWyjACpUtu MUnkK+xPt2KnGNZ9wdCvVKs1O3aM6LiqMH5shVpQ4z9IFgqhlUpMAFBuR3Gbg2Gx V2re2oSFXP1Eb+GS9U+3FeWpbUUryJGWy3Hk2EyhJVbKTpinLXHgveuUJy9GzWwZ E5aLpW8jTeyHaqWRS7qXe7Qvd8OknHPGxSGbMH5eL33arxbcNXuR1qvWaD6BWIht dtFqI0Or7EAmN7UmNPptc1M+6TjxqdOETwRQ0wvPSG6ghCZhpFKjo2V8t7XGoxBE Qlp4tgjHwg1WE4vGK6tqRCQ0mAUE6evILjiwQXr34tRGOE1fNYvN/nN3rAB9rKf4 +1zzrNgl7hrm4G+m7DcgLBsj2S8vfCKgfCtqb0R/hfqgR940fCB19tGHApdnIFrM q/Aq/590ezZX028N+a9oIuOlv1AI9yzFU/BGLgSPGO2ZAUrFBZlJTUIHJ1ABoXR5 tVKg/99rh+xCF73kcJSkR+VysfqBOm3ogHhnmjpu3sbCV1wCxoLb2VQ8QUbBAejI v3PeJVbvwJu4RJ5ezeGiZWZt9m3FAZyKgC1ILjbNkeMBTgYpSQ16ikwSnFd9J2ZR rbw1cdKudaZtn2FVtqSN6at8HoB+cZTO9V2EmpohGWDF4YopwFc=
=xwaw
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Luca Boccassi <[email protected]> writes:

On Mon, 6 May 2024 at 11:33, Barak A. Pearlmutter <[email protected]> wrote:

We have two separate issues here:

a/ /tmp-on-tmpfs
b/ time based clean-up of /tmp and /var/tmp

I think it makes sense to discuss/handle those separately.

Agreed.

I also don't see any issue with a/, at worst people will be annoyed
with it for some reason and can then change it back.

Regarding b/: ...

The tmpfiles rule tmp.conf as shipped by systemd upstream ...
Files that are older then 10 days or 30 days are automatically cleaned up. >>

This seems like a rather dangerous thing to spring on people.

First of all, time can be pretty fluid on user machines.

Then upon reading the release notes, on such a machine, one can simply do:

touch /etc/tmpfiles.d/tmp.conf

And they get no automated cleanups. This stuff is designed to be
trivially overridable, both by end-users and image builders. What I am looking for is, what packages need bugs/MRs filed to deal with this
change, if any.

Isn't this change (as presented) effectively about masking bugs?

We've had people suggesting that implementing this will surprise them
and disrupt their existing use of /var/tmp as scratch storage, and I've
got a lot of sympathy with that, so I'm guessing that the people that
are expected to benefit from this are not those that remember systems
where the main distinction between /tmp and /var/tmp was that /tmp got
emptied at boot time, whereas /var/tmp did not.

That makes me assume that those that would be most likely benefit from
such a change will mainly be users that are never going to type "/tmp"
(with or without a preceding "/var"), and are therefore not going to
have any idea what is being deleting for them, but will be happy never
to get their disk filled.

This makes me wonder what it is that we're expecting to need to delete.

Is this a symptom of sloppy applications that fail to clear up the
debris they create in /var/tmp? If so, is that not a bug in that
application?

I'd suggest that rather than clearing up after the sloppy behaviour of
buggy applications, we instead leave it visible, in the hope that it can
then be fixed.

Of course, that's obviously not worked in some (many?) cases, so where
we know of problematic packages, could we not add per-package tmpfiles.d
files that name the specific paths that those packages are known to
litter the system with, with appropriate deletion timeouts chosen by the Maintainer?

That ought to achieve the benefit you're looking for, without hiding
symptoms of future problems with other packages, and without
inconveniencing anyone that's using /var/tmp as scratch space.

Cheers, Phil.
--
Philip Hands -- https://hands.com/~phil

--=-=-Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEE3/FBWs4yJ/zyBwfW0EujoAEl1cAFAmY6EQYACgkQ0EujoAEl 1cCEkxAAqivrBzpa/tpnWGrJG/65Gb6+tP/byUbkZe2S1Ts2mHmUOEebvNBSdEh/ Mrl3TxQ5QZWkPA4Fi6ZaOcGrDku71On8tipjlvqJsKNuNm+eR0uuxrp0N6Hbpwxr f3E4RWHniW5EESwdX+n9moOWiXphh7hDdLQeSvzvmZqHmUquN9G38bZD5lGgkH82 qpUJJ6CdUd+DErrwbACOcDNUEd7CQ8uGjNBlsRxAan3DTrs8ltWsYF7uTFU1EFZQ WeIecLtsnDqVOZvofqRG0p5IoZHxhnsOKtYSu/9oHi3IAcQSTSFN0XHnqr3+iWbZ BWZmbbNDwom9sGPTXeHUT8o236axmZgkkG1MGb/DypiE5dshH+soTjlAII3YA42A mGte65XyjDYqPJ3JYn/CHbIihTSrI8mp3pSlWYKVLlS9XqgSb1V4YKybm/cMQwS6 AsOqXCcqFBYimWLJjWikJucFT4rCfmSwISTYjCyGuGvGwBlTxMHoVX9nwLX1atEI mveGxddsLl9vmgWv/mK/WZbco1zLJcPNwpjoM7GqalN6VxHbMhHxoBTaqtoqnw2+ ZWqzTibdyJvkScL1N/AEy+eLLINW7lngY28J0bHWjEL7v2wHM5OZTqeWqpHM4y1W ZXWxC2y+7X3zjDfiDxsCZl2XnJCzn6tA5wBIxkK3wPnqJmKPSUo=5Q6O
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gatewa

Rhys, I think you're being unfair. We have a *technical* disagreement
here. But our hearts are all in the same place: Luca, myself, and all
the other DDs discussing this, all want what's best for our users, we
all want to build the best OS possible, and are all discussing the
issue in good faith.

There is an unavoidable tension, and we're hashing it out. Upstream
has fielded a default behaviour which requires adjustment of a variety
of other programs and workflows. Basically, anything that stores stuff
in /tmp or /var/tmp needs to be made might-be-deleted-aware. There are mechanisms for dealing with this, but they're pretty complicated, and
differ wildly for different file lifetimes etc. Other distributions
have adopted that default, and rather than using exposed mechanisms
for avoiding unexpected deletion, are just telling people not to count
on files in /var/tmp/ surviving a reboot if the computer is shut down
more than a month, or whatever. What should Debian do? You can make
arguments both ways, and we are. Generally we follow upstream unless
there's a compelling reason not to. You can suggest various strategies
for making things reliable despite following upstream. You can discuss
why maybe upstream should not be followed in this case. This is
precisely the kind of discussion that leads to good decisions, with
everyone keeping an open mind and sharing information and ideas.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, 07 May 2024 at 07:34:54 -0500, [email protected] wrote:

possibly convince those applications to use their own
scratch space such as /tmp/<package>/ that is more easily identifiable

This would be a denial of service at best, and a privilege escalation vulnerability at worst. To be safe, it would have to be more like /tmp/<package>.XXXXXX where the XXXXXX is replaced by a random string
by mkstemp() or similar.

(For example my system currently has /var/tmp/flatpak-cache-5X58M2/ which
is fine, but using /var/tmp/flatpak-cache/ would be wrong.)

smcv

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

...3) I would put a file in any auto-cleaned space named "1-AUTOCLEAN.txt" that contains some verbage explaining that things in this directory will be wiped based on rules set in (wherever).

You know, that's a pretty good idea!

Put a 00README-TMP.txt in /tmp/ and /var/tmp/ which briefly states the
default deletion policy, the policy in place if it's not the default,
and a pointer to info about altering it. "/tmp's contents are deleted
at boot while /var/tmp is preserved across rebooting." Maybe in
/var/tmp suggest /var/scratch/ or /var/cache/tmp or such as a place
sysadmins might want to set up for not-backed-up but not-auto-deleted
material.

If the contents aren't dynamic, maybe they could be links to files in /usr/share/doc/systemd/.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

"Johannes" == Johannes Schauer Marin Rodrigues <[email protected]> writes:

>> > > If [files can be deleted automatically while mmdebstrap is using them],
>> > > how should applications guard against that from
>> > > happening?
>> >
>> > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
>> > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
>> > fail to take out its own lock on the directory during cleanup, and
>> > respond to that by treating the directory as "in use" and skipping it.
>>
>> That also works, but only as long as mmdebootstrap is actually
>> running, and as far as I understand it is not a long-running service,
>> not sure if it works for this use case

Note that according to the man page, ctime is used as well as mtime.
So for roots that are actually temporary, I don't think much needs to be
done.
It won't matter that the mtime might be old because the ctime should be consistent with when the root is unpacked.

I do wish there were a way to specify for /var/tmp that directories
under /var/tmp should be deleted in their entirety or entirely left
alone.
I realize we'd have a big debate about whether that was a good default,
but I'd find it useful for my systems at least.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, 7 May 2024 at 15:53, Sam Hartman <[email protected]> wrote:

"Johannes" == Johannes Schauer Marin Rodrigues <[email protected]> writes:

>> > > If [files can be deleted automatically while mmdebstrap is using them],
>> > > how should applications guard against that from
>> > > happening?
>> >
>> > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
>> > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
>> > fail to take out its own lock on the directory during cleanup, and
>> > respond to that by treating the directory as "in use" and skipping it.
>>
>> That also works, but only as long as mmdebootstrap is actually
>> running, and as far as I understand it is not a long-running service,
>> not sure if it works for this use case

Note that according to the man page, ctime is used as well as mtime.
So for roots that are actually temporary, I don't think much needs to be done.
It won't matter that the mtime might be old because the ctime should be consistent with when the root is unpacked.

I do wish there were a way to specify for /var/tmp that directories
under /var/tmp should be deleted in their entirety or entirely left
alone.
I realize we'd have a big debate about whether that was a good default,
but I'd find it useful for my systems at least.

This is a reasonable RFE, and it has already been proposed some days
ago (in the right place, upstream): https://github.com/systemd/systemd/issues/32674

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Quoting Andrey Rakhmatullin (2024-05-06 19:14:40)

On Mon, May 06, 2024 at 04:50:50PM +0100, Barak A. Pearlmutter wrote:

tmpfiles.d snippets can be defined to cleanup on a timer _anything_,

It's a question of what the *default* behaviour should be.

For whatever reason, a lot of people who process large data use /var/tmp/FOO/ as a place to store information that should not be
backed up, but also should not just disappear.

To be honest I'm greatly surprised by this idea, and by the suggestion
that a lot of people do this, to me this is very similar by that half-joke about people storing useful files in the Recycle Bin.

I'm doing exactly that. I use paper I already printed stuff on but which were misprints or which are no longer useful to me and are just destined for the recycling bin to write stuff on that is important to me. After I'm done with my work on this scrap paper I decide what I want to keep and copy to permanent storage and what I want to really throw away.

I would not like if (suppose I had a person cleaning up my stuff) a cleaning person came to my desk, saw that there was obvious scrap paper ultimately destined for the bin and threw that away every once in a while. I might be taking my notes on what is ultimately for the trash but I want to be the one to decide when to bring the trash out.

Thank you for this great example! :)

cheers, josch
--==============S87414447516655059=MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Description: signature
Content-Type: application/pgp-signature; name="signature.asc"; charset="us-ascii"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEElFhU6KL81LF4wVq58sulx4+9g+EFAmY6hb0ACgkQ8sulx4+9 g+HWdw//eUuTrciVsgcePDlAG3s5Rv6unSepGdNw7bXcylgpOuW8ZbLE20i7PMeE fN4+BVxiY8qCrqLYkwzrNUnAvyGVCw1NpvLpaTVroIwgeZTQ83si2Gh9Cmfi4cPq PQbCjxiTfKbLkXZrsVixb076pxSyuWA1bGsv+pzkryAyE048xYsrkw/tghGQsINm ataopCVtAwzeZJoZmhaEmr2jKiiazxhWM5kOo2MvjC3TLpKFb5PnoYO4JuLtSFVD ph4jQO0Lq1uDpzjzibWoBrk3ekJuVNoHT6pnyf5q3XelDCQ/hUgxyFzilhCvUlZU s7KJvDBYppHsQAUn0gWO2TBwsPFOFTatqyMqFhjQynd4/THJJrO+mBJHJUDYnXxQ 5x1nl9HA+GwBjxrOj+XlvKEHYZe3VHN/8f2w3F5zXSpecPp5OmrudI8oBD/6c2Qz E6B5lfmwDKv8Lxf1qwUXKJS1zjNOzItWCLhG2trxaJQs95aDsVsXuHhn7FshkpsU tOnJWDe20HAUQhChx9BbjE0tpokUKoCUpF1W/+BcoWyZxG99SCaia5KhDV1frbGC O1V95EL0rOQnmXaJ9r+wPefBMaBWwjjRZdYt5ICSRfiRzP4oUf8I1b/DbeVEkga7 kE/0+2rmUfeGYX8QR3B4uTL9n7rx9i/bZnkafSwCWTBovkXacCg=
=Js2l
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, May 07, 2024 at 09:49:17PM +0200, Johannes Schauer Marin Rodrigues wrote:

Quoting Andrey Rakhmatullin (2024-05-06 19:14:40)

On Mon, May 06, 2024 at 04:50:50PM +0100, Barak A. Pearlmutter wrote:

tmpfiles.d snippets can be defined to cleanup on a timer _anything_,

It's a question of what the *default* behaviour should be.

For whatever reason, a lot of people who process large data use /var/tmp/FOO/ as a place to store information that should not be
backed up, but also should not just disappear.

To be honest I'm greatly surprised by this idea, and by the suggestion
that a lot of people do this, to me this is very similar by that half-joke about people storing useful files in the Recycle Bin.

I'm doing exactly that. I use paper I already printed stuff on but which were misprints or which are no longer useful to me and are just destined for the recycling bin to write stuff on that is important to me. After I'm done with my
work on this scrap paper I decide what I want to keep and copy to permanent storage and what I want to really throw away.

I actually meant the Windows feature :)


--
WBR, wRAR

-----BEGIN PGP SIGNATURE-----

iQJhBAABCgBLFiEEolIP6gqGcKZh3YxVM2L3AxpJkuEFAmY6hlwtFIAAAAAAFQAP cGthLWFkZHJlc3NAZ251cGcub3Jnd3JhckBkZWJpYW4ub3JnAAoJEDNi9wMaSZLh kUAP/ihS4Nz26N7NW36J0bnspEOvToaV58xnSPVn6ivwoO8M1U4WyHiaYR+Lb8nH IEwH8fq5KrzFsrLJ6kMGIMVZCBcm94F2Vv4DDTGZAdHzWAO7v5OApqhjBtN/0wdY LPYj1LrpkfJmzYTKQz2DetvnQDodEwLKnflYPBm9QmtAVo44gGBMWVtRctRgWsxJ CcZvO2TMzzAefCSqH5w2NFli055axbUf8uq1QxZnrn988P0O8v48+b7rPEvvintw AFYHMO9Evfc5J9FIfsWh6M8H1oMaCzQszSUzawHKTyZItldGh3q1C0ujhR7qJJbv bBhtNe78rp9w2WG8wGwpjLaBumS198YwoN72K5GP/DmNuXlUY/b6oKlXLWAHK5bU Y6S9ziS3ivAtQR6SOEwdUggYcL+SCTEPHH3Ai2zBv1bLo3yfQBuDfGjEY1znMGHE S/PWc8ttpUd3cKToeaJyFi8sI/9SjRLBEmAa2pYKALaF4jhLhRqMOyYt//sexBB3 epWPSIRMlbWjEWpUwwEqPJvs3YzmwHa86WFzBj62rl0wUE1QPpqfrml2YWm3HA/5 zaCazSYJDTH4TP8ptwdbYRHvj6HAn11GNa8Y40K2/bGdpT9M3yHjxkZKfY8ZJImP p0EsVWxwlJx4jd4No4Sdu1Wbm6HJM9wAsKgetkbjtn1VwWO3
=0H78
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Le Mon, May 06, 2024 at 11:15:35AM +0100, Barak A. Pearlmutter a �crit :

We have two separate issues here:

a/ /tmp-on-tmpfs

Note that /tmp-on-tmpfs and cleanup-tmp-at-boot are not equivalent.

With cleanup-tmp-at-boot, if your system crashes, you can still backup
/tmp before rebooting.

Cheers,
--
Bill. <[email protected]>

Imagine a large red swirl here.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)