Miriami wrote:

Would you recommend me a mature fuse filesystem, which uses a single file as backing storage, and could self-growing in size?

It's like using a fuse ext4 filesystem, just that with self-growing - I tried the fuse ext4 filesystem, but it seems that fuse ext4 filesystem requires the user to set up a 'disk size' at the beginning, and could not automatically grow.

I would like the fuse filesystem to automatically grow as new files are added in, just like how a SQLite database would grow as new records being inserted. I have found quite a lot of projects seem to meet this function, but seems that those projects

are quite unmature.

In summary, I am looking for a fuse filesystem that:
1. Uses a single file as backing storage, like how SQLite uses one single file for the database.
2. Is mature - if it is listed in the Debian package list that could be queryed with 'apt search', I guess it should be quite mature and reliable - however, it would be totally fine if the project isn't available as packaged binary and building from

source is required.

It would be totally okay if the filesystem temporaily creates files other than the one files it uses as storage - just like how SQLite generates temporary WAL or journal files during database I/O. Just having only one file as storage 'most of the time'

would be good.


I don't think there are any.

However, if you could explain the reason behind your
requirements, we might be able to find something that would fit
what you need, even if it isn't quite what you are currently
asking for.

-dsr-

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, 8 Mar 2025, Miriami wrote:

Hi!

Would you recommend me a mature fuse filesystem, which uses a single
file as backing storage, and could self-growing in size?

It's like using a fuse ext4 filesystem, just that with self-growing -
I tried the fuse ext4 filesystem, but it seems that fuse ext4
filesystem requires the user to set up a 'disk size' at the beginning,
and could not automatically grow.

I would like the fuse filesystem to automatically grow as new files
are added in, just like how a SQLite database would grow as new
records being inserted. I have found quite a lot of projects seem to
meet this function, but seems that those projects are quite unmature.

It would be great if you know that there is a mature one. However, I
haven't found one listed in the Debian package system by trying some
'apt search' commands. During this process, I found a fuse filesystem
called gocryptfs, which seems to be really mature, and could
automatically grow as new files added in - however, gocryptfs stores
in a folder with multiple files, instead of storing all the filesystem
in one single file like SQLite.

In summary, I am looking for a fuse filesystem that:
1. Uses a single file as backing storage, like how SQLite uses one
single file for the database.
2. Is mature - if it is listed in the Debian package list that could
be queryed with 'apt search', I guess it should be quite mature and
reliable - however, it would be totally fine if the project isn't
available as packaged binary and building from source is required.

It would be totally okay if the filesystem temporaily creates files
other than the one files it uses as storage - just like how SQLite
generates temporary WAL or journal files during database I/O. Just
having only one file as storage 'most of the time' would be good.

Why do you want this? There are few if any filesystems that grow like
you're asking. The nearest I can think of is growisofs.

But if, instead, what you want is the backing store to grow then use a
sparse file as the backing store.

# truncate -s 1T file
# mke2fs -t ext4 file
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: done
Creating filesystem with 268435456 4k blocks and 67108864 inodes
Filesystem UUID: 88f11c90-d61b-4394-add2-700968e375d9
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done


# ls -al file
-rw-r--r-- 1 root root 1099511627776 Mar 8 14:46 file
# du -sch file
19M file
19M total

That's a 1T ext4 filesystem that is, when empty, using around 20M of
disk space.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

Miriami wrote:

I would like the fuse filesystem to automatically grow as new files
are added in, just like how a SQLite database would grow as new
records being inserted.

Tim Woodall wrote:

The nearest I can think of is growisofs.

That's mainly a burn program for optical media.
(I praise it for being first to offer free DVD and BD burning and for
giving me code examples for the SCSI specs SPC and MMC when i developed
support for these media types in libburn.)

But it is not fuse based and i never could get it to work with a data
file instead of an optical drive. My own program xorriso does better
with data files as storage medium, but like growisofs it is not based
on fuse.
ISO 9660 is mountable only read-only. Files can be replaced by adding a session, but each file can only be replaced as a whole. Afterwards one
would have to unmount and then mount again, whenever a new session was
written.

But if, instead, what you want is the backing store to grow then use a
sparse file as the backing store.

This idea reduces the search to filesystems which don't hop around and
write without need to new random places on their storage medium.

# truncate -s 1T file
# mke2fs -t ext4 file
That's a 1T ext4 filesystem that is, when empty, using around 20M of

disk space.

Question is how its "du" size grows when you put a few files into it,
overwrite their content, rename and copy them.


Miriami wrote:

I have found quite a lot of projects seem to meet this function, but
seems that those projects are quite unmature.

Would you mind to share a few project names and/or web links ?


Have a nice day :)

Thomas

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Max Nikulin wrote:

On 08/03/2025 19:31, Miriami wrote:

Would you recommend me a mature fuse filesystem, which uses a single
file as backing storage, and could self-growing in size?

Qemu's qcow2 disk images may grow in size while initially they may be
smaller than size exposed to virtual machines. However likely there is no fuse module. I have seen a suggestion to use network block device to mount
an image without starting a VM.

The maximum size of a qcow2 filesystem does not change (without
manual intervention) but it can be started in a sparse file that
only takes up necessary space on disk until it reaches the
maximum size.

There is at least one qcow2 FUSE interface, but it is not
packaged in Debian and looks abandoned to me.

There is a way to export a qcow2 filesystem as a network block
device, and NBD filesystems can be mounted, but this is a
sufficiently complex set of layers that I imagine performance to
be a problem.

If Miriami were to tell us what they want to do rather than how
they want to do it...

-dsr-

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, 8 Mar 2025, Thomas Schmitt wrote:

But if, instead, what you want is the backing store to grow then use a
sparse file as the backing store.

This idea reduces the search to filesystems which don't hop around and
write without need to new random places on their storage medium.

# truncate -s 1T file
# mke2fs -t ext4 file
That's a 1T ext4 filesystem that is, when empty, using around 20M of

disk space.

Question is how its "du" size grows when you put a few files into it, overwrite their content, rename and copy them.

# fallocate -l 100M file
# du -h file
101M file
# mke2fs -t ext4 -E nodiscard file
mke2fs 1.47.0 (5-Feb-2023)
Creating filesystem with 102400 1k blocks and 25584 inodes
Filesystem UUID: 7e49b294-4c51-407a-9b84-557755c7baed
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

# du -h file
101M file
# mount -o loop file mnt
# fstrim -v mnt
mnt: 88.2 MiB (92486656 bytes) trimmed
# umount mnt
# du -h file
5.7M file

It can be cleaned up periodically. Ideally you'd want a fuse fs that
supported fstrim I guess. I don't know whether this is even possible.

ext4 fuse doesn't support SEEK_HOLE/SEEK_DATA (at least not last time I
looked) so I doubt it will support discard.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

Miriami wrote:

About the projects I've found - I searched with
three terms on GitHub - 'sqlite', 'fs' and
'fuse'. It is surprisingly large number of
projects come up with the same idea of using
SQLite as a backing file system storage.

I remeber that filesystems in general have been declared dead about
twenty years ago.
(But who needs filesystems or databases in times of AI ?)

700+ lines of pull request about upgrading
the Rust library used by the project and updating
the API calls to meet the new library version.

What's not written in C will be rotten in foreseeable time.


Grumpy sarcasm aside:

As a result, I decide that I would go and try
with ext4. I would firstly put some files in
and see how much the image grows physically.
I guess it won't overgrow - ext4 doesn't
store files for extra copies, right? I guess
the sparse file mechanism would work fine.

Note that the sparse file behavior depends on the hosting filesystem,
which contains your large sparse file, not on the filesystem inside
that file.
The ext4 inside the file will do what it does with an arbitrary storage
device of the given nominal size (i.e. the size of the file in "ls -l",
not in "du").

I need to fuse a virtual
filesystem for the poor Debian on that device,
with backing storage on the NTFS partition
visible for Windows.
[...]
Due to this, I come up with the idea of only
leaving *one* file visible to the Windows
system - that disk-scanning software - so that
there would be less trouble.

So you will have to test the sparse file behavior of NTFS with an ext4
inside the sparse NTFS file.

that
commercial software frequently performs
full-disk scan 'for the sake of user security'.

I wonder what it will say about a large file with obscure content ...


Have a nice day :)

Thomas

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

visible for Windows. The awful part is that,
I need to run a commercial software on
the Windows system of that device, and that
commercial software frequently performs
full-disk scan 'for the sake of user security'.

If at all possible, you might be better off turning the Windows part
into a VM that you run from within Debian.


Stefan

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)