🧵 2/2 i think, i could tweak the zfs_arc_max(?) variable, decrease its value slightly, in order to have more free RAM on the NAS, for incoming buffers.
Slightly reducing the 1st level RAM ARC, in order to not burn cycles on least-used eviction algos - bigger buffer for incoming writes, which then won't need to be cached in the #NVMe SLOG?
Does that make sense - who "knows" more about #ZFS and #Linux than me?
(Yellow bar = [mostly] ZFS ARC, right?)
Today, for the 1st time, i've transferred files onto the #Linux #NAS / #HomeServer at over 700MB/s (in htop)! 🤩
Source: Ryzen 7840HS, RAM-cached(?), Samsung 990 Pro 2TB NVMe, with LUKS
Target: i5-7500, spinning #ZFS mirror, 2x SATA Toshiba MG09, consumer grade NVMe SLOG mirror, ZVOL with LUKS
Via: #SSHFS over 10gig ConnectX-3, SFP+ DAC, cheap Chinese Realtek switch, one of the CX3 runs via an ASMedia ASM2464PD #Thunderbolt / #USB4 bridge.
I'm pretty chuffed with the #crypto throughput. 🤓
Voilà. C'est mieux. Une partition racine de 12 Go, 8 Go de swap (Proxmox en met automatiquement, j'ai donc limité). Ça me fait un petit 20 Go pour l'OS et tout le reste pour les VM ^^
En parallèle, mon pool ZFS a finit ces vérifications, j'ai pu faire l'export. Demain, je vais donc pouvoir déplacer physiquement les HDD et passer à la préparation des VM.
I've managed to get #OpenMediaVault working on my #RaspberryPi (running #Raspbian Lite) and the performance seems pretty impressive! Despite relying on USB storage for the SSDs.
This is my first time running a #NAS on the Pi, on #OMV, not using #ZFS or #RAID but rather an #Unraid like solution, 'cept, #FOSS called #SnapRAID in combination with #mergerfs (the drives themselves are simply #EXT4).
So far, honestly, so good. I got 2x 1TB SSDs for data, and another 1TB SSD for parity. Don't have a backup for the data themselves atm, but I do have a scheduled backup solution (#RaspiBackup) setup for the OS itself (SD card). It's also got #Timeshift for creating daily snapshots.
I'm not out of the woods yet though, cos after this comes the (somewhat) scary part - deploying #Immich on the Pi lol (using OMV's #Docker compose interface perhaps). I really could just deploy it in my #Proxmox #homelab, and I wouldn't have to worry about system resources or hardware transcoding, etc. but I really wanna experiment this 'everything hosted/contained in 1 Pi' concept.
If you haven't heard, use ecc memory to prevent wrong data from being written, causing a read to later not validate correctly because the checksum does not match.
Yay finally got #ZFS running in a mirror #raid configuration. +1 for data redudancy and hopefully zero data corruption.
So #KDE filemanager #dolphin doesn't search through symbolic links and I had already spent about a year or so cursing #baloo whereas it worked fine on my laptop.
I had to get #fstab bind mounts to mount specific directories from another drive onto my home directory. Everything worked!
Now using #zfs meant I had to force the correct order of mounting else I have empty directories.
fstab's #systemd option - x-systemd.after=datapool.mount came to rescue and everything works again.
#Proxmox #selhosted
Petite amélioration concernant les performances de certains services 😎
Ceux-ci passent sur support de stockage #SSD, et toujours sur système de fichiers #ZFS 👍
Here’s my current hierarchy for this question:
Windows File Server
NFS mounted local rsync
Unencrypted zfs dataset (gets snapped after daily rsync)
syncoid no-sync-snap
Encrypted zfs dataset (key loaded and mounted locally)
syncoid no-sync-snap sendoptions="Rw" recvoptions="u"
Encrypted zfs dataset (no key loaded, unmounted, offsite)
I’m using --nosync-snap because the daily snaps from the original dataset are sufficient and I know that the first replication will nuke any existing sync snaps on the middle (encrypted, keyed, mounted, local) dataset.
When I do the final syncoid send to the offsite untrusted server, how does syncoid know what pieces are already there and to only send what’s new?
Do I need to flatten this out and do my rsync to an encrypted dataset, and sync from that directly to both the second local copy (on another server) and the remote untrusted server using sync-snaps?
I realize the extra local copy isn’t absolutely necessary, but storage is cheap and the data isn’t.
Thanks,
-Pyrroc
3 posts - 2 participants
Got my #ZFS data partition booting of encrypted keys as well. Transferring all the data over to ZFS. Feeling stupid to not have done this earlier, especially when I lost some SQL backups to data corruption earlier. 🤦
Mehr Speicherplatz für den Openmediavault Server
#petermarbaise #tuxoche #openmedaivault #zfs #raidz1 #pool #migration
http://tuxo.li/1mj
I’m looking to finally wipe my final Windows machine and install Linux. I would like the root filesystem to be ZFS so I am looking for distros that include ZFS as an option during installation of the OS.
I know Ubuntu had this in the installer at some point, but took it out. Same with PopOS (I think?). I have gone through the process of installing ZFS Boot Menu and got it working, but I do not feel like I understand well enough how it works to use it for my semi-daily driver.
Coming from TrueNAS/Proxmox on my servers where ZFS is either the default or a common choice, it feels like there aren’t many “desktop” distros with really good ZFS support.
Am I missing anything obvious?
2 posts - 2 participants
I know striping in ZFS is dynamic, and will distribute the data as optimally as possible across the pool.
I’m wondering if setting the recordsize according to the number of vdevs in the pool would be better for performance or avoiding fragmentation?
I remember seeing an example on r/zfs that ZFS will break up a 128K record into 32K chunks when there are 4 vdevs in the pool. But what if the record can’t be divided evenly?
Example: A fresh pool on 3 equal vdevs. For a dataset with large files, would a recordsize of 768K be better than 1M since 768 is divisible by 3?
And how does ZFS distribute the data if the dataset is sent to another pool with 2 or maybe 4 vdevs? 768 is also divisible by those numbers.
1 post - 1 participant
This morning, Brian Behlendorf closed the long standing bug reporting occasional corruption when replicating encrypted datasets using raw send.
This bug’s final dissection and fix were the result of a coordinated community effort, and I’m proud of our own community’s part in that.
Cheers everybody!
github.com/openzfs/zfsZFS corruption related to snapshots post-2.0.x upgrade
opened 01:30PM - 08 May 21 UTC closed 04:55PM - 19 May 25 UTC jgoerzen Type: Defect Component: Encryption Status: Triage Needed### System information Type | Version/Name --- | --- Distribution Name | D…ebian Distribution Version | Buster Linux Kernel | 5.10.0-0.bpo.5-amd64 Architecture | amd64 ZFS Version | 2.0.3-1~bpo10+1 SPL Version | 2.0.3-1~bpo10+1 ### Describe the problem you're observing Since upgrading to 2.0.x and enabling crypto, every week or so, I start to have issues with my zfs send/receive-based backups. Upon investigating, I will see output like this: ``` zpool status -v pool: rpool state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub repaired 0B in 00:03:37 with 0 errors on Mon May 3 16:58:33 2021 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 nvme0n1p7 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: <0xeb51>:<0x0> ``` Of note, the `<0xeb51>` is sometimes a snapshot name; if I `zfs destroy` the snapshot, it is replaced by this tag. Bug #11688 implies that zfs destroy on the snapshot and then a scrub will fix it. For me, it did not. If I run a scrub **without rebooting** after seeing this kind of `zpool status` output, I get the following in very short order, and the scrub (and eventually much of the system) hangs: ``` [393801.328126] VERIFY3(0 == remove_reference(hdr, NULL, tag)) failed (0 == 1) [393801.328129] PANIC at arc.c:3790:arc_buf_destroy() [393801.328130] Showing stack for process 363 [393801.328132] CPU: 2 PID: 363 Comm: z_rd_int Tainted: P U OE 5.10.0-0.bpo.5-amd64 #1 Debian 5.10.24-1~bpo10+1 [393801.328133] Hardware name: Dell Inc. XPS 15 7590/0VYV0G, BIOS 1.8.1 07/03/2020 [393801.328134] Call Trace: [393801.328140] dump_stack+0x6d/0x88 [393801.328149] spl_panic+0xd3/0xfb [spl] [393801.328153] ? __wake_up_common_lock+0x87/0xc0 [393801.328221] ? zei_add_range+0x130/0x130 [zfs] [393801.328225] ? __cv_broadcast+0x26/0x30 [spl] [393801.328275] ? zfs_zevent_post+0x238/0x2a0 [zfs] [393801.328302] arc_buf_destroy+0xf3/0x100 [zfs] [393801.328331] arc_read_done+0x24d/0x490 [zfs] [393801.328388] zio_done+0x43d/0x1020 [zfs] [393801.328445] ? zio_vdev_io_assess+0x4d/0x240 [zfs] [393801.328502] zio_execute+0x90/0xf0 [zfs] [393801.328508] taskq_thread+0x2e7/0x530 [spl] [393801.328512] ? wake_up_q+0xa0/0xa0 [393801.328569] ? zio_taskq_member.isra.11.constprop.17+0x60/0x60 [zfs] [393801.328574] ? taskq_thread_spawn+0x50/0x50 [spl] [393801.328576] kthread+0x116/0x130 [393801.328578] ? kthread_park+0x80/0x80 [393801.328581] ret_from_fork+0x22/0x30 ``` However I want to stress that this backtrace is not the original **cause** of the problem, and it only appears if I do a scrub without first rebooting. After that panic, the scrub stalled -- and a second error appeared: ``` zpool status -v pool: rpool state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub in progress since Sat May 8 08:11:07 2021 152G scanned at 132M/s, 1.63M issued at 1.41K/s, 172G total 0B repaired, 0.00% done, no estimated completion time config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 nvme0n1p7 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: <0xeb51>:<0x0> rpool/crypt/debian-1/home/jgoerzen/no-backup@[elided]-hourly-2021-05-07_02.17.01--2d:<0x0> ``` I have found the solution to this issue is to reboot into single-user mode and run a scrub. Sometimes it takes several scrubs, maybe even with some reboots in between, but eventually it will clear up the issue. If I reboot before scrubbing, I do not get the panic or the hung scrub. I run this same version of ZoL on two other machines, one of which runs this same kernel version. What is unique about this machine? - It is a laptop - It uses ZFS crypto (the others use LUKS) I made a significant effort to rule out hardware issues, including running several memory tests and the built-in Dell diagnostics. I believe I have rules that out. ### Describe how to reproduce the problem I can't at will. I have to wait for a spell. ### Include any warning/errors/backtraces from the system logs See above ### Potentially related bugs - I already mentioned #11688 which seems similar, but a scrub doesn't immediately resolve the issue here - A quite similar backtrace also involving `arc_buf_destroy` is in #11443. The behavior described there has some parallels to what I observe. I am uncertain from the discussion what that means for this. - In #10697 there are some similar symptoms, but it looks like a different issue to me
2 posts - 2 participants
I'm a bit glad to have copied all 12TiB of crud off to another machine running xfs before this kind of thing started happening...
I'm just a bit glad to have taken a complete copy of the entire zpool over to another machine on xfs...
Okay; pretty sure this is a first for me.
I `zpool offline`d a disk in a raidz2 pool, physically replaced it, prepared the replacement disk, and `zpool replace`d the old disk with the new. Resilver ran through to completion, and...
simply restarted all over again!?
No indication in the system logs of anything amiss. No errors in `zpool status` output. Just a class=resilver_start zed event logged.
Hopefully just a temporary glitch and it'll complete normally this time around.
