Barman woes on OVH
So, due to a cascade of Barman’s errors and corrupted backups due to running out of backup space, I had to pretty much clear out my Barman backup storage.
After resetting everything, however, I was not able to make a full backup, because Barman was not able to receive write-ahead logs from PostgreSQL.
2025-06-04 22:00:21,209 [523687] barman.cli ERROR: [Errno 5] Input/output error: '/backup/barman/pg/wals/00000001000000CD/00000001000000CD00000009.tmp'
See log file for more details.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/barman/cli.py", line 2390, in main
args.func(args)
File "/usr/lib/python3/dist-packages/barman/cli.py", line 1600, in archive_wal
server.archive_wal()
File "/usr/lib/python3/dist-packages/barman/server.py", line 2651, in archive_wal
self.backup_manager.archive_wal(verbose)
File "/usr/lib/python3/dist-packages/barman/backup.py", line 847, in archive_wal
archiver.archive(verbose)
File "/usr/lib/python3/dist-packages/barman/wal_archiver.py", line 213, in archive
self.archive_wal(compressor, wal_info)
File "/usr/lib/python3/dist-packages/barman/wal_archiver.py", line 356, in archive_wal
shutil.copystat(src_file, tmp_file)
File "/usr/lib/python3.12/shutil.py", line 388, in copystat
_copyxattr(src, dst, follow_symlinks=follow)
File "/usr/lib/python3.12/shutil.py", line 338, in _copyxattr
os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
OSError: [Errno 5] Input/output error: '/backup/barman/pg/wals/00000001000000CD/00000001000000CD00000009.tmp'
Input/output error? That’s odd, but the stacktrace tells us a lot, as from the function name (os.setxattr), we can deduce that it’s trying to set xattrs on our WAL files. The underlying storage for my backups is OVH’s Backup Storage, accessible over NFS. And NFS, for most of its life, was not able to support xattrs even if the underlying filesystem does. The support has been added to NFS 4.2, while OVH (still) uses 4.1.
So, how to fix this?
Initially, I thought of downgrading Barman, because things had worked before. But that did not help, and so I had to go digging into the source code (which was painful as I am not a Python guy).
# Perform the real filesystem operation with the xlogdb lock taken. # This makes the operation atomic from the xlogdb file POV with self.server.xlogdb("a") as fxlogdb: # If the content has changed, it means the file was either compressed # or encrypted or both. In this case, we need to update its metadata if content_changed: shutil.copystat(src_file, current_file) stat = os.stat(current_file) wal_info.size = stat.st_size
So, if content_changed is true, we use copystat from shutil.py, which copies xattrs from the original file.
# If the bits of the file has changed e.g. due to compression or encryption content_changed = False # Compress the file if not already compressed if compressor and not wal_info.compression: compressor.compress(src_file, tmp_file) files_to_remove.append(current_file) current_file = tmp_file content_changed = True wal_info.compression = compressor.compression # Encrypt the file if encryption: encrypted_file = encryption.encrypt(current_file, dst_dir) files_to_remove.append(current_file) current_file = encrypted_file wal_info.encryption = encryption.NAME content_changed = True
Ah, so therein lies the rub: Barman assumes that it needs to do this if the content is either being compressed or encrypted. And it just so happens that I’ve also enabled GZIP compression so as not to run out of space again. Well, we have to deal with this the old-fashioned way (by lowering the retention policy). After disabling compression, Barman was able to make backups again.
Hope this helps someone, because it sure as fuck would’ve helped me.
#Barman #lighthearted #NFS #NFSv4 #OVH #pgsql #PostgreSQL #SelfHosting
![A screenshot of a grep on Barman logs, showing the error about which the blog post is talking about. /var/log/barman/barman.log:2025-06-04 21:59:04,771 [521188] barman.cli ERROR: [Errno 5] Input/output error: '/backup/barman/pg/wals/00000001000000CD/00000001000000CD00000009.tmp' /var/log/barman/barman.log-See log file for more details. /var/log/barman/barman.log-Traceback (most recent call last): /var/log/barman/barman.log- File "/usr/lib/python3/dist-packages/barman/cli.py", line 2390, in main /var/log/barman/barman.log- args.func(args) /var/log/barman/barman.log- File "/usr/lib/python3/dist-packages/barman/cli.py", line 1600, in archive_wal /var/log/barman/barman.log- server.archive_wal() /var/log/barman/barman.log- File "/usr/lib/python3/dist-packages/barman/server.py", line 2651, in archive_wal /var/log/barman/barman.log- self.backup_manager.archive_wal(verbose) /var/log/barman/barman.log- File "/usr/lib/python3/dist-packages/barman/backup.py", line 847, in archive_wal /var/log/barman/barman.log- archiver.archive(verbose) /var/log/barman/barman.log- File "/usr/lib/python3/dist-packages/barman/wal_archiver.py", line 213, in archive /var/log/barman/barman.log- self.archive_wal(compressor, wal_info) /var/log/barman/barman.log- File "/usr/lib/python3/dist-packages/barman/wal_archiver.py", line 356, in archive_wal /var/log/barman/barman.log- shutil.copystat(src_file, tmp_file) /var/log/barman/barman.log- File "/usr/lib/python3.12/shutil.py", line 388, in copystat /var/log/barm](https://files.mastodon.social/cache/media_attachments/files/114/630/355/225/312/272/original/0f66c4885523538b.png)
