Fixing Noisy Logs with #OpenTelemetry Log #Deduplication

https://www.dash0.com/guides/opentelemetry-log-deduplication-processor

Hab mein PyHardLinkBackup komplett neu geschrieben. Ursprünglich 2015 gestartet und bis 2020 genutzt, schlief es jetzt fast 6 Jahre...

Aber als ich über alte damit erstellte backups gestolpert bin, hab ich mir gedacht, das Konzept ist doch ganz nützlich.

Also kompletter rewrite: https://github.com/jedie/PyHardLinkBackup

#backup #OpenSource #Python #deduplication #hardlinks

TIL: #XFS kann #Snapshots aber keine #Compression, aber dafür #deduplication, wenn auch noch experimental

Wer ein #snapshot artiges Backup für #Linux sucht, könnte sich #kopia ansehen.
Über Regeln sehr fein granuliert einstellbar.
Es hat mich allerdings jetzt fast eine Woche gekostet, es so zum laufen zu bringen, wie ich es mir gewünscht habe. Aber mit viel #script’en hat alles geklappt.
#deduplication und #kompression, schnell und easy.
Sehr zu empfehlen.

And once in a while I cleanup the external libraries with #Czkawka

This is an amazing software for #deduplication of image folders.

https://github.com/qarmin/czkawka

@stratosphere it's always interesting what's constantly trying to hack...

• Glad to see the blocklists reachable again.

Also one little feedback: It would be cool to see some "#deduplication" or rather reduction of entries by using #CIDR notation which should save a lot of lines = table entries compared to the single IPs.

• Tho I'm shure this is not done for ease of statistics and further research down the line.

Sick: Indexed deduplicated binary storage for JSON-like data structures

https://github.com/7mind/sick

#HackerNews #Sick #Indexed #Binary #Storage #JSON #Deduplication #DataStructures

The machine forgets. The Ghost does not.

https://deadswitch.tomsitcafe.com/2025/10/borg-backup-intro.html

#borg #backup #encryption #deduplication #ghostware

Part 1 : #PySpark Data Pre-processing Essentials #filtering || #Deduplication || Data Cleansing.

Learn PySpark data pre-processing with our tutorial! Learn the art of filtering and deduplication, essential techniques for cleaning ... source

https://quadexcel.com/wp/part-1-pyspark-data-pre-processing-essentials-filtering-deduplication-data-cleansing/

Ich hab mal versucht, die Speicheroptimierung durch #Deduplication beim #Backup mit #restic zu quantifizieren. Dies nach einer Laufzeit von knapp 2 Jahren.

Herausgekommen ist: 22,4%

# restic stats latest 
repository d989459c opened successfully, password is correct
scanning...
Stats in restore-size mode:
Snapshots processed:   1
   Total File Count:   438037
         Total Size:   23.271 GiB

# restic stats latest --mode raw-data 
repository d989459c opened successfully, password is correct
scanning...
Stats in raw-data mode:
Snapshots processed:   1
   Total Blob Count:   265960
         Total Size:   18.409 GiB

Hoffe, das richtig interpretiert zu haben.

restic.readthedocs.io/en/stabl…

Some design tests for 🟥 🟩 diff'ing files hosted on Hugging Face.

With the new storage backend (Xet), similar files share many data "chunks".

As HF provides an API to get the list of these chunks, you can now compute the diff and display a chart in the browser without downloading the files. Very fast.

Test here: https://observablehq.com/@severo/parquet-content-defined-chunking-blog-post-reprodu

Details about the three charts in the responses.

#parquet #huggingface #diff #dataviz #deduplication

🔗 𝗞𝗲𝗻𝗻𝗲𝗻 𝗦𝗶𝗲 𝘀𝗰𝗵𝗼𝗻 𝗱𝗶𝗲 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝘃𝗼𝗻 𝗣𝗿𝗼𝘅𝗺𝗼𝘅 𝗕𝗮𝗰𝗸𝘂𝗽 𝗦𝗲𝗿𝘃𝗲𝗿 (𝗣𝗕𝗦) 𝗺𝗶𝘁 𝗣𝗿𝗼𝘅𝗺𝗼𝘅 𝗩𝗶𝗿𝘁𝘂𝗮𝗹 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁?

Wer 𝗣𝗿𝗼𝘅𝗺𝗼𝘅 𝗩𝗘 nutzt, sollte 𝗣𝗕𝗦 einsetzen:
🧩𝗡𝗮𝗵𝘁𝗹𝗼𝘀𝗲 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻
⚙️𝗕𝗮𝗰𝗸𝘂𝗽-𝗠𝗼𝗱𝗶 𝗳ü𝗿 𝗺𝗮𝘅𝗶𝗺𝗮𝗹𝗲 𝗞𝗼𝗻𝘁𝗿𝗼𝗹𝗹𝗲
💾𝗗𝗲𝗱𝘂𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 & 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗺𝗶𝘁 𝗣𝗕𝗦
🚀𝗙𝗹𝗲𝗲𝗰𝗶𝗻𝗴 𝗳ü𝗿 𝗩𝗠-𝗕𝗮𝗰𝗸𝘂𝗽𝘀
📦𝗖𝗼𝗻𝘁𝗮𝗶𝗻𝗲𝗿-𝗕𝗮𝗰𝗸𝘂𝗽𝘀 𝗺𝗶𝘁 𝗖𝗵𝗮𝗻𝗴𝗲 𝗗𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻
🔐𝗞𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗶𝗼𝗻 & 𝗥𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻

📞 𝗦𝗶𝗲 𝗺ö𝗰𝗵𝘁𝗲𝗻 𝗜𝗵𝗿𝗲 𝗕𝗮𝗰𝗸𝘂𝗽-𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗲 𝗺𝗼𝗱𝗲𝗿𝗻𝗶𝘀𝗶𝗲𝗿𝗲𝗻?

Sprechen Sie mit uns 👉 0441-309197-69

#ProxmoxVE #PBS #VMBackup #ContainerBackup #Deduplication #SnapshotBackup

Borg - Deduplicating Archiver with Compression and Encryption

https://www.borgbackup.org/

#HackerNews #Borg #Deduplication #Compression #Encryption #Backup

J'ai lu le très bon billet d' @athoune sur #Kloset, moteur de stockage de backup de #Plakar

https://notes.sklein.xyz/2025-06-16_1639/

#TIL #Backup #Restic #Deduplication

Testing some more concurrent operations on the new #Synology #DS925+ as part of the upcoming #review

Dual #ActiveBackupForBusiness remote #windows11 machines (testing the #deduplication in the process), along a simple file copy onto an external #USB #SSD #OWC enclosure.

(yes 3rd party external drives work just fine!)

The NAS is nice and calm with no performance issues at all.

Happening now! In this month's #CommunityCall, #OpenAIREGraph Training & Engagement Officer Stefania Amodeo takes us on a journey through the Graph #deduplication process that uses a combination of advanced algorithms & human expertise. Missed out? All past #CommunityCall materials can be found on the Graph portal, today's to be uploaded in the coming days.

Past Calls https://graph.openaire.eu/community-calls

anyone know of an implementation of the CPM-SW deduplication algorithm of this paper?

[edited to change the url to the abstract page rather than direct to the pdf]

#deduplication #algorithm

The Problem Isn't Email, It's Microsoft Exchange -- it turns out my 2011-vintage rant still rings true, now also available trackerless: https://nxdomain.no/~peter/the_problem_isnt_email_its_microsoft_exchange.html #inefficiency #timewasted #email #archiving #microsoft #exchange #compliance #deduplication #unsolvedproblems #smtp #mail #annoyances

DB2 Query Deduplication: Optimizing Large Datasets with ROW_NUMBER()
Learn efficient DB2 Query Deduplication using ROW_NUMBER() for large datasets. Optimize your queries with CTEs & indexing for smoother performance. #DB2Query #Deduplication #ROW_NUMBER #SQLoptimization #largedatasets
https://tech-champion.com/database/db2-query-deduplication-optimizing-large-datasets-with-row_number/
Learn how to efficiently deduplicate large DB2 datasets using ROW_NUMBER() and optimize query performance. ...

Testing #Synology #ActiveProtectManager and #DP320 appliance by making the ISO bare metal recovery media.

Let's see how fast this little machine can recover a 50GB setup.

The actual #backup and #deduplication are working really well.