I can't wait to use `extract_if` in #CSVDiff! :awesome:
csv-diff makes use of a "manual" (aka hacky) implementation of it using `drain` (to remove equal csv records) in combination with an "intermediate" HashMap to restore the not-to-be-removed csv records (the ones that are different):
https://gitlab.com/janriemer/csv-diff/-/blob/main/src/diff_result.rs?ref_type=heads#L599
We can _probably_ remove this hacky implementation and replace it with `extract_if`! Very exciting!
2/2
#Fuzzing along in #CSVDiff :awesome:
In the second screenshot I've highlighted some interesting parts:
Key field indices are 2 and 3, so when diffing the records, where key fields are highlighted, they'll be compared as `Modify`, because:
- key fields are equal between left and right record
- other fields are unequal between left and right record
The other two records on the right have no corresponding left record - so those are `Add`ed records
Huh, seems like I really have been living on the bleeding edge (of #FormalVerification):
https://github.com/creusot-rs/creusot/discussions/1477#discussioncomment-12991148
The verification in the prev toot is currently not possible in #Creusot due to missing specs for the `Hash` trait and HashMap more broadly. 😔
Oh well, seems like (at least currently!) I won't be able to fully verify the diffing algorithm of #CSVDiff.🥺
Options I have now are:
- Only verify parts of the algorithm (that don't depend on HashMap ops)
or
- Use fuzzing/property testing
Just published a new version of csv-diff (v0.1.1) 🚀
https://lib.rs/crates/csv-diff
This fixes a nasty bug regarding sort order of modified csv records. 😖
Details in the MR/PR:
https://gitlab.com/janriemer/csv-diff/-/merge_requests/31
Also, two new incoming PRs for #qsv, the #CSV toolkit:
The first updates to the latest csv-diff, fixing aforementioned bug:
https://github.com/dathere/qsv/pull/2456
The second fixes a bug regarding conversion from column names to indices:
https://github.com/dathere/qsv/pull/2457
Ouch, there is another bug and this time it is actually _in #CSVDiff itself_!
It happens with sorting the results of modified rows (urgh, I'm also not happy with the sorting code).😨
Thankfully, datatraveller1 already has found a reproducible example - thank you so much! ❤️
Bug:
https://github.com/dathere/qsv/issues/2443#issuecomment-2598987465
I think I already found a solution, but needs rigorous testing first!
Potential solution:
https://github.com/dathere/qsv/issues/2443#issuecomment-2599681431
Nice, I think I found the bug! 🐛
See all the explanation and possible solution here:
=> https://github.com/dathere/qsv/issues/2443#issuecomment-2597097311
Workaround is also present and explained, so should be no blocker for people.
Will prob provide a fix on the weekend. 🤞
@shuttle I consequently use #TDD, where possible.
Yes, sure, #Rust prevents a lot of bugs at compile time already, but not logic bugs.
For example in #CSVDiff we have ~70 unit tests and ~12 integration tests. The only "bug report" we have ever gotten was due to a corrupted CSV file (being mistaken with a bug in diff):
See here (qsv):
https://github.com/jqnatividad/qsv/issues/1258#issuecomment-1712924932
csv-diff:
https://gitlab.com/janriemer/csv-diff
In the future I'd like to add property and mutation testing as well 🤓
#CsvDiff has finally reached v0.1.0, it's first ever non-alpha/-beta release! 🎉
New features like getting at the headers from the diffresult have been needed for the following PR in qsv (which is in final review):
https://github.com/jqnatividad/qsv/pull/1395
When merged, you'll be able to decide, whether the diffresult should output headers or not (see examples in the PR). :awesome:
Check out csv-diff's Changelog for the full details:
https://gitlab.com/janriemer/csv-diff/-/blob/main/CHANGELOG.md?ref_type=heads#010-30-october-2023
And this is why #UnitTests and #TDD are awesome/necessary (even in #Rust/ #RustLang):
The original requirement:
figure out how many columns the _result_ of diffing two CSVs in #CsvDiff have.
Do you see the error-pattern?
It's
- when we have no diff
&&
- at least one CSV has headers
which makes sense, because I've implemented the feature in the diffing logic, but at that point header information is already lost (in some other thread).
Isn't that beautiful!?🥰
I can't reproduce the bug. ¯\_(ツ)_/¯
Neither in #CsvDiff ...
https://gitlab.com/janriemer/csv-diff/-/commit/0ac6ba724656dbcc620eefda3f8ec5d8064134d6
...nor in #qsv
https://github.com/jqnatividad/qsv/pull/1297
My assumption is that they have forgotten to specify the option --right-delimiter (or --left-delimiter, respectively), when executing `qsv diff`:
https://github.com/jqnatividad/qsv/issues/1258#issuecomment-1712842883
Anyway, we now have additional tests in csv-diff and qsv, so definitely a win, regardless of the outcome! 🎉
@floriann FYI
Oh noes, apparently I haven't considered different delimiters for the left and right #CSV in #CsvDiff.😱
Someone reported a bug in `qsv diff` (which uses csv-diff) with this scenario.
https://github.com/jqnatividad/qsv/issues/1258
I'll have a look at it tomorrow.
Glad, csv-diff is actively used! ❤️
If you want to know, how to provide a large resource (such as an owned String) to a criterion benchmark, you can use the `iter_batched` method:
https://docs.rs/criterion/latest/criterion/struct.Bencher.html#method.iter_batched
See an example of this in #CsvDiff
https://gitlab.com/janriemer/csv-diff/-/blob/main/benches/bench_csv_diff.rs#L67
3/3
Not sure where this will lead to, but it sounds fun and exciting, so let's try! :awesome: :rust: :ferris:
Yay! Sorting the #csv diff result by columns has just been merged into #qsv! 🥳
A new version of csv-diff is out (v0.1.0-beta.2) 🎉
https://lib.rs/crates/csv-diff
This version adds a method, which allows you to sort your diff result by columns (it was already possible to sort by lines).
See the changelog for an example:
https://gitlab.com/janriemer/csv-diff/-/blob/8642a8a7ba14e22d076cee8c3f690c17f41d7528/CHANGELOG.md#010-beta2-19-february-2023
Sorting by columns will soon be integrated into qsv, the #CSV toolkit:
https://github.com/jqnatividad/qsv/issues/714
Thank you @jqnatividad for the idea of this feature! 💚
@hyde Also check out `qsv`. 🙂
It's an actively maintained fork of xsv (xsv is not maintained anymore).
qsv is _very active_ in development.
And shameless plug in the end 😁
Just a few days ago, `csv-diff` got merged:
https://github.com/jqnatividad/qsv/pull/711
csv-diff is a crate for comparing CSVs with ludicrous speed:
https://gitlab.com/janriemer/csv-diff
So the new command `qsv diff` is now the fastest #CSV differ in the world! 🚀
Announcement 🎉 🥳
csv-diff will be integrated into qsv, the CSV toolkit soon! 🎉 :ferris:
PR:
https://github.com/jqnatividad/qsv/pull/711
Comparing the majestic million dataset with 1,000,000 rows x 12 columns takes less than 800ms and only about 150mb of RAM!
With this, it is the fastest #CSV differ in the world!🚀
See the following svg recording for a demo:
csv-diff:
https://gitlab.com/janriemer/csv-diff
#Rust #RustLang #Data #Diff #CsvDiff #Difference #Performance #DataScience #Oxidization
🥳 A new version of csv-diff has just been released! 🚀
https://docs.rs/csv-diff/latest/csv_diff/
csv-diff is the fastest CSV-diffing library in the world - written in #Rust
It can compare two 1,000,000 rows x 9 columns CSVs in < 600ms!
Note that this is still a beta release and the library itself is still very young.
#RustLang #Release #CSV #CSVDiff #Performance #DataScience #Data #Diff #Difference #OpenSource #Crate
