#PropertyTesting

Jan :rust: :ferris:janriemer@floss.social
2025-05-18

#Fuzzing along in #CSVDiff :awesome:

In the second screenshot I've highlighted some interesting parts:

Key field indices are 2 and 3, so when diffing the records, where key fields are highlighted, they'll be compared as `Modify`, because:
- key fields are equal between left and right record
- other fields are unequal between left and right record

The other two records on the right have no corresponding left record - so those are `Add`ed records

#Rust #FuzzTesting #RustLang #PropertyTesting

A screenshot of my terminal that shows output of CSV records that have been generated by a fuzzing library, called bolero (in Rust).

There is one `left` CSV record and three `right` CSV records. The fields of each record are composed of randomly generated bytes with different lengths. However, when looking at the second screenshot......one record on the left and the right stand out in that they share the same value in two fields at field index 2 and 3. One can see that those are the key fields of the CSVs, because at the very bottom of the terminal output, it says "key_fields: {3, 2}".
Jan :rust: :ferris:janriemer@floss.social
2025-05-07

Wow, #FuzzTesting/ #PropertyTesting is actually harder than doing an automatic proof.😓

I didn't expect that!😮

#SoftwareEngineering #Unexpected #Testing #FormalVerification

2025-04-21
Property-based testing in Haskell with QuickCheck falsify

A few days ago, Edsko de Vries of Well-Typed published an in-depth article on property-based software testing, with a focus on the concept of “shrinking.”

In brief, property-based testing is sort-of like fuzz testing but for algorithms and protocols. Like fuzz testing, random test cases are procedurally generated, but unlike fuzz testing, the test cases are carefully designed to verify whether a software implementation of an algorithm satisfies a specific property of that algorithm, such as:

  • “this function always fails if the index is larger than the array”
  • “this function always returns a result in n*log(n) number of iterations for input dataset of size n
  • “the sequence of log messages is guaranteed to obey this rules of this particular finite-state automata: (connect | fail) -> (send X | fail) -> (receive Y | receive Z | fail) -> success .”

Shrinking is the process of simplifying a failed test case. If you have found some input that makes your function return a value when it should have thrown an exception, or produce a result that does not satisfy some predicate, then that input is a “counterexample” to your assertion about the properties of that function. And you may want to be able to “shrink” that counterexample input to see if you can cause the function to behave incorrectly again but with a simpler input. The “QuickCheck“ library provides a variety of useful tools to let you define property tests with shrinking.

Defining unit tests with such incredible rigor takes quite a lot of time and effort, so you would probably do not want to use property-based testing for your ordinary, every-day software engineering. If you are, for example, being scrutinized by the US Department of Government of Efficiency, you would likely be fired if you were to take so much time to write such high-quality software with such a strong guarantee of correctness.

But if you are, for example, designing a communication protocol that will be used in critical infrastructure for the next 10 or 20 years and you want to make sure the reference implementation of your protocol is without contradictions, or if you are implementing an algorithm where the mathematical properties of the algorithm fall within some proven parameters (e.g. computational complexity), property-based testing can give you a much higher degree of confidence in the correctness of your algorithm or protocol specification.

#tech #software #haskell #WellTyped #QuickCheck #UnitTesting #PropertyTesting

2025-02-20

#PropertyTesting Pattern: Some things never change

Examples:

- Adding the same permissions to a user
- Removing a user from a group
- Changing a user preference

Diagram showing Idempotency
2025-02-19

#PropertyTesting Pattern: There and back again

Write properties that pair an operation with its opposite, so that you get the results you started with.

- Serializing and deserializing
- Toggling preferences / settings
- Writing and deleting an object from storage
- Adding and removing a fixed value from a Number

Diagram showing that applying the inverse of a function gives you the original.
2025-02-18

#PropertyTesting Pattern: Different paths, same destination

If order truly doesn't matter, you should confirm that. Notable examples:

- Adding items in a cart (esp w/ & w/o sales)
- Changing user permissions
- Adding data sources

To test, generate a random set of operations to add and then check that the end state is what you expect.

2025-02-17

If you're looking at putting #propertytesting into practice and drawing a blank once you open VSCode, this is the best article for direct advice.

buff.ly/4b6cPSv

I'll dive deeper into it over the week, because the examples can absolutely be expanded to services instead of just functions.

#testing

2025-02-10
2024-01-26

github.com/dubzzz/fast-check - FastCheck is #PropertyTesting in #TypeScript. Amazingly simple and powerful. Great project @ndubien.

2024-01-25

dev.to/mbernard/what-s-wrong-w - a series on getting started with #PropertyTesting in #dotNET. Given a true condition (function) generate random values to validate it always is. Nice series github.com/mbernard.

2023-08-17

Just realized that the implementation of the classic reversal-of-a string Property Test that virtually all the tutorials use is not that correct. Indeed,

rev rev xs == xs

happily passes for

rev xs = xs

That sucks. #propertytesting

2022-11-11

#PropertyTesting is a powerful way to detect #Bugs in code. 🎉 Opinions welcomed on my suggestion to add such tool as first-class citizen in @vitest via fast-check #JavaScript. More at github.com/vitest-dev/vitest/d

2022-11-02

We’ve been using github.com/flyingmutant/rapid on a large project at work recently and I like it (overall). At first I was -1, but it’s really grown on me. #golang #propertytesting #software #development #testing

Client Info

Server: https://mastodon.social
Version: 2025.04
Repository: https://github.com/cyevgeniy/lmst