Why #PropertyTesting Finds Bugs #UnitTesting Does Not

https://buttondown.com/hillelwayne/archive/why-property-testing-finds-bugs-unit-testing-does/

#testing

Why Property Testing Finds Bugs Unit Testing Does Not (2021)

https://buttondown.com/hillelwayne/archive/why-property-testing-finds-bugs-unit-testing-does/

#HackerNews #PropertyTesting #UnitTesting #SoftwareBugs #BugHunting #ProgrammingInsights

#Fuzzing along in #CSVDiff :awesome:

In the second screenshot I've highlighted some interesting parts:

Key field indices are 2 and 3, so when diffing the records, where key fields are highlighted, they'll be compared as `Modify`, because:
- key fields are equal between left and right record
- other fields are unequal between left and right record

The other two records on the right have no corresponding left record - so those are `Add`ed records

#Rust #FuzzTesting #RustLang #PropertyTesting

Wow, #FuzzTesting/ #PropertyTesting is actually harder than doing an automatic proof.😓

I didn't expect that!😮

#SoftwareEngineering #Unexpected #Testing #FormalVerification

A few days ago, Edsko de Vries of Well-Typed published an in-depth article on property-based software testing, with a focus on the concept of “shrinking.”

In brief, property-based testing is sort-of like fuzz testing but for algorithms and protocols. Like fuzz testing, random test cases are procedurally generated, but unlike fuzz testing, the test cases are carefully designed to verify whether a software implementation of an algorithm satisfies a specific property of that algorithm, such as:

• “this function always fails if the index is larger than the array”
• “this function always returns a result in n*log(n) number of iterations for input dataset of size n“

• “the sequence of log messages is guaranteed to obey this rules of this particular finite-state automata: (connect | fail) -> (send X | fail) -> (receive Y | receive Z | fail) -> success .”

Shrinking is the process of simplifying a failed test case. If you have found some input that makes your function return a value when it should have thrown an exception, or produce a result that does not satisfy some predicate, then that input is a “counterexample” to your assertion about the properties of that function. And you may want to be able to “shrink” that counterexample input to see if you can cause the function to behave incorrectly again but with a simpler input. The “QuickCheck“ library provides a variety of useful tools to let you define property tests with shrinking.

Defining unit tests with such incredible rigor takes quite a lot of time and effort, so you would probably do not want to use property-based testing for your ordinary, every-day software engineering. If you are, for example, being scrutinized by the US Department of Government of Efficiency, you would likely be fired if you were to take so much time to write such high-quality software with such a strong guarantee of correctness.

But if you are, for example, designing a communication protocol that will be used in critical infrastructure for the next 10 or 20 years and you want to make sure the reference implementation of your protocol is without contradictions, or if you are implementing an algorithm where the mathematical properties of the algorithm fall within some proven parameters (e.g. computational complexity), property-based testing can give you a much higher degree of confidence in the correctness of your algorithm or protocol specification.

#tech #software #haskell #WellTyped #QuickCheck #UnitTesting #PropertyTesting

#PropertyTesting Pattern: Some things never change

Examples:

- Adding the same permissions to a user
- Removing a user from a group
- Changing a user preference

#PropertyTesting Pattern: There and back again

Write properties that pair an operation with its opposite, so that you get the results you started with.

- Serializing and deserializing
- Toggling preferences / settings
- Writing and deleting an object from storage
- Adding and removing a fixed value from a Number

#PropertyTesting Pattern: Different paths, same destination

If order truly doesn't matter, you should confirm that. Notable examples:

- Adding items in a cart (esp w/ & w/o sales)
- Changing user permissions
- Adding data sources

To test, generate a random set of operations to add and then check that the end state is what you expect.

If you're looking at putting #propertytesting into practice and drawing a blank once you open VSCode, this is the best article for direct advice.

https://buff.ly/4b6cPSv

I'll dive deeper into it over the week, because the examples can absolutely be expanded to services instead of just functions.

#testing

Seems like I found what I was looking for

https://github.com/vouch-opensource/fugato

#propertytesting #test #check

https://github.com/dubzzz/fast-check - FastCheck is #PropertyTesting in #TypeScript. Amazingly simple and powerful. Great project @ndubien.

https://dev.to/mbernard/what-s-wrong-with-the-current-state-of-unit-testing-18gk - a series on getting started with #PropertyTesting in #dotNET. Given a true condition (function) generate random values to validate it always is. Nice series https://github.com/mbernard.

Just realized that the implementation of the classic reversal-of-a string Property Test that virtually all the tutorials use is not that correct. Indeed,

rev rev xs == xs

happily passes for

rev xs = xs

That sucks. #propertytesting

✅ #PropertyTesting is a powerful way to detect #Bugs in code. 🎉 Opinions welcomed on my suggestion to add such tool as first-class citizen in @vitest via fast-check #JavaScript. More at https://github.com/vitest-dev/vitest/discussions/2212

We’ve been using https://github.com/flyingmutant/rapid on a large project at work recently and I like it (overall). At first I was -1, but it’s really grown on me. #golang #propertytesting #software #development #testing