"Beware of bugs in the above code; I have only proved it correct, not tried it." – Donald Knuth
Pitting formal verification against an LLM to generate proofs seems like an obvious use case for AI, but theorem provers are not inherently robust against adversarial agents. Reward hacking theorem provers is possible.
They may use non-constructive axioms to trivialize a proof, or use backdoors like trusted tags and metaprogramming to redefine logical operations, effectively "proving" false statements.
https://www.lesswrong.com/posts/rhAPh3YzhPoBNpgHg/lies-damned-lies-and-proofs-formal-methods-are-not-slopless
#formalMethods