Here's what you might get if you tell GPT3
"Prove or disprove: there exists a nowhere continuous function f whose absolute value is everywhere continuous."
It gave an answer in TeX, which someone typeset.
Because it looks like math, this answer is superficially persuasive. But it's complete bullshit. I soon found three fatal mistakes. For some reason I found the simplest one last.
First I became suspicious of the idea of finding infinitely many disjoint open intervals whose union is the real line. In fact this is impossible! Every open subset of ℝ is a disjoint union of countably many disjoint open intervals IN A UNIQUE WAY, and if that open subset is ℝ itself, that way uses just one open interval: (−∞, ∞).
That was an interesting mistake.
Then I noticed the claim that "|f| is bounded hence continuous everywhere", with no other justification. This is baloney.
Then I actually thought about the function f. Since f = |f|, there's no way |f| but not f can be continuous!
There are probably other mistakes but at that point I lost interest.
Is the moral that chatbots are hopeless at proving things?
No, GPT4 answered the same question quite well. It found the obvious example of a function that works: f(x) = 1 when x is rational and f(x) = -1 when x is irrational. It seems to have correctly proved that f is discontinuous everywhere. And it even acted like a typical mathematician, saying |f| is "obviously continuous everywhere, since it is constant".
Details:
https://nostalgebraist.tumblr.com/post/711802556830089216/update-tried-the-second-example-with-gpt-4-via