At long last, the blog post I've been working on for what seems like forever is finished!
https://cprimozic.net/blog/growing-sparse-computational-graphs-with-rnns/
It's packed with lots of really cool stuff: ML #interpretability, #grokking, #tinygrad, #graphviz, and more
Did some #machinelearning
#benchmarks on my 7900 XTX GPU and wrote up notes: https://cprimozic.net/notes/posts/machine-learning-benchmarks-on-the-7900-xtx/
TL;DR performance on AMD GPUs is pretty bad, but it's likely a software issue.
I tested the raw hardware using some scripts from #tinygrad, and the raw FLOPS are excellent.
Speaking of #tinygrad, I've been hanging out in that Discord server for the past month or so
Very unique project. The creator and maintainer geohot is an interesting person for sure.
Lots of good ideas, and the only place I've actually seen all the way from the top to the bottom of a GPU-powered ML stack before
Not a good place to go to learn though; very much a do-it-yourself vibe and noobs aren't tolerated at all
I've been making more progress on the sparse RNN training and visualization
Working on the blog post now. Lots of cool stuff went into this from custom activation functions, custom regularizers, the new machine learning library #tinygrad, #graphviz, #webgl, and more
Here, it learned a gated 3-state state machine coupled with other neurons that perform a different boolean operation depending on the current state
Have you ever tried https://open-assistant.io/chat ? Two days ago I was listening to George Hotz at the Lex Fridman #podcast and he was talking about putting a 65b 🦙 model in the Tiny box-a PC assembled by the Tiny corp- that would run #tinygrad. I highly recommend listening to him. He ran out of #twitter, he tries to make #selfdriving affordable to everybody and he's into open sourcing #AI. Here is the #podcast -> https://youtu.be/dNrTrx42DGQ
Looking forward to seeing what they do
the tiny corp raised $5.1M
https://geohot.github.io/blog/jekyll/update/2023/05/24/the-tiny-corp-raised-5M.html
Making good progress with my visual search feature in TinyUX.
The simple neural net is stored on the device. It is now only trained on letters and numbers. Will need to train for all icons.
I use the same interface to draw content for the neural network to train on. Still quite number of hours of work ahead for creating that training content.
#neuralnet #ux #icons #pytorch #reactnative #tinygrad #tinyux
