Reflection: learning the fundamentals of AI

Like many people at the start of this year, I was blown away by ChatGPT. I had always had a strong interest in AI but this was something different. I felt like we were watching history unfold.

I read a lot of articles about how LLMs worked but it wasn't enough. How could these new tools work so well?

I decided to learn the topic deeply so that I could beyond "articles" and build AI products myself.

At the start I set myself the goal to work through the textbook (Neural Networks and Deep Learning) and be able to write a simple article that explains what I had learned and how others could follow a similar process. The result is here.

Reinforcing my learning

This mode of Alex(me) reinforcing what I had learned (ARFL) really worked for me. Having to convert ideas and mental models that lived in my head into sentences that other's could understand was a brilliant challenge. It was much harder than I anticipated.

Read the article to get the full picture for youself. This next section is just me talking to me:

I now understand neural network based AI's as very high dimensional computers that have navigated their dimensional space using calculus-driven learning algorithims. There is no magic or black boxes.

I believe in an objective-truth state at the base of the universe. So taking the final example in the textbook, there existed a configuration of the parameters + hyperparameters selected that could create a computer that could detect digits in the MNIST database with around 99.7% accuracy. The local minima already existed, it just took training to navigate to that position. It discovered the location, not created it.

Anywho, if that didn't make much sense then that's okay — these notes are for me — you have you own article!

Reflection

I am quite deterministic when I set myself a goal. I feel like I have complete write-access to my habits/routine. I had a lot going on in my personal life at the time but I still carved out time each day towards this goal. It took me almost four months to finish all the study detailed in the article, write the aritcle, and relaunch my personal site.

The final three weeks felt like a grind and that I was wasting time. Reading Paul Graham's essay on "How To Do Great Work" was a massive motivator that came at the right time. I'm not saying I've done 'great work', but I am motivated by the essay saying great work takes many years of dedicated effort and above all else requires you to be relentless in pursuing your interests. This has been one small step in that direction.

Now I am studying how LLMs work and how to build products with them.

Jemima and I spent many many hours at this park studying, writing, and chewing sticks.