Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog.
Preparing for a brand new year! Spending some time thinking through what I'd like to learn, do, and build -- ideally with a why attached -- followed by how I should go about it.
Transformers -- and large models in general -- have been changing software; I'd like to be able to regularly and comfortably keep up with the state of the art, and train my own models for different purposes. Whether that means training a small model from scratch, fine tuning or applying open source models, or shrinking models to be tiny.
At the same time, building mechanical sympathy for these models seems really important: an intuitive sense of the hardware capacity required, the number of operations running a model takes. I expect I'll be writing about this and spending a lot of time here in 2024.
Of course, this needs to be complemented with actual working code, notes and projects. I expect to level up in Systems Programming, Networking (one of my biggest weaknesses), Cuda and HPC as part of this.
I frequently run into limitations because I don't have a good command over undergraduate - graduate level Maths & Physics. So I'll try and hit some good old textbooks, including Feynman (both on Computation and Physics), and classics on Math.
Information theory, Queuing theory and more advanced simulations are topics I'd like to dive particularly deeply into.
Hy has been making programming extremely fun again because I can bring together a lot of things I enjoy: Lisps, ML, iterating quickly and interactively. At the same time, the available tools have bit-rotted a bit, in progress or just don't exist yet -- nREPL, tree sitter, the emacs mode, jedhy, etc.
Once I have all of that in place, I'd like to play with some Python/Notebook experiments that have been floating around in my head for a really long time, where I mimic SmallTalk while still supporting a production Python environment. With enough elbow grease we can write code like they did in the 80s!
I'm leaning into visualizations, composable tools and also thinking about UIs at the moment: doubling down there and making it easily usable and customizable.
While writing this letter has been valuable to reflect on my week and things I've been learning, I need a better mechanism to easily add and iterate on my notes, particularly with the ability to have an LLM read them for me (though I suspect that's not going to be as valuable as having an LLM read things that I haven't read).
While I've generally kept my slipboxes public, I'll experiment with making a private one this time.
The other thing I learned was to try and use an LSP for organizing and cross linking between the notes easily.
I promised to do this during my talk at PyTorch Con '23, and I'd like to follow through on this. I've structured the code in my head several times over at this point, so it's just a matter of getting it written in a way that works.
As a summary of things I've been learning and paying attention to over the week, ideally cross-indexed with the notebox. At some point I may also tell people I've been writing them once I have a clearer purpose and structure -- including some form of pagination.
— Kunal