Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog.

2024-04-14

A somewhat busy week.

Cleaning up dotfiles

I'm finally getting started cleaning up my dotfiles: there are a couple of decisions I've made and will be implementing as I go:

• Consolidating on $XDG_CONFIG_DIR, with all the configs arranged in .config instead of being spread out throughout my home directory. I need to check how well that works in Emacs.
• Moving away from an org-mode based emacs config: I find the cost of having to Ctrl-' into elisp much more annoying than having a cleaner org file document. If I'm particularly honest with myself, my emacs configuration is nowhere near dense enough to need literate programming.
• Use a git workspace to organize the dotfiles instead of symlinking them; the Arch Wiki delivers on making this true work really well. I wasn't excited by the idea of using stow or more scripts to pull this off.
• Lean into fzf and make it generally available. In the some token, make it very easy to bootstrap the configuration on a new machine. I find myself hopping between devservers very frequently these days, and spending an hour or two configuring and installing is not something I really enjoy anymore.
  • I mitigate this a lot by relying on my local emacs sshing into a machine for most things, but the emacs terminal isn't that good.
• Recording my .zsh_history more intentionally: so far I have a 13 million line zsh history (with duplicates) that I don't use these days, and a more reasonable 63k line history that I do at work. My personal history is only 4k lines. JM strongly recommends using version control: I generally like that approach, want to spend a lot of time maintaining a repo.
  • zsh-histdb seems pretty promising, with the added bonus that it's mainly written in shell too. The problem is that I'm pretty paranoid about this, and want something even more minimal and explicit. Potentially my first real Zig project?
  • My other pet peeve is that my shell sessions don't share the command history live, and I'm not quite sure what the safest way to achieve this is.

Zig, Rust, systems programming & Lisps

Refreshing my Rust knowledge by skimming Rust in Action: I actually seem to remember a lot, but I'll find out when I actually build the thing I'm reading this book for.

I also started playing with Zig: what I really want is a small language with minimal syntax that is still very expressive. Zig seems nice and compact and enjoyable, so I'll spend some time hacking with it.

I'm not sure why all (most?) lisps I've seen follow the Scheme / Common Lisp standards: the homoiconic syntax is decoupled from the other semantics. A small part of me wants to implement my own variant of C (at the very least, compatible with C) that has the syntax of a lisp. Probably emit LLVM IR and have hygienic macros. Then attach batteries by making it trivial to interop with C and Python and other languages.

wiki.c2.com

I frequently find myself hitting the wiki and reading through different discussions. Both the style, speed of loading, minimalism, and depth of the discussion make this particularly enjoyable.

Stanford Transformers Lecture

(Very rough notes from stanford lectures)

Why do LLMS work so well?

• Open API

• Manually inspect data

• Predict probabilities of the next word

• Basically massively multi task learning

• small vs large model -- what they learn (?)

• while overall loss improves

• individual tasks can improve suddenly while others are saturated

• some tasks may move correctly, some may show emergent behavior

• can't predict what a large model will do from smaller models

• Inverse scaling/U-shaped scaling

• Repeat a string

  • All can do

• Fix a quote repeating the same task

  • xs -- 0
  • s -- y
  • l -- y

• Plot scaling curves

Future of AI

• 10x compute every 5 years for same dollars
• Scaling is what contributes to sota research
• Bitter lesson: developing progressively more general methods & scale up
• dominant driving force
• it's easier to get into AI
• encoder - decoder, encoder only, decoder only (least structure)

— Kunal