Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog.
A brief hiatus from writing, and some new plans: I ended up skipping a couple of weeks because I've been busy hacking on TermDex, and really wanted to rearchitect these notes into a much more useful digital garden. I also have some new hardware that I'm enjoying and exploring.
I ended up building a real extension with Go directly talking to sqlite without any third-party Go libraries; CGo made it possible to define the Go functions fairly simply and directly, and I left the template C-code as is while implementing just the pieces I needed in go.
This was fairly satisfying, but there were some initial tradeoffs I made that I didn't quite like, so I'll keep hacking on this: having to define the schema of the virtual table up front, which means I end up scraping all the files on disk up front and generating the table.
Instead I'd like to generate a vtable with JSONB blobs instead: which gives a lot more flexibility and lets me keep the table live without needing to re-create it; and leaves the option open to generate files on demand later. Reading the JSONB spec reminded me a little bit of MessagePack.
For the past several months I've been looking for a powerful ChromeBook that can run a linux virtual machine for me and I've generally been disappointed. Seeing the new Windows Copilot laptops was the impetus to give WSL2 and Windows a shot -- using a Nuphy wireless keyboard gives me a beautiful terminal with an excellent screen and a snappy computer, with the ability to sketch/use Adobe Lightroom when I want.
I'm pretty happy with my decision so far, and Windows has been better than I expected. I'm having a little bit of trouble adjusting to the keyboard shortcuts from Windows, but they do work surprisingly well.
A rabbit hole I fell down recently was trying to understand why torchrun wasn't printing the logs I expected when the underlying program was segfaulting.
The basic reason was process output buffering: if I let the subprocesses print to stdout, they flush instantly and I see the segfault output, but if it's tee'd then the output ends up being buffered by default (C stdlib behavior) and we end up losing logs. (There's also space for much better / simpler implementations of log redirection in TorchRun).
The one way I can think of around this is to use a Pty to spawn subprocesses instead, which fools them into unbuffered output irrespective of the underlying program. The way I got here was looking at Expect, PExpect -- and then finding that PExpect depends on PTYProcess. Python itself also has a much more minimal Pty module that isn't as flexible.
The part I haven't figured out yet is how to distinguish stderr and stdout coming to the subprocess -- with that, I may send out some PRs to torchrun.
Finally, I'm planning to play with Zig this weekend to experiment and explore: Go has been pretty great, and I'll still be using it at work, but I'd love to have something a little less verbose with more explicit control over how things run and work.
Someday I hope I can write my own systems-programming lisp with minimal syntax, great compatibility, and ease of use / iteration.
As I set up the new laptop, I'm also setting up configuration files in dot/2024 as I update each file. Claude has been amazing to achieve this: I had it rewrite my org mode file into a pure elisp .el file to start simplifying and flattening out my configurations.
— Kunal