Bleak month, busy month.
Derivative of top-p (nucleus) sampling, with really good theoretical reasoning as to why this should work better. GPT-4 Technical Report, amazing author list format and impressive results and safety measures. A clean and nifty paper studying memorization in llms, with a notable finding that some things are memorized “right away”. Sander doing a really good explanation of diffusion models as autoencoders which definitely went over my head. Gist Tokens paper, which introduces a great technique to compress a prompt into a single token via masking (it’s so good I read it four times). Transformer Math 101.
Yay, a paper on dataset selection, but it relies on having a target distribution to mimic. Paper on poisoning datasets on the web, though it makes a strange assumption that people don’t keep their crawls and doesn’t make a significant impact. Data mixes in relation to in-context learning, though the paper is explicitly wrong about a few things. Code language mix in relation to performance, comes with some good eval and good but not super impactful results. I really enjoyed that paper because it calls Rust a “low resource” language lol.
Overview of Bing Chat mishaps. Janus on cyborgism, which was impressive in that I thought we were at the boundaries of theoretical safety work. Tangential, the hot mess theory of AI misalignment, which is well-formed though I disagree with it. Anthropic research on moral self-correction came out! Turns out too many RLHF steps actually result in discriminating in favour of minorities. OpenAI on how AI Systems should behave.
Reviewed Universal Distribution Absolute Self-Selection Assumption, from yashkaf and Paul Christiano and it's still not super deep in my head. Ben Kuhn goes to Anthropic, bringing with him tips for new managers. I love to be reminded that diet restriction for climate and welfare are inversely correlated and shrimp suffer.
The History of the Sears Catalog, I also read the catalog of course. Highlight for me was that the low-tier appliance line was called "challenge" as in "challenge toaster", "challenge waffle iron", etc. A surprisingly coherent Boot Boyz Biz article on oddly shaped buildings. BBB cards I failed to acquire for trying times.
A document that is not public and not related to AI, if we’re friends let’s talk about it — it rocked.
Short Books: Letters to a Young Poet and Preliminary Materials for a Theory of the Young-Girl
People cringed at me for reading this commonly remixed text while not being a poetry enjoyer or having read any of Rilke’s writings at all, but I just can’t enjoy poetry! Poetry is one of those things where it hits really hard at the right place and right time, and that comes much less easily to me (or you care about literature). Letters to a Young Poet was really good, my favourite part was the care in all the correspondence, I don’t think there was ever a time or person during which I exercised that care. All the remixes are attempts to provide guidance, but I’d kind of like to see attempts that are more about nostalgia, reflection and reminiscence.
Insofar as I can enjoy poetry, it’s because it sort of performs the secondary purpose of philosophy (the first of which is obviously, being correct) which is to express particular feelings felt in particular times. The Theory of the Young-Girl is certainly not correct, but it really does capture particular feelings at particular times. Every line has its own font!! It lands so well, and does a great job at capturing some particular feelings I have, including the ones that aren’t correct.
Some lines I enjoyed:
Love for the Young-Girl is just autism for two
Look on the bright side, since History’s happening on the dark side [this one really needs to typography to land]
The Young-Girl is never worried about herself, but only about her value. Thus when she encounters hatred, she is struck by doubt: Has her market value dropped?