Thanks for reading my newsletter. This is the first one!
I am not 100% sure what this newsletter is going to end up being, but I want to create one every week, so bear with me as I force myself to write and figure it out.
I convinced Tim Hopper to do an episode of his podcast to let me promote the Fizz Buzz book (as well as my usual pet causes) and it turned out really well. Please listen to it.
I also recorded an episode of Test and Code and of Ben Lorica’s podcast, but those are the kind of podcasts that actually have a release schedule and so on, so maybe they’ll come out in a few weeks.
We’ll do an Adversarial Learning on the topic as well, as soon as we find the time.
I’ve also been making a series of short videos to go with the book, one per chapter. The feedback is that they’re funny and interesting and well-acted, so I encourage you to check them out (and imagine an alternate history in which I went into stand-up comedy instead of math and became a sketch comic and appeared on Saturday Night Live and became good friends with Norm Macdonald).
A lot of chatter on Twitter this week about OpenAI’s GPT-3 language model, which is capable of mind-blowing feats of language generation, especially if you cherry-pick the best outputs and edit them. 🔥
OpenAI has not made the model publicly available, and as a known troublemaker I don’t have strong hopes of getting off the waiting list for the API, so I am reduced to playing with the toys that other people build that tend to stop working after I use them.
I tried asking the “Learn from Anyone” app to teach me how to solve Fizz Buzz (I’m nothing if not topical) and its response was pretty underwhelming:
When I worked at AI2 I spent a fair amount of time messing around with (and trying to break) its predecessor GPT-2, which ingrained in me a deep skepticism of the language-models-as-intelligence viewpoint. The things that GPT-2 and especially GPT-3 can do are very impressive, don’t get me wrong, but I have to see a lot more before I mentally move it out of my “party tricks” bucket.
What’s more interesting to me is an idea that I tried to capture in this joke:
One way to caricature GPT-3 is that it’s a probabilistic model of everything that has ever been written. This suggests a couple of things. (Caveat: I have not gotten to play with GPT-3, I am not an expert on language models, I am speaking well out of my depth.)
First, as a probabilistic model it represents not only things that were written but also things that weren’t written but that conceivably could have been. This line between factual and counterfactual is fascinating to me, and if anyone at OpenAI decided to grant me access to the API this is the sort of thing I would love to investigate.
Second, the more truly novel the text, the more utterly unlike anything that’s come before it is, the more surprising it ought to be to GPT-3. So, for example, GPT-3 can generate simple / canonical React components from descriptions with no problem (“create a counter”). What if you asked it to generate code to solve really novel / unusual problems? What if you asked it to generate really unusual questions? That’s the other area I would love to investigate.
To sum up my speculations, if you are someone whose creative output (code, journalism, poetry, tweets, etc) is utterly predictable (in the conventional sense), then possibly you’re in for a rough go of it as language models get better and better. The moral is clear: stay unpredictable.
One of my favorite Python libraries is Streamlit, which gives you a way of creating interactive web demos using a few lines of Python code. (Full disclosure: I am a small investor in the company, but I would not have invested if I didn’t love the product.)
A couple of weeks ago at work I finished training a machine learning model and wanted to create an interactive demo for it. Back in the day I solved this by spinning up a tiny Flask app, and then creating a janky React front-end for it. (I’m no front-end developer, but I can make janky React front-ends as good as anyone.) This was several hours of work at the least.
Now I can do mostly the same thing in Streamlit in about 10 minutes. (The first time it took me longer, but I’m pretty good at it now.)
Anyway, it’s on my mind this week because they released a new “custom components” functionality that I am dying to play with, but I haven’t had the time. (Between work, selling books, and parenting, I’m pretty overcommitted right now.) But you should check it out on my behalf.
When I want to get away from computers and Twitter, I mostly read fantasy and sci-fi novels, a few pages here and there. Another time I’ll tell you about my all-time favorite books. Today I’ll just mention what I’ve been reading recently.
The last couple of months I’ve been reading a lot of Jack Vance, first the Lyonesse trilogy, and currently the Demon Princes books. (I read Dying Earth a long time ago, I should probably revisit it.) It’s old-school and kind of pulpy and fun if you’re into those sorts of books. I have about a book-and-a-half of Demon Princes left and then I guess I have to figure out something next.
Anyway, that’s the first newsletter. Let me know what you think. Am I wrong about GPT-3? Am I reading the wrong books? Is there some other topic you hoped I’d write about? Send me a mail. And recommend my books to your friends!
Have a great week!
- Joel
I'm subscribing. This is terrific stuff.
Hi Joel! Excited to see you jump on the Substack train. I would just like to say, I think that you are right about GPT-3, but you are reading the wrong books. The right book to read is Arrive at Easterwine: The Autobiography of a Ktistec Machine, by R.A. Lafferty. It is a surprisingly up-to-the-moment take on "artificial intelligence" and computational linguistic creativity, in its own bizarre way. It was published as "pulp" in the early 1970s but is (in my humble opinion) also a great work of literature. It is out of print, but PDFs are out there (hit me up if you are interested).