A while back I had a free evening and a silly idea, and what resulted was an interesting exploration of the large-context behavior and analytical capabilities of modern LLMs (ChatGPT in my case). The idea was very simple:

What would happen if ChatGPT hosted a bookclub with itself?

Now, this is—of course—a very serious question with very serious implications for the trajectory of modern technology, so I set out on a quest to answer mankind's greatest curiosity.

A few hours later, I had my answer, so let's go over it, shall we?

The Why

There has long been a common notion in the Discourse that—since LLMs are trained on public internet data and since that data is increasingly polluted with AI spam content—the general quality of AI-generated results would degrade over time.

As well it's widely known that even the most top-quality LLMs today generally return banal, shallow, and overly friendly responses, even when prompted not to (in some cases because of explicit training to do so), and I was curious to see what would happen if GPT was given the opportunity to converse, long-form, over time with a disparate group of minds. I was interested to see if GPT would overcome its preference for lackluster observations when summarizing text and genuinely discover something novel. To do this, I knew I couldn't simply ask it questions about a given text in its training set, I would need to instruct it to conduct an elaborate play of personalities, each of whom would read and summarize the text in their own way.

Getting it Together

The code itself isn't very exciting. I cobbled together a little Python script that would take in a set of personality files for each of my several bookclub participants and then randomly choose between them for who was going to speak next. To keep things consistent, the script would keep a log of the dialogue as it went and submit that with each request.

As I played with the script, I made two changes, each of which significantly improved the conversation. First, I made it so that each participant would be primed by feeding ChatGPT the given personality and the chapter text, and then I asked GPT to summarize the chapter in the perspective of the given person. That summary would then be added to the given person's mind-state. Second, I added a bias in the random choice of speaker that preferred any names mentioned in the prior response—GPT often referred to other characters by name and it would be natural for that person to respond directly.

Both of these changes inspired several additional ideas for future improvements, but we'll get to that in a minute. For now, let's see what happened when the script did its thing.

Ready, Set, Bookclub!

The book I had originally lined up for this was Pride and Prejudice as it's easily available in the public domain. However that choice proved useless as ChatGPT knew too much about the book from its training set. I needed something it had rarely encountered before so that I could test its observational prowess.

Hence I chose a book I read a few years back: The Purple Cloud, by M.P. Shiel.

All that was left was to craft a series of personalities to enact my little play, and so I set to crafting some backstories and, like a director: I set the scene.

ChatGPT's Scene Instructions & Motivation

I workshopped these quite a bit, but I know there's more to do. Obviously I could have tried custom assistants, but that is discussed more below.

At first, I was pleasantly surprised. Each of GPT's personas invented motivations and expanded on their backstories, and GPT never seemed confused about who was talking and easily tracked the flow of the conversation. Characters routinely invented novel conversation topics and helped to slowly invent the character of the world around them.

However it didn't take long before things went awry.

On several attempts, one of my characters would pour wine for the group (a kind enough gesture), and the group would all toast and thank each other for coming out for the evening. However, it seems no where in GPT's training data does it contain any reference to narrative pacing or what to do in order to finish a toast, and so the conversation would inanely continue on and on as each character added more to the toast, and never once did any of them decide to actually drink. 🥂

In future attempts, I intend to add another personality to act as a sort of stage direction bot, whose only job is to add occasional scene changes that the characters would be assumed to observe. This, hopefully, would alleviate this issue and help to avoid the next one:

Once GPT established a scene, it never changed or evolved. This was surprising as over time all sorts of conversational details were being added to the chat: discussions of the book chapter, character moments, and backstory were ripe for use, but the conversations only ever got duller as they went on. Soon characters were agreeing to visit a consistently invented, newly-opened, vegan cafe with apparently fantastic croissants, but as the small talk progressed characters would endlessly revisit topics, gush pointlessly, and never add anything novel of value.

And that leads me to my most interesting observation: the conversation always died without any insight into or worthwhile analysis of the book chapter provided. Sure, characters would discuss the most superficial of elements and it was all incredibly mundane. Discussion was largely focused on the sense of foreboding mood found in the chapter or on the director's stage direction for some of the characters to be put off by the often unwieldy prose. For all it's might, GPT couldn't even be bothered to bring up specifics of the main character's arctic expedition, comment on any specific character's motivations or plot drama, or even try to predict the ending.

Overall (perceived) conversational quality.

And invariably, one of my characters would ask when the eponymous Purple Cloud would appear, to which another would reply that it "sounded ominous". GPT it seems is about as good at analyzing novel text as a high-schooler who only read the back-of-the-book blurb (read: myself in high-school).

What Does It All Mean?

In general, I was less-than-impressed with GPT's abilities in this task. As I mentioned before, it quickly became apparent that GPT could spout off vapid insights regarding the text of Pride and Prejudice even without the chapter text (I had a bug originally that prevented the text from being included in the chat log, but nevertheless GPT knew about the plot and characters), however with a less-familiar text it proved much less insightful than I expected.

In general, I'm not sure what to make of this little experiment. I'm more certain that I'd like to try it all again with several specific, technical improvements that will hopefully address some of the limitations I encountered.

Bookclub 2.0

While I haven't gotten around to implementing any of these yet, here's the list of features I came up with to improve the process:

Use Stage Direction: As I said, one of the chief limitations of this process was the fact that the scene itself never evolved. That meant that after each character had explored their backstory and current setting, there was no where to go and so they babbled on incessantly about nothing. Adding another agent whose primary job is to insert novel change would hopefully disturb the equilibrium and allow for new insights.

A dump truck drives by and outside a dog barks.

Improve each Character's Mental Model: Here I'd hope to track not only the character dialogue but improve on the priming process I mentioned before by asking (at every step) what each character thinks about the current situation and allow their own mental state to evolve beyond what is present in the spoken dialogue. Currently it's not possible for characters to have separate internal and external states which also likely flattens their complexity. As well I'd hope to include much more detail in each character's backstory and education including other materials they found interesting or recently encountered.

Consider Different LLMs: One critical limitation of this approach is that it only uses one LLM and therefore is biased toward one set of training data and methods. Perhaps if a given character were assigned a different LLM that could help matters. There's bound to be implicit baises in each model that could affect the conversational tone (and who knows! perhaps then it would be possible to have a jerk participate in the conversation).

In addition to these two improvements, I'd hope to investigate some technical changes as well. The sheer volume of tokens being submitted (including the entire chapter text and dialogue history) means that the project quickly racked up a tab. However in order to have richer character personas I'd likely need to investigate training custom assistants rather than feeding it all through the chat log API.

Anyways that's it. I don't really have a through-line for this post. I just tried a thing and thought it was worth sharing. If you have suggestions or feedback, please file an issue or shoot me an email. The project is up on GitHub if you want to play with it yourself.

It's been a while since we've run this little snippet, so let's see what we get!

$ find archive/ -name "*.md"|xargs -I {} cat {} | wc -w
124582

That means that there's roughly 124,582 total words written on this blog, not bad! That means I've managed to increase the total word count on this blog by nearly 24% in two years.

I'm not going for word count here, but it's still an interesting metric mostly because I post so much less frequently than I used to. That realization lead me to a much more interesting question about the overall length of individual blog posts over time. I posted the preliminary results over on Mastodon where I also discussed my reasoning behind the obvious trend of post length increasing over time.

In general, my posts get longer as time goes on, and you can see that even more clearly in this chart which groups posts by the year they were

The general trend for a given post's word count had been slightly upward for a long time, but the trend seems to take off in 2021 (with a huge dip in 2023 due to a lot of quick project posts). 2024 however is the clear outlier.

Post by @sonicrocketman@mastodon.social

View on Mastodon

Of course the early days of this blog were a huge outlier as well, but that's another topic. Excluding the early days, things are trending up.*

Recently, I saw this post from Kottke about so called Foundational Texts, and it got me thinking.

Writer Karen Attiah recently wrote about the pleasure of perusing other people’s personal libraries and then asked her followers what their “personal foundational texts” were…those books that people read over and over again during the course of their lives…

How about you? What are your personal foundational texts?

The question is a super interesting one. What books one reads shape their worldview. All media does this but, at least to me, books seem to hold an outsized influence. A few movies or TV shows stand out to me as being my favorites, fewer still exert a shaping force on me. Books seem to possess a more potent staying power.

The thing is, I don't really re-read books (especially non-fiction). Sure, there's a couple I've revisited over the years, but it's a very small list. That said there are select set of books that, once I'd read them, I've never gotten them out of my head.

So let's get into it (in no particular order):

• The Wizard of Earthsea by Ursula K. LeGuin
• The People vs. Democracy by Yascha Mounk
• The Scientific Revolution by Steven Shapin
• Pale Blue Dot by Carl Sagan
• The Silmarillion by J.R.R. Tolkien
• Zen and the Art of Motorcycle Maintenance by Robert M. Pirsig
• Gödel's Proof by Ernest Nagel

Most of these I read in high school and college (or right after college), which is probably why they are so foundational. What's interesting is that, looking at them now, I clearly see a set of themes that stand out to me as foundational to my thinking about the world: optimism vs. caution, systems vs. the individual, agency vs. limits. Perhaps I'll expound more on these in a future post.

Usually, when I to get interested in a topic, I dive in whole hog, and so my reading list tends to be very single-minded for a while until eventually it takes a sharp detour one day toward something usually unrelated. From 2017-2022 that topic was U.S. Politics and Government. When I'm not in that sort of research-mode, I tend to revert to either some classic American novel or Fantasy.

But this means that what "foundational" means to me is that it is heavily influential and formative as to how I think about a topic. When I think of each of the books in the list above, I can immediately recall the change it had on my thinking.

As I've mentioned before podcasts have had a huge impact on me, especially in my late high-school and early college years. Both the Writing Excuses podcast and the lectures from Astronomy 162 at Ohio State stand out in my memory as shaping my interests.

Now, to continue Jason's call to action, what about you? What are your foundational texts?

At the dawn of the 14th century the Franciscan alchemist Paul of Taranto crouched over the strange lump of metal he'd created. He gaped, in awe of what he had done. It should have been impossible. The scholars told him he was a fool to even try, yet he'd done it. It wasn't gold that he'd created. He was still far from achieving that goal, but he'd made an important step. According to the book he would later publish under the title Summa perfectionis magisterii, Paul—writing under a deliberately confusing pseudonym—had just transmuted Lead into Tin!¹

For those of us in the 21st century, it's second nature to dismiss this sort of claim as obviously ridiculous. There is no known, chemical way to do what Paul intends. But that begs a very obvious question: what had he done? If we take the man at his word, he certainly seems to have done something to his bar of lead, but what? And why did he believe his experiment had succeeded?

These questions will lead us down a very fascinating path, and one that reveals the striking truth about our knowledge of the natural world.

Theory & Practice

We take so much for granted these days about the knowledge of the natural world. We consider obvious and teach to children what took generations of the brightest minds to figure out, and it can be very easy to forget that.

Today we break apart systems to understand them; this is Reductionism. And we use material analysis to understand and manipulate the properties of physical objects: that is we melt, dissolve, chemically alter, and then recombine materials in order to create what we want. But in order to do all that—and better to make predictions about what exactly our methods will accomplish in doing so—we must accept the following assumption: that an object is nothing more than a physical assortment of indivisible components.² This assumption may seem obvious to most people today, but it wasn't always that way! Indeed most serious philosophers in the past considered the idea ludicrous.³

I won't go into the history of Classical & Medieval Matter Theory, but suffice it to say that before a pre-modern version of what we today would call Atomic Theory would emerge, the dominant view of "What is Stuff?" was far more qualitative than quantitative. For the programmers out there, think of their Matter Theory as a sort of Duck Typing.⁴

If it walks like a duck and it quacks like a duck, then it's a duck.

Matter was it's qualities. Gold is yellow, ductile, resists being tarnished. Lead tarnishes easily, it's heavy. Tin is silvery and when you bend it, it squeaks. Therefore the process of turning one metal into another is a matter of giving it the desired qualities!

Photos: mine

As at every point in history, there were differing view points about what exactly matter is made of and scholarly opinions differed greatly. However, one theory—preferred by alchemists like our old friend Paul—held that the four prime elements were bound together into a duopoly of two higher-level substances to make the metals. What were those substances? Why sulfur and mercury!

Yup, literal sulfur (or brimstone) and mercury (or quicksilver).

Now before we make fun of Paul too much (or rather Geber as he called himself in his writing), let's try to understand why he thought sulfur and mercury were the foundational elements underpinning all metals. Under this theory, the difference between say lead and gold was simply in the relative proportions of these primary ingredients!

Paul, like so many alchemists, was seemingly quite the avid experimentalist, and so based his theories on what he could determine by the fire. Among many other experiments, he noticed the sulfurous smell given off by impure metals during refining and assumed that such a smell was due to a volatile sulfur within the metal itself. Additionally, according to Dr. William Newman:

The fact that calcined [i.e. burned] metals often appear in the form of yellow, red, or white powder (what we coul call oxides) suggests to Geber that they also contain...sulfur that remains after the volatile sulfur has been forced out by calcination.
- Atoms & Alchemy by William R. Newman, p. 33

As for the mercury, that brings us back to our earlier tale. Again Newman writes:

Geber proves [his] point by washing lead with quicksilver and then melting it. whereupon the lead gains the creak that the tin had lost—as he puts it, the lead is converted to tin.
- Atoms & Alchemy by William R. Newman, p. 33

According to his theory this worked because tin had more intrinsic mercury than lead. Therefore, to transform it, one simply needed to add mercury into the lead. Sure enough, once lead is washed in this way, it squeaks when it bends. Paul had successfully imparted the quality of squeakability into his lead, thereby transforming it (albeit partly) into tin.

It might sound silly, but consider this: here was a person who formed a hypothesis about the cause of a natural phenomenon, then tested it. It worked, so he built on it. That sounds a lot like science, doesn't it?

What Even is Science?

At it's core, modern science is two things: a process of inquiry and a collection of knowledge. Together those two form a model of the natural world that we use to make predictions and offer explanations into the workings of nature. Science does not, and cannot, tell us how nature actually works under the hood, instead it gives us the tools to develop and test our models.

This point is worth belaboring, because for me, it took a long time to really click.

I studied aerospace engineering, which is basically Newton's mechanics, material design, and a lot of stuff about how fluids flow (air is a fluid, you see). When you study physics in this way, it can be tempting to believe you've learned something about nature, but what you've learned is how to model nature. There are always error terms sticking out, gaps in the theory, losses you approximate (spherical, frictionless cows for example). The math isn't nature, it's an approximation. Crucially, it's an approximation that seems to work, or at least it does within your measure of tolerance.

It's hard to remember that we don't really know how nature works. We know how to approximate it. But that fact becomes easier to understand when you approach the study of nature, not from our current perspective, but from the outside.

What's so fascinating to me about the example of poor old Geber/Jabir/Paul above is that his theories about matter were utterly wrong, and yet they did offer testable predictions that were sometimes correct! Like many scientists before and since, Paul may have very well chalked up his experimental failures to defects in apparatus, hidden variables like mineral origins, or even the incompleteness of his own theories, but nevertheless he worked with and improved his theories so to arrive at testable methods and predictions, and he wasn't the only one!

It fascinates me so much because I find myself wondering: what do we believe about nature that future generations will look back on with the same bemusement that we feel about Paul of Taranto or any other proto-scientist who's theories fell short? What about nature will become so obvious that it's taught to ten-year-olds in four centuries, but that now our brightest cannot see?

In recent years I've embarked on a sort of journey of rediscovery with respect to mathematics. As a kid I was somewhat good at math and I spent my college years doing lots of it—to the point where I grew to really dislike it. But even in those times, mathematics—while becoming increasingly difficult—always seemed like it was trying to tell me something. These days, it fascinates me.

A common observation among those who study classical engineering fields is that many (if not most) of the problem domains are modelled with very similar techniques. The motion of a cantilevered beam, a road bridge, a plane wing, a mass on a spring, a musical note, and even a planet's orbit can be described by very similar mathematics. Indeed so many physical phenomena can be described by these equations that it can make you wonder if you've stumbled into some great secret of the universe.

But that wonderful truth is only a gateway to something much deeper, more fundamental—a true capital-S Secret of the Universe. Mathematics tends to deceive with simplicity, and there's few things in this world that hide more secrets than this simplest of concepts.

Consider the Plane

As children we're taught about the 2-D Plane.¹ At first, students are confused in a manner similar to how they felt when being told was a number now, and oftentimes that confusion grows into consternation. At worst, plotting and manipulating curves on this plane feels pointless, arbitrary, and unnatural. For those who come to know the plane as a tool, it can be quite an enlightening process. As an undergrad engineering student, you become familiar with the plane and its rules; you learn to shape curves and solve perspective puzzles with it. Modern video animations make this so much more intuitive because they're able to show a figure on the plane change over time: you can watch it squish and stretch as you adjust the coefficients of a quadratic.

What's more Geometry holds so much deeper meaning in our world than I think most people imagine. Vast numbers of people can use mathematics to do every day tasks: measure the square-footage of a room, scale up a recipe for a family dinner, or plan their finances, but Geometry does so much more. Simple rules about finding the angles in a triangle can help measure the curvature of the universe, and simple sines and cosines (which form the backbone of many vector operations) enable the ability to encode the meaning of abstract linguistic concepts in modern Large Language Models. Our universe is made of math in truly fascinating ways.

But this isn't my point. So far, this has all been backstory.

Exploring the Plane

What's truly wondrous about the 2-D Plane isn't what you can plot, but what you can discover. There are infinitely many lines, curves, and shapes one can draw on the 2-D Plane and while most of them are useless, quite a few are incredible. Those are the ones we plot to find the motion of a ball in the air, or the speed of a planet's orbit over time. But there are some curves that hold incredible secrets. We know of a few and there are likely many, many more. These magical curves transform our understanding of mathematics itself.

Consider the logarithm:

This is a function that can be used to do all sorts of practical, useful operations, but it can also convert multiplication into addition.

Plotted, the logarithm function makes a curve on the 2-D Plane. This means that there exists a curve on the plane which converts multiplication to addition! What other magical curves exist with incredible properties? Perhaps there's a curve that plots itself? Oh wait! There is!

At it's core, a function is a thing that takes a number(s) and returns another. This sequence of values can be plotted to form a curve on the plane. There's a curve mapping subtraction to multiplication, one for the GDP of the U.S. over time, and (given some encoding trickery) a curve describing the lifespan in seconds of each and every being that will ever exist! They're all in there somewhere; we just have to find them.

This is one of the true wonders of mathematics², and more specifically of geometry. Who would think that simple shapes and curves could tell such great truths? Yet so much hides within the expanse. In this way, the work of finding such curves is one of exploration in a treacherous wilderness. With simple tools, one carves a path of discovery amid the endless vastness, returning to share the knowledge they have found.

The Secrets Within

Held within the depths of this infinite expanse there are perhaps infinitely many fundamental secrets. We only find them through careful study, or occasionally by accident, and we have no idea how many we might be missing—even amidst the sectors of that expanse we've so thoroughly explored.

It's so wondrous to me that such a simple concept could hold such great truths and that shapes and curves upon the plane could so accurately model the natural world. Perhaps it should be the Plane or the Triangle that stands beside Fire as our most powerful invention.

I'm fascinated by the systems by which we, as humans, make and communicate knowledge. From the very beginning of our species we've done our best to devise ways to understand the natural world and our place in it, and we've used a huge number of different systems of thought to organize, piece together, and build a more complete picture of the world for ourselves.

I think a lot of people may misunderstand my point here, so let me be very clear, when I say a system of knowledge-making I mean any system, no matter how misunderstood, how imprecise, or how flawed it might seem to modern sensibilities. Religion is a knowledge system, science is a knowledge system, oral tales and legends are a knowledge system, so is writing, music, folklore, culture, and any other way humans pass down forms of knowledge from one generation to the next.

Every system like this has its flaws and its benefits. Oral tales and folklore for example are excellent ways to pass down traditions and values, but they're terrible ways to ensure later generations understand the precise temperature at which water boils.

But in particular, I'm very interested in the places where systems of knowledge converge on similar or identical truths, even if they do so for different reasons.

An early-modern diagram of atoms. Drawing: me.

For example, an alchemist by the name of George Starkey, living in England in the 1650s, discovered a process by which he could chemically cause water to freeze via the endothermic reaction of sal ammoniac (ammonium chloride) with water, though he explained the phenomenon very differently than we would today—believing instead that he had isolated the quality of coldness and dryness by manipulating the fundamental particles of the compound. Nevertheless, his method worked (and worked well enough for his friends to try persuading him to sell "artificial ice" to Italian nobles in the summer).¹

Nothing, Null & the Void

Likewise the concept of Nothing has puzzled generations of knowledge-makers. Famously the number Zero was absent from Western mathematics until it was introduced around 1200 C.E.. Adding Nothing to a number was considered nonsense. Why even do the math problem then?

In regard to physics, Aristotle argued, rather forcefully, that Nothing (what he called the void) simply cannot exist because such an existence would be paradoxical. Aristotle viewed matter as continuous, and did not accept the idea of the "atom" (that is, a smallest particle of matter) precisely because it would beg a very obvious question: what's between the atoms? It couldn't be nothing, because Nothing couldn't exist.

Another depiction of corpuscles. Drawing: me.

This school of thought lasted for over a thousand years. Eventually though particle theory, or what was called at the time "the corpuscular theory of matter" became increasingly popular right around the time of the Scientific Revolution. All the while, the question that Aristotle posed went unanswered.

Today we know that the vacuum does indeed exist*. (Uh oh)

Indeed the classical vacuum: without air or other gross matter, exists in plenty. Most of space is empty, even most of what we call matter is empty. This is the truth of space and of matter: they are made of mostly nothing. However there are several kinds of nothing, and empty space is only one of those kinds. At the barest level, our current understanding suggests that space, even when devoid of matter or force particles, is still filled with something: namely quantum fields, and even more abstractly with potential.

And this is where the asterisk from before comes in: as far as we know, there is no patch of space that is truly devoid of these fundamental building blocks, so in a way: Aristotle was right.²

Math ado about Nothing

As I mentioned before, Western mathematics had to wait until the thirteenth century to be introduced to the concept of Zero in a formal way. Though once incorporated, Nothing, represented by the number zero, proved to be incredibly useful.

Today the concept of Nothing underpins our entire understanding of numbers. Indeed it seems one can construct all the natural numbers out of it. In this way Mathematics scoffs at poor Aristotle, for unlike matter which is mostly nothing, the counting numbers and therefore all numbers are really nothing at all!³

Nothing, it turns out, is Everything.

Consider constructing ℕ using the Empty Set
(i.e. a set containing nothing):
  0 = {}
  1 = {{}}
  2 = {{{}}}
  ...

Religious Nothing

The Abrahamic Religions (along with many other faiths) also feverishly debate the concept of Nothing. In particular, there is the idea that the Christian God created the universe "ex nihilo", from nothing, as written in the Book of Genesis. Likewise faiths from all over the world deal with the question of Nothing, though usually in the context of the question: Why is there Anything?

Creation stories, like that in Genesis, exist all over the world and come in many truly incredible forms.

All across the world, throughout time and space, humans have needed an answer to the same fundamental questions and we've found a bunch of conflicting and corroborating ways to answer them. Each of these answers have some truth to them and some methods of knowledge-making attempt to answer questions that others simply cannot. Each system is therefore one of partial truths. Science, for example, can answer the question of how matter moves, but it cannot answer what it means to exist.

I think that's what I love about studying knowledge-systems. I find myself trying to inhabit the minds of those who lived under very different presuppositions about the world, to see the universe through their eyes, and in doing so to find some new and partial truths about the world.

Nothing Is

In particular, the concept of Nothing has confounded our reasoning for millenia, and while we seem to have a good grip on it now, it's important to remember that we really only guess at its properties (if Nothing can even have properties—Aristotle, help!).

Indeed, the concept of Nothing has evolved over the past several thousand years, and even still we debate what it really means. That's something I love about questions like this: the debate will likely never end. Questions posed before the Fall of Rome still rattle around in the minds of people today and while sometimes Physics, or Mathematics, or one's Faith can make attempts to answer these questions, we may truly never know. Such truths may be beyond us.

It might be pretty out-of-vogue to host your own servers in 2024, but I still do it and I quite like it. There's a lot of great benefits to building software on basic infrastructure and I always find myself appreciating the flexibility that simple VPS hosting brings. That said, you do need to be careful. Life as a developer on the Internet requires a pretty decent familiarity with security best-practices.

Being self-hosted, there's also a lot of simple design decisions that you need to make when running a system on simple VPSs, and there's really not a lot of good info out there to help with this.

So, here's a simple guide to some of the things I do when setting up my servers.

This post isn't sponsored by Linode. It's just what I know best.

1. Follow Linode/Akamai's Great Guide for Securing Your Server.

Always set up ssh key forwarding and use it. That said, I usually end up leaving Password Authentication Enabled, since I've had problems remoting in from a known machine (i.e. my laptop isn't available). In these cases I use a very strong password (Apple's Keychain app will make up to 31 character random passwords) and synced with iCloud so it's on my phone.

2. Set Up a Cloud Firewall

These days, I prefer using the cloud firewalls provided by the platform (here's a guide for setting this up on Linode). I still do set up simple on-device firewalls on the server itself, but I depend more on the cloud firewall since it's easier to configure.

3. Configure Basic Maintenance Cronjobs

These days, I usually set up a few basic cron jobs to monitor things and keep things running. Here's a list of my typical cron jobs. This list will vary a bit depending on what kind of server you're setting up.

• Postgres nightly/weekly vacuum.
• Disk Usage & Avg CPU Usage monitoring using Nine9s Measures
• Nightly DB Backup & Archive (if needed, see 4)
• Nightly Log Trimming (keep 30 days)
• Automatic Deploys (see 5)

As an aside here, I use Pushover in a lot of my shell scripts. I send a quick API call to the service which forwards Push Notifications to my phone. That way I get notified of things like build failures which normally fail silently.

4. Set Up Automated Backups

If your platform supports it, just use the automated backups for your servers. It's so easy and well worth a few extra bucks per month. If you're not in the position to use those (e.g. Linode when wont't back up encrypted partitions), then make sure you have a cron job that backs up your database. I use simple pgdump | gzip commands in those cases.

If you do the backups yourself, be sure to test restoring from your backups as well!

5. Apps go in Containers

I use Docker, but any container system works. I've been told this setup isn't "how Docker is intended to be used", but that's fine with me. It works. I deploy my apps to my servers using a build script that usually runs automatically. It uses the production git repo for my docker compose files and runs a simple docker compose up with the configuration for production. I even have it set up to boot a backup copy of the current app while it deploys the new one so I get zero-downtime deployments.

It's probably overkill to do this, but it works well and avoids the possibility of corrupting code files on the host machine. The automatic deploy (which I frequently do nightly) is awesome because it means I never have to think about which changes have made it to production. If it's in main it's in prod.

I also run my databases (which is almost always Postgres) in a container as well, only because it makes minor upgrades easier.

6. Run Webservers on the Host

Above, I told you that I use Docker. However these days I always run the webserver (nginx in my case, though Caddy is also very nice) directly on the host. Nginx then proxy-passes to my my containers using the loopback address. That way nothing is exposed (Docker likes to punch holes in firewalls). Running nginx on the host let's certbot work as expected.

7. Set Up Break-Glass Measures

The single most popular post on this blog is about my (in)famous 8GB Empty File break-glass measure. I don't always do this anymore, but it has saved me a bunch of times. To me, SSH password auth is a kind of break-glass fallback too.

What Else?

That's all I can think of at the moment. It's a lot, but it's also not too bad. You don't need to be scared of hosting things yourself, but you do need to be careful. It's a wild world out there on the internet.

As a final note, if you're looking to get a handle on web architecture stuff, I recommend this blog post for a quick rundown on that. Practical Web Architecture is difficult to learn in my experience, and it takes a lot of pondering, trial, and error to do it right. Most resources are intended for enterprise-scale deployments and not smaller projects. In general, don't over-complicate things if you don't need to. One of the best parts about hosting your own apps this way is that you have the flexibility to change things around as you grow (at the cost of some work and likely some downtime).

I've been doing this for a while now, but I'm sure there's something I've missed. This guide is certainly not extensive, but it should at least help you avoid some major headaches. I can say that from experience.

My band, The Fourth Section has been playing more frequent shows lately and along side all of that we've been recording select songs and publishing them on YouTube.

We're well on the way to another album (though the timing is very uncertain). That said though, it feels good to be able to share these live versions of some of our new music. We've even put together a playlist of these edited versions which will continue to grow as we write and perform new material.

I've been playing in a band for nearly a decade now and while my love of writing and playing music has been a crucial release valve and creative outlet for me, one of the best parts of playing in a band is playing live. It's hard work, but it's so fulfilling.

[For nearly all of human history] in order to hear a piece of music you had to be within earshot of someone playing it.
- John Green (vlogbrothers)

Live music is something special. Unlike recorded music—which really hasn't been around all that long in human history—live music has been with us since the beginning. Regardless of form, there's something primal about being in a room where music is being played. It's a communal experience where a group of people sit and appreciate a thing together. Live music is where art meets mind.

And it's not just bands. I'm always struck by just how captivating a solo artist with a guitar can be in a live performance. I love a good acoustic song, but there's something indescribable about being in the room that just can't be replicated by the best recording. A good acoustic song recording can make me feel things, but a good live performance can enthrall me; it can make time stop.

Live music is one of many reminders that we are still very human. Even in this modern and increasingly disconnected world, we still need the same things that drove early humans in times long gone to pound a drum and sing together.

There's tremendous beauty in that, I think.

Yesterday I went hiking at Lake Cuyamaca and took some pictures. Let's look at them as we discuss something I've been wanting to write for a long time.

The Past

When this blog began in 2012 I was in the very early days of learning to code. In fact I built this site back then exactly for the purpose of teaching myself how websites (and the internet) actually worked. Back then I was in school to become an Aerospace Engineer (a subject I never really liked) and I found myself both excelling at and being interested in web development.

For the next almost-decade I hopped between jobs, built several apps and services, joined a startup, founded more than one company, and worked as a contractor. Then, as go most stories in our present era: 2020 happened.

It was a very strange time. I was unemployed for a while with nothing but my side projects to occupy my time. I built a bunch of new things—and even wrote a book about how to build things. I was perhaps the most productive developer I've ever been. To this day there are times when I wax nostalgic for those days (just without the 2020-current-events part).

But when things finally settled down I found myself working full-time, finally, for my own company: SkyRocket Software, doing software contracting. I'm decent enough at the work that I suspect I'll be fine professionally for some time, but during all the 2020/21 craziness something changed. Not just with the world, but with me.

The Present

My thirties came and with that my interests began to shift dramatically. I still love coding and I still do it for a living, but it's not something I obsess over anymore. It's not something I want to do much outside of work (though this isn't universally true). I still run my own servers and all that, but I've stopped dreaming up new projects and instead I've begun to consider shutting most of my existing ones down. As time has gone on I've even found that I care less and less about tech in general. I still exist within this world, but I don't feel anymore that I live in it. As well I feel the industry has drifted away from the things I'm personally interested in—or I'm just getting cranky and old. ¯\_(ツ)_/¯

Either way something has changed.

These days I'm far more interested in reading, writing, and music than coding. I read more books now than I have at any point in my life and I write a lot as well (yes by hand). In part this change hasn't been reflected much in this blog because I haven't wanted to write this very post explaining the shift. This is that attempt.

I've spent a fair bit of time over these past four or five years now thinking about this experience I've had, and I've kept coming back to one particular method of describing what I mean when I say something has changed.

In short, when I think about myself and what I call myself that definition has changed. I used to say I was a software developer who did a bunch of other stuff too. These days, I don't much feel like a developer. I still am of course, but I don't see myself through that lens anymore. My self-conception has changed.

The Future

This blog has always been a general-purpose place where I could write about anything that interests me, but it's always had a technical bent. That is very likely to change in the months and years to come. I have a lot of things on my mind (and a few new projects I'm working on) and while some of those are tech-related, most are not—as such the subject matter on this blog is going to reflect that shift.¹

I hope everyone reading this will stick around and see what's coming. I'm very excited about the future and I'm excited to have a space here where I'll be more willing to write about the things I'm doing these days. It should be fun.

🚀

Recently, John Green announced a now-released podcast, The Universe hosted by himself and Dr. Katie Mack, which attempts to detail the entire history, present, and future of our universe as we currently understand it. While I haven't listened to the podcast beyond the trailer, I am confident it will be quite good. But that isn't really what I wanted to talk about here.

Instead I wanted to talk about how the trailer for that podcast reminded me of my own journey with the topic of Astronomy and Astrophysics.

As a teenager, I spent a fair bit of time in my school library doing one of three things:

1. Playing games on the library computers on whatever sites were as-of-yet undiscovered (read unblocked) by the IT admin.¹
2. Reading fantasy (see Ursula Le Guin's Earthsea Series).
3. Trying to read whatever I could to answer what I thought was a pretty simple question.

What is Time?

Why I was trying to answer this question is lot to my memory, but if I recall correctly the Doctor Who episode Blink came out around that time with its famous "wibbly wobbly" description of time. And Doctor Who was my jam at the time.

Einstein famously quipped: "time is what clocks measure". Which is delightfully unhelpful to the philosophically inclined.

To answer this question I read whatever I could find on the subject including Steven Hawking's A Brief History of Time, The Universe in a Nutshell, and others. Most of what I read went over my head as you'd expect for a teenager reading books on Theoretical Physics, but books like ABHoT lead me eventually to Astronomy. Perhaps if I was going to undertand Time, I needed to understand how it began.

Introduction to Stars, Galaxies, & the Universe

I don't remember exactly how, but eventually I stumbled on the medium of Podcasts and specifically a podcast by Professor Richard Pogge at Ohio State University called Astronomy 162. The podcast is effectively just the lectures for that class which Pogge describes as astronomy for non-science majors. In other words, it's an overview without all the math. A perfect find!

I listened to that podcast (and it's predecessor Astronomy 161) over and over again in order to extract the maximum amount of information from them. I can recall sitting in the backyard of my parent's house for hours devouring these episodes. In particular, the episode that fascinated me the most (and which is the subject of the first episode of The Universe) is titled, The First Three Minutes.

I'd recommend that episode today for anyone who is curious about astronomy. The whole series of lectures is very approachable and understandable to anyone, and but The First Three Minutes is what truly gripped me. As many of my friends know, I love things like that.

An Aside

I remember discovering other podcasts by similarly forward-thinking professors and those podcasts both indulged my curiosity and augmented my own college education later on.

The Internet serving to make more college-level content available to the general public has been one of the coolest things to see happen in recent decades. And it was through podcasting that I also discovered the excellent show Writing Excuses which taught me so much about the craft of writing and continues to be a show I go back to when I need help.

Bringing it Back

The release of The Universe by John Green and Dr. Katie Mack has put me in a bit of a reminiscent mood. I'm thrilled to see a new show attempt to capture the next generation of curious minds and I hope those who listen to it find what I did in Professor Pogge's podcast nearly two decades ago. That podcast was ultimately the reason I wanted to study Astrophysics in college—something I later abandoned to study Aerospace Engineering which turned out to be decidedly less interesting to me—and to this day I am still very interested in Astronomy and Physics.

In a way, perhaps I can credit Doctor Who, Professor Pogge, Steven Hawking, and being a 6th period Library Assistant for the path I eventually took and for kindling the ongoing desire for learning that I keep to this today.

I'm not sure, but it's been fun to think about. What I can be sure of is that podcasts are incredible.