Building a Personalized Newsletter with Bash and a Raspberry Pi

I use Pinboard to save articles I've read and, increasingly, to save articles I want to read. That said I rarely go back and actually read things once they disappear into the Pinboard void. This isn't an uncommon problem, I know, but I think I've devised a simple solution.

I recently set up a Raspberry Pi and mounted it under my desk. I've been playing with RPis for years, but I'd never found a recurring need for them, they've always been toys with fleeting amusement value. But this time around, I've configured it as both a local web server and Samba file share. This allows me to quickly and easily share files with the RPi and, since I configured it to send emails through my Fastmail account, it can now alert me whenever I want.

My Pinboard Weekly Newsletter

Now that everything on the RPi is set up and easily accessable, I wrote up a simple bash script to pull my most recent bookmarks from Pinboard, filter out the stuff I've already read, and draft an email with everything from the past week that I still haven't gotten to.

I've posted a simplified version on Github, but my real script isn't much more complex—all told it comes out to 55 lines of code—and it's run with a simple, weekly cron job.

Pinboard Weekly

Here's a sample of the newsletter email—and yes, my RPi's name is Demin.

Hopefully this weekly newsletter reminds me to actually go back and read the interesting news and articles I've collected during the week (or it will help remind me just how unimportant certain things really are when you've had a week to let them sit).

If you use Pinboard, and you constantly find yourself saving articles and never reading them, give my script a try. If you do, let me know what you think!

Why All My Servers Have an 8GB Empty File

Last night I was listening to the latest Under the Radar, where Marco Arment dove into nerdy detail about his recent Overcast server issues. The discussion was great, and you should listen to it, but Marco's recent server troubles were pretty similar to my own server issues from last year, and so I figured I'd share my life-hack solution for anyone out there with the same problem.

The what and where

Both hosts, Marco Arment and David Smith, run their own servers on Linode—as do I—and I found myself nodding along in solidarity with Marco as he discussed his toils during a painful database server migration. Here's the crux of what happened in Marco's own words:

The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens.

One thing Marco said hit me particularly close to home:

Server administration, when you're an indie, is very lonely.

During my major downtime problem last year, I felt incredibly isolated and frustrated. There was no one to help me and no time to spare. My site was down and it was down for a while. My problem was basically the same: my database server filled up (but for a different reason). And as Marco said, when the disk is full, nothing good happens.

In the days after I fixed my server issues, I wanted to ensure that even if things got filled up again, I would never have trouble fixing the problem.

A cheap hack? Yes. Effective? Also Yes.

On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there's very little you can do to free up large chunks of space. But what if there was a way to free up a large chunk of space on disk right when you need it most? Enter the dd command1.

As of last year, all of my servers have an 8GB empty spacer.img file that does absolutely nothing except take up space. That way in a moment of full-disk crisis I can simply delete it and buy myself some critical time to debug and fix the problem. 8GB is a significant amount of space, but storage is cheap enough these days that hoarding that much space is basically unnoticeable... until I really need it. Then it makes all the difference in the world.

That's it. That's why I keep a useless file on disk at all times: so I can one day delete it. This solution is super simple, trivial to implement, and easy to utilize. Obviously the real solution is to not fill up the database server, but as with Marco's migration woes, sometimes servers do fill up because of simple mistakes or design flaws. When that time comes, it's good to have a plan, because otherwise you're stuck with a full disk and a really bad day.

1 There are lots of tools you can use to do this besides dd. I just prefer it.

All Too Quiet

Other than my last post, it's been pretty quiet here lately. I've spent the majority of the last two months wrapping up the newest update to Pine.blog, which launched silently about a month ago. I meant to do a sort of announcement or retrospective post on the launch, but I just never got around to it.

A new IndieDevLife is coming, but I haven't had the mental bandwidth to record an episode lately. I've been spending a lot of time reading and writing policy proposals for Democracy & Progress and I've been pouring the remainder of my time into a new iOS app.

That's right, I've started yet another new app! It's in a private, early beta now, and I'm expecting the launch in May. No more details just yet, but I promise this one will be particularly special. I'm doing a lot that I've never done before, and playing with APIs I'd never heard of until now. It's fun stuff.

California, Democracy, and Progress

Last week I published the first proposal on Democracy & Progress, my new public policy blog. It's about Democracy Vouchers and how California should adopt them.

You should read the post, then please consider subscribing. The blog is in its initial launch phase, so your subscription means more than it normally would.

I've been wanting to write about policy for a long time, but I could never quite figure out the tone or the topic scope. I finally settled on the idea of discussing California politics through the lens of improving and promoting democracy. It's a big topic, and there's a lot to discuss, so I hope you'll follow along and let me know what you think.

Over the last few years, I've become pretty immersed in the policy world's conversations. I read a lot of policy books, articles and papers, I follow a lot of political writers, and I listen to a lot of politics podcasts. Over time, I started to develop my own policy outlook and then I wanted to participate in the conversation to add what I thought was a different angle on the discussion. Last year I started writing Op-Eds and publishing some in my local paper, but also I wanted to do more than that. I just couldn't figure out what my angle would be, what kinds of topics or ideas I wanted to cover, and through what lens I would cover them.

A few years back, while listening to the Ezra Klein Show, Ezra lamented that we as a society didn't spend more time focusing on local and state politics—where our time and energy is often better spent. Collectively, we don't focus on state and local politics, and yet it's only there where a lot of policy solutions can be done. That conversation stuck with me, and over time the drive to write about state politics has only gotten deeper. It was last summer, when I read All Politics is Local, that the idea for what would become D&P really started to form.

While I was scoping out the policy-blog space, I did a lot of searching around and while I found lots of medium to long-form policy blogs focusing on the national federal government (a lot of which I was already following), I didn't find a lot of the same thing at the state level. It was then I realized I had found my niche.

Politics can be a difficult thing to discuss in public, so that's why I wanted to focus on policy, not politics. Hopefully the blog can stay far away from the concerns of the day and avoid kindling a partisan fervor. At D&P, we're going to focus on solutions. California is a solidly Democratic state (the party not the governing strategy), so partisan squabbling is less of an issue—which is a blessing—but there are still plenty of difficult issues.

As part of my work for D&P, I've started compiling a list of resources for people to help them follow California politics. I know I wish I'd had something like this to get me started, so hopefully I can pay it forward.

Please consider giving D&P a follow in your favorite feed reader via RSS, on Twitter @dem_and_prog, or sign up for the newsletter. Here's to a better, more policy focused California.

Various Goings On

I've been a little scatter-brained over the past few weeks. I've started lots of little projects and finished almost none of them. Hopefully, they'll all start to wrap up soon. I mentioned on a previous Indie Dev Life that I was working on an update to the Pine.blog iOS app, and that is still true. It's coming along nicely. I think I'm about 80% done with it, but I've reached the infamous second 80% and it's become a bit of a slog, so I did what a usually do in this situation: literally anything else.

To that end, I've been experimenting with a few other projects. I've started the process of adding a dark mode to Pine.blog's Web UI, which is coming along nicely. I've also started building out Micropub support for Pine.blog. Both are features I've wanted for a long time, but never gotten to. Hopefully they'll be done around the same time as the iOS refresh. I've always struggled with putting together a coherent design for the Pine.blog web UI, and it shows. With the newest refactor of Pine.blog for iOS though, I've finally developed a coherent design and I've started using the same layout and components on both iOS and the web. It's looking really good, and hopefully this new design will last a while and bring Pine.blog up to modern standards.

On Monday the holidays are officially over and I'll force myself to finish the iOS release for Pine.blog. Usually I try to keep myself pretty focused, but for now I'll keep tinkering — it's a nice thing to do every once in a while.

Breadcrumbs and Pinboard

I use Pinboard, and I have for some time. For years, I've dumped the occasional interesting article there and then largely forgotten about it. Recently though, I've decided to really dive in and start using the service to solve a problem I routinely have.

I read a lot of articles, as I'm sure many of you do, and I often refer to them in conversation or in a blog post. When I do reference an article I like to cite it, even in conversation, but I often can't find the article anymore nor can I remember enough about it to effectively search for it. I think it's important to cite your sources, even casually, because it helps you recognize your own filter bubble, and it helps to ground you in the realm of facts. It also helps the other people you're conversing with better understand and counter your argument if they can tell where you're getting your information from. Plus, if you can find the original source, you can always verify that you're remembering the information correctly.

In an effort to forever solve this problem, I've started archiving essentially every article I read in Pinboard, and I do this whether or not I think the article was any good. This effectively turns Pinboard into a breadcrumb trail that I can use to retrace my path on the Web and hopefully dig up and verify any information that I half-remember.

A while back, I started taking notes when I read books so I can more easily remember and refer back to information I may need, but I don't want to keep a notebook for everything I read online. Enter Pinboard.

There are a few things that my system is lacking so far. Most importantly, I'm still having trouble finding some articles when I know that I've bookmarked them. It seems that either Pinboard's search isn't working how I'd expect or I'm not giving it enough metadata to search through when bookmarking articles.

Hopefully, as I get more accustomed to this new way of using Pinboard I'll drastically increase the frequency when I can recall some useful fact and also produce a worthy citation.

History, Myth, and Talking Cows

I started reading The Early History Of Rome by Livy a little over two years ago, but today I finally finished it. It's a good book and a fun and illustrative read, but there's a reason it took me so long to get through.

Livy, or Titus Livius, as he was actually known, was a Roman writer born in 64 or 59 BC, so his writing style is... strange by modern standards. I would often have to re-read multiple pages after realizing that I had no idea who was doing what. This is compounded by the fact that there are literally hundreds of names and, in a very Roman tradition, they are all incredibly similar. I'm also a fairly slow reader, so between the constant re-reading, my overall slow reading speed, and huge reading backlog, I could only finish 20 pages before getting distracted with another easier book.

That said, Livy's work is certainly worth reading if only for some of the truly amazing stories he tells. Sometimes the stories are so completely outlandish that I have to stop and remind myself that this isn't a fantasy story, it's history. Now, obviously there is myth and legend interspersed with it, and in these early histories they're effectively inseparable. After all, it's claimed —by multiple Roman authors— that the city was founded by two children nursed by a wolf, before the oldest kills the youngest, becomes king, and ascends to heaven in a cloud. It's an interesting read.

I'd like to share just one passage that I earmarked early on in the book. For context, know that the Romans were incredibly superstitious. They were constantly on the lookout for signs from the gods and they rarely did anything without performing some sacred ritual and seeking approval from the gods (see the story of the Sacred Chickens). In this passage, the consuls for the year (461 BC) had just been elected, and war would soon come to the Romans though they didn't know it yet.

The year was marked by ominous signs: fires blazed in the sky, there was a violent earthquake, and a cow talked — there was a rumor that a cow had talked the previous year, but nobody believed it: this year they did. Nor was this all: it rained lumps of meat. Thousands of birds (we are told) seized and devoured the pieces in mid-air, while what fell to the ground lay scattered about for several days without going putrid. The Sibylline Books were consulted by two officials, who found them in them the prediction that danger threatened, from 'a concourse of alien men' who might attack 'the high places of the City, with the shedding of blood'. There was also found, amongst other things, a warning to avoid factious politics.

– Livy, History of Rome, 3.10

There is so much that I love about this passage, but my absolute favorite thing is that Livy reports that a cow talked, but for some reason the first time this happened it was dismissed, and that a cow talking is apparently a warning to avoid factious politics. If that's the case, then I kinda wish a cow would talk today.

I've already purchased the second volume of Livy's work and it's on my shelf ready to go. Hopefully I can get through this one a bit quicker.

Arbitrary Achievement Unlocked

Today marks the 10th episode of my podcast Indie Dev Life, and while that might not seem like a huge milestone, it's an important one because I decided it was. When I decided to start a podcast I, arbitrarily, set a goal for myself to publish at least ten episodes. After that I would decide if I liked doing it and if I would continue. Today I achieved my arbitrary goal.

I do plan to continue the podcast. I like doing it, and it's fun to both record and edit. So far, I like both the format and the length. As is hopefully apparent, I'm trying to model IDL on _davidsmith's Developing Perspective podcast. I enjoyed that show (and it's casual format) a lot and I think a short show works well for the kind of show I'm trying to create. So, if you like the show, don't worry there's going to be more of it.

Arbitrary Deadlines & Achievements

I'm a procrastinator by nature, so I have to set, what are often completely arbitrary and unnecessary, deadlines for myself if I want to accomplish basically anything, plus I have a terrible memory. So, I tend to create task lists for myself and assign tight due dates so that I can remember what I need to do and ensure that I actually get it done. And whenever possible, I try to stick to my arbitrary deadlines no matter the work involved. If I decided something will launch tomorrow, then it will. I do the same with arbitrary goals.

I'm not sure why my brain finds it so much easier to work under a deadline, but it does. Even when I know the deadline is meaningless, simply having one is enough to trick my brain into thinking it's real. The procrastinator in me will always put off work until the last minute (unless it's something I enjoy playing with) so I often make my self-imposed deadlines tighter than they probably should be.

I do the same thing with setting achievements for myself. When I released Going Indie, I set a series of goals for what I'd consider a successful release. My expectations being what they are, I tend to make my first goal incredibly low so that I'm basically guaranteed to hit it. For Going Indie, my first goal was to sell 10 copies, and thanks to you all, I sold many more than that. After the first goal, the rest follow a pretty simple format:

  • What is the bare minimum I'd consider successful?
  • What would I be happy with?
  • What would I want to have happen?
  • What would I dream of happening?
  • What is basically impossible?

This tiered structure helps me understand myself and my own expectations. It also keeps me motivated. If I say I'd be happy with something, and set my expectations accordingly, then when I achieve it, I feel good. Without this structure, I tend to trap myself into believing that my wildest dream scenario is just as likely as the rest and then I'm disappointed when that dream doesn't come true.

I say this in Going Indie, but managing your own expectations is incredibly important, not only to your own motivation, but to your mental health. If you constantly convince yourself that an idea or project is going to make it big, and then it doesn't, you're going to get discouraged. But if you set reasonable goals, and learn to manage your own expectations (of yourself, your capabilities, and your results) then it's a lot easier to appreciate the successes you do achieve and keep going to bigger and better things.

tl;dr I made 10 episodes of my podcast, and I want to make more.

A False Dichotomy

An editorial of mine was published today in a local paper. It's the first in a short series about America's future and how we can work together to solve our most pressing problems, and like most things these days this first essay is about the COVID-19 pandemic.

COVID-19 has disrupted our entire way of life and it seems like that will continue for longer than we would like or hope. The race to a vaccine and mass inoculation is ongoing, but it will take time to see the fruits of that labor. In the meantime, we are not helpless...our current situation is not unsalvageable.

The article ends with a phrase I've repeated over and over again since March.

In short, the current choice between a forced reopening during a pandemic or a safe but devastating shutdown is a false dichotomy — one created by a refusal to help fellow Americans during a crisis.

So far, aside from modest measures (one-time checks, boosting UI insurance, and SBA loans) the federal government has yet to fully respond to the pandemic. Many other nations shut down their economies, but they've also provided aid to individuals and businesses so that they can weather those shutdowns and emerge, raring to go, when they're over. Instead of doing that, our federal government has refused to help states and local governments with revenue shortfalls, failed to provide adequate financial aid to individuals and businesses in need, and overall acted as if the pandemic is nothing to worry about even though, at time of writing almost 300,000 Americans have died of the disease. Instead, the President and Senate Republicans have simply insisted that their hands are tied, that the incredible capabilities of the most powerful nation on earth simply can't be used to help its people.

A lot of this disfunction lies at the feet of Republicans in the Senate. To it's credit, the House passed a new round of pandemic relief months ago, but the Senate never even considered it, and Joe Biden's sizable pandemic plan is probably subject to the same fate.

In an Op-Ed for the New York Times this week, David Brooks wrote about the most recent coronavirus aid talks and Senate Majority Leader Mitch McConnell's efforts to curtail them.

To their great credit, Pelosi and Chuck Schumer embraced the bipartisan framework. Mitch McConnell went on the Senate floor Thursday, pretended to soften, ignored the compromise and did not move an inch.

If [we don’t see a Covid-19 relief measure pass in the next week or two], McConnell should spend Christmas with people thrown out of work and witness the suffering he has caused.

This is what has infuriated me for months. We have a solution to this pandemic. We know how to curtail its spread: shutdown, pay people to shutdown, and encourage masks and distancing. Then and only then: reopen safely. We could open schools while keeping bars and restaurants closed if those businesses could afford it, and we can make sure they could. Instead we choose not to. This dichotomy: reopen unsafely or suffer safely is a false choice. There are plenty of better options, but we won't take them. Instead we chose this route.

I'll leave you with an excerpt from an article I wrote but never published.

..we can't control the virus, we can only mitigate its spread. We can, however, directly effect the economy. We could have built up our institutions, we could have helped those that lost their jobs, we could have woven a net to catch those who fall. We chose not to, but we still can.

Why I Don't Retweet Anything

I still read Twitter (despite my efforts to quit entirely), but I've stopped retweeting. My last retweet was on July 17th and it was of a tweet I wrote for a company I co-own. And if things continue as planned, that will be the last retweet I ever do.

Why No Retweets?

Retweets are Twitter's original sin. Akin to Facebook's Share, or Tumblr's Re-blog, the Retweet is what allows content on Twitter to spread virally. It allows for users to effortlessly spread false or misleading information with impunity and encourages readers to never go past the title of an article (despite Twitter's admittedly laudable attempts to address this issue). Retweets allow users to easily dunk on each other and act as super-spreaders of identity-reenforcing and tribalistic memes that only serve to make Twitter a worse platform.

Quote retweets are a little better (at least they require the user to express some opinion about the content they're spreading), but they still allow for the easy dunking that's a mainstay of the platform.

To their credit, Twitter has long known that the retweet button causes problems on their platform. The original designer of the feature has even said as much.

Chris Wetherell:

“We might have just handed a 4-year-old a loaded weapon,” Wetherell recalled thinking as he watched the first Twitter mob use the tool he created. “That’s what I think we actually did.”

Both Pine.blog and Micro.blog don't have retweets, and Manton does a great job explaining why (I used the same reasoning for Pine.blog).1

Manton Reece:

When you have to put a little work into posting, you take it more seriously. I wonder if fake news would have spread so quickly on Facebook if it was a little more difficult to share an article before you’ve read more than the headline.

Putting in the extra effort to actually articulate an intelligible thought helps me better understand the content myself and it helps me gauge whether or not the given post is even worth sharing. On more than one occasion, I've stopped myself from sharing something because I realized that it just wasn't all that interesting.

How I (Still) Use Twitter

While I still read Twitter, the vast majority of my posts are cross-posted from my microblog. I still favorite posts (because my favorites are automatically saved to Pinboard so I can find them later), but for the most part, my interactions are read-only.

Because I spend most of my "social media time" either reading articles on Pine.blog or browsing Micro.blog, reading Twitter now feels like wading through a nuance-lacking, toxic cesspit rather than acccessing the real-time information platform I once knew and loved, which in turn drives me further away from the platform. I see people I follow there cooking amazing meals, telling fanciful stories, sharing hilarious memes, but all of that is drowned out by the hate and vitriol in every next post.

With all that said though, I do still share stuff on the Web. I just do it by writing a post.

A sample pseudo-retweet
I'd love a browser plugin that removes the Retweet button and hides all retweets. It seems like a few exist, but none that I tried worked. Presumably Twitter tries to circumvent them.

Virality is a fact of life on the Web; neither innately good nor bad. For too long we've seen it as something to be harnessed, to utilize. But a virus is incredibly difficult to control and, like with today's real-world troubles, a pandemic of memes and satire does enormous harm to our individual health and to the health of our society.2

tl;dr Pandemics are bad. Both online and in meatspace. We should limit their spread and stop retweeting.

1 Pine.blog will show you when someone retweets or re-posts your blog posts if you allow for Webmentions, though I'm considering letting users configure this behavior. Retweets also poison the mind of the poster, not just the reader.
2 In an effort to be transparent, I have asked others to retweet my posts before an an attempt to gain some traction outside my bubble. I'm going to stop doing that too.
RSS