BiteofanApple Archive About Code Microblog Photos Links
by Brian Schrader

todolist update

Posted on Tue, 10 Oct 2017 at 03:35 PM

Over the last week, I've made some changes to my todolist script: I've cleaned up the printing a bit and removed the temp file.

todolist terminal output

I had to remove the temp file because it was actually causing performance problems with BBEdit. Since the temp file came into and out of existence every few seconds, BBEdit's project view would dutifully redraw the project file list twice in quick succession, wasting quite a bit of CPU power1, and sometimes causing my MacBook's fan to spin up. Now that the temp file is gone, that problem is too.

1. I'm not sure why BBEdit needs so much power to redraw the project list and I've reached out to their support. Hopefully they can resolve the issue. At the very least my script is a little more well behaved so it's no longer an issue.

todolist →

Mini-Rant About Documentation

Posted on Mon, 09 Oct 2017 at 03:31 PM

I want to talk about documentation. iOS1, Nginx, Python, DRF, Django, Celery, and Postgres have excellent documentation, but documentation only helps when your question is "How does this thing work and what does it do?" Documentation, at least code level docs, are useless when it comes to figuring out what you need in the first place. Celery can tell you how to use Celery, but it isn't as great at telling you why you might need it. I've become convinced that user guides are as, if not more, important than code level documentation, and we as a community need more of them.

1. For their credit, iOS and really all of Apple's developer resources have excellent user guides that explain not only how to use a thing, but why and where you might need it (thinking about it, this could be because iOS and macOS have been around long enough to develop these kinds of docs).

todolist

Posted on Thu, 28 Sep 2017 at 02:28 PM

I've talked before about how I use TODO comments in my code to lay out what I want to do before actually doing it. To help me keep track of all of these TODOs in my code I wrote a little script yesterday and I've put it on Github for anyone who's interested.

The script looks through all of the code (by default Python code) in a given destination directory, greps for the TODO comments, and prints them nicely in a constantly updated list in the terminal. The output looks like this:

Todolist Terminal Window

Writing this script I learned a couple of new things about terminal commands like how to clear the screen without deleting the scrollback or just printing newlines (i.e. what clear does). I've put the script in my /usr/local/bin and called it todolist so now I can invoke it from anywhere and get a nice little list of what I've put off working on.

todolist on Github →

Accidental DevOps

Posted on Tue, 26 Sep 2017 at 12:30 PM

Since I became a developer, I've always worked on small (3-5) or single-person teams. Even at my current job, I'm the the lead and only full-time developer. In more recent projects (including Adventurer's Codex) this means that I'm the DevOps guy and System Admin as well. I'm by no means an expert in either, but I can do both.

I started learning how to manage and administer servers when I started this site back in 2012. Back then I never thought that all of those hours spent configuring Apache and PHP would lead to anything, but those countless hours of frustration taught me the basics. Fast-forward 5 years and I'm developing three major projects (two unannounced) and I'm DevOps and SysAdmin for all three. It's crazy to think about.

I'd highly recommend any new developers to follow the same general path I did: start a project or blog and learn to deploy it yourself. I started with a cheap old-style webhost and FTP, and slowly moved to managing the whole stack on Linode. I'm using Docker on new projects, and for now, I'm scripting my own deploys (though this could change soon if I migrate one project to Ansible).

As developers it's sometimes easy to forget that we write software that actually runs on some actual hardware in some actual datacenter somewhere. Knowing how to do many of the things that DevOps and SysAdmins do will not only make you a better developer, it gives you the ability to do more on your own. You often don't need tons of layers of software to deploy yours if you know how to do it from the ground up (especially if it's a smaller project). Those tools make it easier sure, but they're not required.

Getting Back on the Horse

Posted on Mon, 25 Sep 2017 at 04:20 PM

It's been a while since my last post. This whole summer has been an unusually quiet here, and while a number of personal issues have cropped up this summer that derailed me from blogging, I've really just gotten out of the habit of writing regularly. This is me forcing myself back onto that horse. I've got some exciting news coming, and between work and Adventurer's Codex, I've been keeping myself way too busy.

On the Adventurer's Codex front: we're in the middle of a large refactor caused by our ongoing migration to Webpack which should hopefully fix a bug caused by our fairly primitive current build and deploy system. We're starting to see the light at the end of the tunnel now, and hopefully it shouldn't take much longer before we're back to writing new, cool features. The original build and deploy system was basically a lot of manual work and a shell script. When we wrote it, I had no idea what Webpack was or how to deploy a modern front-end app, now I do. That's what happens when you learn on the go; sometimes you have to step back and fix your past mistakes.

In my dwindling spare time, I've been working on another project that I hope to announce soon, so look for more to come there.

TODOs as a Templating System

Posted on Mon, 31 Jul 2017 at 03:24 PM

When I sit down to start a new feature or project the blank page or empty function can be extremely intimidating; a void of infinite complexity. I'm sure lots of developers do this, and maybe most don't realize it, but I've found that TODO comments are super useful in helping to abstract away nitpicky details and focus on the overall purpose of the code as I'm writing it. Let's say that we want to validate some parameters from an HTTP request and kick off a background task to send an email to a list of requested users. First off, we need to handle the request and kick off the task, but there's a bunch of validation and database queries we need to make before we can do that, and we haven't even written the task function yet, that's where TODOs come in.

class MassEmailView(APIView)
    # TODO: check if user has permission to send mass mail
    def post(self, request):
        # TODO: Get users from the request
        users = []
        for user in users:
            # TODO: send the message
            pass
        return Response(None, status=200)

Right off the bat I know that I need to get a list of users and do something with each of them. In a lot of ways I'm basically writing pseudo-code and slowly filling in the blanks with real code. Next, let's say we write the background task.

# ---- tasks.py ----
@shared_task
def send_email(user, subject, message_text):
    email.send(user.email, subject, text=message_text)

# ---- views.py ----
class MassEmailView(APIView)
    # TODO: check if user has permission to mass send mail
    def post(self, request):
        # TODO: Get users, subject, and text from the request
        users = []
        subject = ''
        text = ''
        for user in users:
            tasks.send_email.delay(user, subject, text)
        return Response(None, status=200)

Slowly the code is coming together. I've written the background task and updated my view. The basic structure is there, but I haven't done the work of parsing the request or any error handling, so let's move on to that.

# ---- tasks.py ----
@shared_task
def send_email(user, subject, message_text):
    email.send(user.email, subject, text=message_text)

# ---- views.py ----
class MassEmailView(APIView)
    @authentication_classes((SessionAuthetication,))
    @permission_classes((IsAdmin,))
    def post(self, request):
        try:
            users = [
                User.objects.get(username=username)
                for username in request.POST['users'].split(',')
            ]
        except ObjectDoesNotExist:
            return Response(INVALID_USER_RESPONSE, status=400)

        subject = request.POST['message_text']
        text = request.POST['message_text']
        for user in users:
            tasks.send_email.delay(user, subject, text)
        return Response(None, status=200)

Now that we're done, it's clear that the TODO comments were hiding quite a bit of complexity, but the overall structure is the same. Just because our code is read by the computer from top to bottom doesn't mean we have to write it that way. Sometimes it helps to start with a rough outline of the whole picture, and slowly color it in bit by bit.

Climate Change and Guns, Germs, and Steel

Posted on Sat, 15 Jul 2017 at 01:42 PM

I've been slowly working my way through Guns, Germs, and Steel for a while now. It's an interesting read overall, but this one section really got me thinking.

What were the factors that tipped the competetive advantage away from [hunting-gathering] and towards [farming]? ...five main contributing factors can still be identified; the controversies revolve mainly around their relative importance.

One factor is the decline in the availability of wild foods. The lifestyle of hunter-gatherers has become increasingly less rewarding over the past 13,000 years, as resources on which they depended have become less abundant of even disappeared... most large mammal species became extinct in North and South America at the end of the Pleistocene, and some became extinct in Eurasia and Africa, either because of climate changes or because of the rise on skill and numbers of human hunters.

I've never heard this part of the argument before: that humans may have turned to farming because of the lack of large, huntable animals after the last ice age. It's fascinating to consider that the recent set of the Earth's cycles of natural climate change have not only been more favorable for humans to develop farming and complex civilization, but that they may have also forced our hands in the first place.1

1. And that man-made climate change could be the end of that same civilization.

Adventurer's Codex: XMPP

Posted on Tue, 11 Jul 2017 at 01:12 PM

You might have noticed in my last post in this series that I mentioned Adventurer's Codex using Ejabberd. A few of you might even be a little abhorred by the idea that a modern piece of software has even a line of XML running through it, but it's true.

But it's XML!

XMPP is an old standard these days, and I think it's safe to say that it's earned a reputation for being a very complex and stateful XML-based protocol. But XMPP has also had years of industry giants testing its limits and adding new and improved functionality to the spec. All of that means that the XMPP spec has tons of recommendations and countless extensions which can make it difficult to approach as a newbie, but it also means that nearly anything you need to do with XMPP has already been done.

In building Adventurer's Codex we wanted to leverage as many server technologies as possible with as little custom logic as we could manage. Nobody on the team wanted to write chat or pubsub functionality, or handle scaling the system with multiple nodes, and we wanted the system to be built on open technologies that support the idea of a standards based Open Web and all of the inter-compatibility that such a web entails. We looked around, and here was this perfectly good, time-tested standard just lying there, waiting for us.

Ejabberd has lots of great resources and chat rooms out there filled with people that can help you and the majority of commonly used specs like Multi-User Chat and PubSub are very well documented. The server software is robust and adaptable, and the defaults are already pretty good at getting you up and running. Perhaps most importantly, the client-side support for XMPP is really good: The web has Strophe.js and all of the native platforms have either built in support or a wealth of Open Source options for speaking XMPP, so expanding support is easy. All of these things came together to make XMPP a worthwhile choice for our needs... even if it's XML.

Adventurer's Codex: The Stack

Posted on Thu, 06 Jul 2017 at 05:28 PM

I figure that if I'm going to write a series of posts about how and why Adventurer's Codex works the way it does, it might be good to first talk about what's actually powering it. It's time to talk about our "stack".

People like to think that a project's internal technologies are chosen by seasoned professionals who carefully weighed the differences between competing options over the course of months, but from what I've seen it's usually just whatever the first developers on the project liked or already knew how to use and Adventurer's Codex is no different. Before we started we asked around for suggestions on frameworks to use, and when in doubt we used what we'd learned at work. We were not seasoned architects and we didn't spend months deciding, we were (are) anxious developers trying to bring our dream to life.

The Stack

On the client-side, we went with KnockoutJS1 and a ton of additional libraries. The trouble with the front-end world is that you essentially have to build up your own OS in the browser. Need notifications, persistence, url parsing, OAuth compatibility, or master-detail view hierarchies? Either you find one, or you build one. In our case, we built a lot. Why we chose to build out so many of our own custom libraries is a story for another time, but we do try to use existing libraries, like Signals.js, Strophe.js, and more as often as we can. But when it came to a few crucial things like Persistence and MVC structure, we went our own way.

The server-side came a lot later in the timeline2, and collectively we have far more experience with the kinds of tech involved. We took the Marco Arment strategy: use the most boring, proven software you can find. We went with Django REST, Gunicorn, Nginx, Postgres, and Ejabberd, all inside of Docker, on CentOS, on Linode. Nothing exciting there (except arguably Django REST) which is good.

Of course, if I could go back in time, there's a few things I'd change about our client-side tech stack, but knowing what we did back then, I think we made decent choices.

1. Don't email me. I know "Knockout is Dead".
2. Until version 1.4, we could get away with hosting on GitHub pages because we had no backend. The app, including it's datastorage, was (and is at time of writing) all done in-browser. More on that in another post.

Measuring Sticks

Posted on Mon, 26 Jun 2017 at 10:37 AM

Humans have a notoriously bad grasp of large numbers. We can see that 10 is 9 more than 1 and we have some feeling as to just how much "9 more" really is, but the difference between 100 and 200 isn't as easy to gauge, let alone 1,000,000 and 1,000,000,000. When I'm dealing with these kinds of huge numbers, I try to boil it down to a simple comparison between things I already know: an iPhone is worth almost 5 months of groceries for example.

The same approach helps when dealing with historical timelines. WaitButWhy did something similar a while back by using generations of people to measure history. For things on a broader timescale, I've started using a few different metrics. For example, Issac Newton died roughly before the founding of the United States1. In fact, if we use the United State's age as a measuring stick, it leads to some interesting comparisons (because it's me, all of the comparisons are going to be to Roman history):

  • The Fall of the Byzantine Empire was (roughly) twice as long ago as the founding of the United States is to today.
  • The Fall of the Western Roman Empire was over six times as long ago as the founding of the U.S.

That means that as ancient as we think the Byzantine Empire is today, we're twice as close to them, as they are to the fall of their western half! It's crazy, but using this method we can also compare historical time periods, and this is the meat of what I'm getting at with all this.

Recently, I've been reading Mary Beard's fantastic book, SPQR, and even though I've been down the rabbit hole of Roman History over four other times2 I keep finding new fascinating things, and the biggest one is always the sheer scale of the timelines involved. We never really stop to think about just how long the "ancient" world was around compared to our "modern" one. To keep this short, here's an example:

The founding of the city of Rome is, mythically, 753 BCE, and the toppling of the kings and the establishment of the republic was in 509 BCE. Using our measuring stick from before means that the age of the legendary 7 Kings of Rome was roughly the same as the age of the current United States, and that the Roman Republic (509 BCE-27 BCE) was, somewhat conveniently, exactly twice as old as the United States is today. For completeness, the Empire lasted a bit longer but was also about double the U.S.'s current age (by traditional dating of the fall of the west). All of that comes together to mean that the Roman state, from fiery birth to limping death, lasted five to nine times longer than the United States has been around, depending on which ending date you use.

Even with all that context, what's really crazy is that the height of ancient Egyptian civilization was over 14 times as long ago as the founding of the United States, or about 2.3 times as long ago as the founding of the Roman Republic. We've all seen that meme that Cleopatra was born closer to the current day than the building of the Pyramids at Giza, but it's also impressive to see just how much closer to us she really is.

Our popular retelling of the founding of the United States is already starting to become somewhat of a mythical folktale and the founding of Rome is shrouded in unsolvable mystery, so I can only imagine the kinds of stories the Romans told each other about the foundings of the even more ancient to them Egyptians, or near prehistoric kingdoms of Mesopotamia.

1. Yes, I know there's 50 years missing in there, but go with it.
2. My Roman history tours in chronological order: The History of Rome podcast, The History of Rome by Livy, The History of Byzantium podcast, and full re-listening of the History of Rome.

Archive

RSS

Creative Commons License