BiteofanApple Archive About Code Microblog Photos Links
by Brian Schrader

TODOs as a Templating System

Posted on Mon, 31 Jul 2017 at 03:24 PM

When I sit down to start a new feature or project the blank page or empty function can be extremely intimidating; a void of infinite complexity. I'm sure lots of developers do this, and maybe most don't realize it, but I've found that TODO comments are super useful in helping to abstract away nitpicky details and focus on the overall purpose of the code as I'm writing it. Let's say that we want to validate some parameters from an HTTP request and kick off a background task to send an email to a list of requested users. First off, we need to handle the request and kick off the task, but there's a bunch of validation and database queries we need to make before we can do that, and we haven't even written the task function yet, that's where TODOs come in.

class MassEmailView(APIView)
    # TODO: check if user has permission to send mass mail
    def post(self, request):
        # TODO: Get users from the request
        users = []
        for user in users:
            # TODO: send the message
            pass
        return Response(None, status=200)

Right off the bat I know that I need to get a list of users and do something with each of them. In a lot of ways I'm basically writing pseudo-code and slowly filling in the blanks with real code. Next, let's say we write the background task.

# ---- tasks.py ----
@shared_task
def send_email(user, subject, message_text):
    email.send(user.email, subject, text=message_text)

# ---- views.py ----
class MassEmailView(APIView)
    # TODO: check if user has permission to mass send mail
    def post(self, request):
        # TODO: Get users, subject, and text from the request
        users = []
        subject = ''
        text = ''
        for user in users:
            tasks.send_email.delay(user, subject, text)
        return Response(None, status=200)

Slowly the code is coming together. I've written the background task and updated my view. The basic structure is there, but I haven't done the work of parsing the request or any error handling, so let's move on to that.

# ---- tasks.py ----
@shared_task
def send_email(user, subject, message_text):
    email.send(user.email, subject, text=message_text)

# ---- views.py ----
class MassEmailView(APIView)
    @authentication_classes((SessionAuthetication,))
    @permission_classes((IsAdmin,))
    def post(self, request):
        try:
            users = [
                User.objects.get(username=username)
                for username in request.POST['users'].split(',')
            ]
        except ObjectDoesNotExist:
            return Response(INVALID_USER_RESPONSE, status=400)

        subject = request.POST['message_text']
        text = request.POST['message_text']
        for user in users:
            tasks.send_email.delay(user, subject, text)
        return Response(None, status=200)

Now that we're done, it's clear that the TODO comments were hiding quite a bit of complexity, but the overall structure is the same. Just because our code is read by the computer from top to bottom doesn't mean we have to write it that way. Sometimes it helps to start with a rough outline of the whole picture, and slowly color it in bit by bit.

Climate Change and Guns, Germs, and Steel

Posted on Sat, 15 Jul 2017 at 01:42 PM

I've been slowly working my way through Guns, Germs, and Steel for a while now. It's an interesting read overall, but this one section really got me thinking.

What were the factors that tipped the competetive advantage away from [hunting-gathering] and towards [farming]? ...five main contributing factors can still be identified; the controversies revolve mainly around their relative importance.

One factor is the decline in the availability of wild foods. The lifestyle of hunter-gatherers has become increasingly less rewarding over the past 13,000 years, as resources on which they depended have become less abundant of even disappeared... most large mammal species became extinct in North and South America at the end of the Pleistocene, and some became extinct in Eurasia and Africa, either because of climate changes or because of the rise on skill and numbers of human hunters.

I've never heard this part of the argument before: that humans may have turned to farming because of the lack of large, huntable animals after the last ice age. It's fascinating to consider that the recent set of the Earth's cycles of natural climate change have not only been more favorable for humans to develop farming and complex civilization, but that they may have also forced our hands in the first place.1

1. And that man-made climate change could be the end of that same civilization.

Adventurer's Codex: XMPP

Posted on Tue, 11 Jul 2017 at 01:12 PM

You might have noticed in my last post in this series that I mentioned Adventurer's Codex using Ejabberd. A few of you might even be a little abhorred by the idea that a modern piece of software has even a line of XML running through it, but it's true.

But it's XML!

XMPP is an old standard these days, and I think it's safe to say that it's earned a reputation for being a very complex and stateful XML-based protocol. But XMPP has also had years of industry giants testing its limits and adding new and improved functionality to the spec. All of that means that the XMPP spec has tons of recommendations and countless extensions which can make it difficult to approach as a newbie, but it also means that nearly anything you need to do with XMPP has already been done.

In building Adventurer's Codex we wanted to leverage as many server technologies as possible with as little custom logic as we could manage. Nobody on the team wanted to write chat or pubsub functionality, or handle scaling the system with multiple nodes, and we wanted the system to be built on open technologies that support the idea of a standards based Open Web and all of the inter-compatibility that such a web entails. We looked around, and here was this perfectly good, time-tested standard just lying there, waiting for us.

Ejabberd has lots of great resources and chat rooms out there filled with people that can help you and the majority of commonly used specs like Multi-User Chat and PubSub are very well documented. The server software is robust and adaptable, and the defaults are already pretty good at getting you up and running. Perhaps most importantly, the client-side support for XMPP is really good: The web has Strophe.js and all of the native platforms have either built in support or a wealth of Open Source options for speaking XMPP, so expanding support is easy. All of these things came together to make XMPP a worthwhile choice for our needs... even if it's XML.

Adventurer's Codex: The Stack

Posted on Thu, 06 Jul 2017 at 05:28 PM

I figure that if I'm going to write a series of posts about how and why Adventurer's Codex works the way it does, it might be good to first talk about what's actually powering it. It's time to talk about our "stack".

People like to think that a project's internal technologies are chosen by seasoned professionals who carefully weighed the differences between competing options over the course of months, but from what I've seen it's usually just whatever the first developers on the project liked or already knew how to use and Adventurer's Codex is no different. Before we started we asked around for suggestions on frameworks to use, and when in doubt we used what we'd learned at work. We were not seasoned architects and we didn't spend months deciding, we were (are) anxious developers trying to bring our dream to life.

The Stack

On the client-side, we went with KnockoutJS1 and a ton of additional libraries. The trouble with the front-end world is that you essentially have to build up your own OS in the browser. Need notifications, persistence, url parsing, OAuth compatibility, or master-detail view hierarchies? Either you find one, or you build one. In our case, we built a lot. Why we chose to build out so many of our own custom libraries is a story for another time, but we do try to use existing libraries, like Signals.js, Strophe.js, and more as often as we can. But when it came to a few crucial things like Persistence and MVC structure, we went our own way.

The server-side came a lot later in the timeline2, and collectively we have far more experience with the kinds of tech involved. We took the Marco Arment strategy: use the most boring, proven software you can find. We went with Django REST, Gunicorn, Nginx, Postgres, and Ejabberd, all inside of Docker, on CentOS, on Linode. Nothing exciting there (except arguably Django REST) which is good.

Of course, if I could go back in time, there's a few things I'd change about our client-side tech stack, but knowing what we did back then, I think we made decent choices.

1. Don't email me. I know "Knockout is Dead".
2. Until version 1.4, we could get away with hosting on GitHub pages because we had no backend. The app, including it's datastorage, was (and is at time of writing) all done in-browser. More on that in another post.

Measuring Sticks

Posted on Mon, 26 Jun 2017 at 10:37 AM

Humans have a notoriously bad grasp of large numbers. We can see that 10 is 9 more than 1 and we have some feeling as to just how much "9 more" really is, but the difference between 100 and 200 isn't as easy to gauge, let alone 1,000,000 and 1,000,000,000. When I'm dealing with these kinds of huge numbers, I try to boil it down to a simple comparison between things I already know: an iPhone is worth almost 5 months of groceries for example.

The same approach helps when dealing with historical timelines. WaitButWhy did something similar a while back by using generations of people to measure history. For things on a broader timescale, I've started using a few different metrics. For example, Issac Newton died roughly before the founding of the United States1. In fact, if we use the United State's age as a measuring stick, it leads to some interesting comparisons (because it's me, all of the comparisons are going to be to Roman history):

  • The Fall of the Byzantine Empire was (roughly) twice as long ago as the founding of the United States is to today.
  • The Fall of the Western Roman Empire was over six times as long ago as the founding of the U.S.

That means that as ancient as we think the Byzantine Empire is today, we're twice as close to them, as they are to the fall of their western half! It's crazy, but using this method we can also compare historical time periods, and this is the meat of what I'm getting at with all this.

Recently, I've been reading Mary Beard's fantastic book, SPQR, and even though I've been down the rabbit hole of Roman History over four other times2 I keep finding new fascinating things, and the biggest one is always the sheer scale of the timelines involved. We never really stop to think about just how long the "ancient" world was around compared to our "modern" one. To keep this short, here's an example:

The founding of the city of Rome is, mythically, 753 BCE, and the toppling of the kings and the establishment of the republic was in 509 BCE. Using our measuring stick from before means that the age of the legendary 7 Kings of Rome was roughly the same as the age of the current United States, and that the Roman Republic (509 BCE-27 BCE) was, somewhat conveniently, exactly twice as old as the United States is today. For completeness, the Empire lasted a bit longer but was also about double the U.S.'s current age (by traditional dating of the fall of the west). All of that comes together to mean that the Roman state, from fiery birth to limping death, lasted five to nine times longer than the United States has been around, depending on which ending date you use.

Even with all that context, what's really crazy is that the height of ancient Egyptian civilization was over 14 times as long ago as the founding of the United States, or about 2.3 times as long ago as the founding of the Roman Republic. We've all seen that meme that Cleopatra was born closer to the current day than the building of the Pyramids at Giza, but it's also impressive to see just how much closer to us she really is.

Our popular retelling of the founding of the United States is already starting to become somewhat of a mythical folktale and the founding of Rome is shrouded in unsolvable mystery, so I can only imagine the kinds of stories the Romans told each other about the foundings of the even more ancient to them Egyptians, or near prehistoric kingdoms of Mesopotamia.

1. Yes, I know there's 50 years missing in there, but go with it.
2. My Roman history tours in chronological order: The History of Rome podcast, The History of Rome by Livy, The History of Byzantium podcast, and full re-listening of the History of Rome.

Adventurer's Codex: Behind the Curtain

Posted on Tue, 20 Jun 2017 at 09:52 PM

Our tale starts as many such tales often do. It was the end of July in the year two-thousand and fifteen by western reckoning and three friends met in a busy tavern in a sleepy neighborhood in San Diego. Their quest: to change the way Dungeons and Dragons would be played forever.

Almost two years ago, I started a project with a few friends that last year we turned into a full-on company: Adventurer's Codex. For no particular reason, I've never spoken about the project here, but I'm going to change that, if nothing but for posterity. Originally I wanted to write a single "What I've learned while building Adventurer's Codex" post, but it got too big and covered so many unrelated topics that I'm just going to have to make it into a series of smaller posts.

A Codex, But For Adventurers!

For those who don't know: Adventurer's Codex is a web-based toolset for playing 5th Edition Dungeons and Dragons (D&D 5e). It has a number of features for both players and DMs (Dungeon Masters, aka Game Masters) with a focus on real-time, collaborative play. I've played D&D for years, and so have the other two founders, and we all think that while the classic pen-and-paper version of D&D is great, there are some definite improvements to be made.

If you're interested in seeing what Adventurer's Codex can do, checkout the site and try it out, and if you don't know what D&D is, or you've never played, then I strongly encourage you to try it.

No Choice but to Rise to the Challenge

Software design has always been interesting to me. I've spent countless hours learning about just how lots of mature projects, like CPython, Cocoa, and iOS, work at the high level, and how their designs limit or enhance their core features. So when it came down to designing Adventurer's Codex, I jumped at the chance. The only trouble was: I'd never actually designed anything so big before; none of us had. And so, like with many projects, the architecture for Adventurer's Codex grew as we did.

When we started, I had no idea how to professionally set up servers, design modern APIs, or design multi-tier web applications. I had this website, which I was proud of, and in-and-out of my day job I'd built lots of different types of web and native software. Coming from iOS/Cocoa and Java/Spring development at the time meant that I was always accustomed to having a very large, opinionated framework to guide the design of whatever I was building, but when it comes to front-end Javascript, that's just not the case. In the browser, we were forced/free to pick and choose our own tools, libraries, and conventions and because of this, we stumbled a lot in the early days: tearing through three different data storage strategies and three more major code architecture changes. Every one of these experiences taught us something about software design, but it also slowed us down, and nearly burnt me out, but all of that is (hopefully) behind us now.

Although getting this far has been a huge undertaking, because of the way we chose to develop Adventurer's Codex, we've been able to slowly roll out bits and pieces of infrastructure over almost a year while still having a working product. If I had to give advice for anyone in the same position: ask people who've done it before. A few invaluable people at my local programming meetup group had experience with all of the things I didn't and were more than happy to point me in the right direction. The internet is great, but it's actually pretty difficult to find out how to design modern web systems from scratch with just a vague notion and Google.

TL;DR: Turns out, designing, architecting, and managing complex software is hard...

* Woo, broke the streak!

The Cloudbleed Bug: An Overview

Posted on Fri, 24 Feb 2017 at 11:06 AM

Tavis Ormandy (Chromium Bug Tracker) →

It became clear after a while we were looking at chunks of uninitialized memory interspersed with valid data. The program that this uninitialized data was coming from just happened to have the data I wanted in memory at the time...

A while later, we figured out how to reproduce the problem. It looked like that if an html page hosted behind cloudflare had a specific combination of unbalanced tags, the proxy would intersperse pages of uninitialized memory into the output...

We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major cloudflare-hosted sites from other users. Once we understood what we were seeing and the implications, we immediately stopped and contacted cloudflare security.

A tweet you never want to see.

Never a tweet you want to see.

tptacek (Hacker News) →

The crazy thing here is that the Project Zero people were joking last night about a disclosure that was going to keep everyone at work late today. And, this morning, Google announced the SHA-1 collision, which everyone (including the insiders who leaked that the SHA-1 collision was coming) thought was the big announcement. Nope. A SHA-1 collision, it turns out, is the minor security news of the day.

Cloudflare Blog →

The bug was serious because the leaked memory could contain private information and because it had been cached by search engines. We have also not discovered any evidence of malicious exploits of the bug or other reports of its existence...

We are grateful that it was found by one of the world’s top security research teams and reported to us.

This broke late last night PST, and while Travis Ormandy and the hardworking team at Cloudflare have resolved the situation, the consequences of this bug are not small. Cloudflare is a very large CDN that sits in front of tens of thousands of sites and all of them are potentially affected.

My report

List of sites affected by Cloudbleed →

Comments with Cited References

Posted on Thu, 09 Feb 2017 at 12:29 PM

A while back I got into the habit of adding links to any source code that I copy from the web. It's a small addition but it's helped me a lot when I need to go back and fix bugs long after I've forgotten what I did or why.

// From: http://bit.ly/2ltsDF6
extension Data {
    /// Create hexadecimal string representation of Data object.
    ///
    /// - returns: String representation of this Data object.
    func hexadecimal() -> String {
        return map { String(format: "%02x", $0) }
            .joined(separator: "")
    }
}

This technique is also really useful if you encounter unsupported or buggy behavior in some framework or library and you write a weird workaround or unconventional solution. In those cases I don't just document that it is a workaround, I try to link to a place that explains or tracks the bug (like the GitHub Issue or Stack Overflow page).

switch identifier {
case "SpecialWaitStep", "OtherSpecialWaitStep":
    // Wait steps aren't backward navigable.
    // https://github.com/ResearchKit/ResearchKit/issues/914
    return nil
default:
    return getStep("...")
}

This way if anyone (future me included) needs to go back to fix that section of code, they'll at least know why that hack is there.1

1. Hell, maybe by then the issue has been solved and they can even remove your hacky code.

So Many Words Written, So Many More to Come

Posted on Mon, 06 Feb 2017 at 07:58 PM

Apparently I'm 236 words short of 50,000 total words written for my blog over 161 posts1. That's about 309 words per post on average. I was playing around with the site just now and just out of curiosity I ran this:

$ find archive/ -name "*.md"|xargs -I {} cat {} | wc -w
   49764

Now curious, I did some more digging to see what I could learn; here's a few different statistics:

  • Longest post:
  • Shortest post:
  • Most average length post:
  • If we count all the drafts I've never published (but still have) the grand total goes up to 64,088 words.
    • That's 14,324 words unpublished.
  • 154 different tags used on posts
  • Most used tags:
    • blog: 21 times
    • web: 14 times
    • space: 10 times
    • open web: 10 times
    • blogging: 10 times
    • python: 8 times
  • Least used tags:
    • D&D, FCC, OS X, alternatives, angular... (and 105 more)

Just for the occasion, I made up a pretty graph.

A graph of word count.

It's hard to believe that I've been blogging for almost 5 years on this site, and if you count my two blogs before this one, then it's been almost 6 years.

And in case you're wondering: yes, this post is just over 236 words. 🎉

1 Technically that 50,000 word count includes handcrafted HTML inside a post's markdown. It's not that much of a factor and I'm defending my claim because technically I wrote that HTML so there.

An Ode to the 13 Inch MacBook Pro

Posted on Sat, 04 Feb 2017 at 01:25 PM

I got my first Mac in 2010: A base model 13" Macbook Pro. Like lots of people I got it in my first year of college with Apple's Education Discount. I had it all the way through college until 2015 when I got my current 13" Retina Macbook Pro. In that time, I'd dropped it, fixed it, swapped out the spinning platter for an SSD, added RAM, and made it my own. I learned to program on it, I played PC games with it, and I built this website with it. I used every ounce of power that 2.4GHz Core2-Duo could spare.

In short: I loved that machine.

That Macbook Pro turned me into a Mac person. I justified buying it by saying that I'd build iPhone apps with it, but that never really came to pass. I did learn about Unix and Bash though, and that knowledge changed how I used computers forever. Back then I looked forward to 2 things every year: the new iPhone OS announcement and the Mac OS X announcement. I even took time off work one year to stay home and watch WWDC talks in a time before I'd ever worked on iOS apps professionally.

In short: I found what I wanted to do because of that Mac and it changed me forever.

My mac at a donut-shop in portland

In 2015 I got my current Mac. When I passed on my old 2010 MBP it was dented, missing a foot, and had no CD drive (I took it out to save weight and then lost it). Fresh out of college I forked over literally all of the money I had for a specced out 13" rMBP. Two and a half years later I can say, it's the best computer I've ever owned or used. I've taken it everywhere, and it's risen to every challenge I've thrown at it (except modern PC gaming but I don't play a lot anyway).

Excluding a few months when I experimented with building PCs in college1 and at my old day-job, my Macs have always been my primary and only machines, and now my 2015 rMBP is my work machine too. They've both been as flexible, durable, extensible, and powerful as I could ask.

The Part About Apple Nowadays

That long preamble was more than a nice stroll down memory lane. I may not have the pedigree of some mac users but I love the Mac and over the years I've seen Apple seem to forget how much people love, use, and need the Mac to do what they do, and it makes me sad. Maybe they're working on something awesome for the Mac and I hope if they are that it's as great as it can be, but it doesn't look that way from the outside.

I could stand here and say that I wish the 13" MacBook Pro had dedicated graphics card and quad-core CPU options (and I really do wish that), but after 7 years I just don't see that happening. I guess what I want is something to prove that the Mac is still worth something to Apple. The Mac community is strong enough to last for years without Apple updating much, but that's not something I look forward to, it's a last resort.

The Circle Be Unbroken

Growing up, my mom had (and still has) a 17" Titanium Powerbook G4, and she used it for almost 10 years, she was the first Mac user in the house, and though I used them in middle and high school I'd always had a Windows PC at home. In 2009 a coworker and long-time Mac user convinced my parents to get me an iPhone, and a few months later I bought my first Mac.

Since that time I've convinced my sister, several coworkers, friends, and many complete strangers to get Macs. I've always said the price was worth it, and even though I think that's less true now, I'd still recommend the Mac over the alternative. All of those people are using Macs today because some enthusiastic college student convinced them to get one, and he got his Mac and iPhone because of a passionate Mac user who convinced him back in 2010.

That old MacBook is till alive; it's my mom's laptop now. It's worked well for her until this year: she got a GoPro and needs to edit the video. That MacBook can do a lot of things, but rendering 1080p and 4K video comfortably isn't one of them. If and when she gets a new computer, I'll probably ask for that MacBook back. There's not much I can use it for, but just like my old iPhones, it'll be a nice thing to keep around for the memories.

1. I also experimented with Linux in college, but that went from recreational use to professional use.

Archive

RSS

Creative Commons License