Automate the Boring Stuff with ChatGPT

There's been a lot of buzz around ChatGPT lately and most recently the discussion followed Tom Scott's video in which he describes how he was able to use ChatGPT to generate working code to solve a problem—including asking it to fix several bugs with its own implementation—and then write an ad spot for him.

In the video Tom expresses his disquiet with the ongoing advances in machine learning and goes so far as to predict that "everything is about to change" in the same way "everything changed" once smartphones came along in the mid-aughts.

I've enjoyed using text-to-image AIs like Midjourney for a while now, but I'd largely stayed clear of things like ChatGPT. Tom's video though got me curious, and so I logged on to see what all the fuss was about.

tl;dr I was pleasantly surprised.

Introducing DeveloperGPT

There are so many things that developers do day-to-day that are at the same time important and dreadfully dull, and like many devs I find myself searching for ways to both save time and automate away as much menial work as I can. It's one of the reasons I love Bash so much. And while at some companies such work is often pawned off to junior developers in need of experience, as a one-man shop I have definite use for a sort of algorithmic junior developer who could take on some of that less-engaging work. And after watching Tom's video I started to hope that perhaps ChatGPT could be exactly that.

In order to see if ChatGPT could fill this role, I did what anyone looking to hire a junior developer would do: I gave it an interview.

Giving ChatGPT a Coding Interview

In total I gave ChatGPT five questions to answer. These questions were almost entirely development related and spanned the range of what I'd expect a junior developer to be able to do given time and access to the internet.

from django.urls import reverse
from rest_framework.test import APITestCase
from rest_framework import status
from .models import User

class UserTests(APITestCase):
    def test_create_user(self):
        """
        Ensure we can create a new user object.
        """
        url = reverse('user-list')
        data = {'username': 'testuser', 'email': 'test@example.com'}
        response = self.client.post(url, data, format='json')
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        self.assertEqual(User.objects.count(), 1)
        self.assertEqual(User.objects.get().username, 'testuser')

        # ... (mine: omitted for brevity) ...

The first test I gave ChatGPT was simply to write a suite of unit tests to verify that a Django User REST API allowed for standard CRUD operations. In general it did very well, even including a test for partial updates via the PATCH method, but it did assume I also wanted it to use Django REST Framework and its implementation of the PUT and POST tests don't include all the correct values. The assumption to use DRF is a fair one, but the other mistake is a red flag.

Still though it would have saved me from writing the bulk of the code—assuming it works. I didn't actually test the code.

    import boto3
    import csv

    def delete_files_from_s3(file_path):
        # Initialize S3 client
        s3 = boto3.client('s3')

        # Read the CSV file containing file names
        with open(file_path, 'r') as f:
            reader = csv.reader(f)
            files = list(reader)

        # Delete each file from S3
        deleted_files = []
        failed_files = []
        for file in files:
            try:
                s3.delete_object(Bucket='your-bucket-name', Key=file[0])
                deleted_files.append(file[0])
            except Exception as e:
                failed_files.append(file[0])
                print(f"Failed to delete {file[0]}: {e}")

        # Return the list of deleted and failed files
        return deleted_files, failed_files

    # Test case 1: All files are deleted successfully
    deleted_files, failed_files = delete_files_from_s3('file_list.csv')
    assert len(deleted_files) == 5
    assert len(failed_files) == 0

    # Test case 2: Some files cannot be deleted
    deleted_files, failed_files = delete_files_from_s3('file_list_with_errors.csv')
    assert len(deleted_files) == 3
    assert len(failed_files) == 2

    # Test case 3: CSV file does not exist
    deleted_files, failed_files = delete_files_from_s3('nonexistent_file.csv')
    assert len(deleted_files) == 0
    assert len(failed_files) == 0

The second test involved writing a simple Python function to clean up stray files in S3 using a list provided by a CSV file. ChatGPT scored pretty well here. It included the asked-for error handling and API interface, but its tests are not correct. The third test especially would crash when the file was opened, but the script does not account for that.

Once again I didn't run the code, but assuming the use of boto3 is anywhere close to the actual API, it was faster to have ChatGPT write the code than for me to do it.

import React from 'react';
import { Container, Row, Col, ListGroup, Card } from 'react-bootstrap';
    function AppointmentList({ appointments }) {
      return (
        <Container>
          <Row>
            <Col sm={4}>
              <ListGroup>
                {appointments.map((appointment) => (
                  <ListGroup.Item key={appointment.id}>
                    {appointment.title}
                  </ListGroup.Item>
                ))}
              </ListGroup>
            </Col>
            <Col sm={8}>
              <Card>
                <Card.Body>
                  <Card.Title>Appointment Details</Card.Title>
                  <Card.Subtitle className="mb-2 text-muted">Title</Card.Subtitle>
                  <Card.Text>
                    Lorem ipsum dolor sit ...
                  </Card.Text>
                  <Card.Subtitle className="mb-2 text-muted">Date</Card.Subtitle>
                  <Card.Text>
                    January 1, 2023
                  </Card.Text>
                  <Card.Subtitle className="mb-2 text-muted">Location</Card.Subtitle>
                  <Card.Text>
                    123 Main St, Anytown USA
                  </Card.Text>
                </Card.Body>
              </Card>
            </Col>
          </Row>
        </Container>
      );
    }

    export default AppointmentList;

This test surprised me. I had ChatGPT create a two column layout in React for an appointment management application. Explaining a complex UI in words is difficult without visual aids, so I expected some misunderstandings, but it powered through and did exactly what I asked. It failed to add any navigation to the app nor did it use the data inside of the appointments array, so it's not really usable as a component, but it's certainly a good starting point and one I'd be happy to use as a starting template should I ever need such a thing.

That said, without tooling to make ChatGPT more available during development it would be easier to just write the layout myself.

This last test was by far the most involved and open-ended. I asked ChatGPT for a Swift-based RSS Feed Parser and it certainly gave me something close. It used the correct Foundation APIs and even understood the need to concatenate the element text, but it failed to generate a valid test fixture (or even valid XML) and there's absolutely no error handling to speak of.

That said, it did produce valid XML once I called it out and there's something to be said for such humility. I would certainly use this code as a starting point if I needed to build something like this in the near future.

Should I Hire ChatGPT?

In general I'd say it performed fairly well as a developer, though its ability to imitate bloggers is very lacking. For the fifth test I tried to have it write this post, but the output didn't sound like me at all and it kept trying to sell itself using features it doesn't have. Overall I'd give ChatGPT a tentative recommendation and would probably hire it contingent of course on a 30-day probationary period to see if it can keep its tendency toward exaggeration in check.

It's Not an AI Apocalypse, It's Better

I don't really buy the idea that ChatGPT and its ilk will do much to disrupt our everyday lives in a dramatic way. I do however see both it and it's text-to-image cousins as being more and more useful as tools to automate away so much of the boring or tedious work of being a developer or a person on the internet in general.

Like I said, I already use Midjourney to create images for my D&D campaign and those images have had a significant impact in improving the gaming experience. We were never going to contract an artist for four once-a-week landscapes and so economically speaking the AI has had no external effect. Our D&D games just got marginally better. And that's great!

ChatGPT seems to me like it could be yet another advantage that enterprising developers would use to enhance their own productivity. If something like ChatGPT ever shipped inside of an IDE like Xcode, I could imagine a whole suite of amazing features (automatic complex refactors, automating test writing, and more). GitHub's Copilot does some of this, but it's in its infancy. In general I'd expect to see tools like ChatGPT improve a developer's productivity and let us focus more on the design and nuances of our code rather than on the tedious implementation; and for that I am very excited.