computer science, math, programming and other stuff
a blog by Christopher Swenson
2014-10-26

What I Learned As A Teen Programmer

I recently stumbled across the source to a NES emulator I wrote when I was a teenager. Reading through the source code, I thought a lot about how much I had grown as a developer and engineer since that long-forgotten summer. For instance, I now understand things like (ha!) having a consistent style (some of my code was indented over 24 spaces so that it would be in the middle of the monitor), writing comments (there are basically no comments in this code), and tests (I didn't even know that testing was a thing). But, I also realized that writing that emulator, and all of the development I did at the time, laid a strange foundation that colored how I've thought about programming since.

Up to that point, I'd puttered around a little in BASIC and C, but I didn't fall in love with programming until I discovered assembly language. Assembly was the first thing that really clicked for me: it was so simple, its semantics were so clear, and I found some great resources to help me learn (e.g., a nice tutorial I found on a BBS). So, in my heart, assembly language is my first language.

When I tell this to people now, they give me strange looks. It's almost unfathomable to teach people assembly language first. But, in doing so, I learned an awful lot about how software works, how computers work, and what foundation programming languages are built on top of.

My masterpiece as a teen programmer was the NES emulator I wrote entirely in x86 assembly language: almost exactly 10,000 lines of it. This was my first taste of a real program: it was potentially useful, it had meaning, and it was a moderately large software project (my first). Programming this was what made me dig deep and learn how to program.

Here's some things that writing that emulator, and assembly language in general, taught me:

  1. Data is just memory. I remember trying to understand pointers in C and being incredibly frustrated with the strange syntax and the weird semantics. On the other hand, dealing with memory in assembly is so clearcut: memory is just a giant flat array (more or less), and pointers just point to stuff in that giant array. Indirect references, arrays of pointers, etc., make so much more sense when boiled down like this.

    Even today, when trying to visualize many problems, I think to myself: what does this look like in memory? What structures are being used and how are they laid out? This helps me understand how algorithms and data structures work together.

  2. How to write everything yourself. There were some limited BIOS and DOS kernel calls, but for the most part, if I needed some capability, say, reading from a floppy drive or writing to the screen, I had to write it from scratch: calling I/O ports, reading and writing to RAM addresses, etc. If I wanted graphics, I had to draw every single pixel myself. These days, I know that this is just reinventing the wheel, something normally regarded as a bad habit; however, I also learned an awful lot about inventing wheels doing it. I still have the bad habit of writing too many things myself, and not leveraging existing libraries as often as I should, so this lesson wasn't a complete victory.

  3. Sometimes you just have to buckle down and program all of the tedious parts. A big part of the NES emulator is writing a cycle-accurate CPU emulator. I'm sure that you can imagine how daunting it must be to start writing CPU.ASM and have an empty jump table with 256 entries, knowing that you have to write all 256 functions to handle every single operation. Even worse, you have to get every single piece exactly right.

    This taught me an extremely valuable skill: being able to focus on tedious, repetitive tasks. Programming is absolutely chock-full of tedious programming that is absolutely necessary. A great example is date and time functionality (especially time zones): incredibly important code to get correct, but so incredibly tedious and detail-oriented. Being able to dive into thousands of pages of specifications and turn them into working libraries and APIs has paid great dividends in my career as a software engineer, and I first learned that here. Being methodical and untiring is a great skill.

    And really, sometimes there is just no elegant solution. Often I see programmers strive to come up with a beautiful, simple, clean way to solve a problem, only to realize that the solution doesn't stand up to reality. Sometimes reality is ugly, and sometimes you have to write ugly code to solve it: you just need to program your way out of it.

  4. Assembly is useful. Learning assembly has come in handy countless times: reading GDB disassembly of my code for degugging, understanding and optimizing performance, writing compilers. Every time I see an assembly dump on a screen, it feels like home.

  5. You have to fail a lot to succeed. When writing code as a teenager in assembly language, I spent a lot of time crashing my computer. I mean, assembly is really easy to get wrong. When you get it wrong, you often do so catastrophically.

    But more so, in order to learn how to build things, you first have to learn to build them wrong, and learn to recognize your mistakes. And this code is just full of mistakes.

Overall, revisiting old code like this helped me reflect on my programming: where I've been, where I am, where I'm going. I encourage everyone to save as much of their code as they ever write, and come back to it in a few years, and to reflect.

2014-10-19

ttyrec conversion

I'm an armchair fan of NetHack and other roguelikes, though I don't play them too much. I've always wondered why there wasn't, say, a NetHack screensaver that would just play through the game, since it seemed to be ripe for screensavering.

I'd found some pieces of tools that might do the job around the Internet: TTY "recordings" of NetHack ascensions (essentially, replaying through the terminal sequences while they are playing), pyte (a Python terminal emulator), and images2gif (a mostly working animated GIF maker).

It's almost there, so I wrote the missing piece that takes the screen buffer (as a matrix of character codes) and converts it to an image using an old DOS bitmap font, and then all of the glue code to make it all work together. I sped up the results by 5× or so, and then run the whole thing through ImageMagick in the end to shrink the animated GIF.

I open sourced the whole thing as ttyrec2gif. It produces nice GIFs like:

There's one last piece: how to actually turn this into a screen saver.

In OS X, it's a matter of writing a little Quartz Composer program to randomly pick GIFs from a directory and play them as movies. This has hard-coded paths in it, because I haven't figured out how to make OS X screen saver configuration parameters, so I'll just post a screen shot of what the program looks like:

(Having Quartz pick a random GIF and then reload and pick another one after 5 minutes is a fun challenge. The best solution I came up with involved using a wave generator that, when rounded, would trigger an event only when it reached its apex, and this triggered a sampler to resample and pick a new random GIF. Kind of a Rube Goldberg way of doing it, but it was fun.)

Anyway, at the very least, there are some cool GIFs out there to look at now. I'm considering doing a big run and converting a more runs into GIFs, but it's very time-consuming (it takes many hours to convert a game to an animated GIF for now).

2014-10-12

Pyway or the highway

Pyway is a database migration tool that I wrote to solve the problem of: I want database migrations, but I don't want to bring Rails or the entire JVM with me just to do them.

So, I wrote a little script that does the most simple thing you can do with database migrations: keep track of them by file name, and run any new ones in sorted order. Python is pretty much universally present at this point in server environments, so that's the tool that I used.

This is nothing sophisticated, and currently has sqlite3 hardcoded in it, but it should be trivial to modify for any other databases. I didn't even use Python bindings for sqlite3: I just called the command-line program to do the migrations. That way, it should be easy to modify it for other databases in the near future.

I'll probably end up adding Postgres or MySQL support sometime soon (when one of my projects that uses those technologies needs to be updated).

2014-10-05

This Holacracy Thing

I've had a busy year where I've been trying out a lot of new ideas for working and collaborating, and learning a lot. One thing that I learned about and experienced for a few weeks was the system of Holacracy. The goal of Holacracy is to have a managerless, self-organized team of people rather than having a traditional top-down management tree. First, I'll state that my view of Holacracy comes after only about a month of working within it, and I don't feel that even after that I fully got it. So take this post with a grain of salt.

It's an AI for a boss

The initial appeal of Holacracy was that the group within the startup I was working at had no boss. The Holacratic system has enough processes in place to theoretically allow decisions to be made without a central leader: a true peer-to-peer sort of system.

Essentially, we had several Holacracy meetings a week. These meetings were incredibly structure-heavy. People would propose ideas at point A, discuss them at point B, and these discussions and round robins followed rules that theoretically ensured fairness.

An insight I had to this was that your team is controlling an NPC "boss", and you have a little script that you run through to control its AI.

This sounds like a good thing: if you have a very small team, maybe only a couple of people, it might seem like there would not be enough work for a full-time manager. So you can have a cardboard cutout that you all pretend is your boss, and you have a system through which your cardboard captain makes decisions. (I like to think of my cardboard cutout boss as Captain Picard.)

It's too complex

But, this AI is quite complex. Holacracy has a lot of rules. And I mean a lot.

There was a discussion I had with someone at the startup where they indicated that Holacracy was probably impossible to understand (and certainly to get right) without hiring a special Holacracy consulting company to come in and help you adopt the system — helping you go learn the rules and make your Captain Picard make decisions, basically.

This kind of put me off. A system that's so complex that it takes weeks of training to be able to even use it to have a constructive meeting? It smells a bit fishy to me. Meetings are hard, but they're not that hard, even for a cave troll programmer like me. There are entire consulting companies who would love to sell you tons of expensive training for Holacracy.

To make it even worse, it's also nearly impossible to run with Holacracy without using special Holacracy meeting software to help you facilitate your meetings. There's a lot of structure and rules, and the software helps enforce these rules. This also smells a bit fishy.

Being a newcomer to this incredibly confusing set of rules was also incredibly unpleasant. I felt lost in a maze of rules. I felt that I couldn't contribute to meetings because I didn't even know when I was allowed to talk. And when I was allowed to talk, I often didn't know what to say. (There are lots of magic words you use to indicate your status, and I sort of had to learn these all from context.) Given that I was on a feeler contract (before I came on full-time), I was not given any training in Holacracy.

Finally, with such elaborate rules, there's inevitable rules-lawyering. The rules are meant to be used as a weapon to combat inefficiency, but sometimes people get wrapped up in them to the point that legitimate, necessary work was dropped on the floor because it did not fit into the system well. I felt that at times this did a disservice to the team, and certainly to the users of our software.

It's time-consuming

Given the AI manager and the complex rules, I think it's about a wash so far. It's nice not to have a manager, but annoying to have a large RPG rules book to memorize before starting work.

But, those meetings to make Captain Picard dance were ridiculously time-consuming. I'm of the leave-me-alone-so-I-can-code-in-my-cave School of Programming. I can be coaxed out of my cave for the occasional meeting, and sometimes even enjoy it. But the level of meetings required for the Captain Picard gears to run was debilitating. Hours and hours each week were spent. Several people had a significant amount of work before and after meetings dedicated to cleanup and prep work. (And even during the meeting, a couple of people were often not allowed to participate in discussions, or supposed to limit such contributions significantly, as they played the role of facilitator, secretary, etc.)

In conclusion

Coming in as a new hire is daunting. You are adjusting to a new team, learning their quirks, and learning their work. When you add on a whole new system of thinking about meetings and management, and give no training (when a lot of training is required), you are setting people up to fail. It certainly was just too much for me. Holacracy was a huge contributor to my deciding not to work for this particular company long-term.

The company that I worked with clearly prefers Holacracy, and that's their prerogative. And if it works for them, that's great. But, I didn't feel comfortable in that system, and I don't think I'm the only one. I don't expect to in the future, and I plan to avoid working for companies that embrace Holacracy.

Or, you know, maybe Holacracy could be rebranded to use an actual Captain Picard cardboard cutout and a real RPG-like rulebook. I might have had more fun then.

2014-01-25

Adobe Source Code Pro

Just thought I would mention, since I haven't talked about fonts for a while, that I would give a short update.

About a year ago, I decided to try Adobe Source Code Pro, a new fixed-width font.

Since I kind of forgot that I was using it, and the experiment has gone on a year, I'll consider it a success.

I definitely notice that Source Code Pro is better suited to the desktop and laptop world: the font has a bit higher heights and feels a little more natural at lower dpi, larger screens (compared to Droid Sans Mono).

It's free. Give it a shot.

P.S. Also, the web site got a bit of a facelift!

2013-08-25

A Minimalistic RSS Reader

Saddened by the demise of Google Reader a few months ago, I completely gave up on RSS feeds and just cut myself off from the world.

A few days ago, I finally decided it was time to move on. Unfortunately, I haven't been super pleased with the options available out there. Plus, I realized I didn't want to get burned again when they inevitably close and break my heart.

So, like dozens of other people, I just decided to write my own RSS feed reader.

One user

One fundamental design decision I made early on is that this system would only support one user, and so I can throw out 95% of the complexity of designing the system right off the bat.

Examples:

  • No security. I can throw web page up under an Apache proxy with SSL and basic auth to allow only me in. I don't need to mess with logins or anything.
  • No database. The entire state of the program will just be written out as a JSON blob periodically and read in on startup.
  • No frills. I only use a few basic features, so I'm only going to implement those. This also means that the page looks incredibly ugly.

Open

I wrote it in Go (requires 1.1+) and open sourced it at https://github.com/swenson/littlereader under the MIT license. Feel free to use it, adapt it, whatever.

It's not finished, but it has nearly every feature I need right now.

2013-07-21

Day Camp 4 Developers

I'll be talking at this week's Day Camp 4 Developers on version control (or source code control, or source code management, or whatever we want to call it this week). It'll be an introductory talk introducing why you should use version control, how it works in general, and then going into some specific workflows with git, the notorious version control system. I've had a lot of fun writing the talk, so it should be a lot of fun to hear it.

There are several other great talks that are going to happen:

  • Lorna Jane Mitchell: Practical Software Estimation Techniques, or "How Late?"
  • Thursday Bram: How to Make Good Documentation a Regular Part of Your Day
  • Brendan Wovchko: How to Speak Business & Eradicate Confusion in Software Development
  • Me: Source Code Management and Version Control

It's only $40 to sit on this great set of talks. Friday, July 26, 2013. 8am PST to 12:30am PST.

2013-06-01

Configuration files in Go

The other day, I was starting to port an existing service I had into Go. There were a lot of issues that I had to tackle to get the functionality I wanted, including being able to run in at least four different environments: test, dev, stage, and prod.

There are a lot of "standard" ways to do this, most focusing on some sort of text or structured file that you load at runtime using file I/O.

However, in dynamic languages, a somewhat common practice is to use a file in that programming language as your configuration. So, in Python, you might have a settings.py file that is actual executed Python.

In non-scripting languages, like Java, you normally have an XML, YAML, INI, or JSON file that you read in. But, I've seen at least one non-scripting language, Clojure, that encourages using an executable Clojure file for configuration.

The primary argument against using a file in your programming language itself is that the compile time may be long, and deploying a brand new binary just to change the config file is laborious and slow.

But, I thought, Go doesn't have this limitation: Go compiles super fast, and the binaries tend to be reasonably sized, so deploys won't be that big of a deal.

So, can we just use Go code to be our configuration file?

Definitely. I wrote up a quick template (released under CC0, so feel free to copy and use) for a configuration file in Go. There's a small amount of boilerplate, but it is super easy to compromise.

https://github.com/swenson/goconf

There are four key parts:

  • var config = getConfig() – this triggers the configuration file to be read at initialization time. You can also use an init() function to do this.
  • type Config struct { ... } – specify all the variables you want in your config file
  • func readConfig() Config { ... } – populate a Config struct based on your environment, which I do via a switch statement.
  • Set your environment (ENV environment variable) when running

That's it. This is a pretty straightforward and easy way to do config files in Go.

2013-04-07

When to Start Testing and Why

I didn't begin to really appreciate the value of proper testing in code until well into my current software engineering career. Testing wasn't something that came up during my formal education in computer science, since it isn't really relevant to the academic pursuit (except for perhaps a course in software engineering, most code you write academically is more about learning or exercising something than creating production code). Now, though, I love testing, and love how it helps ensure stability in my code.

Two questions that come up when I talk to other people, especially relatively new programmers, are when should you learn how to test in a language or environment, and when in your project do you start testing on a given project?

To answer the first question, the best time to learn to test (in my opinion) is after have started to get a good handle on the language and environment you're in, especially if this is the first language you've ever learned. The simple reason for this is that testing is often almost like an entirely new sub-language you have to learn, and it can be a little confusing to dive right into it.

However, you really don't want to put it off for too long, especially if you've never written test code before. Writing good, idiomatic, testable code is an important skill, and living too long in a vacuum of untested code might make it harder to learn to write testable code.

What do I mean when I say "testable code"? Generally, to me, testable code means that you've designed each class or function to be easily tested in isolation. Often, this means making it slightly more explicit than you might naturally write, writing more, smaller functions and extracting out dependencies (such as database connections) so that you can test each small piece.

After all, if your function uses a database connection, you aren't probably too concerned with launching an entire database to test that function and testing that the database writes worked – you are more interested in validating your logic that is wrapped around it. When you call insert_entry, does it convert values correctly, does it call the appropriate methods that would invoke the database, etc.? In general, you can accomplish testing independent of the database by mocking out the database layer when testing out the checkbook logic.

Going back to our second question, on when do you start testing a given project? I like the philosophy of test-driven development (TDD), where you write the tests before you write the code. This is doubly true if this is code for a client, for your job, or some other code that is meant to be consumed by people other than you, or running any kind logic dealing with other people's data. Furthermore, writing tests first helps you to figure out the architecture of your project a lot better, and to make sure that your architecture is testable from the beginning.

Writing tests before you write every piece of your software is a good way to do practice defensive coding: ensure that every piece of your programs and libraries are extraordinarily well tested before they're even written. As you write more functionality, keep growing those tests as to ensure that everything cooperates well and that nothing new is going to break the old.

2013-01-01

Scala and Clojure

My primary language for the past few months has been Scala, with some occasionaly bouts of Java.

I've also started to learn Clojure. The company I am working for is into the JVM stack, and these languages are both based on the JVM.

As a long-time Java programmer, I appreciate the cleanliness of Scala, and I've grown pretty comfortable with it over the past few months. It's sort of the language that Java really wanted to be when it grew up: all that OO goodness that we love, but without so many mistakes as Java has, and with a helpful smattering of functional programming.

That said, Scala has some flaws, including some fairly deep ones.

  • IDE support is pretty terrible. I tried to use Eclipse and IntelliJ early on, and they were both incredibly broken. I've never gotten them to work well. I just use Sublime Text 2 these days, and I find that Scala is a clean enough language that this isn't too big of a deal.
  • Implicits are just a bad idea. The standard library uses them sparingly and fairly well, but this is such a dangerous feature. (For those of you who don't know Scala, implicit functions are used to automatically convert an object of one type to another, under the hood.) It makes sense in a few limited cases, but I groan when I see them now.
  • The standard fundamental data types are wasteful and have poor performance. The List and Map data types are optimized for adding things to them, but in doing so create lots of extra copies and have very poor access properties. List, the most fundamental structure, is a damned linked list! This makes me want to cry. OpenHashMap is better, at least.

That said, I much prefer Scala to Java at this point.

Clojure, on the other hands, I am much newer to, and don't have nearly as solid experience with yet, so take my points with a grain of salt.

Clojure's obsession with immutability is nice from the point-of-view of thread-safety and (theoretical, yet mostly unrealized) concurrency, but at a high cost. Its memory footprint seems to be the worst of the JVM languages, and it defaults to using reflection for most method calls, so it is significantly slower. (Even with proper type annotations, it is slower.)

Furthermore, I think Clojure is just fundamentally harder to read. I've done my fair share of Lisp programming in the past, and I think that Lisp is just a lot less natural to read, probably because we spend so much time with infix languages.

Plus, I would posit that Clojure's premise is doomed to failure. A Lisp that doesn't support tail-call optimization and proper recursion seems broken. (And yes, I know about loop/recur, but that is essentially a functional for-loop.) Scala does some rudimentary TCO, though it doesn't always tell you the function you wrote can use it.

The biggest plus for Clojure is that it is JVM (which is also a weakness). It's a plus because it means you can take advangate of the Java ecosystem (logging, libraries, experience, etc.). Which is probably the primary reason we are using it instead of some other functional languages.

That said, I will be spending quite a bit of time this year learning both in greater detail, so perhaps I will update with further opinions on them.

(I'm also thinking about getting more into Go this year, and perhaps finally learning Erlang.)