computer science, math, programming and other stuff
a blog by Christopher Swenson
2014-10-19

ttyrec conversion

I'm an armchair fan of NetHack and other roguelikes, though I don't play them too much. I've always wondered why there wasn't, say, a NetHack screensaver that would just play through the game, since it seemed to be ripe for screensavering.

I'd found some pieces of tools that might do the job around the Internet: TTY "recordings" of NetHack ascensions (essentially, replaying through the terminal sequences while they are playing), pyte (a Python terminal emulator), and images2gif (a mostly working animated GIF maker).

It's almost there, so I wrote the missing piece that takes the screen buffer (as a matrix of character codes) and converts it to an image using an old DOS bitmap font, and then all of the glue code to make it all work together. I sped up the results by 5× or so, and then run the whole thing through ImageMagick in the end to shrink the animated GIF.

I open sourced the whole thing as ttyrec2gif. It produces nice GIFs like:

There's one last piece: how to actually turn this into a screen saver.

In OS X, it's a matter of writing a little Quartz Composer program to randomly pick GIFs from a directory and play them as movies. This has hard-coded paths in it, because I haven't figured out how to make OS X screen saver configuration parameters, so I'll just post a screen shot of what the program looks like:

(Having Quartz pick a random GIF and then reload and pick another one after 5 minutes is a fun challenge. The best solution I came up with involved using a wave generator that, when rounded, would trigger an event only when it reached its apex, and this triggered a sampler to resample and pick a new random GIF. Kind of a Rube Goldberg way of doing it, but it was fun.)

Anyway, at the very least, there are some cool GIFs out there to look at now. I'm considering doing a big run and converting a more runs into GIFs, but it's very time-consuming (it takes many hours to convert a game to an animated GIF for now).

2014-10-12

Pyway or the highway

Pyway is a database migration tool that I wrote to solve the problem of: I want database migrations, but I don't want to bring Rails or the entire JVM with me just to do them.

So, I wrote a little script that does the most simple thing you can do with database migrations: keep track of them by file name, and run any new ones in sorted order. Python is pretty much universally present at this point in server environments, so that's the tool that I used.

This is nothing sophisticated, and currently has sqlite3 hardcoded in it, but it should be trivial to modify for any other databases. I didn't even use Python bindings for sqlite3: I just called the command-line program to do the migrations. That way, it should be easy to modify it for other databases in the near future.

I'll probably end up adding Postgres or MySQL support sometime soon (when one of my projects that uses those technologies needs to be updated).

2014-10-05

This Holacracy Thing

I've had a busy year where I've been trying out a lot of new ideas for working and collaborating, and learning a lot. One thing that I learned about and experienced for a few weeks was the system of Holacracy. The goal of Holacracy is to have a managerless, self-organized team of people rather than having a traditional top-down management tree. First, I'll state that my view of Holacracy comes after only about a month of working within it, and I don't feel that even after that I fully got it. So take this post with a grain of salt.

It's an AI for a boss

The initial appeal of Holacracy was that the group within the startup I was working at had no boss. The Holacratic system has enough processes in place to theoretically allow decisions to be made without a central leader: a true peer-to-peer sort of system.

Essentially, we had several Holacracy meetings a week. These meetings were incredibly structure-heavy. People would propose ideas at point A, discuss them at point B, and these discussions and round robins followed rules that theoretically ensured fairness.

An insight I had to this was that your team is controlling an NPC "boss", and you have a little script that you run through to control its AI.

This sounds like a good thing: if you have a very small team, maybe only a couple of people, it might seem like there would not be enough work for a full-time manager. So you can have a cardboard cutout that you all pretend is your boss, and you have a system through which your cardboard captain makes decisions. (I like to think of my cardboard cutout boss as Captain Picard.)

It's too complex

But, this AI is quite complex. Holacracy has a lot of rules. And I mean a lot.

There was a discussion I had with someone at the startup where they indicated that Holacracy was probably impossible to understand (and certainly to get right) without hiring a special Holacracy consulting company to come in and help you adopt the system — helping you go learn the rules and make your Captain Picard make decisions, basically.

This kind of put me off. A system that's so complex that it takes weeks of training to be able to even use it to have a constructive meeting? It smells a bit fishy to me. Meetings are hard, but they're not that hard, even for a cave troll programmer like me. There are entire consulting companies who would love to sell you tons of expensive training for Holacracy.

To make it even worse, it's also nearly impossible to run with Holacracy without using special Holacracy meeting software to help you facilitate your meetings. There's a lot of structure and rules, and the software helps enforce these rules. This also smells a bit fishy.

Being a newcomer to this incredibly confusing set of rules was also incredibly unpleasant. I felt lost in a maze of rules. I felt that I couldn't contribute to meetings because I didn't even know when I was allowed to talk. And when I was allowed to talk, I often didn't know what to say. (There are lots of magic words you use to indicate your status, and I sort of had to learn these all from context.) Given that I was on a feeler contract (before I came on full-time), I was not given any training in Holacracy.

Finally, with such elaborate rules, there's inevitable rules-lawyering. The rules are meant to be used as a weapon to combat inefficiency, but sometimes people get wrapped up in them to the point that legitimate, necessary work was dropped on the floor because it did not fit into the system well. I felt that at times this did a disservice to the team, and certainly to the users of our software.

It's time-consuming

Given the AI manager and the complex rules, I think it's about a wash so far. It's nice not to have a manager, but annoying to have a large RPG rules book to memorize before starting work.

But, those meetings to make Captain Picard dance were ridiculously time-consuming. I'm of the leave-me-alone-so-I-can-code-in-my-cave School of Programming. I can be coaxed out of my cave for the occasional meeting, and sometimes even enjoy it. But the level of meetings required for the Captain Picard gears to run was debilitating. Hours and hours each week were spent. Several people had a significant amount of work before and after meetings dedicated to cleanup and prep work. (And even during the meeting, a couple of people were often not allowed to participate in discussions, or supposed to limit such contributions significantly, as they played the role of facilitator, secretary, etc.)

In conclusion

Coming in as a new hire is daunting. You are adjusting to a new team, learning their quirks, and learning their work. When you add on a whole new system of thinking about meetings and management, and give no training (when a lot of training is required), you are setting people up to fail. It certainly was just too much for me. Holacracy was a huge contributor to my deciding not to work for this particular company long-term.

The company that I worked with clearly prefers Holacracy, and that's their prerogative. And if it works for them, that's great. But, I didn't feel comfortable in that system, and I don't think I'm the only one. I don't expect to in the future, and I plan to avoid working for companies that embrace Holacracy.

Or, you know, maybe Holacracy could be rebranded to use an actual Captain Picard cardboard cutout and a real RPG-like rulebook. I might have had more fun then.

2014-01-25

Adobe Source Code Pro

Just thought I would mention, since I haven't talked about fonts for a while, that I would give a short update.

About a year ago, I decided to try Adobe Source Code Pro, a new fixed-width font.

Since I kind of forgot that I was using it, and the experiment has gone on a year, I'll consider it a success.

I definitely notice that Source Code Pro is better suited to the desktop and laptop world: the font has a bit higher heights and feels a little more natural at lower dpi, larger screens (compared to Droid Sans Mono).

It's free. Give it a shot.

P.S. Also, the web site got a bit of a facelift!

2013-08-25

A Minimalistic RSS Reader

Saddened by the demise of Google Reader a few months ago, I completely gave up on RSS feeds and just cut myself off from the world.

A few days ago, I finally decided it was time to move on. Unfortunately, I haven't been super pleased with the options available out there. Plus, I realized I didn't want to get burned again when they inevitably close and break my heart.

So, like dozens of other people, I just decided to write my own RSS feed reader.

One user

One fundamental design decision I made early on is that this system would only support one user, and so I can throw out 95% of the complexity of designing the system right off the bat.

Examples:

  • No security. I can throw web page up under an Apache proxy with SSL and basic auth to allow only me in. I don't need to mess with logins or anything.
  • No database. The entire state of the program will just be written out as a JSON blob periodically and read in on startup.
  • No frills. I only use a few basic features, so I'm only going to implement those. This also means that the page looks incredibly ugly.

Open

I wrote it in Go (requires 1.1+) and open sourced it at https://github.com/swenson/littlereader under the MIT license. Feel free to use it, adapt it, whatever.

It's not finished, but it has nearly every feature I need right now.

2013-07-21

Day Camp 4 Developers

I'll be talking at this week's Day Camp 4 Developers on version control (or source code control, or source code management, or whatever we want to call it this week). It'll be an introductory talk introducing why you should use version control, how it works in general, and then going into some specific workflows with git, the notorious version control system. I've had a lot of fun writing the talk, so it should be a lot of fun to hear it.

There are several other great talks that are going to happen:

  • Lorna Jane Mitchell: Practical Software Estimation Techniques, or "How Late?"
  • Thursday Bram: How to Make Good Documentation a Regular Part of Your Day
  • Brendan Wovchko: How to Speak Business & Eradicate Confusion in Software Development
  • Me: Source Code Management and Version Control

It's only $40 to sit on this great set of talks. Friday, July 26, 2013. 8am PST to 12:30am PST.

2013-06-01

Configuration files in Go

The other day, I was starting to port an existing service I had into Go. There were a lot of issues that I had to tackle to get the functionality I wanted, including being able to run in at least four different environments: test, dev, stage, and prod.

There are a lot of "standard" ways to do this, most focusing on some sort of text or structured file that you load at runtime using file I/O.

However, in dynamic languages, a somewhat common practice is to use a file in that programming language as your configuration. So, in Python, you might have a settings.py file that is actual executed Python.

In non-scripting languages, like Java, you normally have an XML, YAML, INI, or JSON file that you read in. But, I've seen at least one non-scripting language, Clojure, that encourages using an executable Clojure file for configuration.

The primary argument against using a file in your programming language itself is that the compile time may be long, and deploying a brand new binary just to change the config file is laborious and slow.

But, I thought, Go doesn't have this limitation: Go compiles super fast, and the binaries tend to be reasonably sized, so deploys won't be that big of a deal.

So, can we just use Go code to be our configuration file?

Definitely. I wrote up a quick template (released under CC0, so feel free to copy and use) for a configuration file in Go. There's a small amount of boilerplate, but it is super easy to compromise.

https://github.com/swenson/goconf

There are four key parts:

  • var config = getConfig() – this triggers the configuration file to be read at initialization time. You can also use an init() function to do this.
  • type Config struct { ... } – specify all the variables you want in your config file
  • func readConfig() Config { ... } – populate a Config struct based on your environment, which I do via a switch statement.
  • Set your environment (ENV environment variable) when running

That's it. This is a pretty straightforward and easy way to do config files in Go.

2013-04-07

When to Start Testing and Why

I didn't begin to really appreciate the value of proper testing in code until well into my current software engineering career. Testing wasn't something that came up during my formal education in computer science, since it isn't really relevant to the academic pursuit (except for perhaps a course in software engineering, most code you write academically is more about learning or exercising something than creating production code). Now, though, I love testing, and love how it helps ensure stability in my code.

Two questions that come up when I talk to other people, especially relatively new programmers, are when should you learn how to test in a language or environment, and when in your project do you start testing on a given project?

To answer the first question, the best time to learn to test (in my opinion) is after have started to get a good handle on the language and environment you're in, especially if this is the first language you've ever learned. The simple reason for this is that testing is often almost like an entirely new sub-language you have to learn, and it can be a little confusing to dive right into it.

However, you really don't want to put it off for too long, especially if you've never written test code before. Writing good, idiomatic, testable code is an important skill, and living too long in a vacuum of untested code might make it harder to learn to write testable code.

What do I mean when I say "testable code"? Generally, to me, testable code means that you've designed each class or function to be easily tested in isolation. Often, this means making it slightly more explicit than you might naturally write, writing more, smaller functions and extracting out dependencies (such as database connections) so that you can test each small piece.

After all, if your function uses a database connection, you aren't probably too concerned with launching an entire database to test that function and testing that the database writes worked – you are more interested in validating your logic that is wrapped around it. When you call insert_entry, does it convert values correctly, does it call the appropriate methods that would invoke the database, etc.? In general, you can accomplish testing independent of the database by mocking out the database layer when testing out the checkbook logic.

Going back to our second question, on when do you start testing a given project? I like the philosophy of test-driven development (TDD), where you write the tests before you write the code. This is doubly true if this is code for a client, for your job, or some other code that is meant to be consumed by people other than you, or running any kind logic dealing with other people's data. Furthermore, writing tests first helps you to figure out the architecture of your project a lot better, and to make sure that your architecture is testable from the beginning.

Writing tests before you write every piece of your software is a good way to do practice defensive coding: ensure that every piece of your programs and libraries are extraordinarily well tested before they're even written. As you write more functionality, keep growing those tests as to ensure that everything cooperates well and that nothing new is going to break the old.

2013-01-01

Scala and Clojure

My primary language for the past few months has been Scala, with some occasionaly bouts of Java.

I've also started to learn Clojure. The company I am working for is into the JVM stack, and these languages are both based on the JVM.

As a long-time Java programmer, I appreciate the cleanliness of Scala, and I've grown pretty comfortable with it over the past few months. It's sort of the language that Java really wanted to be when it grew up: all that OO goodness that we love, but without so many mistakes as Java has, and with a helpful smattering of functional programming.

That said, Scala has some flaws, including some fairly deep ones.

  • IDE support is pretty terrible. I tried to use Eclipse and IntelliJ early on, and they were both incredibly broken. I've never gotten them to work well. I just use Sublime Text 2 these days, and I find that Scala is a clean enough language that this isn't too big of a deal.
  • Implicits are just a bad idea. The standard library uses them sparingly and fairly well, but this is such a dangerous feature. (For those of you who don't know Scala, implicit functions are used to automatically convert an object of one type to another, under the hood.) It makes sense in a few limited cases, but I groan when I see them now.
  • The standard fundamental data types are wasteful and have poor performance. The List and Map data types are optimized for adding things to them, but in doing so create lots of extra copies and have very poor access properties. List, the most fundamental structure, is a damned linked list! This makes me want to cry. OpenHashMap is better, at least.

That said, I much prefer Scala to Java at this point.

Clojure, on the other hands, I am much newer to, and don't have nearly as solid experience with yet, so take my points with a grain of salt.

Clojure's obsession with immutability is nice from the point-of-view of thread-safety and (theoretical, yet mostly unrealized) concurrency, but at a high cost. Its memory footprint seems to be the worst of the JVM languages, and it defaults to using reflection for most method calls, so it is significantly slower. (Even with proper type annotations, it is slower.)

Furthermore, I think Clojure is just fundamentally harder to read. I've done my fair share of Lisp programming in the past, and I think that Lisp is just a lot less natural to read, probably because we spend so much time with infix languages.

Plus, I would posit that Clojure's premise is doomed to failure. A Lisp that doesn't support tail-call optimization and proper recursion seems broken. (And yes, I know about loop/recur, but that is essentially a functional for-loop.) Scala does some rudimentary TCO, though it doesn't always tell you the function you wrote can use it.

The biggest plus for Clojure is that it is JVM (which is also a weakness). It's a plus because it means you can take advangate of the Java ecosystem (logging, libraries, experience, etc.). Which is probably the primary reason we are using it instead of some other functional languages.

That said, I will be spending quite a bit of time this year learning both in greater detail, so perhaps I will update with further opinions on them.

(I'm also thinking about getting more into Go this year, and perhaps finally learning Erlang.)

2012-10-10

Things I've Learned At Google, Part Two

I've spent the past 19 months working as a software engineer at Google. They have adopted a lot of great practices, some completely their own, and I certainly learned a lot from working them. I thought I would share some of the particular insights I had while working there.

This first post was about software engineering. This post is more about general workplace practices.

If you want to have the best software engineers, then hire smart people and treat them well.

I've worked at places in the past where they are good at hiring smart people, but then tend to treat them poorly. Then, they wonder why the leave in droves for companies, like Google, who aren't afraid to buy you a nice chair and keyboard.

If you want to have great employees, then you have to treat them like it.

Open source licensing is important.

Open source code is a lot harder than you'd think, and even the most competent software engineers will sometimes try to get away with not following the rules. The license implications in things like the GPL and AGPL are serious business. Before using any code that touches GPL or AGPL (or other similar licenses), educate yourself well on them, or designate someone at your company to be the open source czar to make sure you aren't messing this stuff up. It's not really hard or anything: it's just a little tedious, and incredibly important.

Shoes are optional.

I worked at mostly "business casual" places prior to Google, but Google goes straight for "casual", bordering on "who cares". And for all that, I never noticed it as a problem. Unless you are violating the health code, do what makes you productive and happy. (For instance, I am more productive when not wearing shoes, but wearing pants (or at least a kilt). YMMV.)

Working from home can be fine.

Personally, I work better in an office. But, my Google office is about 1 or 1.5 hours away. After a few weeks, I figured out a system where I could be very productive working from home — certainly efficient enough so that there is a net gain in productivity due to commuting time. So, I spent a lot of time at Google working from home, and did quite well. (Early on, I spent a lot more time in the office, as I was still bright-eyed and bushy-tailed and was still trying to understand all of Google's inner workings, and this required being able to communicate with my coworkers more.)

Don't be a gatekeeper unless it is really important.

As organizations grow, certain groups start to feel important, entitled, or some other emotion, and begin injecting themselves into every process in the business. Some of these are really important, like privacy and security. For instance, a lot of managers believe that they are not doing anything unless they change something, and so they sometimes like to start sticking themselves in processes they don't belong in.

My favorite instance of this at Google is that one of founders has to approve the application package of every single new hire (at least, for software engineers). Even though hiring is important, I am a little doubtful that his review does anything other than make him feel better.

Becasue of gatekeepers, you can end up with a giant checklist of people who have too much of a say in your project because someone in the past didn't have due diligence. This can really eat at the velocity of your project, especially small projects (where a just a few people have to deal with all of the gatekeeprs), and definitely at the morale of the team.

Google is large, and is brimming with gatekeepers. Which I find interesting, because Google has a really neat system for getting rid of gatekeepers: readability.

Let me explain readability. If you are new to Google or new to a programming language, most of your first code needs to be reviewed by an expert in that field — a Java expert for example, who can say that your Java code meets all of the standard guidelines to maintainability, readability, and general sanity. After a while, you will become recognized as an expert as well (through a formal process), and you won't need to have this be part of every review anymore.

Perhaps a solution to gatekeeping would be to have more readability-like things: if you've launched a few projects that tackled certain thorny areas very well, then maybe you now have "readability" in those areas, and we can trust you a little bit more, and remove some bureaucracy from your plate. It's an idea at least. But the moral of the story is: don't insert yourself into the bureaucracy and become part of the problem unless there is no other choice.

Gatekeepers are a symptom of trust and communication mismanagement. When your product is ready for launch or has made some significant progress, all of a sudden, a bunch of gatekeepers (like your senior managers) take an interest in your project. Some of them will have looked at your project in its early phases to greenlight the concept, but to many of them, they may not have ever heard of you before. And, especially at the very highest levels, they only have 1 bit of communication down to you: launch or cancel.

This low-bandwidth communication is bad, especially if they see problems with your product or direction, and unfortunately, you may have just wasted a lot of effort (perhaps months or years of engineering) on something that is going in the scrap pile. This is perhaps avoidable by trusting the management below you to make the right decision early on. A senior manager having so much drastic control over a project sends a mixed message about trust.

Writing is still incredibly important.

Don't get me wrong, I understood this pretty well before. But even at Google, writing is still important. For one thing, any large company probably has a convoluted promotion process, and Google's, while leaner than some, still requires a lot of paperwork. Paperwork means writing, and if you can't communicate effectively about your own accomplishments, then you are just not going to get promoted, at Google or anywhere.

I've seen people time and time again tell me that "they just aren't good at writing about themselves" — this has been nonsense every time. They were fine at writing about themselves, they just didn't like to. In which case, too bad: if you want to get promoted, then just do it.

If you don't write about it, it didn't happen.

Kids aren't nearly as good at computers as I thought they were.

While not strictly workplace-related, this is something fun I learned at Google.

While working from Google, I was often the token Google engineer that was paraded around for local high schoolers that we would do some outreach for. I would answer questions from "how does search work?" to "do they really let you drink at work?". It was rewarding and I really enjoyed talking with them.

However, they were all just terrible at technology, despite all that "kids these days" should be masters of all computers, cell phones, and whatnot. They should be tumblring circles around my desktop-loving self when it comes to, well, everything, but I'm pretty sure most of them couldn't program a VCR, or whatever the modern equivalent to that is. (For my younger readers, programming a VCR is perhaps the second most trivial "techie" thing to do in my generation, only slightly above having an alarm clock with the time set correctly (rather than blinking 12:00)).


I'm sure I learned more at Google, but those are perhaps some of the important things I felt I learned while working there.

I can't wait to see what I learn at my next job. :)