Adam @ Bitscribe

Back in the bad old days of web development, I used JSP to create page content. Java is a terribly language for doing front-end web work, because it’s bad at processing text, and which usually comprises the majority of what web apps do. (And that’s also why Perl was the reigning king in this realm for so long.)

One of the things that drove me absolutely batty about JSP was the need to check every string for both null and blank, which meant your code was littered with stuff like:

if (request.get("quantity") != null && request.get("quantity") != "")
{
   quantity = (int)request.get("quantity")
   doSomething(quantity);
}

Yikes. When I moved on to PHP, I was immediately struck by the improved readability of the same code:

if (strlen($quantity) > 0)
   doSomething($quantity);

An empty string and a null will both give you zero on a string length. (You can always use === or isset() if you want to check for null explicitly.) A similar guard clause if often used for integers and arrays, too. In Java you have to check both whether a value is null, and whether it is zero or empty, respectively. In PHP you can just do:

if ($quantity != 0)

and

if (!empty($array))

Now that I’m living in Ruby land, I’ve found that my code is generally much prettier than PHP for most things… except this one. Ruby is a bit more strongly typed that PHP, so I’m back to the JSP syntax:

if params[:value] != nil and !params[:value].empty?

Hardly the succinct readability one has come to expect.

Ruby’s base classes provide some nice convenience functions to make your code more readable: Array.empty?, String.blank?, and Fixnum.zero?. The solution here seems obvious: make all of these methods exist on NilClass.

Thanks to Ruby’s metaprogramming, it’s easy to add this to your own application:

class NilClass
   def empty?; true; end
   def blank?; true; end
   def zero?; true; end
end

Great! Now we can just do:

if params[:value].empty?

Furthermore, I would tend to think that all of these should be collapsed together into a single method. I think empty? is the most applicable. So empty? means a blank string, a zero, or an empty array or hash. This reflects what the code really wants to know: is there anything in this variable? Should I bother doing anything with it? 95% of the time, the difference between nil and non-nil but empty is irrelevant.

Update: aidanf suggests using .blank? instead, which I like a bit better than the names suggested above.

Update 2: A coworker informed me that Rails extends most of the basic Ruby types with blank?. So you can do [].blank?, {}.blank?, nil.blank?, 5.blank?, etc. Apparently this is pretty unknown, because not one person mentioned it in the many dozens of comments on this post appearing on various news aggregators. Note that 0.blank? does return false, but in most cases I’m more interesting in strings anyway.

Let’s imagine that you had never before tried to open any canned food. Someone hands you a canopener. What do you think of this device?

It appears completely pointless. You can speculate on what it might be useful for, but mostly it will just seem like a contrivance of gears, blades, and levers without purpose or meaning. Probably you won’t even be able to remember the details of what it looks like if asked a day later to describe it.

Now further imagine you are handed a can of food and asked to open it. After variously trying to twist off the top, cut it open with a knife, and beat it against the countertop to open it, you give up in frustration. Then someone hands you the canopener - the same one you saw yesterday - and some basic instruction on its use (perhaps just a drawing of the right position to hold it in).

A light goes on in your head. The formerly pointless contrivance transforms into a magic wand that allows you easy, neat access to the delictable contents of the can. After using it you marvel at the clever contraption. Asked the next day to describe the can opener, you breathlessly go on and on about how this amazing series of gears and circular blades can be used to perform a most impressive feat.

The canopener prior to trying to open the can was a solution without a problem. Meaningless, arbitrary, an apparent waste of materials and complex design. But once you’ve grappled with the problem that the tool is designed to solve, you gain a unique insight and appreciation into the tool’s usefulness.

And so my clumsy metaphor closes, leading us to my true point: software tools. If you’ve never written a program that needed to manipulate data records with hundreds of thousands of entries, a SQL database seems like bloated, overly complex, contrived tool. If your web applications have always been limited to a few short PHP scripts, objects and model-view-controller frameworks probably look like nothing more than huge amounts of red tape.

Here’s the interesting bit: these observations are actually true. Canopeners without cans are just some much useless metal and plastic. SQL is not worth all the necessary infrastructure and specialized knoweldge for a program which is going to store a dozen phone numbers. And MVC on a simple web script is a hinderance rather than a help.

The solutions-without-problems concept is good to keep in your head for two reasons. One, when you show a useful tool to someone and they don’t get it, chances are they just have not grappled with the right problem yet. If you wish to enlighten them, then what you want to do is show them the problem instead of the tool. Once they have fully come to grips with the problem, it should only take a casual inspection of the tool to see its value.

In a parallel fashion, when you read or otherwise learn about a new tool and it seems pointless, then you should assume that haven’t come face-to-face with the problem yet. To understand the tool you’ll need to find them problem first, then get more details on the tool.

The worst offender of solutions-without-problems is academia. In high school I failed to grasp trigonometry in any serious way because I couldn’t see what possible benefit it was to compute the sine or cosine of an angle. Later on when I began writing programs to calculate geometry in 3D games, the usefulness of trig dawned on me like a revelation. “Why the hell didn’t my teachers tell me that this was actually _useful_?” I thought. Well, they probably tried, but without a (real) problem to solve, sine and cosine were just arbitrary functions.

I see this pattern heavily in CS courses: for example, teaching object-oriented programming on a small, one-person project where the benefits of OO are nearly non-existent. (In fact the extra overhead imposed in terms of writing header files probably makes it a worse alternative to procedural code.)

How can we get away from this approach to learning? I like to take what I call the Throw Them To The Wolves approach. Also known as the School of Hard Knocks. Throw yourself at a hard problem, let yourself get intellectually torn to bits, and then retreat to seek possible tools designed for the problem.

Catapult is a side project I’ve been working on this past few weekends, with the goal of bringing a command-line style interface to web navigation.

Think about how one normally navigates to the web apps one uses every day. If I want to compose a new post on my blog, for example, I go to the blog homepage (by entering in the complete url, maybe utilizing my browser’s autocompletion), then grab my mouse and click on the admin link, then click on the write link, then switch back to the keyboard to start typing. This is no big deal for just one app, but think of all the actions like this we perform every day. Web searches, Gmail, Yahoo yellow pages, Google Maps, Craigslist, eBay, your bank’s website, and so on. Typically we do all this with bookmarks, urls typed in from memory, and lots of clicking on navigation links. It’s not very efficient.

Catapult lets you create commands that act as shortcuts, in some cases dynamic ones. So for example a Google search would be “g [search terms]”. I can launch Gmail with “gmail”, but I can also jump straight to the compose window with a recipient already entered by typing “gmail someone@somewhere.net”. Searching my inbox is just as easy: “gmail /thing to search for”.

It’s interesting to note that using the Gmail launcher has increased my productivity in an unexpected way. I often need to send an email about something, but opening my inbox and seeing some new messages that have come in distracts me from that task. By jumping straight to the compose window I don’t even get a chance to see the inbox and thus avoid unintended distraction.

Check it out, and post any feedback here; or drop me an email with your comments.

The (Misguided?) Faith in Unit Tests stands as a rare counterpoint to the flurry of pro-TDDers (one of whom is me). The author makes some great points; but they have more to do with incorrect applications of TDD than flaws in the approach itself.

Let’s separate the technical design of software applications into two layers: high-level, which is an outline of all the working parts and how they fit together; and low-level, which is a description of the details of each part. A high-level design of a shopping cart, for example, might be something like: “Products represent items the user can buy, and belong to one Category. Cart is a container for CartItems, each of which specifies a quantity and is linked to a Product.” Design decisions you would make at this level are things like: can a Product can belong to more than one Category? Or: is multicurrency implemented through its own class, or just a currency_type enumerated field alongside any price field on an object?

The low-level design is a description of the interface to each object. For example, how does one add items to the cart: cart.addItem(item, or cart.getItems().add(item), or item.addToCart(cart)? Do you copy the price of the Product onto the CartItem when it is added to the cart, to prevent updates to the Product from affecting the customer’s checkout price? Does the shipping calculator connect to the shipping carrier’s API every time it needs to calculate a cart’s shipping price, or does it cache a local table of weights and shipping values to avoid slowdown from the network cost of the transaction? Most of these decisions affect either the internal implementation of the objects, or the APIs they export.

Tests, and unit tests in particular, help somewhat with low-level design, but not at all with high-level design. The latter is informed by the needs of the application. High-level design defines the mapping between software and the real world. Tests have a limited scope: their whole universe is the application, and anything beyond the boundaries of the code cannot be addressed. (It should be noted that business logic tests are one attempt to bridge this gap; we use this for some of our apps, but it isn’t appropriate in all situations.)

Furthermore, TDD is not so much a way to help create your design as a way to enforce it. Yes, writing the unit test first is a good way to discipline yourself to create a clean, straightforward, and orthogonal API for any given class, and I make this very point repeatedly in my screencast on test-driven development. But this is only half the story, maybe less. TDD’s real benefit over the long term is documenting and enforcing the interface and functionality of an object. A unit test says: “This is how the object is supposed to work, and I can prove it: just run me.”

So if someone new is working on the project, they can find out how multicurrency support is handled or what you have to do to add an item to a shopping cart by looking at the tests. Better yet, if they change an object’s implementation in a way that is inconsistent with the low-level design - for example, tinkering with the pricing code for products without properly handling multicurrency - this will break the tests, thus forcing the developer to confront the fact that they have a misperception about the design.

Where is the right place for the high-level design to live? For this I don’t think we have any solution other than traditional documentation. To some degree a good UI communicates many aspects of the design: for example, a breadcrumb trail shows the relationship between the primary objects (e.g. Store > Category > Product). But there’s usually still plenty of behind-the-scenes design that can’t be communicated this way. For that, some text descriptions entered into a wiki, along with some diagrams showing class or table relationships, is still your best bet.

People never liked technical documentation because the bulk of its maintenance is related to low-level design. If you’re documenting that this object has these ten methods and this first method is named this and takes these three arguments, each which have this type - well, that’s going to get out of date quickly. It’s a pain to maintain which means it won’t be up to date, and when it’s not up to date no one uses it and thus it becomes worthless. Unit tests free you from this painful cycle. (If you’re making a complex, publicly-used support library, reference documentation generated from the real code with something like RDoc should complement the tests.)

Cut out documentation of low-level design, though, and you’re left with much less documentation to write. I suspect most good technical designs - and by “good” I mean “free of excessive complexity” - could be summarized in a few paragraphs and a diagram or two, backed up by a larger document to cover domain-specific knowledge for the data tracked by the app.

Recently I recorded a few of my developer workshop presentations for public consumption on the Bitscribe site:

http://bitscribe.net/screencasts

Sure, screencasts are trendy, but you gotta admit they are fun. (Acknoweldgements to pyvnc2swf, which as near as I can tell is the only real option right now for recording cross-platform screencasts.)

I get bored easily. As such, I always want to move around to different projects to keep myself challenged, break up the routine, and get fresh perspectives. Luckily my current line of work, consulting, gives plenty of opportunity for this sort of intellectual wanderlust.

One surprising side-effect of my thirst for new projects is that it seems to result in higher quality output, especially in terms of code maintainability. I’m going to call this effect cross pollination. By moving developers around within the team to work on different components, no area of the code becomes the exclusive domain of just one person. This in turn results in healthier code, because different people are examining and trying to understand any given module. They ask questions like “Why does this bit of code work this way?” - a question very rarely asked of code one is familiar with.

If something is hard to understand or change right now, it’s probably going to be fragile over the long run. (It’s like the old technique for programmer job security: “If it was hard to write, it should be hard to understand and even harder to modify.”)

This article advocates a common approach to this problem: get programmers to train other programmers on the quirks of the codebase. I see this is little more than a bandaid. The real solution here is not to let software turn into quirky, only-one-or-two-people-know-how-it-works code. Cross-pollination of developers ensures that people are always writing code that is understandable to any other developer. The important thing is not having a certain number of developers that know the code, it’s making sure that you’re getting new eyes on the code from time to time.

A truly well-written piece of software can be easily understood and modified by any component developer familiar with the underlying tools. They need not “know” the code. Reading it - and especially reviewing the unit tests - will give them all the information needed to fix bugs or add features.

Mark-Jason Dominus’ Why I Hate Advocacy draws attention to common fallacies made when arguing for any particular tool or method. He makes many excellent points, but my favorite part was this hilarious quote:

“Perl’s motto is “There’s More Than One Way To Do It”, and sometimes that means that one of the ways is to use PHP.”

I’ve been using TextMate on my Mac for a while now, trying to train myself back into using (gasp!) the arrow keys for navigating a text document. After a decade of using the mighty vim to edit source, emails, and all other manner of text, it’s not easy to make the transition, even though TextMate is very easy to use and learn.

Ultimately TextMate is not as powerful for editing text as vim - nothing is, except probably emacs. But vim is a very focused piece of software (in the fine UNIX tradition of small, sharp tools). It absolutely cannot be beat for actually editing text, especially code. But for anything beyond that scope, forget it. For example, something as simple as search-and-replace through multiple files is not included. Programmers used to development in IDEs would probably be shocked at this apparent omission.

So TextMate can’t compete on editing text, but it makes up the difference elsewhere, by providing a broader set of tools. The most immediately useful feature I’ve discovered is Apple-T, which allows you to open a file anywhere in your project tree by typing an abbreviation, learned by the editor based on your historic habits. For projects with many files and especially a multi-level directory hierarchy, this is a much faster method of getting to individual files, especially those you access frequently, than traditional approaches.

But using it has left me feeling that this is only the tip of the iceberg: editors could go much further with the same idea. For example, in the Rails directory layout, you have many files named index.rhtml. Each is in a subdirectory named for its component, e.g. views/account/index.rhtml or views/product/index.rhtml. But TextMate searches on the filename only, not the directory, so you’re reduced to using the arrow keys to scroll through the list of index.rhtml entries.

Why go by filename at all? Who really cares what source file our code resides in? Instead we should be thinking in class / object space. I want to go to Account::sendOverdueNotice(). Figuring out that that resides in classes/account.php, opening that file, and then searching for or scrolling to the method definition is a very indirect method of navigation. Extending the TextMate approach, this would work something like hitting Apple-T and then typing “acct:sendover”.

The astute reader might point out that ctags, a fairly standard UNIX tool, has offered something like this for decades. With a modern version like Exuberant Ctags (on Ubuntu: apt-get install exuberant-ctags), type “ctags `find . -name *.rb`” in the top level of your Rails project. Then you can use a command like “vi -t send_overdue_notice” at the shell prompt, or “:tag send_overdue_notice” from within vim to jump straight to that method. (The latter offers tab completion, which is very nice, though not quite as nifty as the TextMate abbreviation thing.)

Ctags covers about 75% of what I’m suggesting. But it hasn’t changed much in recent years, and that last 25% is pretty crucial for this to really be a pleasure to use. In some ways ctags is worse for the wear today, because (as near as I can tell) it doesn’t support any object space mapping. If you have two identically named methods on different classes you can’t access more than one. You also have to regenerate the file periodically by hand… which could be automated with a cron job, but long story short is that I’ve never found myself using ctags long-term, despite the incredible potential it promises.

Take my idea above and put it in a nice friendly editor like TextMate, along with the niceties that you’d expect from such an app, like automatically generating the index, mapping the class space as well as method names, and autolearning abbreviations. Then allow the user to use this same interface to create a new method. i.e. if you try to jump to one that doesn’t exist, it starts you on a blank slate in the right place.

With an interface like this, you won’t even care how your files are stored. They could be all in one giant directory, or spread out across a tree like Rails, or whatever. The editor would decide that according to some sort of basic criteria for the language, defined by some sort of internal scripting, similar to how syntax highlighting is handled.

Furthermore you won’t care about the layout within the file. Historically I’ve spent a lot of time worrying about how the methods should be ordered. This is not something that can be solved satisfactorily for any decently large object. Method interactions are in a web, two-dimensional at least, perhaps more. Files are one-dimensional, so trying to group like methods in a logical order is a pointless and frustrating exercise. Eventually I gave up on worrying about this much, but I wonder how many other coders waste brain cycles on this without even realizing that they are doing so.

What’s more, your editor should behave sort of like a web browser: every method call is a hyperlink (activated by the keyboard, of course) to its class definition. I realize that many IDEs have something like this already, but it never feels like a first-class citizen in the navigation tool family, more of a tack-on feature. Done right, and in concert with the class/method name abbreviation jump, I think that developers could be way more efficient in our day-to-day work of navigating source code.


Update: It seems that this behavior is no longer exhibited in the current version of Rails (1.1.6), as each test is wrapped in a transaction. So this entire post is pretty much moot now.


Rails’ tests (unit, functional, and integration) use a database with read-only values. Anything which modifies the database is not saved. This allows the tests to run in isolation from each other, which is a Good Thing.

In the real world, however, I often find myself needing to write methods which go and massage a bunch of data in the database. A simple example would be a nightly cron job which creates invoices for accounts whose billing cycle are due.

In Rails this is done by creating a static method like Account.create_invoices, and invoking it from the crontab via “script/runner -e production Account.create_invoices”. Since it is a method, it can be tested in a unit or integration test. (I think the latter is more appropriate, since a high-level method like this often touches a number of models.)

But this method’s main “output” is not a return value, but rather an adjustment (or lots of them) to the database. For example, I may want to go find every account with an open billing item, create an invoice in the invoices table, and then mark the item closed. What I really want to test when this method is done is that there are some number of new invoices in the invoices table, that the billing items have been marked closed, and that the account has no open billing items. But you can’t check most of this output, because it exists as changes to the database, not direct return values.

My impression is that the Rails way views this as a Bad Thing. Every method should return its results, rather than going in and massaging a bunch of data in the database. This makes it more orthogonal and easier to test.

I agree with this philosophy in theory, I’m not sure that this is realistic for real-world applications. I’m like to think that that’s because I’m still trapped in the SQL paradigm, so maybe someone can enlighten me on the pure Rails way to do this.

Here’s a concrete example:

class Account < ActiveRecord::Base
        has_many :invoices
        has_many :billable_items

        def open_billable_items
                items = []
                billable_items.each do |item|
                        items << item if item.open?
                end
                return items
        end

        def create_invoice
                open_billable_items.each do |item|
                        item.close   # creates the invoice and marks the item closed in the db
                end
        end

        def self.create_all_invoices
                Account.find(:all).each do |account|
                        account.create_invoice
                end
        end
end

class BillingTest < ActionController::IntegrationTest
        fixtures :accounts

        def test_create_all_invoices
                assert_equal 5, accounts(:first).open_billable_items.length
                assert_equal 0, Invoice.count

                Account.create_all_invoices

                assert_equal 0, accounts(:first).open_billable_items.length
                assert_equal 5, Invoice.count
        end
end

Maybe the right thing to do here is not to test at such a high level, but instead test only BillableItem.close by having it return the created invoice. This bothers me though. It needs to work at the high level, so why can’t I test that?

And this is a very simple example. In reality, the nightly cron job may be touching dozens of tables and thousands or even millions of rows. Returning all affected rows as a result doesn’t make much sense, and may be completely impossible due to memory limitations. (The whole reason we use a database is so that we can operate on large sets of data without having to instantiate every record at once!)

The concept of a repository is a simple one on the surface, but it has surprisingly deep and subtle ramifications. Putting data in a repository encourages or even enforces unification, normalization, and consistency.

Source repositories were the first type of repository I encountered. I started using them for their archival capabilities, but as soon as I did I got a surprising side-effect: normalization of my project directory and file layouts. Almost overnight, my source trees turned from sprawling, unpredictable jungles of files into well-organized and consistently-named file hierarchies.

But it’s not just source. How about media? Many people have a movie “collection,” which is usually a pile of DVDs, VHS tapes, and maybe even laserdiscs which are not organized in any particular fashion. Some have covers, some don’t. Some are copies, some are store-bought copies. It’s a mess, but most people don’t think of it that way. It’s just the default, because that’s the way collections tend to be.

Repositories, on the other hand, tend toward the opposite. It’s actively difficult and in some cases impossible to have a repository that disorganized. iTunes is a great example of a media repository. Not to say that it can’t be a mess, it certainly can. But you have to sort of work at it. By default things are pretty well-organized.