Alternately, learning to program should require some real learning by doing that isn’t informed by some gold-star method or gamification. I am of the opinion that these approaches only develop a false sense of mastery among its users. This is the kind of pernicious marketing that Jeff Atwood is railing against. You’re not suddenly a PHP expert because you spent ten hours on Codecademy.

Learning to program encapsulates a lot of different skills. It’s an iterative process, and you should approach it as with any new skill of similar profundity: with care, interest, and a knowledge of your own limitations. Instead of assuming you can tackle any task related to your achievement badges, you should aim to constantly question whether what you’re doing now is adequate, performant, scalable, and extensible. This is a great behavior to get into for any programmer, at any time. It’s the reason why I look at diffs before ever committing code, and it’s one reason why I get better at what I do as time goes on — because I take the time to review my practices and look for better ones.

But this isn’t just good advice for programming as a skill. There are plenty of great skills that you could do professionally that you can learn at home. Look at the vast array of talents required in home improvement. There are certified individuals who have the requisite education and experience to build you a house, set up your wiring, install insulation, or configure your home network. But that doesn’t mean that we should leave all aspects of these skills exclusively to the pros. That’s where Atwood oversteps his bounds with respect to programming as well. While it’s not a good idea to wire your entire house if you don’t know what you’re doing, it’s well worth knowing how to switch out your outlets, or to reset your router if it’s acting up. Not everyone may need to have the programming chops to develop a database schema or roll their own MVC platform, sure. But that doesn’t mean that they might not get good use out of some text processing that a basic education in regular expressions with a lightweight interpreted language could lend.

Most of the real bare-bolts stuff I’ve learned about software, I have learned on the job, by doing. I’ve probably aced more job interviews based on my knowledge of design patterns –which exclusively involved reading every design pattern listed on Wikipedia, and then trying to identify them in real code — than talking about the technical aspects of my Master’s thesis.

The availability of information on the Internet means that the empowerment of passionate amateurs is a trend in all skilled professions. It shouldn’t be railed against; we should just be happy that it’s easier for people who want to learn to find resources that get them adequately situated. Sure, you can watch a YouTube video about how to use a smoker, but you’re not a pit master until you’ve made it a lifestyle. And there will always be that dichotomy for all hobbies or professions that require learning and practice.

I would even go so far as to say that some skilled professions such as law and medicine should start looking for ways to better incorporate this additional tier of semi-experts into the profession. The issue here, as programming has feared in the past, is that it is likely to drive down the low end of the market. But I think that software in general has become better for having people from different backgrounds with a passion in the practice, if not so much the college major, and the availability of new technology for these fields, as with programming, allows individuals to more easily sit on the shoulders of giants. But again, there will always be a difference between those who can identify the signs of a flu, and those who know what other symptoms symptoms to look for that could result in a more grave diagnosis.

This introduces a new UI, grouped inter-wiki search for the global site, and improved relevance. If you’re curious about more details, please take a look at today’s search announcement as well as a long-term view of search at Wikia.

I’ve had a lot of fun developing this stuff, and I’m looking forward to our next big moves!

I’m always looking for ways to combine things that excite me. Considering the fact that I spend most of my work day bouncing around in my chair to one mixtape or another while coding, it was only a matter of time before I was hit with the following epiphany: I should make a language model for MediaTakeOut. MediaTakeOut is an interesting website, because its headlines have a lot of entertaining, edgy constructions that are relatively unique to its site. I wanted to use a language model to see whether the prevailing patterns I saw could actually be reconstructed using the laws of probability. Also, let’s be honest, here: language models aren’t really that useful, but they can be hilarious. The reason why they’re funny is that they decontextualize the underlying qualities of a given textual corpus (thus generating grammatical nonsense). This is particularly telling when that corpus uniquely subsumes a specific author or domain. A lot of the headlines on MTO are already written to be funny or outrageous, so one could imagine what you might get from inserting a little bit of randomness into the mixture.

So a couple of nights’ worth of work, and I wrote a MediaTakeOut Headline Generator, lovingly styled after the website it parodies. I open-sourced the code at GitHub under the project name MTO-ON-BLAST. Though maybe that’s not a fair name. Putting someone or something “on blast” means to give it a hard time, such as “R&B singer Monica gets put ON BLAST”. I used the term since those were the kind of entertaining constructions I was looking to generate in the language model. I would posit that the Chomsky Bot does a better job putting Chomsky on blast than my project does putting MTO on blast.

Anyhow, I’m really writing this blog post because I wanted to describe the components of the project, and how each worked. I first cut my teeth on real programming using Python in grad school, so this was an opportunity to play around with Python after years in PHP and a couple of months slogging through some intense Perl. Let’s be honest, though: this isn’t much more advanced than a graduate Computational Linguistics I project from an implementation perspective. I could have written my own language model rather than use an off-the-shelf implementation if I wanted to impress my thesis advisor, but I was really only looking to have some fun.

First off, I wrote a script called mto-scrape.py, which uses the lxml library to record headlines in the MTO archives by accessing elements in the DOM. We iterate through each page of the archive, scraping the page’s HTML, parsing it into a DOM object, and pulling out headlines from appropriate DOM elements. I also ended up using this basic approach to scrape comics from Nothing Nice To Say, an old favorite webcomic of mine, for offline viewing. So the same approach is fairly reusable.

Since the above script is relatively naive, I stripped escape character slashes and excessive whitespace using a a command line script. You can diff headlines.txt and headlinesprepped.txt if you want to get a feel for the changes. I should probably add a bash script to automate this step, but I trust that anyone who actually wants to repeat my results would know enough to handle this rather quickly.

The next step is the script mto-analyze.py, which uses the Natural Language Toolkit to consume the prepped text file and store the necessary data to create the language model. It also displays the top 100 unigrams, bigrams, and trigrams for the corpus. For the uninitiated, these are N-grams, where N is the number of adjacent tokens to constitute a unique type. Consider the following tweet from Lil B:

I MAKE SURE TO TURN OFF MY LIGHTS WHEN IM NOT USING THEM TO SAVE THE ENERGY OF THE WORLD, WE ARE VERY LUCKY RIGHT NOW #BASED – Lil B

The first three unigrams would be: [I], [MAKE], [SURE]. The first three bigrams would be [I MAKE], [MAKE SURE], [SURE TO].The first three trigrams would be [I MAKE SURE], [MAKE SURE TO], [SURE TO TURN]. These tokens are then stored as types, with individual counts associated them. For instance, the unigram types [TO] and [THE] would both have a token count of two, since they are seen twice in this tweet. This means that they are more likely to appear than other words, from a probabilistic standpoint. This is, without getting too academic, the prevailing concept behind a language model. The more data that we have, the larger the constructions we can attempt to generalize against. With almost 37,000 headlines scraped from MTO, with an average of 16 words (or about 96 characters) per headline, it’s relatively safe for us to attempt to build a trigram model. That’s a model that uses information from the previous two words to select a relatively probable next word.

Many language models use conditional probability based on the prior probability of observed events. Finding the conditional probability when constructing a trigram language model would be answering the question, given the last two words provided, what is the probability distribution of the all possible next words? We use the prior probability of observed trigrams in our headlinesprepped.txt file to inform this decision. This is what offers the illusion of grammaticality when using a relatively flat approach to sentence construction. By inserting randomness into which probability-weighted type is selected, we make each generated sentence variable, rather than just always generating the most probable sentence.

The script mto-languagemodel.py uses the data stored from the analyze script to create a language model using a smoothed bayesian approach available in NLTK. It spits out N sentences, which are then stored on my site’s server. For performance purposes, I don’t do any headline generation on the fly. Instead, I grab a random line out of a large flat file. When searching for a subject, I’m basically adding a grep step before pulling out the random line. A post-processing regular expression handles appropriate spacing for punctuation tokens, which are treated just like normal tokens in the language model.

So there you have it. Now all you need to do is go to the generator and search for a particular term. The fun part about language models is how they help you discover the emergent characteristics of a given corpus in a more novel way that simple charts or graphs (like what the analyzer component provides). So under all the fun, this is some real computational linguistics in action.

Preface

I’m cranky. I complain about a lot of things. There’s a lot in the world of natural language that I don’t like. Most natural languages were invented by complete amateurs who didn’t have half a clue what they were doing when they started joining noun with verb. Combine that with the Sapir-Worf hypothesis, and you’ve got a whole gang of self-congratulating numbskulls who can’t even think past the foolish paradigms they’ve constructed to truly subject their puny minds to logical thinking.

This is not the same. English is an aberration. It’s not merely awkward to speak or ill-suited for what I want, or sub-optimal, or against my religion. I could tell you something I like about most languages I don’t speak, even though I have good reasons not to speak them. But English is the lone exception. Every time I try to compile a list of gripes about the English language, I get stuck in this depth-first search of discovering more and more appalling trivia. (Hence, fractal.)

English is an embarrassment, a blight upon my Broca’s area. It’s so broken, but so lauded by every empowered amateur who’s yet to learn anything else, as to be maddening. It has paltry few redeeming qualities that I would prefer to forget it exists at all.

But I’ve got to get this out of my system. So here’s one last try.

An Analogy (not really)

Say you were learning English for the first time, and someone told you that the rule for forming the past tense of a verb would be to add -ed to the end of a word. That’s a productive rule that applies to all persons and numbers. I walked; John slipped; they skipped; we flipped. Now try to apply that to be, do, buy, eat, etc. You think to yourself, “Hey, this is ridiculous! There are all these rules I have to learn that don’t work for all of verbs I want to use the most. What a stupid language. How on Earth did the people speaking this make such a mess out of their most common verbs?”

Now in order to use English proficiently, you have to memorize all of these exceptions. Most of the time, you’re dealing in these exceptions, and not the rules. However, since you took the time to memorize the productive rules, you find that ultimately, you’re able to generalize for a large set of vocabulary you never even memorized. The more you practice, the better you get. Eventually, you’re able to explain the subtleties of some of these stranger subsets of rules. Some non-native experts can even explain to you why there are subsets of verbs that exhibit ablauting, and why certain subsets of Latin-inherited words exhibit prosody and tonality that violates iambic pentameter. And the fact that some native speakers can’t shouldn’t be too much of a surprise, either. Sometimes, it’s hard to articulate the hard stuff, but it doesn’t make them any worse at speaking English. It just confirms their status as an empowered amateur.

But in the end, it’s still English. And I would never speak English, because I’m better than that. I have a degree in communication sciences and am very active on ChatHub.

Stance

English is a mess. It’s a mess because long ago, a population of Celtic people were overrun by a few populations of Germanic people. Sometime thereafter, a portion of the island they were living on became occupied by the Roman empire. Later still, in 1066 A.D., William the Conquerer effectively made England a French colony. For much of its history, English was a language of subservience, or a language in flux due to one or more events of occupation or contact.

English became a language of historical significance, and a tool with critical mass, when a perfect storm of events, beginning in the 15th Century, culminated into today. The discovery of coal reserves, the establishment of British Empire, the Protestant Reformation, the colonization of the New World and India, and ultimately a scientific history that reaches from Newton through to Turing poised English as a major international language for trade, science, and engineering.

This is terrible. The concept of linguistic relativity suggests that using a language subjects you to the subconscious paradigms expressed in it. The fact that English misses a number of aspects, tenses, person and number distinctions screams for a solution created by people who are too smart to use a natural language as their lingua franca.

What it really all boils down to is the fact that English is a great language for complete amateurs to learn. The overwhelming majority of people who speak English as their only langauge learned it as their first language and stopped there. Most of them are terrible at it and make egregious mistakes in its grammar and usage every day. Do we really want to continue to allow these groundlings of such low acumen to continue to pollute linguistic landscape?

My position is thus:

• English is full of surprises: ox -> oxen; be / was / am; the pronunciation of the word “subtle”
• English is inconsistent: food vs. good; read (present) vs. read (past); one sheep two sheep…
• English is flaky: English in Canada is like a completely different language from the English spoken in India or South Africa. And I’m still trying to figure out what the lady in the beginning of “Let Me Ride” from Dr. Dre is saying, but it’s allegedly also English.
• English is opaque: I before E except after C doesn’t even freaking work most of the time

Don’t comment with these things

I’ve been in arguments about the English language a lot. I hear very generic counter-arguments for why we shouldn’t scrap English all together:

• Don’t tell me you were born speaking it. Learn something else. Yeesh.
• Don’t tell me everybody’s using it. If everyone jumped off a cliff, would you? Yeah, I didn’t think so.
• Don’t tell me it’s an international lingua franca for science, engineering, and trade. If it was so great, then why did they have to borrow the word lingua franca from Latin? And why would lingua franca mean “French language”? See, you peel back the onion, and you’re just left with more and more layers of absurdity. And tears.
• Don’t tell me a subset of its rules were inherited from Latin and French. Really, what’s the point of speaking some weird wrapper around Latin when we can just speak Latin? Then we wouldn’t have to worry about appropriately converting noun declensions to their place in English (type safety). How else will anyone understand a Latin loan word, if we’re not properly declining the term into its ablative case? And the Germanic part is like Perl. And Perl is hard.
• Don’t tell me that Shakespeare, Milton, Twain, or Meyer (yes, I went there) wrote in English. I’m aware! They could write in pictograms, for all I care. You’ll always find smart people who can overcome the shortcomings of their platform.
• Ideally, don’t tell me anything. Hearing or reading too much English on any given day is literally enough to send me into a flying rage. So I wrote a simple script in Erlang that filters out all email messages not written in Esperanto, so odds are that I won’t even see any emails you send me.

Side observation: I loooooove Esperanto. It’s got a great spec; it’s completely logical; there’s this very smart committee of fantastically wealthy, extremely popular people who spend all of their waking hours monitoring its usage to prevent it from getting too illogical. I mean, look at George Soros. That guy was raised speaking Esperanto and he gets by just fine.

ENOUGH!

Have I made my point? Almost any sufficiently utilized system develops irregularities, particularly around areas of high frequency of interaction. It’s one of the reasons that we have irregular verbs and irregular plurals. PHP’s development, without any kind of an academically or professionally defined specification, is as organic as the growth of natural language. It’s only recently stopped being a pidgin of C and Perl over the last decade and entered a status as a well-used but poorly regarded creole. That’s exactly the status of English before the Globe Theatre or the first translation of the Bible into what we refer to as Modern English.

Specifications, or the lack thereof, do not define the utility of a language. Prestige does not define the value of a language. If you don’t like a language — either natural or programming — don’t use it, and leave it at that.

Could you imagine if I wrote this about the fundamental shortcomings of a language from a developing nation, or a low-prestige dialect of any language? Everyone would rightly call me a bigot. I’m suggesting that squelching about the shortcomings of a language without trying to offer solutions does roughly amount to technological bigotry. It’s not productive, and it’s not welcome. Participation and dialogue is always welcome. But you’re not opening a productive dialogue by denigrating the subject of discussion.

The word ‘search’ is a negative word. It fairly reeks of loss and effort. You lose your car keys and you search for them. Your pet runs away and you search for her. Having to search implies loss. It implies effort. Search is a means to an end. You search to rescue; you seek to find. There is little that is pleasant about the process itself. The only time to feel good about a search is when it ends, successfully.

With a heavy sigh, I continue reading. I sludge through a subjective, nebulous paragraph espousing the author’s opinion on the connotations of the term “search” versus the term “find”. He cites “searching” for keys when you lose them and “searching” for pets when they run away as reasons why the term “search” is so awful. Let’s pause for a minute. The difference between calling an application a “search engine” versus a “find engine” is even more trivial than pragmatics or semantics; it is a marketing problem, and has no effect on the ultimate functionality of such a utility.

So beyond marketing issues, which sites like Bing and Ask have previously attempted to tackle (in my opinion, rather unsuccessfully), what is the problem with search? Dr. Whittaker feels that search is broken, in summation, for the following reasons:

• Search engines serve up search results pages — they don’t send you directly to what you’re looking for
• Search engines do this because their revenue is largely based on ad delivery hosted on the search results page
• Large companies that are centered on a search application have no incentive to innovate beyond this paradigm, because doing so would cause a loss of revenue. This means we must look to outsiders for further innovation on search (e.g. Apple’s Siri)

This line of reasoning is patently absurd. This assumes that for most queries:

1. There is only one correct page for any given query.
2. The user knows exactly what result they want.
3. The user is 100% right in that this result is appropriate both for himself or herself, as well as for all other users. Furthermore, they are right in that there are no results remotely as relevant as their desired best result for same query.

I’m a little surprised that the kind of ambiguities that make serving a results page necessary escape the author. He is a self proclaimed “former Googler, former professor and former startup founder”. Dr. Whittaker readily conflates search, which is as much about discovery as it is about navigation, with mind-reading. To his credit, I probably should have just stopped reading after he devoted more than one melodramatic paragraph to the semantics of the term “search”.

Here’s an excellent example of everything I find wrong with this article: after having some questions about the actual usage statistics in these contexts, I used Google to get some numbers on how many documents match these exact phrases. I used the verbatim tool and found numbers identical to the default setting, so let’s naively assume that it’s applicable across all of the hidden variables for every other user in the search engine.

“looking for my keys”: 313,000 results
“searching for my keys”: 104,000 results

“looking for my pet”: 4,460,000 results
“searching for my pet”: 538,000 results

I may not have combed through every result to make sure that they relate to the context of lost pets and keys, but at least I’m using real data to inform my opinion. Using this information, we can see that the exact straw men the author constructed don’t hold under their own weight. If he had tried to be in any way experimental about his argument, he could have saved at least two people from writing an acerbic blog post. Dr. Whittaker makes the same mistake that the generative linguists have made for decades: informing what they believe to be objective arguments on intuition alone.

The fact that I was able to access this information through a search engine unravels Whittaker’s perceived intentions of the general purpose of search. Let’s not even get into the “discovery” use case, whereby an individual seeks to learn about a topic by accessing multiple relevant documents on an identical term (really, what is the one right page for “Napoleon” or “The Beatles”?). I think one example of what makes search a robust and useful platform in its current state is counterpoint enough.

In my use case, I made a search not in hopes of accessing a single page, but in search of information about the term itself, and its presence on the web — search metadata, if you will, that is extremely helpful in making decisions about future search behavior, and the importance of each document relative to the other. Google is not just a “finding” engine; it’s a tool for research, and a warehouse of knowledge. If you want to get a single page and skip the SERPs, just click the “I’m feeling lucky” button and quit complaining.

I’m currently working with Time Doctor, LLC, a company working on the sites TimeDoctor.com and Staff.com, as a retained technical advisor. It’s a great opportunity to work on different software in my spare time and continue honing my craft. I get the opportunity to analyze code quality and application architecture, and solve some interesting problems on a useful stack of productivity and hiring tools. I’d see myself continuing on in this capacity for the foreseeable future, since I really enjoy doing it in my free time.

Here and there, I’ve also been providing some additional time to Earmilk.com, a really cool cutting-edge music blog that’s got some interesting new features on the up and up. Keep an eye out for those changes to come into effect real soon.

My big announcement is that I will be changing my full-time employment from Software Architect at AetherQuest Solutions to Senior Software Engineer at Wikia. I’m making this change for a lot of reasons. Working at a top-50 website is definitely up there. And I’m really excited to get into Big Data again now that the technology has progressed a bit. I’m excited to use my search knowledge again, too, after spending so much time on other sort of enterprise-level concerns in my current role.

I’ll be continuing strengthening my skills in application architecture, technical definition, and project management on the side as I work through some important issues with Global Workforce. This is important to me, because I don’t want to lose any of the momentum I’ve achieved over the last year and change.

I had a lot of great opportunities to learn and grow at AQS, and I’m really proud of what I accomplished there. But it’s about time I put the “language” back in “Language Hacker”!

But we don’t have “clients”. Why should we have an SLA?

SLAs are useful for everyone. They give your team an opportunity to sit down and decide what is a sensible turnaround for important issues with your software. And if you feel like you don’t have any clients, think again; if your software is a product that fuels other business within your organization, you are your own client! This goes even more so for a client-oriented business — you are your foremost client, since unsuccessfully serving external clients is a disservice to your own business.

You may already have a release process. Well, you should. You may already have a process for prioritizing issues. But an important component to introducing an SLA into the mix is the line of communication it creates. It gives you an opportunity to introduce a dialogue between all groups affected over what your team’s priorities should be at any given time, what is a reasonable time for arriving at a solution, and what sanctions (if any) exist against failure or to promote success. This brings me to my first guideline:

Keep It Collaborative

Think of all of the people involved in your product: developers, managers, users, designers, QA engineers, etc. If you don’t have an SLA, try this simple exercise: walk up to one of each of these different roles and ask them what a reasonable amount of time it should take to fix a bug that completely broke the ability to make money with your product. Now ask them what will happen if it takes longer than that to fix that bug. For as many people you ask this line of questioning, you’ll get as many different answers.

With an SLA in place, everyone who interacts with the use or development of the product should be on the same page with respect to what the expectations are. If they’re unfamiliar with the specifics, they will at least know they can reference a document and find out without asking around. When was the last time you got everyone in a room together, or looking over the same document, to agree on how long it should take to fix something that could make or break your business? Wouldn’t you feel a bit safer if you knew everyone had an idea of how long was too long? And what if, perhaps, certain scenarios were incentivized for or against in advance? If for instance, you were to give developers a nominal bonus (say $25) any time a bug of the highest severity was addressed within the SLA (or perhaps even within a shorter time limit), would you feel more or less comfortable that the issue would be addressed in time to prevent harm to your business?

Now think about the different kinds of people involved in the software from another aspect: there are the users or clients, obviously; then there are those involved in implementing user requests — technical or managerial. Obviously, in most organizations, a layer of customer- or client-facing individuals interface the need of the users, who generally want everything right away, with the group tasked with delivering what the users want — preferably at a pace that doesn’t keep them burning the midnight oil or rushing their work. Everyone needs an open line of communication when developing an SLA — either as an agreement within a single organization or a contract between more than one. This is a method of achieving mutual understanding and reaching a compromise on what can be determined a reasonable turnaround. It also improves the likelihood that the process in the SLA will be followed, since fewer parties will feel that the agreement has been forced upon them, but rather reached through a negotiation they were involved in.

Keep It Granular

A well-written SLA should have a description of the various scenarios from the reporting of a bug, to its prioritization, all the way to the concrete turnaround time for delivery of its solution. This means you need at least three separate classifications for reported bug:

• A high-severity issue
  • Usually financially impacting, but can also be highly time-sensitive
  • Needs a high turnaround time
  • Generally requires a hot-fix or off-cycle release, and sometimes a patch
  • These may require additional process in advance of the solution (such as reverting to a previously stable release) in order to ensure the product’s integrity
• A medium severity issue
  • A broken, but not financially vital or time-sensitive functionality
  • Should be addressed during the current in-development release
• A low severity issue
  • Minor UI issues, or an unexpected result in an edge case
  • Not financially impacting in any way, still usable
  • Can be prioritized to the backlog, to be addressed within an agreed-upon number of releases

If you have N levels of granularity, you should have N different turnaround times, and N different criteria. So make sure that you’re not overcomplicating things, but also make sure that the scenarios you describe effectively capture the issues you encounter from the origination of the bug report to its resolution and incorporation into a stable release.

Keep It Reasonable

This really only applies to issues of a higher level of severity. Anything that can be prioritized for the current in-process releases is not that difficult to handle. If there’s one positive thing the Agile methodology has brought to software development, it’s the philosophy that an in-development feature with a threatened deadline has the option of changing deadline or changing scope. Don’t be afraid to iterate more or delay a new feature if it means addressing a medium-severity issue of greater importance to the client or the organization. Make sure your SLA includes language describing the risk to meeting deadlines for agreed-upon feature changes or requests.

Different groups may have different expectations on what is a reasonable turnaround time to address an issue of a given severity level. However, in the words of Lil Wayne, numbers don’t lie. Promising to deploy a hotfix within 4 hours of an initial bug report, if your average issue resolution time is 3 hours and time to prepare a release is a half hour, is a dangerous game to play; it will only set you up for failure.

It only takes one vexing bug to leave your team looking incompetent. It’s better to under-promise and over-deliver here. Don’t give yourself an excess of padding, but make sure that you’ve taken into account the worst-case development time for your highest-severity issues, and the time it might take to QA the solution and manage a release for it.

Since an issue of the highest severity is a stop-the-presses kind of issue, it’s okay to entertain options here that may otherwise be outside of process — for instance, patching the solution in advance of deploying a hotfix or off-cycle release — in order to meet the SLA. You should include language that enumerates these options and explains the risk for each scenario. If a solution is found within the SLA, and the difference between on-time delivery is opting for one of the riskier scenarios or not, the parties involved should be again be informed of the choices. At this point, the SLA should be considered met, whether or not the riskier option is invoked.

Incentivize Your Team

An SLA isn’t much more than a mission statement unless there is language that explains what happens if the terms are not met for a given issue. This portion of the document might differ between what’s provided to, say, a developer compared to a client. For a client, it may involve future discounts or credits. For a developer, it may affect something like bonuses or vacation time. That’s not to say that every time a developer does his or her job, you should be shelling out extra cash. This is really just an example of one way to put some spurs on your agreement.

For the good kind of developer, the kind you should hope to have in your organization, meeting the agreement and maintaining the integrity of the product and the company that builds it should be incentive enough. However, if you’re in an organization that does performance bonuses, it may be worth using the SLA as an indicator of performance. This is a good way to introduce a level of transparency that can serve as a motivating factor. Another passing thought on this is perhaps to include language in the “developer version” of the SLA that comps time spent outside of an eight-hour work day to deliver a high-priority issue at a certain rate — for instance, one hour of paid vacation for every four hours spent.

Personally, we don’t do any of that for the developers in our group. None of this kind of stuff is going to buy a sense of ownership, but there are a lot of organizations that modulate their compensation based on performance, so I thought I’d give the option. My opinion is that paying a fair wage and making sure the people who do the brunt of the work have an opportunity to contribute to the process of developing the agreement should be a better motivator. Involvement builds ownership. A bonus program in the wrong hands can be executed poorly or arbitrarily, and is just as much a dividing factor as it is a motivating factor.

It’s All About Integrity

A service-level agreement is, in the end, an opportunity to put some real concrete language around what constitutes scenarios that derive from the normal process of maintaining a software product. It also sets expectations for deliverability on issues of routine maintenance. By further defining how issues are evaluated, and agreeing on a reasonable turnaround time, you can set your team up for success — both in the eyes of stakeholders, as well as in their own eyes. Another great thing about this exercise is that it will make developers feel protected from the whims of an emotional client. Doing this for your team will allow them a sense of breathing room, and the peace of mind to know that they are being looked out for in the grand scheme of the process of product development.

Think about it: you have an opportunity to turn “I need it here yesterday” to “I should expect this up and running in 12 (or 24, or 48) hours”. You’re protecting the integrity of your organization by preventing the attitude that deliverability should be instant. Without something in place to modulate this, you’re already fighting a losing battle at the initiation of a bug report.

You have an opportunity to keep middling UI issues that a single stakeholder thinks is the end of the world from causing yet another late night in the office for no reason other than his own peace of mind. Expect it to be handled by noon tomorrow — not midnight tonight. Remember that SLA you helped us develop? You’re protecting the morale of your team, and thus its integrity as a working unit. There’s no need to fatigue your employees with long hours and the fear that they may be kept late, so long as expectations are adequately set between all groups internally.

Every organization can use an SLA. It’s the starting point at which all issues, and their solutions, are prioritized. It introduces transparency, and it gives everyone a bit of breathing room. And, executed effectively, it’s probably the closest to collective bargaining we’re going to see in the world of software development. So take advantage of it.

The challenge of coordinating productivity between locations up to three hours apart can be rough at times — particularly when it comes to the days approaching the creation of a release candidate from the development branch, or when a “should have been done yesterday” hotfix floats to the top of the ticket pile. However, there are some great perks. You get to show up to work in the morning with more work collectively done than when you left.  For clients on the East Coast, we can work further into the night to get them what they need when they get in the door first thing the next morning. My favorite part of having a distributed dev team, though, is the technologies we have implemented in order to keep communication active and instant. Here are a few guidelines that I have found really enable a development team to get a lot done and stay in the know.

Get on IRC

At least for my use case, there’s no need to pay money for something like BaseCamp or HipChat when you’ve got a little spare server space. It should take you maybe a few hours to set up an IRC channel for your developers. Require that they stay on this channel any time they’re working. I have a bot I call Hal (cliched, I know) in the channel that keeps the channel open and logs all conversations. Bots are generally easy to extend, and come in most languages you should be comfortable in working. For instance, I added a module to Hal so that he will poll our Git repository to report when code has been pushed to a branch, and announces it. We use Jira for version control, and any time a ticket is mentioned by its key, he will give a link to the ticket’s page as well as its title. If we’re worried about our various servers, I can have it poll them for load average and CPU usage. IRC is great productivity tool, because it’s a simple protocol with plenty of extendable, open-source bots. You can tailor your communication experience to make sure everyone can stay abreast of what’s going on throughout the day with a minimum of distraction.

Ditch the Standup and Do Your Daily Scrum in IRC

Even if you’re all in the same office, what are you doing making people get out of their seats, stand up, and provide their uninterrupted attention to something that they’re only half listening to and will likely forget anyway? Get what people are up to in the logs. Let them keep working on what they’re working on. Meetings kill developer concentration. I hold a 9:15am Pacific scrum meeting every day. That’s 12:15pm on the East Coast. If my developers are busy working on something important over there, I’m not going to make them get out of their seats, walk into a conference room, and dial into a GoToMeeting call just to chat about their day. By digitizing the daily scrum, you keep all the value of a constant contact point at a steady interval without breaking anyone’s stride. You also get the benefit of logging as well as the additional value of customizing the scrum experience with an IRC bot. And of course, the best benefit of all: it doesn’t really feel like a meeting, and we developers loathe meetings, don’t we?

Continuous Integration on the QA Instance

Since we’re all distributed, I think it’s a much better idea to cut out the risk of multiple people attempting to deploy development code on the QA server at the same time. This is, of course, where all the code for the next release is going, and we want to make sure that it can build by itself at all times. I use Hudson to poll the development branch of our Git repository, which will then update the code and automatically build everything — rolling back and running new database migrations, clearing the cache, etc. I prefer to keep an eye on Hudson’s builds via RSS, since Hal is already announcing pushes to the development branch. However, it’s your IRC channel; do what you want with it! You can easily consume Hudson’s RSS feed to create a bot module that announces whether a build was successful or not. Most continuous integration solutions accommodate some way of automatically gleaning build data, so don’t feel like you have to use Hudson to get this level of value.

Code Reviews and Release Notes

Because everyone should be aware of the direction of the product and have a stake in its level of quality, we encourage code reviews for all code submitted. Another developer has to pass your code before it can go into a release candidate — present company included. I also keep other developers updated on the details of the release notes so that they can skim functional changes and recent fixes in the product. Internal release notes are available as a plaintext file in the code base, but I also send an email out both for the release candidate and the final internal release. This has helped catch snafus or missed must-have items on more than one occasion.

Daily Summaries

In a perfect world, you could go into Jira and get an hourly breakdown of what everyone did that day. The reality is that most developers are not consistent time recorders, even when it’s important, and even when you hammer it into them — again, present company included. The reality is also that sometimes developers work on things that do not have or require a ticket. Encourage your developers to record what they worked on in a short summary via some kind of internal publishing tool. In our case, each developer has an internal blog they will record a daily summary into, detailing what tickets were worked on. This is useful for splitting out whether a developer worked on maintenance release items, or maybe a major functional item that will be worked on in parallel with multiple sprints. They may have even had all their time diverted to a hotfix. It’s hard to know — especially when you’re filtering things by fix version — without the “executive summary”.

Leverage The Time Difference; Don’t Bemoan It

With developers a few time zones ahead of you, sometimes it’s hard to make sure everything is going to get done. It’s not cool to require people to ask permission to head home at a reasonable time, and they shouldn’t have to. If something you wanted to get done by the end of the day Eastern Time doesn’t get done, don’t expect someone to stay late just because it’s only 2:00 or 3:00 Pacific. Set goals that are reasonable for the time zone of the person working on it. Make sure you’re utilizing your resources with the time difference in mind. If a developer is working on something that they know should be completed that day, make sure that they’re in contact about its status at the end of the day. If it’s not completed, and it’s part of a release that has to get shipped out ASAP, have a set protocol, or at least a contingency in place to coordinate a hand-off. A good guideline here is to make sure that all in-progress branches get pushed at the end of the day, and that tickets get commented with details on the work done thus far so that anyone else can pick it up.

Conclusion

For my team, these guidelines have done a lot in terms of improving the quality of our code and the quality of our product. It has also improved the opinion our users have not only of the product, but the people behind it. You can use most of this information without having more than one office. If you have staggered shifts or people on different schedules, you can even use the last guideline. The most important part here is that we’re actually improving how we manage developers’ priorities and output while reducing the amount of time they’re not working. We’re not breaking anybody’s stride, and we’re actually encouraging active communication without ever having to leave your desk. The payoff is clear: developers are happy because their meeting load is minimized; you can actually have a higher frequency of meetings since you don’t have to coordinate a bi-coastal call; everyone knows the immediate status of a release, so anyone can work on it at any time — shortening development time and increasing client responsiveness.

If you’re stuck slogging through call after call trying to coordinate developers in more than one place, or if you’re just sick of having to get out of your chair for your daily standup meeting, give these methodologies a shot, and see if it works for you.

It’s 2005. I had just graduated from SUNY Albany, and had my grad school plans all figured out. I’d be going to UT Austin to continue my studies in linguistics. With a few months before my move to Austin, Texas, I picked up some temp work that landed me at a Verizon Wireless store at Crossgates Mall, just outside of Albany, New York. It was a mall job, but as far as mall jobs go, it wasn’t terribly bad. Sure, I still felt over-qualified, but that was before I had much of an idea of the value of a bachelor’s degree in liberal arts (or the lack thereof, as we’ve all come to learn since).

The official role was greeter; I would catch people as they came in and try to get them time with either a sales associate or a customer service rep. If necessary, I would sign them up for tech help from one of a revolving squad of anti-social metalheads in the back room. It was a lot of standing, and a lot of schmoozing.

Verizon not-so-subtly used their temporary labor service as a cheap HR department. Most of the non-temporary employees were at one time temps, and management regularly put individuals designated as greeters into training for customer service roles. I got a lot of experience with the variety of roles available at a cell phone store, and even had an opportunity to do a little off-the-cuff salesmanship when things got a little busy.

All of my exposure to the different roles and responsibilities got me thinking about what I could do to make the place run smoother. This was at the forefront for me, as I was primarily responsible for corralling people to different sections of the store. As a result, customers would often hold me responsible for not getting individuals with their salesperson or CSR in a timely manner.

The issue that I first targeted involved the potential logjam that occurred in the sales queue. Salespeople had to spend a lot of face time with potential new contracts or new phone purchases. For every individual they spent time with, they had to fill out a large form about their encounter, outlining what items were bought, and other sundry details about the interaction. Each form took about five minutes. It was something they could stop and go back to, and often would develop a significant backlog if they had a a full queue of customers.

The customer queue was tracked by a simple form on a clipboard, at the podium where I would stand. Names were written in pen, and then, once handed off to a salesperson, were crossed off. For a salesperson to begin serving a customer, they had to leave their desk and closely inspect the form. This meant that they would often have to stop filling out their back log of forms just to check if the scribbles on the clipboard were crossed out names or new customers awaiting service.

My “big idea” was idiotically simple, if you think about it. Grab two highlighters from the the back room — one pink, and one blue. For a new customer, who had not yet been served, highlight their name pink. The color could be seen from the other side of the room easily, so a salesperson could just catch a glimpse of the clipboard, and, if there were any pink lines available, they would then know to stop what they were doing and start helping customers. Customers would be taken off the queue by highlighting the pink lines with the blue highlighter, which resulted in a dark purple, providing an excellent contrast to the light pink.

This idea was a big hit with the sales staff. It helped them stay on top of their paperwork, and kept them almost instantly available to customers at all time. However, one of the younger managers —  whose primary responsibility was sales — took severe issue with it. He felt that their motivation should be the commission they stood to make. I wasn’t sure how these two concepts clashed. In the end, it may have just been that he didn’t like the idea. He told me to stop doing it.

And then a regional manager came by for a visit. The sales staff told him all about my great idea, and how much easier it made everything for them. This manager loved my idea so much he literally took two highlighters, put them in my hand, and told me to keep doing what I was doing — after the assistant manager had already told me to stop. The pecking order seemed pretty clear, so I went back to doing what I had been doing, and witnessed a notable improvement in efficiency return to the sales queue.

The next week, I showed up at work only to immediately be summoned to the back office. The head manager was literally holding up the highlighters as if I had brought contraband to the workplace. I wasn’t fired, because I was an employee of a temp agency, but my contract as a greeter with Verizon was terminated prematurely due to insubordination.

At the time, I was pretty dismayed. It was the first time a professional engagement ended in such an ignominious way for me. Now, I just joke about it whenever the topic of innovation or cell phones comes up. Of course I didn’t want to be a lifer at Verizon Wireless. I was just hoping to spend a few more weeks making money before I moved across the country — and maybe even transfer to a store there for some part time work during my studies. I suppose telling them I would be moving in a month didn’t really give them a reason to keep me around anyway.

I could spend this paragraph using this my personal experience as an anecdotal example of our failing broadband and wireless infrastructure. I could vehemently rail against the big telecommunications companies in this country, how they wantonly absorb government subsidies without any returns in the form of improved end-user service, but I’ll skip it. I’d rather not contribute to the pool of vitriol you’ll see in most hackers’ blog posts about wireless phone or broadband service and the companies that purvey it. But I will say that these are organizations that clearly have a cultural problem of discouraging innovation. There are bigger companies that do better, and there are smaller companies that do worse. It’s not their industry; it’s not their size.

I learned a great deal from this fiasco — lessons about the interpersonal and bureaucratic sides of working in a company that stick with me to this day:

1. Just because an idea is good doesn’t automatically give it traction with management. This speaks for itself. If you implement something useful without getting approval in a workplace that values management over traction, no amount of justifying yourself will keep you from getting read the riot act.
2. Multiple managers make for noisy signals. If you absolutely need more than one person managing someone else, you need to make absolutely sure that the employee knows who’s in charge of them for what aspect of their job. Otherwise, you’re just confusing people, and making it harder for them to do their job effectively. If you’re a co-manager of an employee, make sure you’re staying in your sphere of influence. Contradicting another manager could make more trouble for the employee than for you, and that’s particularly problematic from an ethical standpoint.
3. Efficiency is not the only motivation. The sales manager’s interest was in motivating the sales staff to stay on top of their queue of customers, and to be proactive in servicing customers. He valued customer service over effective data entry. It was his prerogative to identify and rank the sales team’s priorities, and I can understand how staying up on your forms could take a back seat to making sales. There are countless applications for this lesson in the software industry. Your approach may optimize for all the vectors you’ve identified, but that doesn’t mean there isn’t one you haven’t taken inventory of. That “unknown unknown” may have an untenable cost introduced to it due to your approach. Collaboration usually helps catch these before they become a problem.

When all is said and done, I’m kind of glad I got “fired” from this job. As a student, I hadn’t really encountered real failure before then, and it was my first exposure to real rejection therapy. It was also my first experience learning that things aren’t always fair in the workplace, and sometimes a paycheck is better than being right.

I’ll still never use Verizon though.

Object-Oriented Rollbacks

I’m a big believer in using some kind of database change log for any project you’re working on. As someone firmly entrenched in the PHP world, I’m used working with serial migrations such as you’d see in the DbDeploy functionality of Phing. So the form our DB change log table takes basically lists the major-level component upon which the logged migration interacted, and the migration number. Furthermore, with each release, we have a Version class that always lists the current version of the product and the most stable migration number for each type of migration.

This basic table allows us a couple of different operations:

• Update to latest migration available: This is used when we deploy stable code, or when we’re updating our local databases to the current iteration of development code
• Update only to the latest stable migration This is used for both updating to a stable version of the database schema as well as returning the database schema to its last stable state.

For both of the above actions, if the database’s change log shows that it is at a migration later than the most stable one, we roll back all migrations, removing the latest row from the change log to reflect where we are in the state of the database’s schema. Rollback logic was originally a just a required method of each migration class that implemented a “Reversible” interface. The migration would run a deploy() method when rolling forward, and a rollback() method when rolling back. Here are some issues with utilizing this methodology off the shelf:

1. Before pulling down new code in your development branch, you’ll have to roll back to the stable branch in case you need to, for instance, ameliorate a code conflict. Sometimes, this unnecessarily destroys test data.
2. If you forget to roll back to stable, you now might try to roll back code that never ran if migrations have been reordered or changed
3. Your database only knows whether it ran Migration_N, and not whether calling Migration_N::rollback() will actually roll back the logic created by Migration_N::deploy(). It’s entirely possible that Migration_N is now Migration_N+1 or even Migration_N-1

So how do we mitigate against these possibilities? Ideally, we would still like to tie rollback logic to the same migration class that is deploying the changes to the database schema. However, we don’t want to make the mistaken assumption that calling Migration_N::rollback() will universally roll back what posted the original increment to the database change log.

Pairing Deployments to Rollbacks

My solution to this problem is to insert rollback logic directly into your database change log. I added a TEXT field called “rollback” to each database change log. I then changed the Migration’s rollback() method to a getRollbackSQL() method that returns the fully valuated query that we store in the change log. We now take the following steps when rolling back a migration:

1. Check if the change log entry contains rollback SQL. If so, run the query, delete the row, and continue iterating
2. If there’s no rollback SQL in the change log entry, we try to call Migration::getRollbackSQL() for the migration number we’re on.
3. Worst case scenario: there’s nothing available for us to do. For instance, the change may have been irreversible, data cleanup, or just asymmetrical. Just remove the entry from the DB change log, and hope that we

This behavior is encapsulated in the parent Migration class’s rollback() method. So how about the old migrations that may have utilized rollback() in a way that (here’s hoping rarely) flouts the paradigm of just executing a query against the database? And how about future cases where a simple DB query may not be all we have to do during the migration? Well, for these cases, we can just extend the rollback() method itself instead of the getRollbackSQL() method. This allows us freedom when we need it, and simplicity when we don’t.

Application

Let’s walk through a scenario where this methodology provides a significant value against a process that requires we always update our database to a stable state before updating your development branch.

Joe Developer creates a migration in a branch to address a ticket during a maintenance sprint. He finishes development on the ticket, and merges his completed work, including the migration, into the development branch. As he’s finishing this up, Joe gets a call from his boss. Someone needs a hot fix, pronto. Joe immediately switches over to work on said hot fix, branching off of the most stable tag of the software.

As it turns out, the fix also needs a migration, but Joe never rolled his changes back. Switching back to the development branch just to roll back to stable would break his stride, but it’s something he’d have to do now that he’s working on the hot fix, which doesn’t have an instance of a development-level migration. However, regardless of what branch he’s got checked out, if he’s working under the new paradigm, he can easily roll back any unstable migrations (provided they all use the getRollbackSQL() methodology), and can be sure that he’s rolling back the old migrations, and not any migration he’s currently coding under the same migration number. Furthermore, when he merges his completed hot fix into the development branch, if he’s using continuous integration, he can be assured that the migration deployment process won’t get confused by any migration reordering. Otherwise, there’s a chance that on some instances that use development code, Migration_N_hotfix (the real Migration_N) would never run, or that the rollback method would be called against Migration_N_hotfix when we’d actually want it to run against Migration_N+1_Dev now.

Conclusion

Obviously, this is just one way to handle the issue — particularly, if you’re using migration classes rather than simple SQL scripts. Reconciling server-side application code with database-level logic will always introduce areas where you can improve your coordination. Some people just opt to avoid this trouble all together and write their migrations as .SQL scripts rather than as classes, but there is a value in treating your approach to migration at the application layer, through objects. For instance, in our application, one type of migration is actually iterated over a variety of databases, so we can just act over the same table on a number of databases by just creating one class that manages providing the appropriate values for each dynamic script.

Whenever there’s an opportunity for iterating your software, there’s often an opportunity for iterating your process. The original approach was developed for lack of having any database rollback whatsoever — changes were in effect permanent, leading the shared development instance to diverge significantly from the stable instance in unexpected and irreversible ways. I’m sure there will be opportunity in the future to improve things even further. Altering your database schema invariably introduces some level of risk. So what are some of the tricks you use to safely iterate your database?