19 Frog Driven Development

The frog and the boiling pot
Remember the frog in the pot that is boiled to death because it doesn’t realize the water’s temperature is rising and slowly but surely it is boiled to death instead of jumping out?

We were that silly frog during the development of one of our products: slowly getting boiled without realizing that we were getting more and more screwed. The symptoms: customers complaining about buggy software, release schedules slipping horribly out of control and costs rising beyond triple the budgeted costs.

If only we’d asked ourselves if we were being a frog and had jumped out of the boiling pot we wouldn’t have gotten burned as badly as we did.

This is the story of our efforts at developing an Outlook plugin. It’s a story about frogs, boiling pots, outsourcing, weird randomness and mismanagement¹. Hold on tight for something that reads like a good ol’ BOFH story.

Getting into the pan
Development started off like any other development process. We knew that building a sync plugin for Outlook would require platform-dependent expertise. We had a great experience with our OSX client development done by an external freelancer who was eager, dependable and got the job done fast (thanks Gernot). So we made the decision to outsource the job. We quickly found a suitable list of developers and chose one.

The developer we chose had the right credentials. He wasn’t the cheapest but seemed the best value for money. Let’s call him Kev². Kev was an independent developer, we had direct and frequent contact via email, chat and phone. Kev was very keen to take the job and was excited about starting. We mutually agreed that Kev would be online every day in IRC so we could answer questions quickly and he would commit SVN daily.

We, the frog, are loving this new pot. Nice temperature by the way. Yes, the water’s perfect.

Things start off good, we based the project on an hourly rate. But the risk was low because every iteration would provide running software. Agile. Risk-free. Perfect. Up till now, all is good. We’re excited, eager and ready for a challenge.

The water is getting hotter
Then development and SVN commits started to slow down, Kev didn’t join us in IRC anymore and we had trouble getting any replies to emails. Occasionally we would get an email with a new build and at this time it seemed like we were nearly there.

We switched the project from an hourly rate to a fixed fee after seeing that the costs would be more than we budgeted. This would move some of the risk to Kev and be the right incentive to have him deliver both a working release and the source code.

Getting new build releases slowed down even more and we wouldn’t get the corresponding source code. We’d get responses about him using multiple PC’s, not having “the code on the internet connected machine”. And IRC just “wasn’t his way of working”. Very different language from when we started.

We remained in the pot. The water? Comfortable. Nice. Hot? Nah, we have releases! Working software. Teh Awesome!

Boiling?
After 3 months of hacking, including delays and the increasing communication difficulties — Kev finally emailed a working alpha version. We were relieved and excited to have something and I guess actually happy that we’d stuck with Kev through the bad times and persevered through the difficulties. All we needed now was the source code.

In hindsight this is where things really started to suck the fat kumara. As soon as we finally got the source code we found out Kev had used an open source project in one of the parts of the Outlook plugin. Open source? Yeah, of the kind that you cannot use commercially, and thus we couldn’t release it. Crisis! Death! Nazgul! Badgers with small guns shooting you in the eyeballs.

Ok, you’ve got to give it to Kev, he did agree this was wrong. And he would do his utmost to fix it. We decided it was better to use a licensed version of another library and Kev would only have to integrate things. Easy as cake. Crises averted.

We of course remain in the pot. Still ok, I guess. Hot? Yeah a little. But, we’re still in control. We’re fixing it. Crisis averted. Just a little more time…

Code held for ransom
During the next months Kev managed to fix the Outlook plugin and we were able to release an alpha version and subsequently a beta version. We kept having to fight to receive the source code.

And never got it.

Then more than a year after starting the process and still not having received the latest source code we found out another kicker. This finally woke us up and we realize we’d not just gotten into boiling water but we’d been there for a long time.

Kev admitted to us that he’d gotten into great financial problems and that the IRS had confiscated his gear and that there was no other way to get our source code than pay the total that he owed the IRS. You can imagine our reaction. There’s no way we would or could do that. Kumara. And. Suck.

It’s boiling!! Jump! Jump! Get out! Get out! Anywhere but here…

The consequence was of course that we didn’t have the source code. 12 months of time, energy and for us a huge pile of money was lost to this process. And thus we had to rewrite the whole thing! We were close to being boiled alive, without source code and thus forced to jump out.

Jumping from one boiling pot into another
So we found an alternative. This time the perfect solution. We thought. Everybody’s doing it. You know, outsourcing. To India. Cheaper, more developers, thus faster… right? Tight management to be the issue. Clear budgets and all. We found an outsourcing firm, with project management nearby, literally 1km away. So it would be tight. We agreed on a budget upfront. And all would be good. Heaven. Did I mention tight?

After 4 weeks the close-by-our-office “Management” came back to us, telling us they wouldn’t be able to stick to the budget and planning. They told us it would be based on “Time and materials”. So I assumed (stupidly — like a frog) this would only take 1 or 2 extra months. At this point even a frog would get suspicious but I’m not a frog and still it ended up taking another 4 months and we still had no working client.

It was similar to the Kev experience but this time I made sure they kept sending us the source code. After pushing hard we received weekly and monthly updates including source code. Thankfully we had direct access to the team and developers working on our project via IRC and MSN. However progress was still too slow and the time and materials was getting expensive.

Jump out! It’s going to boil here too! Jump before it’s too late. But this time, find a nice pot NOT on a stove.

Out of the boiling water
In the course of developing our Outlook plugin we’ve hired an external single developer in Sweden, an off-shore development house in India and recently we’ve moved our Outlook development 100% in-house and on location. We are now back on track with our development and extremely pleased with how we’ve managed to turn this situation around. This process has cost us dearly, both in money, time and possibly an investment round — but definitely parts of my own personal sanity.

Introducing Frog Driven Development
Of all the lessons learned in this process the most important one is never to stop asking yourself if you are being a frog. Keeping going on the false promise of “we’re nearly there” is lethal. It leads to the equivalent of a startup killing death spiral that can suck up all your most valuable resources: morale, money, focus and worst of all, time.

So keep asking yourself “Am I being a Frog, right now?”. If the answer is yes, jump out. The only way to tell the difference in temperature is by comparing and testing the waters. That is Frog Driven Development. Compare the waters and find the one that has a temperature that doesn’t kill you. How do you tell? Ask experts, get people that have done this before. Ask your wife. Ask your dog, the neighbor. The answer is likely to surprise you. Get more people to tell you: you are a frikkin’ frog dealing with badgers. Are you on that slippery slope? Are details killing you? Get out while you can.

Jump.

Now.

1. I don’t mean to blame anyone (except myself) and I’ve tried as much as possible to remain neutral and objective — even though I probably failed at this. I apologize in advance that this post is a one-sided affair but decided it’s important to give at least one side of the story.

A big reason not to share this story is that it’s ugly. We managed to get in quite a mess and I’m sure there are things that others would have done differently. However you live and learn and we’ve learned so much from others sharing their experience that we like to try to give something back and hope others can learn from our mistakes. Which is a much better reason to share the story.

2. Name changed for privacy reasons. Kev was from Sweden. So you know.