It’s Monday morning. I check my email. I see we’ve had a major systems issue overnight while trying to return from our secondary data center to our primary.
This move was planned in advance. It was communicated to our customers. It was run on schedule. And it still went wrong. The best part? The retreat to our secondary data created another major issue wherein we duplicated some credit card transactions, so a handful of our customers were charged twice.
You can read all the gory details here…but honestly, it doesn’t get much worse than this.
FreshBooks has been at this for 7 years. In the process we have created the category we now lead: online invoicing – it didn’t exist before we breathed life into it. Now we’re managing data on behalf of millions of people. We take this responsibility seriously. We’ve built a team of extraordinarily dedicated professionals to ensure our success and that of our customers. And yet, still, mistakes, errors and issues happen…and it hurts like hell when they do.
Right now I am watching the faces of our most talented people. They are moving swiftly to action, measuring every step, taking the whole load on their shoulders and methodically doing what needs to get done. They are heros in my books – I’ll tell them when the dust settles. Why? Because, as I watch them make their moves from within the war room, I know in my heart it could not be done better than they are doing it.
Still, we are here. We’re a little guy, though I don’t accept that as any kind of excuse. It’s no comfort that even the big guys have trouble. I just wonder openly: why, despite the best efforts of hugely talented people does it have to be so hard?
The world is a better place because FreshBooks helps businesses get paid so they can focus on what they love to do. The issues that have occurred this morning are not due to carelessness, more than anything they feel as though they are rooted in growing pains, and that despite our best efforts, things still go wrong.
So I’m reminded that it’s not how you handle things when times are good that matters, it’s how you handle them when they go pear shaped. I’m also reminded how good it is to have real pros working on your problems…and that sometimes being a pro is all you’ve got.