When choosing data tools, there's no replacement for getting hands-on. But if you've ever found yourself taking out a trial, logging in, and thinking "Well, this seems ok, but I'm not really sure," I feel your pain.
That's where a data tool bake-off comes in. Whether you're picking out a data warehouse or need to choose between ETL providers, a data tool bake-off is a great way to ensure you end up with the right tool for your business's unique needs. In this article, I'll show you how and why to do it...with the help of a few friends.
What is a bake-off and why should you do one?
The idea behind a bake-off is simple: Just like chefs vying to see who can bake the better Victoria sponge, you're going to have teams do a quick head-to-head test of two or more tools and/or solutions to see which one wins the prize. Only instead of comparing the data equivalent of tasty pastries, you’re comparing the tools for making the pastries.
Bake-offs are particularly a good idea when you need to make a high-stakes decision where getting it wrong will cost you time and money you can't afford. For example, once you've committed to a new ETL tool, the pain of switching to another one three months later is not something you want to experience.
Bake-offs are also useful when you're choosing between radically different architectures. In that case, it can be almost impossible to figure out which solution is better unless you try them out on the real world problems that your business is facing.
For example, I once managed a team where after we'd gotten through the first two painful iterations of a complex data solution we discovered a new, super easy to use drag-and-drop data transformation tool that looked like it would let advanced power users take care of many of their unique needs.
But when we ran a bake-off, we quickly discovered that the new tool only worked well for simple data models. With our organization's incredibly complex data model, the tool produced a snarled rats nest that made our top power users ask, exactly how early in the day are we allowed to start drinking?
There's one more reason why you might want to do a bake-off: internal politics. Whether it's squabbling folks on your team or two Big Dogs who've decided to turn a decision about tools into a pissing contest, you can end up in a situation where there's no way you can get to a reasonable decision everybody can live with.
Putting potential solutions to a head-to-head test may not solve all your problems, but it can greatly increase your odds of success.
How to structure a bake-off
While technical challenges will probably abound in your bake-off, you don’t have time for a summer-long slog that eliminates competitors one by one and ends with a celebratory fête. Here are four steps to planning a bake-off that’s sure to be a showstopper:
Step 1. Set the goals and judging criteria
Good bake-offs require clear goals. A goal of "learning about the tools," for example, is not a recipe for success.
That's why it's important that everyone is crystal clear exactly what you are trying to find out and why. If you can't confidently fill in the blanks in the sentence, “If I know ___, it will help me make the decision because ___," it's time to go back to the kitchen.
It's especially important to be clear on what trade-offs you're willing to make. For example, a tool that can easily scale up to handle tens of millions of users may not be a great tool for quickly whipping up a new solution.
The bakers also need to be clear on how complete the solution has to be. In general, bake-offs work better when the teams are trying to produce a data product that’s small enough to finish in a few weeks at most.
Once you've nailed the details, use your goals to create the criteria for judging the bake-off and share it with your bakers.
Step 2. Decide who should be involved
Next, you need to figure out the bake-off's participants. This includes:
- Bakers: What mix of experience do you need on each team to meet your goals?
- Judges: Who will decide which tool/solution wins the bake-off? One thing to keep in mind is that it's common for a data bake-off's judges to include some or all of the bakers—especially when they are the ones who will be living with the tool every day.
- Stakeholders: Who else needs to be at the table? And do they need to be involved throughout the bake-off or just brought in at the beginning and the end?
Finally, if your stakeholders don't include any Higher Ups, make sure you get the relevant senior staff to buy in. If they've never participated in a bake-off, they may need a little convincing that it's a good use of time and money.
Step 3. Decide on the test data
Just as the Great British Baking Show judges set technical challenges for their bakers, a data solutions bake-off is all about challenging your team of bakers—and your test data sets are one of the most critical ingredients.
You'll want to choose a mix of data sets that are straightforward and tricky: straightforward so you get to see what the tool can do when it's on the equivalent of a test track, tricky so you get to see how well the tool handles a rougher ride.
When choosing test data, be mindful of your bakers’ skill sets. It’s okay to push them a bit, but the point is to put the tools through their paces, not to crown your company’s star (data) baker.
Step 4. Set a timeline
The last component of a good bake-off is the timeline. In general, a few weeks is a good length for a bake-off. That’s long enough to really see what the tools can do, but not so long that you end up investing a lot of staff time that won't pay off.
Once you’ve time-bounded the bake-off, make sure everyone is clear on where the bake-off work fits with staff's other duties. Are you setting aside a team's time for one sprint's worth of work? Is it 20% of several power users or developers time? Then double check with any relevant managers to ensure that the team members who are going to be involved will have the time they need to participate.
How to make a data tool bake-off a success
No one likes feeling like their valuable time has been wasted. Just as with baking, some careful planning and close attention to detail can help keep you on the right track.
Here are a few tips for ensuring you get the most out of your data tool bake-off:
Keep your eye on the prize
A small amount of focus can make a big difference in getting the most out of a bake-off:
- Choose a problem that’s critical to the success of the data product. For example, if you’re choosing an ETL tool and the teams are working with dirty data, are you trying to test ease of setup or how difficult it is to clean up the data? If an issue isn't that crucial, don't have folks spend a lot of time on it.
- Bake-offs usually have more unknowns than a regular project—that’s kind of the point. So be sure no one is spending too much time on unknowns that won’t pay off.
- Have regular check-ins to so folks don’t go down rabbit holes—unless they’re going after a rabbit you want them to catch.
- Don't forget the bake-off’s audience. Are you ultimately going to sell the tool to power users or devs? Are you looking for something for self-serve stakeholders? You may not be able to please everyone at the same time, but creating a challenge and scope that will satisfy your core users is key.
For expensive solutions, combine a bake-off with a pilot project
If you are trying to decide between solutions that are quite costly, a bake-off by itself might not be enough to reduce the risk of your decision. So, you might want to run a bake-off to pick a solution, then run a pilot project to further test the proposed solution. That way, you've got two bites at the (baked) apple before you're fully committed.
It's okay to call it early
Because nerding out on a new tool can be lots of fun, and because the stakes of getting it right may be high, it's easy to feel like you have to run the bake-off for the full time allotted.
But sometimes you know within a few days that one of the tools/solutions is the champ. If that's the case, stop the bake-off early and declare a winner. And if you're testing two tools and you have a clear loser but aren't sold on the remaining contender, you can always do more research and pit it against a new rival.
If you're an army of one, heavily involve your power users
If you are a solo analyst, it's hard to do a high-quality bake-off all by yourself. So lean on your power users. You might even end up finding that it makes sense for you to play more of an advisory role and let your power users lead—especially if you've built a network of data allies.
It might also be worth involving a developer even if devs aren't going to use the tool/solution you choose. An experienced but user-friendly developer could act as a sounding board, giving you a perspective from someone with a deep understanding of data and who also understands what you’re trying to accomplish.
Foster friendly competition
Depending on the folks involved, you might want to turn the bake-off into a competition. If there's a little friendly team rivalry, the contestants are likely to push themselves and the tool/solution a bit more. And so long as the competition stays friendly, it can make the bake-off more fun and help build camaraderie.
The best bake-offs are competitive enough so people push themselves but keep the competition reined in enough so everyone is focused on making the right decision. You want everyone to end up happy with the tool you ultimately choose, not salty about the one that didn’t get picked.
Have fun with it
You don't have to get playful with a bake-off. But it's a really good excuse to let loose a little.
So, get aprons and chef's hats. Or give an engraved cake plate to the winner. You know you want to, and it helps to foster important camaraderie among your bakers...and maybe even a little FOMO among their peers.
Conclusion
A lot of the time, making a choice between tools or solutions is pretty straightforward. But for situations where it isn't, a bake-off can be a great way to get an answer you can feel confident about. And having a little fun in the process? That’s just the icing on the cake.
