We often come across new products aiming to help us with our lives or improve our productivity. It’s in our best interests to find the best products that will minimize tedious work to none, but still most of the time we don’t properly and thoroughly test them
We often come across new products aiming to help us with our lives or improve our productivity. It’s in our best interests to find the best products that will minimize tedious work to none, but still most of the time we don’t properly and thoroughly test them. The crux of the matter is the emphasis on properly and thoroughly. First, testing a product properly refers to how its designers think that it should be tested, and we all know that humans have a knack for doing the unexpected 🙂 . Second, testing anything thoroughly is impossible as we would have to evaluate a product based on 100% of its features to claim that. Usually it will be difficult to complete an evaluation only based on necessary features (from our perspective of necessary of course). This stems from our time limitations, either we have work to get done and therefore have limited time available and/or because of a limited trial period.
Unfortunately there’s no yellow brick road to testing as products vary wildly in features. That said, here are some of my top tips on how to get the most out of the little time you have available. I’m going to focus on data oriented products and split my tips into two groups: data visualization tools and data management tools (Seriously..you really thought that I wouldn’t tie it back to data somehow? Shame on you 😉 )
Data Visualisation Tools
As every tool has different capabilities the first question you need to ask yourself is what you are looking to get out of using the product. In the data world this could be anything from reports to dashboards, data mining, sharing insights and so forth. Once you know what you’re testing for go ahead and select a few products that match your need (2-4 products is a good number). Before diving into the testing you should set out a few goals for the trial. For example, a comprehensive set of goals in data visualizations would be to test their connectivity to your data sets and create 2-4 reports/questions of various difficulty. DO NOT TAKE SHORTCUTS – you must test the product from start to finish(based on your goals) so when you finally make your pick you don’t encounter issues which should have been uncovered during the trial. With any new tool you’ll discover unexpected nuances down the road, it’s unavoidable, so test as much as possible ahead of time.
It’s important to be consistent in your testing as much as possible. If you test for the same set of features and capabilities and run the exact same queries on identical data sets then you have a point of reference for eliminating the tools that under-perform or cannot execute at all..
When you start testing you’ll need to track the following criteria:
How complex is the tool?
This criteria is subjective as it’s dependant on your team’s proficiency level in query languages. Are you willing to write queries? Do you need to use the tool’s UI? Is the tool’s UI is easy to use? All of these questions are important but don’t forget that most tools come with an inherent learning curve and that’s fine and expected. Just make sure that the learning curve isn’t too steep for you and your team both in difficulty and time required to learn and adapt.
What is the query complexity range?
Questions and queries can vary in complexity. There are simple metric + grouping queries which each and every tool I’ve encountered will execute perfectly. On the other hand there are complex queries that require subqueries, variables, loops, etc. These types of queries are usually difficult to describe in a UI and therefore most tools omit them by design (or at least most of them do), but you’ll always have the option to write your own query from scratch. Honestly, I’ve yet to find a tool that masters both simple and ALL of the complex queries in my arsenal. There was one that came exceptionally close but no spoilers 😉 maybe next blog I’ll tell you.
How consistent should my results be?
The results should be 100% as you expected them to be. This should always hold true and is the single point you should never compromise on.
Are the results being displayed according to your preferences and needs?
If the results are displayed to your preferences is entirely dependent on your personal preferences. However, your needs are a derivative of your point of view and the story you want to tell.
Star Wars Dot Gif
Obi-Wan could have told Luke that basically he cut off his father’s legs and left him to burn on a lava planet but that wouldn’t have served the jedi’s needs. Sanddance is an amazing visualization tool but there are cases where all its dynamics could detract from the story you’re trying to tell.
Is the price right?
Budget versus perceived value. First, ask if it’s in your budget. Second, ask if it includes all of the feature on your list. Third, ask if it includes additional features not on your list, if those features are even relevant in your case and how it impacts the price. Fourth and last, ask if your budget fits your requirements of the solution.
The best way to get the most out of these tests is to score each tool based on the points above and keep track of the scores. The tool with the most points at the end of your tests is your winner. Obviously make an exception if it scored the highest overall but bombed on one specific feature or point.
Normally these tests take a few hours or max a couple of days. If it’s dragging out more than that then most likely you’ve selected too many tools or are testing too many features with each tool. After all you set out to obtain reports and/or dashboards. The journey is important but so is getting there timely.
Data Management Tools
Choosing data management tools is trickier than choosing data visualization tools because there are risks switching in between tools. Management tools usually dependent on your database/s infrastructure . You use them to handle data ingestion and governance inside your database and data warehouse (some also handle storage) so changing them repeatedly is painful and ill advised. Pick a few tools (but not too many) that you’d like to test and check off the following points :
How complex is the tool?
You want to check how much manual work you need to put in to get the tool to ingest data. Is it simply a matter of entering credentials and clicking a button or will you need to write code, flatten the data, etc. The best thing to do is count the steps taken for the initial connection. Next, try to understand if the tool eliminates or creates complexity over time. Will you constantly need to update your settings to account for changes in your databases or is the tool built for change? Is it easy to change, add, drop, alter and so on anything in your data?
How agile is the tool?
Is it possible to manipulate and change the data and metadata before or after ingestion? This will have a huge impact on when you can start working with your data not to mention long term implications on when your data will become available for use.
How stable is the tool?
This one should be an obvious point to look into. What happens in cases of errors? Are there any downtimes?
How much $$$?
Is the price in your budget and is it worth it based on the different features and capabilities the tool offers ? Also examine the pricing model so if you suddenly scale up you don’t get hit with a massive bill.
These tests will take more time and must simulate real business issues and concerns. For example if the tool needs to connect to your production database and other/external data sources (social media, files systems and so on) you need to make sure it’s capable doing that. Each point (except for price) should be split into 4 sub points:
- Ingestion – Does the ingestion method fit your needs? Is there data enhancement on top of the original data? What’s the speed of the ingestion?
- Metadata – Do you need to manually set the metadata? Is it possible to change the metadata at any point of time?
- Extraction – This only affects tools that offer storage as well. How easy it is to connect other tools and extract the data.
- Extra features – Does the tool execute and store backups? How easy it is to scale up/down and how does it affect price and time?
After scoring all the points and their sub-points pick the tool most suited to your needs. Keep in mind which points and sub-points are relevant for you. Testing data management tools takes between a couple of days to a few weeks. I know that seems like a lot of time but think about it, these tools will have a major impact on your company’s data. Any wrong choice and poof, here goes your entire understanding of the data. Best case scenario you get saddled with slow ingestion or incorrect schemas. Worst case scenario you lose data or end up with corrupt data.
Some important notes to keep in mind while testing both data visualization and data management tools:
- Be consistent! Compare apples to apples so you don’t get mixed up. After comparing all the similar features, capabilities and needs you can scope the unique abilities and score them separately (usually they will be nice to have features otherwise they would have been part of your initial needs list).
- Some of the tools might have features that are not available during trial so speak with a representative and schedule a demo in order to understand what features they have.
Once you’ve made your choice stick it out and give the tool a chance to deliver. We are so used to immediate gratification that we forget that our brains need to catch up with the tool. Expect facepalm moments when you discover you’ve been doing something wrong and expect @!#$%& moments when the UX doesn’t deliver.
Basicaly, tools are relationships. They take time, effort and communication. It is not a catholic wedding so if it wasn’t meant to be you have a way out but dating isn’t fun. Just saying.
