Until recently, data warehouses were largely the domain of big business. With a data warehouse, you could consolidate and analyze vast amounts of information, deriving new insights that gave you an edge over your competitors. But creating a data warehouse was like building a skyscraper from scratch—it often took a staggering amount of time and money.
Now that's changing. With Data Warehouse As a Service (DWaaS) online platforms, even a seven-person startup can begin to unleash the power of a data warehouse. But the field of DWaaS is still relatively new, and there are still significant challenges it needs to overcome.
In this article, I'll give you an overview of why DWaaS has such promise, describe some of the remaining obstacles facing this field, and explore how some DWaaS platforms are breaking down the barriers to fulfilling DWaaS' potential to help companies use their business data to succeed.
What is data-warehouse-as-a-service and why do you need it?
One of the big headaches of a traditional data warehouse is its hardware and software infrastructure. Data warehouses are big beasts, and they usually require a lot of data storage and computing power. With Data Warehouse As a Service (DWaaS), you get to outsource those infrastructure headaches to someone else.
Here's what that means for your bottom line:
Infrastructure setup is much easier
Setting up a typical data warehouse isn't fun. First you have to figure out roughly how large it's going to be and how much your users will use it. Then you need staff or consultants who have the expertise and time to set up the hardware and software.
For example, if your data warehouse is built on Microsoft's SQL Server, the engineers who are going to set it up need to be fluent in the nuances and oddities of how to install and configure the particular version of SQL Server. This can be a real pain.
And that's assuming that you can afford to set up your data warehouse on a separate server. If you can't, you will need to configure the gazillions of server options so they accommodate the needs of both your data warehouse's and every other product running on that server.
You also need someone who understands the ins and outs of Microsoft's current pricing scheme for different SQL Server licenses. If you don't get it right, it can cost you big bucks.
And that's just the server license. You're going to face an additional set of hardware and software headaches for your data storage.
With DWaaS, all that work and suffering goes away.
Near-zero infrastructure maintenance
DWaaS not only reduces the costs of getting started, it also can drastically cut or even eliminate the headaches of maintaining your data warehouse's infrastructure.
Take the problem of security.
If you are running your own server, one of the things you have to do regularly is install any new security updates. That's because there are lots of people on the Internet who use tools to scan the horizon of the digital Savanna for signs of prey—servers whose admins haven't installed the latest patch to fix a security bug.
Here's the thing about installing security patches: It's critical to protecting your business. And no one else is happy you're spending time on it instead of taking care of their critical business needs or new business opportunities.
So unless your company has a large IT staff, sooner or later you're going to slack off. Is it really a big deal if you wait just a little longer before you patch the server given that somebody has a crisis that has to be handled right now?
With DWaaS, nobody in your business has to constantly balance the importance of security patches with other mission-critical needs.
And that's just one of dozens of infrastructure maintenance headaches that are now no longer yours.
Scaling up infrastructure as your needs grow is a snap
Where DWaaS really comes into its own is in scaling up your infrastructure.
Suppose your business is starting to take off, and staff are becoming more sophisticated about how they use data to drive your company's success.
Your data warehouse's hardware wasn't designed to handle your new storage and compute needs. But your needs only require about a third of the capacity of a new server or a new hard drive.
Too bad! If you want to keep up with demand, you've got to go through all the pain and suffering of setting up new hardware and software. Once that's set up, you'll have to migrate your data warehouse over to new infrastructure, and that can be a lot of work if you have a large data warehouse.
Unless, of course, you're using a DWaaS. Then all you do is click a few buttons, plunk down some more dough, and you're good to go.
Salesforce data warehouse is a prime example of a DWaaS that has a solid reputation for scaleability.
For that reason, many startups tend to abandon their data warehouse for Salesforce.
It's easy to scale your infrastructure to handle fluctuating demand
DWaaS doesn't just help you scale up. It also lets you dynamically modify the scale of your data warehouse operation as your circumstances change.
Odds are your data warehouse's server load isn't consistent throughout the entire year. For example, there's a good chance that near the end of the year, users hit the data warehouse a lot harder as they do more analyses for annual reports as well as for planning for next year. With an old school data warehouse setup, either staff are going to have a miserable user experience when they need the data warehouse the most, or you're going to have to spend a lot of money on excess capacity you don't need the rest of the year.
The more valuable your data warehouse is to your business, the worse this problem will become. Did you spend the last year convincing more managers and staff that leveling up the sophistication of their data analyses will help fatten the bottom line? Congratulations! Now there are two or three times as many people fighting to get a share of limited data warehouse compute resources during crunch time.
With DWaaS, it's pretty straightforward to scale up your capacity when you need it, then scale it down when you don't.
That's one of many reasons why startups can now afford to use data warehouses—DWaaS can put the power of a data warehouse in their price range.
Most DWaaS platforms only address half of your pain
That's the good news about DWaaS: it takes away most of your infrastructure headaches.
The bad news about most DWaaS platforms is that you're still stuck with several migraine-inducing problems.
You still have to structure your data to fit your business needs
One of the single biggest costs of creating a data warehouse is that you have to know in advance what business needs your data warehouse will be used to address. A data warehouse can allow you to quickly crunch through vast amounts of data to answer questions, but only if the data in your data warehouse is structured so it optimizes its ability to answer those questions.
Wrapping your head around the super geeky topic of structuring data can be a bit tricky. So let's use an analogy: a sandwich shop.
Most sandwich shops use a similar layout for the counter where staff make your sandwich. The counter is organized as a row of several clusters of bins of pre-cut ingredients. The person making your sandwich doesn't need to do any chopping or any other food prep. They don't have to open a jar of mustard or mayo. Everything is neatly laid out to minimize the time it takes to make a sandwich.
In other words, these sandwich shops structure their ingredients to optimize solve a problem: creating lots of sandwiches while still giving customers a decent range of options to choose from.
In contrast, most restaurants are laid out to solve a very different business problem. Like sandwich shops, they too are creating food for their customers. But unlike a sandwich shop, restaurants are organized so they can produce a wider range of meals with even more room for customization.
What if a restaurant creates a particularly amazing burger and a ton of extra customers start showing up at lunchtime to buy it? Now the restaurant has a line around the block at noon, and most customers are stuck waiting a long time for their lunch.
So while most DWaaS will protect you from having to figure out exactly how much infrastructure you'll need to handle your users, you're still stuck trying to figure out in advance what their needs will be. If you guess wrong, you're going to have a lot of very frustrated, pissed off users. Because you—or they—didn't really understand their needs, queries that would take only seconds to perform given the right data structure are now taking an hour or more.
And that's just one query. An effective dashboard may have to run anywhere from 5 to 10 queries—sheer torture if you guessed wrong.
You have to continually optimize queries and data structure
Even if you guess right the first time, your users' business needs aren't going to stand still.
If you get lucky, your IT staff or consultants can address these new needs by making small changes to the data warehouse (e.g., adding indexes that speed up access to a particular field so queries run faster). But inevitably, some of these needs will require IT staff to substantially restructure your data. To put it another way, originally all you needed was a sandwich shop. Then you needed a restaurant. Now you need a restaurant, a falafel shop, and a burrito truck.
Even if you don't need to massively reengineer your data, your users' new needs can still require a lot of time or money to address. For example, sometimes you can speed up analyses by throwing more computing power at them. But most DWaaSes are designed so if you need more server power, you can't add it incrementally—you need to pay for a whole new virtual server.
In short, as your users' needs change, your IT staff will still end up spending far too much of their time scrambling to fix yesterday's problems rather than preparing for tomorrow's opportunity.
You still have to spend a lot of money on data engineers
Speaking of IT staff, while most DWaaSes eliminate the need to hire people who can handle your data warehouse infrastructure, you're still going to need data engineers. And data engineers don't come cheap.
And the more intensively your staff take advantage of what a data warehouse has to offer, the more data engineers you'll need. As many businesses have learned the hard way, whenever you decide you can afford to hire more analysts, odds are you'll also have to hire data engineers so the analysts don't sit around twiddling their thumbs. Any analyst can easily install the latest BI tool. But that tool is useless if the data in the data warehouse isn't structured so an analyst can pull the data they need to solve the problems they were brought on to fix.
Even if you can afford to keep hiring data engineers, you'll end up paying substantial opportunity costs. It can take a considerable amount of time to find a skilled data engineer and train them so they are fluent in how your system is set up. Time spent hiring and training means lost business opportunities for you and more chances for your competitors to gain an edge.
Panoply: Full-service DWaaS
Luckily, we're starting to see the rise of full-service DWaaS platforms like Panoply that automate most the work of a traditional data warehouse engineering team. As a result, Panoply is one of the first cloud data platforms that give you the analytic power of a traditional data warehouse on a budget and timeline you can afford.
Here's what that means for you:
If you're an analyst or marketing manager:
- Adding new data sources is easy. No more waiting on IT, data engineers, or consultants to prioritize adding a new source of data that you need for a crucial analysis. With the click of a few buttons, you're good to go.
- You can spend more of time on analysis. Everyone knows that analysts spend the majority of their time on data wrangling. Panoply automates much of that work. As a result, you get to focus your work on unlocking new insights that will help your company thrive.
If you're a product manager or a CTO:
- Getting set up is simple and quick. Panoply is designed to get you up and running quickly and easily. Our built-in data integrations make it super easy to bring in all the data you count on. And it's just as easy to connect Panoply to your favorite analysis tools, whether you're working with Python notebooks or user-friendly BI.
- You can cut down on maintenance. With Panoply, you don't need to hire a bunch of engineers to repeatedly optimize your data warehouse. Panoply seamlessly scales as your business and your data needs grow. Because Panoply automates this work, you and your team can put your energies into your company's greatest opportunities for success.
- You don't have to spend all your time fighting fires and dealing with pissed off users. Instead, you can focus on helping your company execute the strategies that will give you an edge over your competitors.
These and other advantages are why Panoply is ranked #1 in for Ease of Implementation and Ease of Use on sites such as G2.
What’s next?
Talk with a Panoply solutions specialist to learn more and to get a personalized demo. Or start your free trial today!
