Panoply.io - A blog about data management and data infrastructure

Technical insights and creative ideas about data management, data infrastructure, and data analysis.

Subscribe to our emails and get Panoply updates on the fly.

Being Data Driven Data Technology

Data Warehouses Vs Data Virtualization—and the Winner is...

Data analytics allows organizations to gain insight into their business operations and find new ways to increase their revenue. However, the data sources used to provide these insights are often....

[+]
Written by An Bui |December 12, 2017

Data Industry and Trends

Three Data Warehousing Trends From AWS re:Invent 2017

re:Invent, Amazon Web Services' yearly conference for data scientists, analysts and business intelligence (BI) professionals, provides information, direction, and guidance for businesses and....

[+]
Written by An Bui |December 07, 2017

Data Industry and Trends Product

Panoply, Stitch and Chartio Partner to Offer the World’s First Automatic Cloud Data Stack

Panoply, Stitch and Chartio today announced the availability of the world’s first Automatic Cloud Data Stack. The combined offering enables data analysts and business users to connect to disparate....

[+]
Written by Yaniv Leven |December 05, 2017

Data Technology

Data Warehouse Design: The Good, the Bad, the Ugly

Good Business Intelligence (BI), allows your organization to query data obtained from trusted sources and use the answers to gain a competitive edge in your industry. The first step to achieving....

[+]
Written by An Bui |November 28, 2017

From raw data to analysis in under 10 minutes.

Sign up now for a demo or a free trail of the Panoply.io platform.

Learn more about platform features

Being Data Driven Data Technology

Data Warehouse or Data Lake: How to Select the Right Solution

Data warehouses and data lakes are two different types of data storage repositories. Data warehouses integrate data from various sources and structures them for business reporting. Data lakes....

[+]
Written by An Bui |November 21, 2017

Product

Panoply Moves to General Availability on the World’s First Smart Cloud Data Warehouse - Fully Self-Driving and Autonomous

Following the steady growth trajectory of customer onboarding, with a select-user rollout in late 2016 and a quiet launch in February 2017 that has ballooned to over 100 active customers to date,....

[+]
Written by Paul Friesen |November 14, 2017

Data Technology

Database or Data Warehouse - Which is Right For You?

The future of any successful enterprise will hinge on its ability to harness and use massive amounts of data currently available, to improve productivity, decrease costs, and increase profits. How....

[+]
Written by An Bui |November 07, 2017

Data Industry and Trends

Don’t Just Create Data – Learn How to Use It!

There are some big things happening around big data. I’ve had the chance to work with various types of cloud, data center, and virtualization systems through the years. One of the biggest trends....

[+]
Written by Bill Kleyman |October 31, 2017

Data Technology

PostgreSQL vs. MariaDB

PostgreSQL and MariaDB are open source databases, which boast of vast community support and development, as well as enterprise versions. While PostgreSQL has withstood time and continued....

[+]
Written by Cheryl Adams |October 31, 2017

Being Data Driven Data Technology

Joint Analytics with Power BI and Azure

Business Intelligence (BI) has, by tradition, been a long and drawn out process. In the early days of computers, BI tools such as SAP/Crystal Reports and IBM/Cognos came along to speed things up.....

[+]
Written by Cheryl Adams |October 24, 2017

Data Industry and Trends

Cloud Security Best Practices

As your organization moves its data into the cloud, you face the challenge of balancing productivity and efficiency gains with tremendous compliance and security concerns. While securing your....

[+]
Written by Cheryl Adams |October 24, 2017

Data Industry and Trends

Top Tweets from Tableau Conference, #Data17

TL;DR -- whether you were there or not, these interesting tweets provide food for thought until #Data18.

[+]
Written by An Bui |October 23, 2017

Data Industry and Trends

Big data is getting bigger

Every single day we, as a connected – mobile – workforce, are creating vast amounts of data. In fact, a recent Cisco report indicates that by 2020, data center storage installed capacity will grow....

[+]
Written by Bill Kleyman |October 17, 2017

Data Industry and Trends

ETL will it survive big data?

Managing big data is a never-ending challenge. Extract, transform, and load (ETL) processes serve as the traditional foundation for enterprise data warehousing. However, with the emerging....

[+]
Written by Cheryl Adams |October 17, 2017

Data Industry and Trends

Panoply LIVE at Tableau Conference 2017

The Tableau Conference brings together the best and the brightest in data analytics from across the country under one roof in Las Vegas, Nevada. Tableau is a leader in BI reporting tools with....

[+]
Written by Cheryl Adams |October 11, 2017

Being Data Driven Data Technology

Connecting to Data Warehouses in Tableau

Is it easy to connect to a data warehouse? Selecting the right tool can help make this connection attainable. Tableau Desktop makes this connection easy, from opening the software to connecting to....

[+]
Written by Cheryl Adams |October 05, 2017

Data Industry and Trends Being Data Driven

The Basics of Data Warehouse Automation

In the past, designing a data warehouse has taken too long to complete. Executing numerous semi-automated steps results in a data warehouse that was limited and inflexible. This limitation caused....

[+]
Written by Cheryl Adams |October 05, 2017

Data Industry and Trends

The power struggle in the Cloud DB War just shifted to a new paradigm (and not everybody knows it)

I gotta hand it to Oracle and Larry Ellison, the guy knows how to put on a show! They also know how to do business. When they announced their quarter highlights on September 14th, Larry reported....

[+]
Written by Yaniv Leven |October 05, 2017

Data Industry and Trends Being Data Driven

How To Get People to Share Data

If you ask a group of big data analysts what are some of the top professional challenges they face, you can easily imagine which challenges are consistently shortlisted. One is the amount of time....

[+]
Written by Cheryl Adams |September 27, 2017

Data Industry and Trends

Top Cloud Security Threats, Risks, and Concerns

There are many risks involved with data security. Storing data in the cloud might seem like a safe bet, and for most users it is. But risks will always exist. Below we have identified some serious....

[+]
Written by Cheryl Adams |September 27, 2017

Being Data Driven

ELT vs. ETL by Design

The traditional approach for handling data warehousing as an analytical task has been Extact , Transform, and Load (ETL). However, in recent years, there has been an explosion in the types and....

[+]
Written by Cheryl Adams |September 19, 2017

Data Industry and Trends Being Data Driven

Data Management Best Practices for Cloud Computing

As more organizations begin their journey to the cloud, they need to plan how they will apply the best practices of data management to ensure that cloud-based, data-driven use cases are successful....

[+]
Written by Cheryl Adams |September 19, 2017

Data Industry and Trends

Enterprises Eye Big Benefits from Cloud Data Warehouses

There’s no question that the cloud offers many potential benefits. Major companies everywhere are lining up to reap the benefits by moving their back-office systems to cloud-based data warehouse....

[+]
Written by Cheryl Adams |September 12, 2017

Data Industry and Trends

Defining the Role of the Business Intelligence Architect

Are you considering a career as a BI Architect? You have made an excellent choice. Business Intelligence Architects are some of the leading minds in business and information management.

They use....

[+]
Written by Cheryl Adams |September 12, 2017

Data Industry and Trends Being Data Driven

Top 10 Questions to Ask When Selecting a Big Data Solution

Long before there was big data, there were computing solutions designed to store, organize, and analyze data. But just like the job of a big data architect, it's both bigger and different than ....

[+]
Written by Cheryl Adams |September 05, 2017

Being Data Driven

The Organization of Data Management

Designing a database schema is the first step in building a foundation in data management. Ineffective schema design can creates databases that are heavy consumers of memory and other resources,....

[+]
Written by Cheryl Adams |August 31, 2017

Data Industry and Trends Being Data Driven

How To Become a Great Big Data Architect

Organizations looking to leverage big data impose a larger and different set of job requirements on their data architects versus organizations in traditional environments.

[+]
Written by Cheryl Adams |August 24, 2017

Data Technology

Redshift Pricing: A Data Engineer's Guide

Although there are several data warehousing systems on the market, Amazon Redshift and Google BigQuery are the industry behemoths. There is a “prevailing consensus that Amazon is leading the pack.....

[+]
Written by Alon Brody |July 06, 2017

Data Industry and Trends

Top 5 MongoDB alternatives

When it comes to NoSQL databases, MongoDB is definitely in the lead as the world's fastest-growing database. But the days when a single database was enough are behind us;

[+]
Written by Yaniv Leven |June 15, 2017

Data Technology

CouchDB vs MongoDB

Whether your database needs to run on a mobile device or scale to a humongous size, this post has you covered with CouchDB vs. MongoDB. One of these databases can run on your phone while the other....

[+]
Written by Matan Sarig |June 12, 2017

Data Industry and Trends Being Data Driven

4 Traits of Outstanding Data Engineers

A data engineer conceives, builds and maintains the data infrastructure that holds your enterprise’s advanced analytics capacities together.  

[+]
Written by Yaniv Leven |June 08, 2017

Data Technology

Redis vs MongoDB

The battle of the NoSQL databases continues. We first analyzed Cassandra vs MongoDB. This time we will dig in Redis vs MongoDB.

[+]
Written by Matan Sarig |May 29, 2017

Data Industry and Trends

The Rise of Infrastructure as a Service (IaaS)

In today’s fast-paced competitive market, there are critical advantages to utilizing infrastructure-as-a-service, especially for businesses concerned about fast-tracking their analytics....

[+]
Written by Yaniv Leven |May 25, 2017

Data Industry and Trends

9 Must Attend Events for Data Engineers in 2017

To be an outstanding data engineer you must constantly keep up to date with the latest developments and tools for building data infrastructure, learn about the real world examples from top....

[+]
Written by Yaniv Leven |May 18, 2017

Data Technology

Redshift Data Types. Day to Day Challenges.

Let's talk about data. The sheer volume of the data collected today is enormous. When I try to even imagine the amounts ... it's staggering.

[+]
Written by Alon Brody |May 15, 2017

Data Industry and Trends

Data Jobs in High Demand: 2017 statistics

Across industries, a rapidly rising demand for data scientists and engineers by far outstripps supply. In fact, the shortage of skilled IT professionals is one of the main factors that is slowing....

[+]
Written by Yaniv Leven |May 11, 2017

Data Technology

Cassandra vs MongoDB

When it comes to NoSQL databases, MongoDB and Cassandra may seem similar but are actually quite different. Both have similar use cases, but are not intended for transactional data (i.e. accounting....

[+]
Written by Matan Sarig |May 08, 2017

Data Industry and Trends

What Star Wars Can Teach us about Analytics Infrastructure

Big Data can be a seductive master. How can you maximize time-to-insight without going over to the Dark Side?

[+]
Written by Yaniv Leven |May 04, 2017

Data Industry and Trends Being Data Driven

How To Become A Data Engineer: A Guide

The demand for skilled Data Engineers is projected to rapidly grow.  No wonder that’s the case: no matter what your company does, to succeed in today’s competitive environment, you need a robust....

[+]
Written by Yaniv Leven |April 27, 2017

Data Technology

Redshift Query Queues: The Complete Guide

Setting up a Redshift cluster that hangs on some number of query executions is always a hassle. When users run queries in Amazon Redshift, the queries are routed to query queues. Usually the....

[+]
Written by Alon Brody |April 25, 2017

Data Technology

AWS Spectrum, Athena and S3: Everything You Need to Know

Yesterday at AWS San Francisco Summit, Amazon announced a powerful new feature - Redshift Spectrum. Spectrum offers a set of new capabilities that allow Redshift users to seamlessly query....

[+]
Written by Roi Avinoam |April 21, 2017

Data Industry and Trends Data Technology

Amazon Athena Does Battle: Comparing Athena and Redshift

Data warehouse technologies are advancing towards interactive, real-time, and analytical solutions. In particular, cloud-based data warehouse technologies have reached new heights with the help of....

[+]
Written by Alon Brody |April 18, 2017

Data Industry and Trends

Cloud data warehousing trends 2017

It is an exciting time for cloud data warehousing with significant disruptions expected throughout 2017. As Gartner, in March 2016, showed, warehousing enterprises continue to race against one....

[+]
Written by Yaniv Leven |April 13, 2017

Data Technology

MongoDB vs MySQL: the differences explained

MySQL has become an inexpensive option for organizations around the globe that need a relational database. However, as the variety and volume of data has increased in recent years, non-relational....

[+]
Written by Matan Sarig |April 10, 2017

Data Industry and Trends

AI, NLP and Machine Learning: Data Engineering in 2017

For many organizations, their ability to collect data has long surpassed their ability to organize it quickly enough for analysis. As companies of all sizes are increasingly pressured to leverage....

[+]
Written by Yaniv Leven |April 06, 2017

Data Technology

Lessons Learned From the Recent Amazon S3 Outage

On Feb 28th, Amazon Web Services (AWS), went down for approximately 4 hours. The outage occurred in Amazon’s US-East-1 facility in Virginia, their biggest region, and disrupted business continuity....

[+]
Written by Yaniv Leven |March 30, 2017

Data Technology

MariaDB vs MySQL. A Comparative

Healthy competition can bring out the best in organizations. Just look at companies like Coke and Pepsi or General Motors and Ford, all of which were obsessed with outdoing the other while the....

[+]
Written by Matan Sarig |March 28, 2017

Being Data Driven

Five Data Tasks That Keep Data Engineers Awake at Night

As a data engineer, I enjoy my work, but when it comes  to integrating and managing data  there are quite a few tasks that are downright tedious. From data remodeling to juggling a hodgepodge of....

[+]
Written by Yaniv Leven |March 23, 2017

Data Industry and Trends Being Data Driven

Which type of data warehouse is right for you?

I loved this line from an article I recently stumbled upon:  “Choosing between the different types of data warehouse platforms can be simplified once you know which deployment option best meets....

[+]
Written by Yaniv Leven |March 16, 2017

Being Data Driven

SQL or NoSQL, That Is The Question!

We all know that in the database technology world, it comes down to two main database types – SQL (relational) and NoSQL (non-relational). The differences between them are rooted in the way they....

[+]
Written by Alon Brody |March 09, 2017

Product Data Technology

Introducing Panoply Multi Zone Redshift Service

Today’s considered data management best practices, advocate building resilient architectures that span multiple data centers, regions or even continents.

[+]
Written by Roi Avinoam |March 02, 2017

Data Technology

Redshift Security - Encrypting Loads with S3/KMS/Lambda

One of the main goals of this blog is to help developers and data architects, just like us, with their Amazon Redshift operations. Starting from a full comparison with Google BigQuery, explaining....

[+]
Written by Alon Brody |February 22, 2017

Being Data Driven

Tableau & Redshift: How to Visualize Amazon Redshift Data

Data analyst? You must be familiar with both Tableau and AWS Redshift. And you must know, or at least you should believe me that integrating the Redshift columnar-based database speed with the....

[+]
Written by Alon Brody |February 08, 2017

Being Data Driven

Step-by-Step: Loading Data from Google Analytics to Redshift

The basic free Google Analytics (GA) functionality is great for small and medium websites and mobile applications, but as your online business expands, you will start looking to get more of this....

[+]
Written by Alon Brody |January 24, 2017

Data Industry and Trends

Future of Data Warehouse is Cloud: This 2017 Survey Proves It

Amazon re:Invent is a great place to gather feedback from industry professionals, and especially gauge cloud-industry trends. This past year, we used the opportunity to collect and analyze....

[+]
Written by Peter Gorne |January 18, 2017

Data Technology

11 Great ETL Tools, and the Case for Saying “No” to ETL

Extract, Transform, and Load (ETL) is a data warehousing process that uses batch processing to help business users analyze and report on data relevant to their business focus. The ETL process....

[+]
Written by Alon Brody |January 09, 2017

Product

Panoply PGproxy: Smart Routing of Your Data Warehouse Queries

A smart, proactive PostgreSQL connection pooler, we are proud to announce our PGproxy offers the advantages of query routing and rewriting, making it optimal for data engineers who need to query....

[+]
Written by Roi Avinoam |January 02, 2017

Data Industry and Trends Data Technology

Apache Spark: Promises and Challenges

If you’re looking for a solution for processing huge chuncks of data, then there are lots of options these days. Depending on your use case and the type of operations you want to perform on data,....

[+]
Written by Alon Brody |December 20, 2016

Being Data Driven

Data Augmentation: Bringing New Life to Your Data

If you recognize your data as an asset, than augmenting it simply means growing your business assets. With data augmentation, you can run manipulations on existing data, use multiple sources from....

[+]
Written by Yaniv Leven |December 13, 2016

Data Industry and Trends

Data-Oriented Takeaways from AWS re:Invent 2016: Query S3, Batch, Glue and More

Attendance at this year’s AWS re:Invent conference almost doubled, from 18,000 people last year to 32,000 people last week. The large, international cloud computing event attracted professionals....

[+]
Written by Yaniv Leven |December 05, 2016

Being Data Driven

How to Move Your MySQL to Redshift

 

Data analytics is a requirement for virtually every business today. However building an analytical data warehouse requires vast computing power to maintain performance and quick retrieval of....

[+]
Written by Roi Avinoam |November 21, 2016

Data Industry and Trends

AWS re:Invent 2016 - The Top 5 Amazon Redshift Breakout Sessions

We’re sure you’re already beginning to build your re:Invent schedule for 2016. As the largest cloud conference in the world approaches, it’s time to make sure that you’re maximizing your time and....

[+]
Written by Peter Gorne |November 16, 2016

Being Data Driven

ETL vs ELT: The Difference is in the How

For the last couple of decades ETL (extract, transform, load) has been the traditional approach for data warehousing and analytics. The ELT (extract, load, transform) approach changes the old....

[+]
Written by Roi Avinoam |November 06, 2016

Being Data Driven

Load and Transform: How to Prepare Your Data for Business Intelligence

Preparing data for Business Intelligence (BI) can be a very tedious and time consuming process. You want the data to turn into the best reports for analysis. But, the raw data needs lots of....

[+]
Written by Alon Brody |November 01, 2016

Being Data Driven

Castles in the Cloud: Tips for Cloud Infrastructure

The cloud, unlike the Force, is not a mystical energy that surrounds us and binds us.

[+]
Written by Ken Saggy |August 15, 2016

Data Technology

Redshift vs. BigQuery: The Full Comparison

As we speak the future of cloud computing is being decided. Amazon and Google, as well as Microsoft and a few others, offer multiple cloud solutions for practically everything. Despite the....

[+]
Written by Roi Avinoam |July 04, 2016

Being Data Driven

Data Tools: How to Ace Product Evaluation

We often come across new products aiming to help us with our lives or improve our productivity. It’s in our best interests to find the best products that will minimize tedious work to none, but....

[+]
Written by Alon Brody |June 07, 2016

Data Industry and Trends Being Data Driven

Never judge a book by its cover: data infrastructure and data visualization, mutually exclusive or not?

This is the third and final part in the 101 series covering big data concepts, terminology and technology

[+]
Written by Moran Gilad Halevi |May 23, 2016

Being Data Driven

Data Dithering: How White Noise Can Improve Data Importing

Data and music have much in common given that both tell stories with points and notes, yet each on its own lacks the coherence and context that distinguishes music from noise. Paradoxically, in....

[+]
Written by Alon Weissfeld |May 09, 2016

Being Data Driven

The Syntax of Semi-Structured Data

The byproduct of the ever increasing integration of the internet in our daily lives is massive growth of data transitioning between applications and servers every minute. In order to make these....

[+]
Written by Oshri Bienhaker |April 25, 2016

Being Data Driven Data Technology

Redshift Benchmark: A First Test of the Entire Data Funnel

Amazon Redshift is a very powerful data warehouse, optimized for analyzing massive amounts of data in a blink of a second, when configured right.

[+]
Written by Roi Avinoam |April 12, 2016

Being Data Driven Product

Introducing Panoply.io’s Upsert Mechanism

One of the most critical issue in data warehouse management is avoiding duplications. When you are periodically loading data into your data warehouse you will want to skip the rows that have been....

[+]
Written by Alon Brody |March 28, 2016

Being Data Driven Data Technology

Blitz Analyzing with Microsoft SandDance

Microsoft Research just released SandDance, a free web-based data visualization tool.

[+]
Written by Alon Brody |March 25, 2016

Being Data Driven

Data Warehouse Automation. A Question of When Not If

The premise that to build a perfect data warehouse you must have perfect business understanding is simply irrelevant. We must look to technology to empower us to keep pace with our rapidly....

[+]
Written by Yaniv Leven |March 14, 2016

Being Data Driven

How Data Structures Impact the Data Warehouse

This is the second part in a 101 series covering Big Data concepts, terminology and technology.

[+]
Written by Moran Gilad Halevi |February 29, 2016

Being Data Driven

The fallacy of One Data Technology to Rule Them All

As a species we’ve invented some pretty nifty things. It’s practically consensus that the wheel, the printing press and the internet top the list but we don’t need to look that far back to find....

[+]
Written by Roi Avinoam |February 16, 2016

Being Data Driven

Go Raw: Why Raw Data Reigns Supreme

ETL processes (extract, transform and load) might be fundamentally flawed.

[+]
Written by Alon Brody |February 01, 2016

Being Data Driven

Build KPIs not One Time Reports

Businesses need to deliver insights fast if not immediately. If this were not true, we would not be writing this post, you wouldn’t be reading it and over all, big data would be less of a diamond....

[+]
Written by Alon Brody |January 19, 2016

Being Data Driven

Data Warehouse What?

This is the first part in a 101 series covering Big Data concepts, terminology and technology. Starting with the data warehouse.

[+]
Written by Moran Gilad Halevi |January 04, 2016

Data Industry and Trends

Why we Built Panoply.io and what Ray Kurzweil’s Law of Accelerated Returns has to do with it

“An analysis of the history of technology shows that technological change is exponential…So we won’t experience 100 years of progress in the 21st century — it will be more like 20,000 years of....

[+]
Written by Yaniv Leven |December 23, 2015