“An analysis of the history of technology shows that technological change is exponential…So we won’t experience 100 years of progress in the 21st century — it will be more like 20,000 years of progress (at today’s rate)”
The Law of Accelerating Returns, Ray Kurzweil, 2001
When I first read Kurzweil’s 2001 essay “The Singularity is Near," I was heading operations and analytics and Roi was leading technology at a mobile gaming company. The exponential growth of technology was evident all around us but I wanted to assess what kind of immediate implications it had on our company’s operations. So I took a step back and reviewed my team’s ability to pinpoint and extract business insights for decision makers in the company.
At first glance, it looked good. We were collecting and storing 10 times more data events than at the previous company I had worked at and with better and sexier analysis tools. But were we getting faster, superior insights? Had our ability to integrate data and extract information improved over the course of the last few years?
Paradoxically, as our technological capabilities grew, we were collecting more data but from a decreasing percentage of our total data sources. The number of utilized data sources was multiplying faster than our IT departments capacity to integrate them. Even placing that aside, the amount of time dedicated to ETL and internal data maintenance byproducts was beyond inefficient.
Today, looking in from the outside, it seems an oxymoron that the same technological advancements that made it possible to process and store more data made it harder to convert that data into information. All my experience has led me to the conclusion that Ray might actually be right, and if he’s right, the universe of data management is heading straight into a brick wall.
…the emergence of the first technology creating species resulted in the new evolutionary process of technology. Therefore, technological evolution is an outgrowth of—and a continuation of—biological evolution…A specific paradigm…provides exponential growth until the method exhausts its potential. When this happens, a paradigm shift…occurs, which enables exponential growth to continue.-Ray Kurzweil, The Law of Accelerating Returns
A paradigm shift in the data universe is near. The world of Big Data is mutating as we speak; In 2020 the rate of data production will be 44 times what it was in 2009. The explosion of new data sources and unstructured data sources is beginning to exhibit an exponential curve.
Business Intelligence and Analytics need to scale up to support explosive growth in data sources
- Gartner, 2013
From collecting to connecting data, the tectonic plates are shifting!
Behind these numbers are organizations of all sizes harnessing more and more data and struggling with the task of turning data into insightful information. Moreover, not only are smaller and smaller organizations gathering more data but also smaller business units inside the organization are starting to embrace analytics as a viable form of decision making.
The internal processes of harnessing data for smarter data oriented decision making were not meant for the use of smaller and smaller entities and will hit a bottleneck at the IT/DBA in charge of the data to information pipeline.
Today, developers need to construct their data infrastructure, including data warehouses and ETL processes, and cast all of their business logic processes inside. Processes such as upsert operations, data encryption, historical data treatment and slowly changing tables, indexing, nested fields, vacuuming and the list goes on and on. For whatever reason, all these processes are being developed and handled manually. And as the fragmentation in both data and needs grows manual processes will just not cut it.
This is why we built Panoply: to be the enabler of the much needed paradigm shift. Even if you don’t believe in Ray’s theories, it is clear that technology and data are tied together in a noose.
For one to experience exponential growth we must understand the data associated with it. It is evident in the day to day struggles of CRM manager striving to understand campaign information but waiting in line for IT to prioritize its events against bug fixes. A CTO itching to build a new feature in a mobile application but reluctantly sending his engineers to index a new table. A CEO, a CMO a product manager or whatever business owner, needing to analyze some small piece of information but use partial data because new sources are proliferating too fast for their IT to keep up.
We believe in the full democratization of data across the organization. Connect any data source to any insight tool with a click of a button. We refuse to believe in difficult learning curves and we believe that if the company has data somewhere, it should be accessible for insightful analysis anywhere, by anyone.
We don’t care about big data or fragmented data and neither should you. We care about enabling you to get fast insights from your data wherever it may sit, with whatever BI/ Data mining/ CRM/ CS/ ERP/ etc. tool you want, so you can focus on your core not your data infrastructure.
