Big Data 101 – The Rise and Fall of Greenplum
Last time we explored Massively Parallel Processing (MPP) databases as a way to run SQL queries on Big Data at Rest. One of the leading commercial products in this category is Pivotal’s Greenplum, which, at its peak, achieved annual revenues of over $100 million. The rise and fall of this product provides an illustrative lesson in the industry trends for Big Data.
The original idea was brilliant: take an existing OpenSource relational database (Postgres), with all of its great community-developed infrastructure and adapters, and graft on an MPP back end for storage and query execution.
EMC’s Paul Maritz took this product, along with GemFire, an in-memory caching system, and spun off a company named Pivotal to address the growing Big Data market.
Thanks to Pivotal’s great marketing and sales team, Greenplum was adopted by a number of high-profile enterprise customers. Many of these customers soon discovered that the product was very difficult to tune. Like a kamikaze pilot, Greenplum wanted to use all the available system resources for every single query, which led to instability when executing multiple queries in parallel.
The solution, according to Pivotal, was to define queues for different query types, and assign resource limits to each queue. But, this was not a long term fix, as the mix of queries often changes over time, rendering yesterday’s resource allocation policy useless. In response, Pivotal bought MoreVRP, a startup that specialized in dynamic database configuration based on the current state of the database. This product provided little relief for customers, as it was difficult to configure by anyone except a handful of Pivotal experts.
Other problems with Pivotal emerged over time. In a pilot project at Barclays Bank, it was discovered that Greenplum actually slowed down when disk caching was enabled. Like a one-trick pony, Greenplum wanted to do full table disk scans for each and every query.
Other customers discovered that under high write load, Greenplum caused Linux journaling errors that required rebuilding the entire database, a two-day process.
Perhaps Greenplum could have ridden out these issues. Meanwhile, larger market forces were at work in the form of Hadoop. Hive performance within Hadoop improved by leaps and bounds, and Cloudera released Impala, an MPP database based on Hadoop’s file system. Within a few short years, Pivotal faced competition from Hadoop-based products that were far cheaper, more stable, equally performant, and involved far less vendor lock-in.
Faced with a tsunami of RFPs that required OpenSource solutions, Pivotal tried to counter with HAWQ, which stored Greenplum’s proprietary data format on Hadoop’s HDFS file system. But customers were not fooled – if the data format was proprietary, no other Hadoop application could process it, so there was no tangible benefit to storing the data in HDFS.
In the end, Pivotal had no choice but to raise the white flag. In March 2015 they announced that they would OpenSource Greenplum, and move their focus to vendor-independent cloud platforms.
So where does that leave existing Greenplum customers? Clearly, Pivotal has not OpenSourced Greenplum in order to expend more energy improving the product, and it is unlikely that the OpenSource community will embrace it.
Unfortunately, Greenplum now sits in the IT ledger as part of technological debt, and should be replaced as part of a carefully planned modernization effort. Ness can help, based on our expertise in Big Data technologies and our extensive experience with IT modernization projects.
