Databricks opened a R&D centre in Amsterdam, The Netherlands, earlier this year to leverage the database engineering talent there and to strengthen their collaboration with CWI. The San Francisco based company leads the development of open source software Apache Spark, the most widely used software tool to analyze large amounts of data. Databricks offers Spark as a Cloud service to companies and organisations, such that those customers are able to run their data analysis in a managed environment efficiently and effectively.
Databricks funds research in the Database Architectures research group of CWI, which previously developed the well-known database systems VectorWise and MonetDB. The CWI researchers work on database techniques that Databricks is interested to incorporate. The CWI methods allow users to analyze large amounts of data, not limited to tables, but also including (social) networks with growing and changing data.
Peter Boncz, senior researcher in the Database Architectures research group, coordinates the collaboration with Databricks. Peter Boncz stated: "The arrival of Databricks to Amsterdam enriches the local data science ecosystem, and underlines the reputation of the CWI in the field of Big Data technology."
Peter Boncz saw opportunities for the fundamental research at CWI: "For CWI, the collaboration provides the chance to look behind the scenes with Databricks. We will gain insight in the great diversity of data analysis problems that users encounter, and can thus discover some of the open questions in the field of data analysis. Also, professors from Berkeley and Stanford are involved in Databricks, so new scientific collaborations will arise."
"Databricks is excited about growing our R&D presence in Amsterdam and doing some of the most innovative engineering work in Big Data analytics and data science", stated Ram Sriharsha, interim site manager for the new Databricks R&D center in Amsterdam. "The collaboration with CWI and the expertise of CWI in high-performance databases was decisive in the selection of our new location. With this collaboration, we aim to make Spark and Databricks faster and more scalable."