Revitalizing Atigeo's Data Stack
Atigeo was a caring technology company that transformed scientific concepts into big data products and services. They developed xPatterns, a cloud-based platform for big data analytics (PAAS) that offers abstraction tools for creating analytical applications, available either as a managed cloud service or for deployment on the customer’s or partner’s infrastructure. Their infrastructure evolved from using Cloudera, Hadoop, and Hive to adopting Spark, Shark, Tachyon, and Mesos, providing enterprise support for the Berkeley Data Analytics Stack (BDAS). They also contributed to the development of new REST SparkSQL/Shark Server (Jaws) and a Spark Job Server (which is open source at www.github.com/atigeo). At times, they needed to modify the open-source code (Spark, Mesos, Tachyon) to enhance the latest version of the stack for product development. Additionally, they created the ETL pipeline for various analytical applications using custom Hadoop/Spark jobs and Hive/Spark SQL scripts, along with distributed tasks to transfer data between HDFS/Tachyon and Cassandra/Solr utilizing Hadoop-MR/Spark.