Our solution to the Hadoop small files problem — In this post I’ll talk about the problem of Hive tables with a lot of small partitions and files and describe my solution in details. A little background As you may know by now (if you’ve read my previous posts), in my organization — we keep a lot of our data in HDFS…

Why is it so important to track your users’ queries, and how do we do it? — In this post, I’ll explain how we used the ELK stack (actually just Elastic and Kibana) to analyze the usage of our users on top of the Hadoop cluster. As you might know if you’ve read my previous posts, our SQL-over-Hadoop solution is Apache Impala. …

A short description of Firebolt, for not-so-technical people — Recently, we started using Looker for ad-hoc queries of business users in the company (PMs, managers, marketing, and basically everyone). One main problem we have though, is that our Looker performance is pretty bad — queries just take too long. The performance isn’t bad because of Looker of course, but…

A human-readable stock analysis, from a rational perspective — Recently I noticed how much I love Spotify’s product, and every month when I see the monthly charge of 19.90 NIS (the Israeli currency), I think to myself: “that’s fucking worth it”. I wonder how many of you who have Spotify (Premium) agree with me. …

Q&A session on specific issues that bothered us — Yesterday I had a 90 minutes e-meeting with Greg Rahn, a product manager on the team at Cloudera that contributes to Apache Impala. I want to thank him here for his time, it’s truly awesome to be the user of a product with a PM like Greg. To our conversation…

A letter to the developers and product manager of Impala — In this post I’m going to write what are the features I reckon missing in Impala. We take Impala to the edge with over 20,000 queries per day and an average HDFS scan of 9GB per query (1,200 TB scanned/week). …

Why Small Frequently-Queried Tables Shouldn’t Be Stored In HDFS? — In this post I’ll describe a weird problem we had with our Impala service and how we investigated it, solved it and the conclusions from the whole experience. In my opinion this is relevant not only for Impala but for every processing platform that operates over HDFS. Something’s wrong We have a…

How to make your selective queries run 100x faster? — In this article I’m going to explain how did we solve the problem of selective MPP (Impala, Presto, Drill, etc.) queries over >10TB tables without performing a full table scan. This article will describe the idea of what we call “partition index” in a very simple way. The detailed architecture…

How did we make our Impala run faster? — So you have your Hadoop, terabytes of data are getting into it per day, ETLs are done 24/7 with Spark, Hive or god forbid — Pig. And then after the data is in the exact shape you want it to be (or even before that) and everything is just perfect…