FAQ

• How is Azure Data Lake Storage Gen2 different from Blob Storage?

  Azure Data Lake Storage is optimized for running analytic workloads on unstructured data. Azure Data Lake Storage Gen2 is optimized for fast I/O of high volume data, thereby making analytic workloads run faster and lowering the TCO for analytic jobs. Further, Azure Data Lake Storage Gen2 provides the added flexibility of organizing data either in a flat or hierarchical namespace.

• What is the different between hierarchical and flat namespaces?

  With hierarchical namespaces, you can organize data into structured folders and directories. With a flat namespace, your files will be organized in a flat structure just like Blob Storage. A hierarchical namespace allows operations like folder renames and deletes to be performed in a single atomic operation, which with a flat namespace requires a number of operations proportionate to the number of objects in the structure. Hierarchical namespaces store additional meta-data for your directory and folder structure. However, as your data volume grows, hierarchical namespaces keeps your data organized and more importantly yields better storage performance on your analytic jobs – thus lowering your overall TCO to run analytic jobs.

• When should I use a Blob Storage versus Data Lake Storage flat namespaces?

  While Blob Storage and Data Lake Storage with flat namespace are similar in the way data is stored, Data Lake Storage with flat names performs better for analytic workloads. You should put data into Blob Storage, if you are confident that there is no need to run analytics services like Azure Databricks, Azure HDInsight etc. However, if you need to run occasional analytics jobs, you should put such data into Azure Data Lake Storage Gen2 with flat namespaces. For data that will be use constantly for analytics, we recommend that you put such data into Azure Data Lake Storage Gen2 with hierarchical namespaces.

• Are larger files better than smaller files?

  Yes, larger files are more cost effective and yield better analytic performance. For files > 4MB in size, Azure Data Lake Storage Gen2 offers lower price for every 4MB block of data read beyond the first 4MB. To read a single file that is 16 MB is cheaper than reading 4 files that are 4MB each. In both cases, the total number of transactions are 4. However, the last 12 MB, read as three transactions, from the 16MB file are cheaper, there by making the total cost of reading a 16MB file cheaper than reading 4 files that are 4MB each.

  More importantly, Azure Data Lake Storage Gen2 is highly optimized to perform faster on larger files. This means that your analytics jobs will run faster, when operating on larger files, thus further your TCO for running analytics jobs.