Download files from Blob storage account - Azure Cloud & AI Domain Blog

Introduction

For every test run, we had a requirement to download application log files from Azure blob storage, parse and generate performance metrics (response times).

Problem statement/Challenge

Our performance tests run for longer duration like (3hours/24hours/5days) and write thousands/lakhs of logs files to blob storage.

We have to filter and download logs based on different criteria’s like

Download Logs only after particular Timestamp in UTC.
Download all logs.
Download only those logs, in which file name contains given string.
Download only those log file with specific extension.

Solution

I had written below C# code snippet that will automatically download logs based on different criteria. This code snippet can be further customized and reused based on our needs.

Code snippet along with explanation in comments

App.Config

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <appSettings>
        <add key = "StorageAccountConnectionString"
            value = "DefaultEndpointsProtocol=https;AccountName=testaccount;AccountKey=testaccountkey+sy5AthQhOBRw==" />
        <add key = "ContainerName" value = "testcontainername" />
        <add key = "TestExecutionStartTimeinIST" value = "9/18/2015 10:30" />
        <!– To download all files, use * in value
        Example: add key = "NameContains" value = "*”
        To download file that contains a give string in name, use string name
        Example: add key = "NameContains" value = "Application”
        To download file with extention, use extention
        Example: add key = "NameContains" value = ".txt”
        – >
        <add key = "NameContains" value = "Application" />
        <add key = "FileExtention" value = ".txt" />
    </appSettings>
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
</startup>
</configuration>

Program.cs

// Read configuration from app.config
    string strgConnString = ConfigurationManager.AppSettings["StorageAccountConnectionString"];
    string containerName = ConfigurationManager.AppSettings["ContainerName"];
    string fileNameMatch = ConfigurationManager.AppSettings["NameContains"];
    string fileExtension = ConfigurationManager.AppSettings["FileExtention"];
    string dateInputinUTC = ConfigurationManager.AppSettings["TestExecutionStartTimeinIST"];

// Create a Storage Account
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(strgConnString);

// Create the blob client
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

// Get reference to a container
CloudBlobContainer container = blobClient.GetContainerReference(containerName);

string filename = string.Empty;

// Convert time to UTC
DateTime configtime = DateTime.Parse(dateInputinUTC).ToUniversalTime();
Console.WriteLine("*******************Downloading Files from blob after :{0}", configtime);

// Traverse and download each item matching filter criteria
    foreach (IListBlobItem item in container.ListBlobs(null, false))
    {
        if (item.GetType() == typeof(CloudBlockBlob))
        {
            CloudBlockBlob blob = (CloudBlockBlob)item;
            // Condition to download files based on last modified
            if (blob.Properties.LastModified.Value.Date >= configtime.Date)
            {
                    if (blob.Properties.LastModified.Value.TimeOfDay >= configtime.TimeOfDay)
                    {
                            // Download all files
                            if (fileNameMatch == "*")
                            {
                                Console.WriteLine(blob.Uri.AbsolutePath);
                                string path = @"DownloadFolder" + filename;
                                using (var fileStream = System.IO.File.Create(path))
                                {
                                    blob.DownloadToStream(fileStream);
                                }
                            }
                            // Download all files that contains given string
                            else if (blob.Name.Contains(fileNameMatch) == true)
                            {
                                Console.WriteLine(blob.Uri.AbsolutePath);
                                filename = blob.Uri.AbsolutePath.Split('/')[2];
                                string path = @"DownloadFolder" + filename;
                                using (var fileStream = System.IO.File.Create(path))
                                {
                                    blob.DownloadToStream(fileStream);
                                }
                            }

                            // Download all files with extension
                            else if (blob.Name.EndsWith(fileExtension))
                            {
                                Console.WriteLine(blob.Uri.AbsolutePath);
                                filename = blob.Uri.AbsolutePath.Split('/')[2];
                                string path = @"DownloadFolder" + filename;
                                using (var fileStream = System.IO.File.Create(path))
                                {
                                    blob.DownloadToStream(fileStream);
                                }

}

}
}

Hope this will help you. Suggestion and Feedback is welcome.

Happy Coding!