How to use Azure BLOB storage and ML Studio notebook

Richard Horvath at Unsplash

Yes, you can use Azure Blob storage as the dataset source for your Azure Machine Learning notebook. Azure Blob storage is a scalable and cost-effective storage solution provided by Microsoft Azure.

To connect and process data from Azure Blob storage in an Azure Machine Learning notebook, you can follow these steps:

1. Create an Azure Machine Learning workspace: If you haven’t already, create an Azure Machine Learning workspace in your Azure subscription. This workspace will serve as the central hub for your machine learning experiments and resources.
2. Upload data to Azure Blob storage: Upload your dataset to Azure Blob storage. You can use the Azure portal, Azure Storage Explorer, Azure CLI, or Azure SDKs to upload data to Blob storage containers.
3. Connect to Azure Machine Learning notebook: Open the Azure Machine Learning studio and navigate to your workspace. Create a new notebook or open an existing one.
4. Access Azure Blob storage in the notebook: In your notebook, you can use the Azure SDKs or libraries like azure-storage-blob to access the data stored in Azure Blob storage. Install the necessary packages if they are not already installed in your notebook environment.
5. Read data from Azure Blob storage: Use the appropriate code to read the data from your Blob storage containers. For example, you can use the azure-storage-blob library to create a BlockBlobService object and then use its methods to read the data.

Here’s a sample code snippet that demonstrates reading a file from Azure Blob storage:

from azure.storage.blob import BlockBlobServic


# Create a BlockBlobService object
blob_service = BlockBlobService(account_name='<storage_account_name>', account_key='<storage_account_key>')


# Specify the container and blob name
container_name = '<container_name>'
blob_name = '<blob_name>'


# Download the blob to a local file
local_file_path = 'local_file_path'  # Specify the path to save the downloaded file
blob_service.get_blob_to_path(container_name, blob_name, local_file_path)


# Read the file from local storage
with open(local_file_path, 'r') as file:
    data = file.read()


# Process the data as needed
# ...

Remember to replace <storage_account_name>, <storage_account_key>, <container_name>, <blob_name>, and local_file_path with your specific values.

By following these steps, you can connect to Azure Blob storage from an Azure Machine Learning notebook and process your data for AI/ML applications.

Here’s an example code snippet that shows how to load image data from Azure Blob storage into an Azure Machine Learning notebook using the azure-storage-blob library and the ImageDataGenerator from Keras:

from azure.storage.blob import BlockBlobServic
from keras.preprocessing.image import ImageDataGenerator


# Create a BlockBlobService object
blob_service = BlockBlobService(account_name='<storage_account_name>', account_key='<storage_account_key>')


# Specify the container and blob directory
container_name = '<container_name>'
blob_directory = '<blob_directory>'


# Download the images from Blob storage to a local directory
local_directory = 'local_directory'  # Specify the local directory path to save the downloaded images
blob_list = blob_service.list_blobs(container_name, prefix=blob_directory)
for blob in blob_list:
    local_path = f"{local_directory}/{blob.name}"
    blob_service.get_blob_to_path(container_name, blob.name, local_path)


# Create an instance of ImageDataGenerator
data_generator = ImageDataGenerator(rescale=1./255)


# Use the flow_from_directory method to load the images from the local directory
image_data = data_generator.flow_from_directory(
    directory=local_directory,
    target_size=(IMAGE_WIDTH, IMAGE_HEIGHT),
    batch_size=BATCH_SIZE,
    class_mode='categorical'
)


# Use the loaded image data for further processing and training
# ...

In this code snippet, you need to replace <storage_account_name>, <storage_account_key>, <container_name>, <blob_directory>, local_directory, IMAGE_WIDTH, IMAGE_HEIGHT, and BATCH_SIZE with the appropriate values for your scenario.

The code downloads the images from the specified Blob storage container and directory to a local directory. Then, the ImageDataGenerator is used to load the images from the local directory, rescale their pixel values, and generate batches of image data. Finally, you can use the loaded image data for further processing or training your AI/ML models.

Make sure you have the azure-storage-blob and keras packages installed in your Azure Machine Learning notebook environment before running this code.

Thank you.