Azure Blob Storage → AWS S3 Migration

Overview

Migrate from Azure.Storage.Blobs to AWS S3 while preserving extension method signatures for storage operations.

Package Migration

Azure Packages (Remove)

<PackageReference Include="Azure.Storage.Blobs" Version="12.x.x" />
<PackageReference Include="Azure.Identity" Version="1.x.x" />

AWS Packages (Add)

<PackageReference Include="AWSSDK.S3" Version="3.7.x" />
<PackageReference Include="AWSSDK.Extensions.NETCore.Setup" Version="3.7.x" />

Configuration Migration

Azure Configuration (appsettings.json)

{
  "AzureStorage": {
    "ConnectionString": "DefaultEndpointsProtocol=https;AccountName=...",
    "ContainerName": "documents"
  }
}

AWS Configuration (appsettings.json)

{
  "AWS": {
    "Region": "us-east-1"
  },
  "S3": {
    "BucketName": "my-documents-bucket",
    "KeyPrefix": "documents/"
  }
}

Concept Mapping

Service Registration

Azure (Before)

services.AddSingleton(sp =>
{
    var config = sp.GetRequiredService<IConfiguration>();
    var connectionString = config["AzureStorage:ConnectionString"];
    var containerName = config["AzureStorage:ContainerName"];
    
    var blobServiceClient = new BlobServiceClient(connectionString);
    var containerClient = blobServiceClient.GetBlobContainerClient(containerName);
    containerClient.CreateIfNotExists();
    
    return containerClient;
});

AWS (After)

services.AddDefaultAWSOptions(configuration.GetAWSOptions());
services.AddAWSService<IAmazonS3>();

services.AddSingleton<IStorageService, AwsS3StorageService>();

Extension Method Preservation

CRITICAL: Keep storage API surface identical.

Example Extension Methods (Keep Signatures)

public interface IStorageService
{
    Task<bool> UploadAsync(string blobName, Stream content, string contentType, CancellationToken cancellationToken = default);
    Task<Stream?> DownloadAsync(string blobName, CancellationToken cancellationToken = default);
    Task<bool> DeleteAsync(string blobName, CancellationToken cancellationToken = default);
    Task<bool> ExistsAsync(string blobName, CancellationToken cancellationToken = default);
    Task<string> GetDownloadUrlAsync(string blobName, TimeSpan expiresIn, CancellationToken cancellationToken = default);
    Task<IEnumerable<BlobItem>> ListAsync(string prefix = "", CancellationToken cancellationToken = default);
}

public class BlobItem
{
    public string Name { get; set; } = string.Empty;
    public long Size { get; set; }
    public DateTimeOffset LastModified { get; set; }
    public string ContentType { get; set; } = string.Empty;
}

Azure Implementation (Before)

public class AzureBlobStorageService : IStorageService
{
    private readonly BlobContainerClient _containerClient;
    private readonly ILogger<AzureBlobStorageService> _logger;

    public AzureBlobStorageService(BlobContainerClient containerClient, ILogger<AzureBlobStorageService> logger)
    {
        _containerClient = containerClient;
        _logger = logger;
    }

    public async Task<bool> UploadAsync(string blobName, Stream content, string contentType, CancellationToken cancellationToken = default)
    {
        try
        {
            var blobClient = _containerClient.GetBlobClient(blobName);
            var options = new BlobUploadOptions
            {
                HttpHeaders = new BlobHttpHeaders { ContentType = contentType }
            };
            
            await blobClient.UploadAsync(content, options, cancellationToken);
            _logger.LogInformation("Uploaded blob {BlobName}", blobName);
            return true;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to upload blob {BlobName}", blobName);
            return false;
        }
    }

    public async Task<Stream?> DownloadAsync(string blobName, CancellationToken cancellationToken = default)
    {
        try
        {
            var blobClient = _containerClient.GetBlobClient(blobName);
            var response = await blobClient.DownloadAsync(cancellationToken);
            return response.Value.Content;
        }
        catch (RequestFailedException ex) when (ex.Status == 404)
        {
            _logger.LogWarning("Blob {BlobName} not found", blobName);
            return null;
        }
    }

    public async Task<bool> DeleteAsync(string blobName, CancellationToken cancellationToken = default)
    {
        try
        {
            var blobClient = _containerClient.GetBlobClient(blobName);
            await blobClient.DeleteIfExistsAsync(cancellationToken: cancellationToken);
            return true;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to delete blob {BlobName}", blobName);
            return false;
        }
    }

    public async Task<bool> ExistsAsync(string blobName, CancellationToken cancellationToken = default)
    {
        var blobClient = _containerClient.GetBlobClient(blobName);
        return await blobClient.ExistsAsync(cancellationToken);
    }

    public async Task<string> GetDownloadUrlAsync(string blobName, TimeSpan expiresIn, CancellationToken cancellationToken = default)
    {
        var blobClient = _containerClient.GetBlobClient(blobName);
        var sasBuilder = new BlobSasBuilder(BlobSasPermissions.Read, DateTimeOffset.UtcNow.Add(expiresIn))
        {
            BlobContainerName = _containerClient.Name,
            BlobName = blobName
        };
        
        return blobClient.GenerateSasUri(sasBuilder).ToString();
    }

    public async Task<IEnumerable<BlobItem>> ListAsync(string prefix = "", CancellationToken cancellationToken = default)
    {
        var items = new List<BlobItem>();
        
        await foreach (var blobItem in _containerClient.GetBlobsAsync(prefix: prefix, cancellationToken: cancellationToken))
        {
            items.Add(new BlobItem
            {
                Name = blobItem.Name,
                Size = blobItem.Properties.ContentLength ?? 0,
                LastModified = blobItem.Properties.LastModified ?? DateTimeOffset.MinValue,
                ContentType = blobItem.Properties.ContentType ?? "application/octet-stream"
            });
        }
        
        return items;
    }
}

AWS Implementation (After)

public class AwsS3StorageService : IStorageService
{
    private readonly IAmazonS3 _s3Client;
    private readonly ILogger<AwsS3StorageService> _logger;
    private readonly string _bucketName;
    private readonly string _keyPrefix;

    public AwsS3StorageService(
        IAmazonS3 s3Client,
        IConfiguration configuration,
        ILogger<AwsS3StorageService> logger)
    {
        _s3Client = s3Client;
        _logger = logger;
        _bucketName = configuration["S3:BucketName"]!;
        _keyPrefix = configuration["S3:KeyPrefix"] ?? string.Empty;
    }

    public async Task<bool> UploadAsync(string blobName, Stream content, string contentType, CancellationToken cancellationToken = default)
    {
        try
        {
            var request = new PutObjectRequest
            {
                BucketName = _bucketName,
                Key = $"{_keyPrefix}{blobName}",
                InputStream = content,
                ContentType = contentType
            };

            await _s3Client.PutObjectAsync(request, cancellationToken);
            _logger.LogInformation("Uploaded object {ObjectKey} to S3", blobName);
            return true;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to upload object {ObjectKey} to S3", blobName);
            return false;
        }
    }

    public async Task<Stream?> DownloadAsync(string blobName, CancellationToken cancellationToken = default)
    {
        try
        {
            var request = new GetObjectRequest
            {
                BucketName = _bucketName,
                Key = $"{_keyPrefix}{blobName}"
            };

            var response = await _s3Client.GetObjectAsync(request, cancellationToken);
            
            // Copy to MemoryStream to avoid disposal issues
            var memoryStream = new MemoryStream();
            await response.ResponseStream.CopyToAsync(memoryStream, cancellationToken);
            memoryStream.Position = 0;
            
            return memoryStream;
        }
        catch (AmazonS3Exception ex) when (ex.StatusCode == System.Net.HttpStatusCode.NotFound)
        {
            _logger.LogWarning("Object {ObjectKey} not found in S3", blobName);
            return null;
        }
    }

    public async Task<bool> DeleteAsync(string blobName, CancellationToken cancellationToken = default)
    {
        try
        {
            var request = new DeleteObjectRequest
            {
                BucketName = _bucketName,
                Key = $"{_keyPrefix}{blobName}"
            };

            await _s3Client.DeleteObjectAsync(request, cancellationToken);
            return true;
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Failed to delete object {ObjectKey} from S3", blobName);
            return false;
        }
    }

    public async Task<bool> ExistsAsync(string blobName, CancellationToken cancellationToken = default)
    {
        try
        {
            var request = new GetObjectMetadataRequest
            {
                BucketName = _bucketName,
                Key = $"{_keyPrefix}{blobName}"
            };

            await _s3Client.GetObjectMetadataAsync(request, cancellationToken);
            return true;
        }
        catch (AmazonS3Exception ex) when (ex.StatusCode == System.Net.HttpStatusCode.NotFound)
        {
            return false;
        }
    }

    public async Task<string> GetDownloadUrlAsync(string blobName, TimeSpan expiresIn, CancellationToken cancellationToken = default)
    {
        var request = new GetPreSignedUrlRequest
        {
            BucketName = _bucketName,
            Key = $"{_keyPrefix}{blobName}",
            Verb = HttpVerb.GET,
            Expires = DateTime.UtcNow.Add(expiresIn)
        };

        return await Task.FromResult(_s3Client.GetPreSignedURL(request));
    }

    public async Task<IEnumerable<BlobItem>> ListAsync(string prefix = "", CancellationToken cancellationToken = default)
    {
        var items = new List<BlobItem>();
        
        var request = new ListObjectsV2Request
        {
            BucketName = _bucketName,
            Prefix = $"{_keyPrefix}{prefix}"
        };

        ListObjectsV2Response response;
        do
        {
            response = await _s3Client.ListObjectsV2Async(request, cancellationToken);
            
            items.AddRange(response.S3Objects.Select(obj => new BlobItem
            {
                Name = obj.Key.Replace(_keyPrefix, string.Empty),
                Size = obj.Size,
                LastModified = obj.LastModified,
                ContentType = "application/octet-stream" // S3 doesn't return content type in list
            }));

            request.ContinuationToken = response.NextContinuationToken;
            
        } while (response.IsTruncated);

        return items;
    }
}

S3 Bucket Infrastructure (Terraform)

resource "aws_s3_bucket" "main" {
  bucket = "my-documents-bucket"

  tags = {
    Environment = "production"
  }
}

# Versioning (optional - equivalent to Azure blob versioning)
resource "aws_s3_bucket_versioning" "main" {
  bucket = aws_s3_bucket.main.id

  versioning_configuration {
    status = "Enabled"
  }
}

# Encryption (server-side)
resource "aws_s3_bucket_server_side_encryption_configuration" "main" {
  bucket = aws_s3_bucket.main.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

# Lifecycle rules (equivalent to Azure access tiers)
resource "aws_s3_bucket_lifecycle_configuration" "main" {
  bucket = aws_s3_bucket.main.id

  rule {
    id     = "archive-old-files"
    status = "Enabled"

    transition {
      days          = 30
      storage_class = "STANDARD_IA" # Infrequent Access
    }

    transition {
      days          = 90
      storage_class = "GLACIER" # Archive
    }

    expiration {
      days = 365
    }
  }
}

# Block public access (security best practice)
resource "aws_s3_bucket_public_access_block" "main" {
  bucket = aws_s3_bucket.main.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

IAM Requirements

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject",
        "s3:ListBucket",
        "s3:GetObjectMetadata"
      ],
      "Resource": [
        "arn:aws:s3:::my-documents-bucket",
        "arn:aws:s3:::my-documents-bucket/*"
      ]
    }
  ]
}

Testing with Localstack

docker-compose.yml:

version: '3.8'
services:
  localstack:
    image: localstack/localstack:latest
    ports:
      - "4566:4566"
    environment:
      - SERVICES=s3
      - DEBUG=1
    volumes:
      - ./localstack-data:/var/lib/localstack

Test configuration (appsettings.Development.json):

{
  "AWS": {
    "Region": "us-east-1",
    "ServiceURL": "http://localhost:4566"
  },
  "S3": {
    "BucketName": "test-bucket",
    "KeyPrefix": "documents/"
  }
}

Migration Checklist

• Remove Azure Blob Storage NuGet packages
• Add AWS S3 NuGet packages
• Create S3 buckets (Terraform/CloudFormation)
• Configure bucket policies and encryption
• Set up lifecycle rules (archival, expiration)
• Update service registration
• Convert BlobContainerClient to IAmazonS3
• Update configuration (connection string → bucket name)
• Add IAM policies
• Migrate SAS URLs to pre-signed URLs
• Update integration tests to use Localstack
• Verify extension method signatures unchanged
• Test multipart uploads for large files (>5 MB)

Common Pitfalls

⚠️ Object keys: S3 doesn't have folders - use prefixes (e.g., documents/2024/file.pdf) ⚠️ Multipart uploads: Files >5 GB require multipart upload API ⚠️ Pre-signed URLs: Expiration max is 7 days (vs Azure SAS which can be longer) ⚠️ Storage classes: S3 Glacier retrieval takes hours (vs Azure Archive instant with rehydration) ⚠️ Listing performance: Large buckets may require pagination (use ContinuationToken) ⚠️ Metadata: S3 metadata keys must be lowercase

Advanced: Multipart Upload for Large Files

For files >100 MB, use multipart upload:

public async Task<bool> UploadLargeFileAsync(string blobName, Stream content, string contentType, CancellationToken cancellationToken = default)
{
    var transferUtility = new TransferUtility(_s3Client);
    
    var request = new TransferUtilityUploadRequest
    {
        BucketName = _bucketName,
        Key = $"{_keyPrefix}{blobName}",
        InputStream = content,
        ContentType = contentType,
        PartSize = 10 * 1024 * 1024 // 10 MB parts
    };

    await transferUtility.UploadAsync(request, cancellationToken);
    return true;
}

Success Criteria

✅ All Azure Blob Storage code replaced with S3 ✅ Extension method signatures unchanged ✅ Bucket policies and encryption configured ✅ Lifecycle rules migrated ✅ IAM policies applied ✅ Pre-signed URL generation working ✅ Localstack tests passing ✅ Large file uploads working ✅ No breaking changes to consuming applications