Pages - Menu

The Quickest Way to Bulk Create AWS S3 Buckets

I am in a mission of creating a bunch of s3 buckets in bulk. I could just use the web interface and do it manually. I reckon I can get the job done in around 30 minutes, but I am looking for a more an IT way to do it.

Solution

After a quick look around, I think I can use the Command Line Interface.

Download and Setup


Configure


Run the aws configure command to create a default profile. Fairly straight forward.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

A list of region name can be found here.

Commands


http://docs.aws.amazon.com/cli/latest/reference/s3/index.html

I use the ls command to pull out a list of my buckets.

$ aws s3 ls

In order to create buckets, the command I need is mb.

$ aws s3 mb s3://mybucket

I prepared my list of commands by using Excel / Find and Replace.

$ aws s3 mb s3://sunnyw.net.dummy1
$ aws s3 mb s3://sunnyw.net.dummy2
$ aws s3 mb s3://sunnyw.net.dummy3

And I just copy and paste into the console.

PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell> aws s3 mb s3://sunnyw.net.dummy1
make_bucket: s3://sunnyw.net.dummy1/
PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell> aws s3 mb s3://sunnyw.net.dummy2
make_bucket: s3://sunnyw.net.dummy2/
PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell> aws s3 mb s3://sunnyw.net.dummy3
make_bucket: s3://sunnyw.net.dummy3/
PS C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell>

I didn't need any fancy API calls or build any C# console. If the requirement was more complicated, I could put those logics in PowerShell script. Simple is Good!!

Amazon Web Services - AWS101


Recently had a meeting with the Amazon expert discussing about our AWS needs. We had a brief overview of all the relevant AWS services. There were many jargon involved, I already know some of them, but not all. Mostly are just product names.

Region and Availability Zone

Availability Zone is a physically isolated location or what traditionally called data centre. In an ideal setup we would split our instances into 2 AZs for redundancy. I was told the connection between the AZs are dark fiber and latency at single digit millisecond.

Amazon region can have multiple AZs. See the difference between Region and Availability Zone.

Elastic Load Balancing

ELB is the amazon version of network load balancing. It sits in front of AZs and redirects traffics to different AZs.

CloudWatch

Amazon CloudWatch is normally what I refer to as website monitoring. It does health check on the system and sends alert to ELB for traffic control.

EC2 (Elastic Compute Cloud)

EC2 are virtual machines. We can setup IIS on EC2 instance(s) to host our websites. When we spin up an EC2 instance, we can choose from a range of predefined images with different operating system and applications.

Amazon Machine Image

I will interpret AMI as a fancy name for a machine snapshot image or known as ghost image in old fashion. We can use it to scale out our system or simply for backup / restore strategy.

Relational Database Service

RDS is what I would call a managed database instances. It is an already setup database instance and I can utilize without managing a real database instance that I need to rdp in or patch.

S3 (Simple Storage Service)

I am a little more familiar with S3 because my site is hosted by using the S3 bucket. S3 is a plain cloud file storage. Some people misinterpret S3 as CDN, because it can potentially be utilized as CDN for saving bandwidth, but there is a difference.

S3 does not utilize edge servers for faster content delivery. It only has one copy of the files in one location. It is however designed for high performance to serve static files (eg. images, video etc), and also cheaper rate as no compute hours are charged.

CloudFront

The Amazon version of CDN that does edge caching is actually called CloudFront. It is designed to deliver S3 contents to end users via edge servers.

Simple Email Service

SES is a SMTP service for sending out emails.

CloudFormation

I will describe CloudFormation as a non-UI way to create and instantiate different kind of Amazon Services. For example, we can create a EC2 instance by just writing a script in CloudFormation format and run it via the console.

C# .Net Cannot close stream until all bytes are written.

Scenario

We are dealing with the Stream class in .Net and run into an inner exception of "Cannot close stream until all bytes are written."

Code

try
{
    using (var ms = new MemoryStream())
    {
        xElement.Save(ms);

        // Create a PutObject request
        var request = new PutObjectRequest
        {
            BucketName = bucket,
            Key = key,
            InputStream = ms
        };

        // Call S3 ....
    }
}
catch (Exception ex)
{
    _logger.Error(ex.Message, ex);
    throw ex;
}

By using the keyword using, we are expecting .Net will handle the close connection for us.

Rather than unable to close the stream, it was actually a problem where the position of the MemoryStream was at the end after the XElement.Save(). We need to manually reset the position of the stream to zero, so that it can be written to the PutObjectRequest object.

xElement.Save(ms);
ms.Position = 0;