Granting billing access to IAM users

If you are the root account holder, then IAM users in that a/c by default will not be able to view billing or account preference information (even if you give them Administrator access). As mentioned in the AWS docs:-

http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/control-access-billing.html

the below checkbox under (My Account -> Account Settings) needs to be enabled if you want IAM users to view billing. This can further be restricted by user level IAM inline policies or group level policies.

Sorting contents of S3 bucket via AWS CLI since AWS console doesn't allow sort today

In AWS console, the columns don't sort if we click on them :-

So the current way to sort would be to use AWS CLI using "sort_by" function as below:

$aws s3api list-objects --bucket sample-bucket --max-items 3 --query "sort_by(Contents, &LastModified)[*].{Key:Key,Size:Size,LastModified:LastModified}"
[
    {
        "LastModified": "2015-02-04T10:55:29.000Z",
        "Key": "sample1.csv",
        "Size": 27848
    },
    {
        "LastModified": "2015-02-04T10:55:29.000Z",
        "Key": "sample2.csv",
        "Size": 469266754
    },
    {
        "LastModified": "2015-02-04T10:55:44.000Z",
        "Key": "sample3.csv",
        "Size": 469529478
    }
]

Since s3api does not have a "sort-order" switch unlike some of the other api's, you will have to use the "reverse" function specified as below:-

$aws s3api list-objects --bucket sample-bucket --max-items 3 --query "reverse(sort_by(Contents, &LastModified)[*].[Key, LastModified])" --output table

sample3.csv 2015-02-04T10:55:44.000Z
sample2.csv  2015-02-04T10:55:29.000Z
sample1.csv  2015-02-04T10:55:29.000Z

For more details on jmespath json query language specification, pl. refer to

http://jmespath.org/

Four ways to secure your S3 object storage

1. IAM user policies - http://docs.aws.amazon.com/AmazonS3/latest/dev/example-policies-s3.html
2. ACLs - http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html
3. Bucket policies - http://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html
4. Query String authentication - http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html

IAM policy to allow AWS console view to "Preferences"

If you are using tags on your EC2 instances, you will want to look at "manage resources and tags" option under preferences. If you would like to give a particular user (e.g a linked user a/c) access to consolidated billing account, but restrict all other views, you can use an IAM policy like below to allow access:-

*******************

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "aws-portal:ViewBilling",
                "aws-portal:ModifyBilling",
                "aws-portal:ViewAccount"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Deny",
            "Action": [
                "aws-portal:ViewPaymentMethods",
                "aws-portal:ModifyPaymentMethods",
                "aws-portal:ModifyAccount"
            ],
            "Resource": "*"
        }
    ]
}

****************************

Debugging a http 504 (Gateway Timeout) issue from Amazon ELB logs

Sometimes when you are testing an application that is hosted on AWS and front ended by ELB, you may see "504 Gateway Timeout" as below in the browser:-

The sequence of events that lead up to 504 client side error is as below:

1) Client connects to the ELB and submits HTTP request
2) ELB Picks a backend and sends the request onto the backend.
3) The backend receives the request and starts processing it
4) While the backend is still processing the request the client gives up waiting and ends the connection (this sometimes can be reported as 504 by proxies and other HTTP libraries)
5) The backend finishes processing the request and replys to the ELB
6) As the client has ended the connection the ELB can't pass the request on to the client and instead logs a 408 error.
7) A network packet capture on the client machine should indicate a client socket getting timed out.

For details on the various ELB error codes, you can refer to

http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/ts-elb-error-message.html

In the ELB logs, you will see

*************
2015-03-18T14:55:24.507386Z <ELB NAME> <Client IP>:51305 - -1 -1 -1 408 0 0 0 "GET <URL> HTTP/1.1"
*************

If you note that "-1" value reported by the ELB, it indicates that client socket was closed and the timing for the response and content-length are reported incorrectly as "0". AWS ELB team is looking into addressing this in future. A similar log for HTTP 404 will look like

************
2015-03-18T14:00:15.618471Z <ELB NAME> <Client IP>:50855 <Instance in ELB pool IP>:<Listen Port> 0.000071 0.010108 0.000021 404 404 0 2096 "GET <URL> HTTP/1.1"

************

Current Limitations of the Amazon ELB

If you are using Amazon ELB for handling production workloads for your SaaS/PaaS customers, you would have encountered the below limitations with Amazon Elastic Load Balancers

• No iRULES mapping - If you have worked with F5 load balancers, you would already be familiar with iRules (https://devcentral.f5.com/articles/irules-101-01-introduction-to-irules). It allows for inspecting the incoming packets and execute rules that can allow for redirection of traffic. Today, ELB is not "application aware", which means that the load balancing feature is pretty basic and can only do "round-robin" balancing (as opposed to weighted load balancing etc). The options to send the traffic to a particular instance based on the client-ip header do not exist. NOTE - The good news is that Amazon ELB team is working on a potential solution with high priority and should be available before end of this year. 

• No Fixed IP for ELB - There are situations where your customer might outbound traffic from their firewall to access only certain ip addresses on port 80 or 443. ELB's today have a domain name and have public ip addresses (could be multiple if ELB is available across zones). If the below public ip's are whitelisted in the corporate firewall, then these ip addresses will change when ELB's are scaled out or rebooted for maintenance. NOTE - Amazon's workaround for today is to whitelist the entire CIDR block of EC2 addresses (which includes ELB addresses - https://aws.amazon.com/blogs/aws/aws-ip-ranges-json/). The other option is to pre-warm the ELB's to estimate the load and then whitelist upto 10 public ip addresses for the 10 ELB instances.

***********

$ host Test-ELB-669438560.us-east-1.elb.amazonaws.com

Test-ELB-669438560.us-east-1.elb.amazonaws.com has address 107.23.x.x

Test-ELB-669438560.us-east-1.elb.amazonaws.com has address 54.236.x.x

***********

• No Mutual Authentication for SSL - One of the nice features of the ELBs is that they can be made as the SSL termination point instead of your reverse proxy or application. There are several benefits of this approach including reduced configuration on the application/reverse proxies. However, the downside is that currently we cannot perform mutual authentication as part of the SSL handshake, i.e., we cannot perform client certificate verification, which is necessary to make sure that only valid clients are accessing the application. NOTE - Amazon indicated that they are aware of this limitation and working on a potential solution before the end of this year.

IOPS as measured by Amazon

AWS doc - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-characteristics.html
EBS benchmarking for piops -  - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/benchmark_piops.html

"IOPS are input/output operations per second. Amazon EBS measures each I/O operation per second (that is 256 KB or smaller) as one IOPS. I/O operations that are larger than 256 KB are counted in 256 KB capacity units. For example, a 1,024 KB I/O operation would count as 4 IOPS. When you provision a 4,000 IOPS volume and attach it to an EBS-optimized instance that can provide the necessary bandwidth, you can transfer up to 4,000 chunks of data per second (provided that the I/O does not exceed the 128 MB/s per volume throughput limit of General Purpose (SSD) and Provisioned IOPS (SSD) volumes).

This configuration could transfer 4,000 32 KB chunks, 2,000 64 KB chunks, or 1,000 128 KB chunks of data per second as well, before hitting the 128 MB/s per volume throughput limit. If your I/O chunks are very large, you may experience a smaller number of IOPS than you provisioned because you are hitting the volume throughput limit. For more information, see Amazon EBS Volume Types.

For 32 KB or smaller I/O operations, you should see the amount of IOPS that you have provisioned, provided that you are driving enough I/O to keep the drives busy. For smaller I/O operations, you may even see an IOPS value that is higher than what you have provisioned (when measured on the client side), and this is because the client may be coalescing multiple smaller I/O operations into a smaller number of large chunks.

If you are not experiencing the expected IOPS or throughput you have provisioned, ensure that your EC2 bandwidth is not the limiting factor; your instance should be EBS-optimized (or include 10 Gigabit network connectivity) and your instance type EBS dedicated bandwidth should exceed the I/O throughput you intend to drive."