Installing 32 bit glibc on 64 bit linux OS

When you are trying to start your applications, it is possible that you encounter a startup error in your applications that require native library support. It could be because you don't have 32 bit glibc package installed on your 64 bit OS. Things to confirm,

• Confirm if your system is indeed running 64 bit OS

$ uname -a

Linux ip-10-98-0-41 2.6.32-431.11.2.el6.x86_64 #1 SMP Mon Mar 3 13:32:45 EST 201

4 x86_64 x86_64 x86_64 GNU/Linux

• Confirm the distribution version

$ cat /etc/system-release

Red Hat Enterprise Linux Server release 6.5 (Santiago)

• Confirm the glibc version installed

$ sudo yum list installed glibc

Loaded plugins: amazon-id, rhui-lb, security

Installed Packages

glibc.x86_64          2.12-1.132.el6           @rhui-REGION-rhel-server-releases

• Once you have confirmed that you don't have 32 bit glibc installed, you can install it using package manager specific to your OS

$ sudo yum install -y glibc.i686

• Once 32 bit glibc package has been installed, you can confirm by running "yum list installed"  again

$ sudo yum list installed glibc

Loaded plugins: amazon-id, rhui-lb, security

Installed Packages

glibc.i686            2.12-1.132.el6           @rhui-REGION-rhel-server-releases

glibc.x86_64          2.12-1.132.el6           @rhui-REGION-rhel-server-releases

How to determine if your Amazon EC2 image has PV-GRUB bootloader?

If you would like to load a custom kernel in your Amazon linux instance, you will need an Amazon modified version of bootloader called PV-GRUB (1.04 or higher). Typically, you can determine that by running a set of AWS CLI commands

$aws ec2 describe-instances --instance-id <i-xxxxx> --profile <your profile>
{
    "Reservations": [
        {
            "OwnerId": "",
            "ReservationId": "",
            "Groups": [],
            "Instances": [
                {
                    "Monitoring": {
                        "State": "disabled"
                    },
                   ....
                   ....
                    "Architecture": "x86_64",
                    "KernelId": "aki-1eceaf77",
                    "IamInstanceProfile": {
                        "Id": "",
                        "Arn": ""
                    },
                    "RootDeviceName": "/dev/sda1",
                    "VirtualizationType": "paravirtual",
                    "Tags": [
                       ...
                    "AmiLaunchIndex": 0
                }
            ]
        }
    ]
}

$aws ec2 describe-images --image-ids <aki-xxxx> --profile <your profile>

{

    "Images": [

        {

            "VirtualizationType": "paravirtual",

            "Name": "pv-grub-hd0_1.04-i386.gz",

            "Hypervisor": "xen",

            "ImageOwnerAlias": "amazon",

            "ImageId": "aki-xxxx",

            "State": "available",

            "BlockDeviceMappings": [],

            "Architecture": "i386",

            "ImageLocation": "amzn-ami-us-east-1/pv-grub-hd0_1.04-i386.gz.manifest.xml",

            "RootDeviceType": "instance-store",

            "OwnerId": "",

            "Public": true,

            "ImageType": "kernel",

            "Description": "PV-GRUB release 1.04, 32-bit, configured for (hd0)/boot/grub/menu.lst"

        }

    ]

}

In a VPC, if you are unable to connect to internet through NAT from your instances in private subnet check the Route table and security group configuration

If you have a simple VPC with 1 public and 1 private subnet and you are able to connect to internet from ec2 instances running in your public subnet but not your private subnet instances, then check the following:-

• Check your NAT instance to see if "source/destination" check is disabled

• Check to see if your VPC route tables have been created correctly with main route table not being explicitly associated with a subnet

• Check to see if your secondary route table has association with the internet gateway (igw-*) and is shown as "Active"

• Check to see if your secondary route table has "subnet association" to the public subnet CIDR as you have defined when you created the VPC

NOTE - If you need the instances in your private subnet to use IGW instead of NAT for outbound traffic, then you will need to add your private subnet to this "subnet association" table above. e.g. (10.98.1.0/24). 

• Once all of the above have been checked, then most likely it an ingress port issue on your NAT instance's security group. You have to make sure to add the private subnet(s) CIDR in the "inbound" port rules of your NAT instance security group (assuming your NAT instance is not part of "default" VPC security group)

NOTE - In future if you add additional subnets to your VPC, you will have to explicitly add the CIDR block of the new subnet in the ingress security rules of the NAT sg. You may also add the entire CIDR block of the VPC if you want to automatically add all subnets to connect through NAT instance.

Running a simple telnet could confirm whether you now have access to internet after making the above changes.

On RHEL, sometimes you get the error when running yum - ""Another app is currently holding the yum lock; waiting for it to exit..."

This error could be because some previous yum command did not complete or has been stopped. In which case, you have to find the pid of that "defunct" process. To do that you can do

$ps aux |grep yum
root      2510  0.0  0.0 175512  2660 pts/0    T    21:13   0:00 sudo yum list mysql
root      2511  0.0  0.0      0     0 pts/0    Z    21:13   0:00 [yum] <defunct>
root      2561  0.0  0.0 175512  2656 pts/0    T    21:20   0:00 sudo yum list mysql
root      2562  0.3  0.2 313968 20388 pts/0    T    21:20   0:00 /usr/bin/python /usr/bin/yum list mysql
tdo       2566  0.0  0.0 103252   824 pts/0    S+   21:21   0:00 grep yum

You would have to remove the lock associated with that lock held by that pid first

$sudo rm -f /var/run/yum.pid 2511

Once you kill the lock, you can kill the defunct process using "kill"

$sudo kill -9 2511

and then you should be able to run yum again.

Deleting a VPC sometimes fails because of dependencies

Recently, I was trying to delete a VPC both through console and through awscli and encountered failure because some dependency. The awscli was throwing the error below:-

$aws ec2 delete-vpc --vpc-id <vpc-id> --profile <profile-name> --debug --region us-east-1
2014-05-06 14:24:30,107 - awscli.clidriver - DEBUG - CLI version: aws-cli/1.3.8 Python/2.7.3 Windows/7, botocore version: 0.42.0
2014-05-06 14:24:30,108 - botocore.service - DEBUG - Creating service object for: ec2
2014-05-06 14:24:30,216 - botocore.hooks - DEBUG - Event service-data-loaded.ec2: calling handler <function signature_overrides at 0x0286C630>
2014-05-06 14:24:30,216 - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x0286C530>
....
ClientError: A client error (DependencyViolation) occurred when calling the DeleteVpc operation: The vpc 'vpc-xxxx' has dependencies and cannot be deleted.
2014-05-06 14:24:32,667 - awscli.clidriver - DEBUG - Exiting with rc 255

A client error (DependencyViolation) occurred when calling the DeleteVpc operation: The vpc 'vpc-xxxx' has dependencies and cannot be deleted.

The error message simply says that there is a dependency but doesn't state what the dependency is. If it were a subnet, network acl, RDS instance still connected to VPC, it is easy to determine through the console. However, the network interface is shown only in the "EC2" page rather than VPC page. From the AWS console, you can go to EC2->Network Interfaces to see if there are any eni-* that is still present. If present, you can delete it from the console or from the awscli like

$aws ec2 delete-network-interface --network-interface-id eni-xxxx --profile <profile-name> --debug --region us-east-1
2014-05-06 15:09:57,334 - awscli.clidriver - DEBUG - CLI version: aws-cli/1.3.8 Python/2.7.3 Windows/7, botocore version: 0.42.0
2014-05-06 15:09:57,335 - botocore.service - DEBUG - Creating service object for: ec2
2014-05-06 15:09:57,421 - botocore.hooks - DEBUG - Event service-data-loaded.ec2: calling handler <function signature_overrides at 0x02902630>
2014-05-06 15:09:57,423 - botocore.hooks - DEBUG - Event service-created: calling handler <function register_retries_for_service at 0x02902530>
....
2014-05-06 15:10:00,976 - awscli.formatter - DEBUG - RequestId: 7a23cfd5-4c40-4797-8bd7-5902a6a7f309
{
    "return": "true"
}

Once you delete the unused eni-*, you can now run the "delete-vpc" command to get rid of the unused VPC.

OnDemand vs Reserved Instance pricing of EC2 instances

If you are considering purchasing reserved EC2 instances for your project, you should first consider whether you will need a light utilization, medium utilization or heavy utilization machines. Once you decide on the type of utilization, you have to consider the region of availability. Easy to compare the pricing is freely available from PromptCloud -

ec2-ondemand-vs-reserved-instance-pricing

Getting public X509 certificates from Amazon SES SMTP server

Using openssl "s_client" command to get the certificates that is needed to import into SSL truststore in your java application

$openssl s_client -connect email-smtp.us-east-1.amazonaws.com:587 -showcerts -crlf -starttls smtp