Tuesday, August 5, 2014

Can't SSH into one machine running in Tokyo region from jump host

I ran into a strange issue were one of the machines running in an Amazon VPC in ap-northeast-1b zone can't be SSH'ed into from jump host in our data center. However, another NAT instance in the same CIDR block can be SSH'ed into from the same jump host.

Timeout from jump host:-

$ ssh -i key.pem user@ec2-54-x-x-x.ap-northeast-1.compute.amazonaws.com
ssh: connect to host ec2-54-x-x-x.ap-northeast-1.compute.amazonaws.com port 22: Connection timed out

After checking the security group to make sure that the correct natted external ip address is allowed for ingress port, a simple nmap command showed that port was not in "open" status

$ nmap ec2-54-x-x-x.ap-northeast-1.compute.amazonaws.com -p 22 -P0
Starting Nmap 5.21 ( http://nmap.org ) at 2014-08-05 10:16 PDT
Nmap scan report for ec2-54-x-x-x.ap-northeast-1.compute.amazonaws.com (54.x.x.x)
Host is up.
PORT   STATE    SERVICE
22/tcp filtered ssh

Nmap done: 1 IP address (1 host up) scanned in 13.09 seconds

Next is to check wether there are valid routes to that particular machine. In order to run "traceroute", you will need to request "sudo" privilege from your jump host administrator

$ sudo traceroute -T -p 22 54.x.x.x
[sudo] password for user:
traceroute to 54.x.x.x (54.x.x.x), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *

Since there are no routes available, most likely cause will be firewall behind which the jump host sits is blocking connections to this particular instance. To confirm you can run packet capture such as

$sudo tcpdump -nnvv host 54.x.x.x -w capture.pcap

Now you can try the SSH again and then inspect the packets like

$ sudo tcpdump -r capture.pcap
[sudo] password for user:
reading from file capture.pcap, link-type EN10MB (Ethernet)
23:52:41.715663 IP jumphost.54540 > ec2-54-x-x-x.ap-northeast-1.compute.amazonaws.com.ssh: Flags [S], seq 310869746, win 14600, options [mss 1460,sackOK,TS val 1367537756 ecr 0,nop,wscale 7], length 0
23:52:44.715767 IP jumphost.54540 > ec2-54-x-x-x.ap-northeast-1.compute.amazonaws.com.ssh: Flags [S], seq 310869746, win 14600, options [mss 1460,sackOK,TS val 1367540756 ecr 0,nop,wscale 7], length 0
23:52:50.715717 IP jumphost.54540 > ec2-54-x-x-x.ap-northeast-1.compute.amazonaws.com.ssh: Flags [S], seq 310869746, win 14600, options [mss 1460,sackOK,TS val 1367546756 ecr 0,nop,wscale 7], length 0

As we can see about the 3-way TCP handshake does not proceed further than SYN packet sent from the jump host but there is not SYN ACK followed by ACK. So the connection is not established. Now we have enough information to talk to the relevant folks in IT that manage the jump host.

No comments:

Post a Comment