Fun with ESX: Learning about Bridge in Linux using VMs

Linux kernel has an inbuilt support for providing switching capability. In this post, I am going to define a simple a simple network to setup and then create the same using Linux VMs on ESX host.

Problem definition:-
Create a basic bridge(switch) and allow communication between the two VMs connected to that switch/bridge. In this, the switch will be created on a VM. Below mentioned diagram lists the network setup details:-

Requirements:-
Since we are using ESX host as the hypervisor, we need an ESX host to run the virtual machines, and 3 linux virtual machines on it. 

Machine(Bridge VM) which will act as bridge:-
This VM needs to have 2 vnics.

Rest 2 Traffic VMs(VM1 and VM2):- 

These VMs need to have atleast one vnic.

Setup Part:-
 Create 3 VMs on the ESX host. Below is the snapshot from my setup:-

Next install bridge-utils on the bridge-vm(Switch_Test VM as per my setup). These bridge-utils provide the functionality to create/delete/modify the virtual switches on the linux host.

 root@switch-test# apt-get install bridge-utils  

Now make sure that no ip-address is assigned to the two nics of switch-vm. Instead provide the following config in "/etc/network/interfaces" file(my linux OS is ubuntu).

 # The loopback network interface  
 auto lo  
 idace lo inet loopback  
   
 auto eth0  
 iface eth0 inet manual  
   
 auto eth1  
 iface eth1 inet manual  

Once this is done, you need to create your first switch. You can create that using the following commands:-

 root@switch-test# brctl addbr br0  

For more details, see the manpage of brctl.

Now you need to add the two nics of switch-vm to the bridge.

 root@switch-test# brctl addif br0 eth0  
 root@switch-test# brctl addif br0 eth1  

Now you need to zero-ip the interfaces added.

 root@switch-test# ifconfig eth0 0.0.0.0  
 root@switch-test# ifconfig eth1 0.0.0.0  

Once added, you need to bring up the br0.

 root@switch-test# ip link br0 up

 OR

 root@switch-test# ifconfig br0 up

Once done, your bridge is ready. You just need to plug the machines as mentioned in the setup diagram(problem statement diagram).

Main challenge here is to have the TrafficVM's connected to the individual nic of switch and making sure they cannot talk with each other directly. To do this, I have created 2 port-groups, each having separate VLAN and then applied them to the VM's vnics. Mentioned below is the pic of vswitch network on ESX host.

Simplifying the above network connectivity diagram, I have a network setup as depicted in the diagram below:-

Make sure that both the port-groups created are in promiscous mode, otherwise packets will not reach to the interfaces of switch-vm.

Now assign IP addresses to VM1 and VM2 such that they both are in the same subnet. Send a ping from VM1 to VM2. You will be able to communicate from VM1 to VM2 and vice-versa via switch running inside switch-vm.

Fixing Cisco's VPN Client Error on Windows 8

Today, working on my Windows 8 machine, I found that Cisco's VPN client was giving me "Error:442" and was unable to connect to VPN. After trying a lot and searching on web, I found the solution to the problem. Here are the steps mentioned below to get it working.

Step 1:-
Goto Run by pressing "Windows + R" and type "regedit". It will open the registry editor.

Step 2:-
Navigate to the following key: "HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\CVirtA"

Step 3:-
Here you will find that "DisplayName" key has some garbage value followed by VPN client name. You need to modify it.

Select the DisplayName to modify, and remove the leading characters from the value data upto "%;" i.e.

• For x86, change the value data from something like "@oem8.inf,%CVirtA_Desc%;Cisco Systems VPN Adapter” to "Cisco Systems VPN Adapter”

• For x64, change the value data from something like "@oem8.inf,%CVirtA_Desc%;Cisco Systems VPN Adapter for 64-bit Windows” to "Cisco Systems VPN Adapter for 64-bit Windows"  Once this is done, you will be able to connect to VPN using VPN Client.

Other Errors which might also lead to Error 442:-
If you have enabled "Internet Connection Sharing", even then you will find this error. For this, you need to open "services.msc", navigate to "Internet Connection Sharing" and disable it.

Playing with ESX: Setting up NAT in Linux (Basic Setup)

This post is about how to setup NATing in a Linux environment. If you have one central Gateway or Public IP and want all the internal machines to communicate to the outside world using Gateway's IP or the public IP, you need to setup a NAT for the same. Below mentioned steps will tell you how to configure the same.

Requirements:-
1. Gateway should have 2 nics, one to access internal network and other to access public network.
2. Internal machines should be connected via switch/hub to private nic of Gateway/Machine A.

In the setup mentioned below, I am using 2 VMs, config of each VM is as shown below:-

Config of Gateway:-
2 nics attached to VM.
First nic attached port-group "VM Network" which can access the public network.
Second nic added port-group "TMP" which allows only VLAN 85 traffic. Hence its a private network.

Config of internal VM:-
Internal VM is having only 1 nic. This nic is assigned a port-group TMP such that its a part of private network and can access nic1 of Gateway VM.

Network setup for the above is as follows(Note that Gateway VM is named as RST1 and internal VM is named as RST2):-

Assign IP address to Nic1 on the Gateway machine which is a part of the private network.

 # This file describes the network interfaces available on your system  
 # and how to activate them. For more information, see interfaces(5).  
   
 # The loopback network interface  
 auto lo  
 iface lo inet loopback  
   
 # The primary network interface  
 auto eth0  
 iface eth0 inet dhcp  
   
 auto eth1  
 iface eth1 inet static  
 address 10.3.1.1  
 netmask 255.255.255.0   

To enable routing functionality on Linux, one needs to enable IP forwarding. You can do the same by entering the following command:-

 # echo “1″ > /proc/sys/net/ipv4/ip_forward  

To make the changes persistent across reboots, just add the following line in "/etc/sysctl.conf". Open this file in your favourite editor and add the following line:-

 net.ipv4.ip_forward = 1  

Reboot your machine.

Now you need to add Gateway functionality to this VM. For this, you need to add IPTable rules such that traffic coming on nic1 is sent out through nic 0. You can enter the following commands on the Linux terminal:-

 /sbin/iptables -P FORWARD ACCEPT  
 /sbin/iptables --table nat -A POSTROUTING -o eth0 -j MASQUERADE  

You can also add the above two lines at the bottom of "/etc/rc.local" file so that changes are present even after reboot. Now your basic NAT gateway is up and running.

To test the functionality of the Gateway, just provide the gateway address as the address of eth1 of Gateway and you will be able to connect to the outside world using Gateway. Below mentioned is the "/etc/network/interfaces" file of inside VM.

 # This file describes the network interfaces available on your system  
 # and how to activate them. For more information, see interfaces(5).  
   
 # The loopback network interface  
 auto lo  
 iface lo inet loopback  
   
 # The primary network interface  
 auto eth0  
 iface eth0 inet static  
 address 10.3.1.2  
 netmask 255.255.255.0  
 gateway 10.3.1.1  
   

Now if you will try to ping to outside world using this internal VM, you would be able to do so using the gateway.

Somethings you can try:-
Try configuring the Gateway such that you can SSH to the VMs behind the Gateway. You need to enable Port-Forwarding for the same. Happy configuring. :)

Last but not the least, I would like to thank my special friend for coming up with the idea to learn together and do this setup. Hopefully we both would do a lot more funny experiments in future. Thanks again to that friend. :)

Communication between VM’s in ESX on ESX setup

VMware ESX is a hypervisor from VMware which helps you to virtualize your infrastructure. Refer this link for more details.

Virtualizing the ESX hypervisor (i.e. running it as a VM) itself is a great way to experiment with different setups and scenarios without needing physical lab setup to perform the same. One can create his own mini-lab on the virtual ESX setup, add networking and storage to it, and thus can simulate a wide variety of test scenarios. Virtualizing the ESX also gives user the freedom to clone/snapshot the ESX. Thus, user can simply bring up this virtualized ESX on some other host and he has his whole lab setup, up and running within a few minutes.

VMware ESX has the capability of running “ESX as a VM” on top of the physical ESX host. This virtual ESX host has all the capabilities as that of the physical host. Thus, one can have a VM running inside a VM (virtual ESX) present on physical server.

 

VM’s communicate with each other through the virtual switches (vswitch) provided by ESX.  Considering the case of Virtual ESX on the physical ESX, one gets the following connectivity between the vswitches:-

Looking at the above setup, let’s have a scenario where machine A wants to communicate to machine B. Packets from machine A needs to travel through vswitch1 and vswitch2 to reach to machine B. Assuming there are no VLANs configured on the switches and all ports are a part of default VLAN (native VLAN), one might say that A will be able to communicate with B. But in reality, A is unable to communicate with B. If A sends a ping request to B, then only ARP resolution takes place (mac address of B gets learn at A). Ping’s reply is not received by A. The workaround to facilitate this communication is by putting vswitch1 on promiscuous mode. Let’s discuss about the cause of failure and how promiscuous mode solves the issue.

Cause of failure:-

VMware vswitch does not have the capability of mac learning. It maintains a table of the port to mac binding only for the VM’s present on the vswitch. Anything except that, it forwards through the uplink (physical NIC).

Promiscuous mode:-

It is a mode of operation which allows network device to intercept and read each network packet that arrives in its entirety. Enabling promiscuous mode on a switch makes the switch work like a hub.

How vswitch works while in non-promiscuous mode (default):-

For ARP request, since it’s a broadcast, following steps take place:-

1.  Machine A sends an ARP broadcast.
2. vswitch1 forwards this broadcast packet to all members present in that VLAN as well as on Physical NIC. This broadcast packet is received both by vswitch2 and a Physical Switch.
3. On receiving the ARP request vswitch2 again broadcasts it and now it is received by machine B.
4. Machine B sends a unicast reply. This reply is seen by vswitch2.
5. Since the destination is not in its table, vswitch2 forward this response through physical NIC and is received at vswitch1.
6. Vswitch1 finds the destination mac-address entry in its table and it forwards the reply to machine A. Hence, machine A learns the mac-address of machine B.

Next, if one tries to ping to machine B from machine A, following steps take place:-

1. Machine A sends a ping request with destination mac-address of machine B (which it learnt from ARP broadcast).
2. Since it is a unicast request, vswitch1 tries to find the destination mac-address entry in its table. It is unable to find that entry and hence it sends that packet out to physical switch through the NIC. This packet is not forwarded to vswitch2 and hence machine A is unable to communicate with machine B.

When vswitch1 is in promiscuous mode:-

ARP request is resolved in the same way as it’s resolved for non-promiscuous mode discussed above.

For ping request/response, following actions take place:-

1. Unicast ping request is sent by machine A to machine B. This is received by vswitch1.
2. Since vswitch1 is in promiscuous mode, all the ports on the switch are able to see the traffic of other ports who are part of the same VLAN. Therefore, this packet is sent out to both physical port and vswitch2.
3. Vswitch2 finds port to mac entry for destination machine B and forwards the packet to machine B.
4. Machine B replies to ping request. This packet is received at vswitch2.
5. Since no entry for destination mac is found on vswitch2, it is forwarded to vswitch1.
6. Vswitch1 is able to find the entry for destination mac in its table, and hence the response is forwarded to A. Thus, the response from B is received by A.

With promiscuous mode enabled, VM’s on the internal ESX host are able to communicate with the outside world.

However, drawback of promiscuous mode is that it will lead to increase in the traffic on vswitch and it will degrade the performance of the switch. One solution could be to put only the port which is attached to vswitch2 in promiscuous mode. But then, A needs to be a part of the same VLAN which that port allows. Another solution could be to use the advanced virtual switches available in the market which also perform mac learning. Some examples of the advanced switches are VMware DVS, Cisco’s N1KV, IBM DOVE, etc.

References:-

http://www.vkernel.co.uk/?p=14

http://blog.nlitee.com/virtualization/vmware/configuring-vlans-on-vswitches/

http://www.cisco.com/en/US/products/ps9902/index.html

Allowing FTP through Firewall by Inspecting traffic

Problem statement:-

FTP traffic needs to be allowed through the firewall. Basic details about FTP can be found here. This cannot be done by simply allowing the traffic for control connection ( by default port 21 ) and not taking care for the data-connection. For this, one needs to analyze the FTP control path (control connection) data to get the details about the data-path (data connection). Once the data-path details are known, one just needs to create embryonic flow entries for the data-connection.

Embryonic flow: A flow entry created for the traffic-flow before there is actual traffic flowing which matches the flow specifications.

flowID: This is a five-tuple field to differentiate the various connections.

Control path activities for a particular FTP request are as follows :-

For Active Mode

For Passive Mode 

  

Out of these 8 packets on control-path, ignoring ACK, we have ‘Five’ generic packets in our flow. We can list these requests/responses in terms of states.

These states are:-
1.    Opening
2.    Opened
3.    command_sent
4.    execution_started
5.    execution_complete

One needs to keep track of FTP control connection’s state and the data-connection details negotiated. This can be done by storing the details in a table called “flow table”.

Flow table column’s are as follows :-
1.    Command Type
2.    IP Address exchanged for data-connection( required for creating embryonic flow )
3.    Port number exchanged for data-connection( required for creating embryonic flow )
4.    Response_code of the command
5.    State of the FTP connection

When a new FTP connection is initiated (is in opening state), one entry is made in the flow table. It then goes through all the states; when it reaches to the state of execution_complete, that entry is removed.

STATE TRANSITIONS FOR PORT COMMANDS:-

Now lets consider the case for PORT command. Five request/response actions occur in this. These are:-

( Client[XYZ] to Server[FTP 21] ) – FTP Request: PORT          -> OPENING
( Server[FTP 21] to Client[XYZ] ) – Response: 200 PORT      ->  OPENED
( Client[XYZ] to Server[FTP 21] ) – FTP Request: <CMD>    ->  CMD_SENT
( Server[FTP 21] to Client[XYZ] ) – Response: 150                ->  EXEC_STARTED
( Server[FTP 21] to Client[XYZ] ) – Response: 226                ->  EXEC_COMP

Steps followed for each of the above requests/responses:-

Packet1:-

First packet sent is the packet with PORT command from client to server.

 

When this packet is received, a flow-entry is made and IP_Address and PORT negotiated are calculated from the message. Also the state is changed to OPENING.

Packet 2:-

Second packet is the response to the PORT command.

Once it is received, Response code is calculated from the message. If  the response-code is positive ( like 200), a flow-entry is present in the flow_table for this packet’s flowID and state is OPENING, then the state is changed from OPENING to OPENED.

Packet 3:-

On the next packet if flow table has a entry for its flowID and is in the OPENED state with the direction from Client to server, then it’s a request from Client to server to perform a specified operation.

After receiving this packet, the state is now changed from OPENED to CMD_SENT.

In case of PORT commands, TCP 3-way handshake for the data-connection occurs after this packet is received by the server. Therefore, embryonic-flow-entry is made in the allowed-connections table on the firewall so that this data-connection traffic is not blocked by the firewall.

Packet4:-

Next packet for the particular flow is the response from the server. Based on the response code received, the state will be changed.

If the response from the server is positive preliminary reply and the state for the particular flow-entry is CMD_SENT, then the state is changed to EXEC_STARTED. If the response is negative, then the flow-entry is deleted, embryonic-flow-entry is also deleted.

Packet5:-

This is the last packet received indicating the status of the FTP command.

If the response code is ‘226’, this means the transfer has occurred successfully. If the previous state was EXEC_STARTED, the state is now changed to EXEC_COMP and the embryonic-flow-entry and flow-entry can be deleted from the respective tables. If some negative reply is received, even then the embryonic-flow-entry and the flow-entry are deleted from the tables.

STATE TRANSITIONS FOR PASV COMMANDS:-

For passive commands, it again goes through the five states mentioned below:-
( Client[XYZ] to Server[FTP 21] ) – FTP Request: PASV                   ->   OPENING
( Server[FTP 21] to Client[XYZ] ) – Response: 227 (port_no)         ->   OPENED
( Client[XYZ] to Server[FTP 21] ) – FTP Request: <CMD>             ->   CMD_SENT
( Server[FTP 21] to Client[XYZ] ) – Response: 150                         ->   EXEC_STARTED
( Server[FTP 21] to Client[XYZ] ) – Response: 226                         ->   EXEC_COMP

Steps followed for each of the above requests/responses:-

Packet1:-

First packet sent is the packet with PASV command.

Once it is received, a flow-entry is made in the flow_table. Also the state is changed to OPENING.

Packet 2:-

Second packet is the response to the PASV command. If a flow-entry is present in the flow_table for the packet’s flowID and its state is OPENING and the direction is from server to client, then its a response from the server.

Once it is received, Response code and PORT negotiated is calculated. According to the response code received, a particular action is taken. Suppose if its 227, then the state is changed from OPENING to OPENED.

In case of PASV commands, TCP 3-way handshake on the data-connection occurs after this packet is received by the client. Therefore, an embryonic flow-entry is made in the firewall’s allowed-connections table with the data-connection details.

Packet 3:-

If the next packet for the particular flowID is from Client to server and state is OPENED, then it is a request from Client to server to perform some operation.

After receiving this packet, the state is now changed from OPENED to CMD_SENT.

Packet4:-

Next packet for the particular flowID would be response from the server.

If the response from the server is positive and state is CMD_SENT, then the state is changed from CMD_SENT to EXEC_STARTED. If the response is negative, then the flow-entry and embryonic-flow-entry are deleted  from the respective tables.

Packet5:-

This is the last packet received indicating the status of the FTP command.

If the response code is ‘226’ and state is EXEC_STARTED, this means the transfer has occurred successfully, state is changed to EXEC_COMP and the flow-entry and embryonic-flow-entry can now be deleted from the respective tables. In case of negative replies also, the same action is taken.

Conclusion:-

This approach can be followed in all the cases where you have the data-path created dynamically and its information is negotiated on the control-path. Above example listed how this can be applied to FTP. Similarly, one can use it for RSH, MSRPC with different set of columns in flow_table( as per the protocol needs).

 

How FTP works

FTP stands for File Transfer Protocol which is used to transfer files from one host to another over TCP. RFC 959 lists all the details of FTP. It is an application layer protocol and is used widely across the world for transferring files.

FTP connections (what we read and know):-

FTP consists of 2 connections:-

1.    Control Connection on port 21 of server (Standard, can be changed)

2.    Data Connection on port 20 of server (Standard, can be changed)

Control connection is established between port 21 of server and random port of client. Once opened, this connection remains there unless closed by user or idle timeout occurs.

Data connection is established every time client sends a request using control connection to server for data transfer. For every new Data connection, port number changes. The new port number is negotiated on the control connection. 

NOTE:-

Not all the FTP commands use the Data Connection…

Here is the list of FTP commands. Commands with (√ ) indicates that the command uses data connection and commands with (X) indicates that the command doesn’t use data connection.

 

Looking at the above diagram, one could see that of all the commands available for FTP, ratio of commands which use data connection to all commands is very less.

FTP data transfer modes are of 4 types:-

·        1. Active Mode (Also known as PORT based)

·        2. Passive Mode (Also known as PASV based)

·        3. Extended active mode to support NAT (EPRT)

·        4. Extended passive mode to support NAT (EPSV)

Active Mode:-
In this, client tells to the server on which client-port the data connection will be made. So client sends the port number with the command to the server. In PORT commands, port 20 is used by server for Data transfer.

Passive Mode:-

Sometimes client might be behind the firewall. In such cases, server cannot directly connect to the client on the negotiated port. So for that Passive mode is used. In Passive mode, any random port is used by server for Data connection.

        

In this, Server sends the data connection port no. to client and then client connects to that portnumber for data connection.

EPRT and EPSV modes:-
These modes are to support FTP in case of IPv6 and NATing. More details about them can be found in FTP’s RFC document.      

Traffic flow examples for PORT and PASV modes:-

Now we will try to see what traffic flows in case of FTP connection.

Server’s IP: 192.168.10.10

Client’s IP:  192.168.10.11

Now let us see what traffic flows when some request is sent by FTP-client to FTP-server which uses data-connection for both Active and passive modes. Thing to note here is that control connection is already established between client and server. We will discuss what happens after that.

­For Active mode:-

Once the user enters the FTP request, ftp client needs to create a data connection to get the output from the server. It first needs to negotiate the port. So it sends the command “PORT” with client machine’s IP and port to which the data connection should be made.

As seen from the above diagram, client sends request to server and server responds back. Port is successfully negotiated between them.

Next client sends the FTP request on the control channel. Now server performs 3-way handshake with the data port negotiated and then responds back with the output on the data channel. Once the data-transfer is complete, the data-connection is terminated.

For each new FTP request, a new data-connection is created.

For Passive mode:-

In passive mode, the client first informs server that it wants to open a passive mode data-connection with the server. It sends “PASV” command to server on control channel. Server responds back by sending the port to which it will open the data connection.

As we can see from the above diagram, server responds back with the IP and port number. 3-way handshake occurs at this time for the data-connection.

Next client sends the FTP request on the control-connection and server sends back the output on the data-connection. Once the data-transfer is complete for that request, the data-connection is terminated.

Conclusion:-

Considering the active and passive modes, we can re-design the basic diagram of FTP to more advanced and appropriate one as follows:-

This diagram now appropriately lists about the FTP and the various connections in FTP.