Copy-DatastoreItem - Understanding the Traffic Flow

Posted on 3 mins read

I brought this up on Twitter a while ago.

I studied the traffic flow – and would like to share it with you here, but first here is the architecture of the testbed – which will help explain in more detail

Architecture

Well the environment is pretty simple. One vCenter server, four ESXi Servers – each with a local datastore and a shared datastore among the 4 hosts. My laptop is the one that was running the Copy-DatastoreItem cmdlet.

Copy-DatastoreItem has the following parameters

  • Item - Specify the datastore item you want to copy. You can use a string to provide a relative path to the item in the current provider location.
  • Destination - Specify the destination where you want to copy the datastore item.

The PowerCLI syntax:

Copy-DatastoreItem -Item $ISOPath -Destination $Destination

The first case was copying to a local datastore on ESX3 (x.x.x.173)

So I fired off the command and the ISO started to copy.

But how does the file get from my laptop to the ESX3? From my laptop directly to the host? Some other way?

In order to check this I chose a file that was reasonably large (~3.0 GB) so I could see the network activity. I opened up the Resource Monitor and sorted the columns to see the network traffic and saw the following.

Traffic from laptop

The network traffic was going from my laptop directly to the vCenter Server – using Powershell of course.

Next – from the vCenter – is had to get to the host of course.. – but how?

Here is the Resource monitor from the vCenter Server

Traffic to & from vCenter - local datastore

On the top right what you see is that the vpxd.exe (vCenter server process) is the one that has the highest amount of network traffic.

In the bottom window you can see on the first line that it is sending out traffic to ESX3 (.173) over port 902 and on the second line it is receiving traffic from my laptop (.187) over port 443. This makes perfect sense.

PowerCLI is communicating with the vCenter – and the vCenter is sending the traffic over to the host. So if we were to look at the original architecture again , the traffic flow will look like this

Architecture with flow

Next I tried the same thing, but this time to the Shared datastore.

Based on what we had before the flow should be:

Laptop –> vCenter –> ESX host.

But the question is though – which ESXi host? The destination parameter to where I want to copy the file is as follows:

vmstore:/<datacenter_name>/<datastore_name> – in my case – vmstore:/UCS/VMGuestDatastore

There is no indication to which host this will go to, there are a number of hosts in the cluster and they are all connected to same datastore. So how does the traffic actually flow in the end?

To answer that we need to look at the Resource Monitor on the vCenter again.

Traffic to & from vCenter - shared datastore

As you will notice – the same traffic flow – but in this case – it changed to ESX2 (.172) and was not going through ESX3 anymore.

I tried the process a number of times – and the result was always the same, it always went to ESX2 (.172)

I could not find the logic behind this.

It did not take the hosts sorted by Name nor according to HostId as you can see below.

Name and HostId

This one has me intrigued – and if anyone has any ideas – what the logic is behind how vCenter chooses which host the traffic will flow through – please do share.