Issues with E-mail Stuck in the Drafts Folder on Exchange 2007/2010

This is one of the cases where you get the feeling that things got messed up, and you simply do not know why or how to even begin to fix them. Story goes like this:
In preparation for an upcoming Exchange Server 2010 course that I'm scheduled to teach next week, I sat at home with my Dell XPS laptop (running with 8 GB RAM and a 258 GB SSD), installed VMware Workstation, and prepared a few base images of Windows Server 2003 SP2 (to be installed with Exchange Server 2003 in preparation for the migration labs), and Windows Server 2008 R2 (to be installed with Exchange Server 2010).

To prepare the images I installed 2 virtual machines, one with each operating system. I then updated each VM, installed some of the stuff I use in order to customize and prepare the machines for my personal preferences, and got ready to clone them.
As you probably know by now, and if you don't, it's about time, Microsoft-based operating systems use SIDs (Security IDs) that are generated as part of the initial setup of Windows. If you have more than one computer with the same SID, this could cause problems, and cloning a computer (either physical or virtual) without re-generating this SID can cause SID duplication.
Anyway, knowing this issue I was careful to run SYSPREP on both VMs before shutting them down and cloning the virtual hard disk files. However, as you'll soon find out, it seems that I have neglected one little aspect of using SYSPREP on Windows Server 2008 R2.
However, at this moment I was not aware of any issues. I was under the impression that SYSPREP did what it was supposed to do - remove the machine's SID and other settings, preparing it for cloning.
So, I cloned the VMs, and created this configuration:

• WIN2008-SRV1 - to be a Windows Server 2008 R2 Domain Controller, DNS server, and an Exchange Server 2010 running the CAS and HUB Transport Roles.
• WIN2008-SRV2 - to be a Windows Server 2008 R2 member server, and an Exchange Server 2010 running the Mailbox Role.
• WIN2008-SRV3 - to be a Windows Server 2008 R2 member server, and an Exchange Server 2010 running the Mailbox Role for my DAG demo.
• WIN2003-SRV4 - to be a Windows Server 2003 R2 SP2 Domain Controller for a second separate domain in a separate forest, DNS server, and an Exchange Server 2003 SP2.
• WIN2008-SRV5 - to be a Windows Server 2008 R2 member server in the second  separate domain in the separate forest, and that would eventually be promoted to become a Domain Controller to that domain, and that should eventually be installed with Exchange Server 2010 running the Mailbox, CAS and HUB Transport Roles for my migration demo.

All was set to go. I installed the right Exchange Server 2010 roles on WIN2008-SRV1 and WIN2008-SRV2, and started to customize the Accepted Domains, E-mail Policies, Send and Receive Connectors and so on.
I opened Outlook Web Access (OWA) or what they call now as "Outlook Web App". All seemed to be working like a charm.
And then it hit me.
I created a simple e-mail message to myself.
The e-mail message got stuck in the Drafts folder.
I did it again, and now I had more messages  stuck in the Drafts folder.

I rebooted the Mailbox role server. I rebooted the CAS/HUB/DC server.
I looked and searched for more information on the Internet. I remembered that in Exchange Server 2007, if you had less than 4GB free space in the partition that held the Exchange database and queue, this could happen. If that is true, one needs to make the disk larger, if possible, or move the Mailbox databases, Queue Database and Queue Log to another empty drive. But in my case, the virtual disks were considerably larger (80GB) and had plenty of free disk space.
I even found this:
An e-mail message is not sent as expected when you send the e-mail message from Outlook Web Access and the Microsoft Dynamics CRM 3.0 client for Outlook is open
http://support.microsoft.com/kb/946653
Not related to my case.
I continued to search on, and finally I got it.
Error: Not able send mail from OWA 2010. When I send mail, the emails go to Drafts Folder.
http://social.technet.microsoft.com/Forums/en/exchange2010/thread/e7d72510-3d86-41ea-ba1a-b742aeb4ccc8
However, I knew that this was not my case, because I know for 100% that I used SYSPREP before cloning MY systems.
And then, like a hammer in the head, I remembered that I forgot to select the “GENERALIZE” option in SYSPREP. Unlike previous versions, it seems that this version will NOT change the SID unless you pick that option.

I immediately used PsGetSID to see if I was correct or not.
PsGetSid
http://technet.microsoft.com/en-us/sysinternals/bb897417.aspx
When I ran it on WIN2008-SRV2 I got this SID:
S-1-5-21-1253556042-3608898735-3328301812

And when I ran it on WIN2008-SRV1 (the DC), I got the same SID:
S-1-5-21-1253556042-3608898735-3328301812

There goes my lab!
Luckily, I still had my base images cloned, so I could scrap everything and re-build, this time, properly running SYSPREP with the GENERALIZE option!

The Essential Guide to Creating and Cloning Virtual Machine Images

As you probably know by now, Microsoft-based operating systems use SIDs (Security IDs) that are generated as part of the initial setup of Windows. If you have more than one computer with the same SID, this could cause problems, and cloning a computer (either physical or virtual) without re-generating this SID can cause SID duplication. Please read the following article if you need to learn more about this issue:


 you can use PsGetSid (by Microsoft Sysinternals) to find out if you're using computers with duplicate SIDs:

Have you performed a rollout, only to discover that your network might suffer from the SID duplication problem? In order to know which systems have to be assigned a new SID (using a SID updater like our own NewSID), you have to know what a computer's machine SID is. Up until now, there's been no way to tell the machine SID without knowing Regedit tricks and exactly where to look in the Registry. PsGetSid makes reading a computer's SID easy, and works across the network so that you can query SIDs remotely. PsGetSid also lets you see the SIDs of user accounts and translate a SID into the name that represents it.

Assumptions

This guide assumes that you’ve got some sort of virtualization infrastructure in place – this could be a VMware product, Microsoft Hyper-V, Citrix XEN and so on. It also assumes that you’ve got some sort of virtualization management tool, and a library where you store all the virtual machine templates and master images. This guide is not product specific.
In addition, it's important that you have a basic knowledge about how to set up and run your virtualization product, that you are knowledgeable about setting up virtual machines, and about the proper procedure to install and configure a Windows-based operating system on these virtual machines.
Lastly, this guide assumes that you’re knowledgeable about the proper procedures needed to be taken prior to creating a virtual machine clone, how to use SYSPREP (the system preparation tool from Microsoft), and how to create proper answer files for the preparation procedure.
Note: SYSPREP is a tool that prepares an installation of Windows for duplication, auditing, and customer delivery.
To download SYSPREP for Windows Server 2003/R2 and Windows XP, please use one of these links:
Download details: System Preparation tool for Windows Server 2003 SP2 Deployment
Download details: System Preparation tool for Windows Server 2003 SP2 Deployment (x64)
Download details: Windows XP Service Pack 3 Deployment Tools
Note that in Windows Server 2008/R2, Windows Vista/7 – the SYSPREP tool is already included in the operating system, therefore there’s no need to download it.
To create the proper answer file under Windows Server 2003/R2 and Windows XP, you need to either manually edit an existing SYSPREP.INF file, or create one for your needs. To create a SYSPREP.INF answer file for Windows Server 2003/R2 and Windows XP you can use the SETUPMGR.EXE tool found inside the Deployment Tools. Use the above links to get the proper version for your needs.
To create the proper answer file under Windows Server 2008/R2, Windows Vista/7, you need to either manually edit an existing answer file, or create one for your needs.  . To create an answer file for Windows Server 2008/R2, Windows Vista/7, you must use the tools available in the Windows Automated Installation Kit (AIK).
Download details: Windows Automated Installation Kit (AIK)

Preparing the System for Cloning

Prior to cloning the virtual machine there are several steps that you must accomplish. For example, some of the preparation tasks should include:
Note that unlike cloning physical machines, since we will be cloning virtual machines, there is not hardware abstraction layer that we need to worry about, no mass storage devices, and no other devices that need to be detected and installed.

• Log on to the computer as an administrator.
• Install and customize applications, such as Microsoft Office, Internet Explorer favorite items, and so on.
• Customize the Default User profile.
• Update Windows and other software components.
• Clean temporary files.
• Defragment the disk, and compact the VHD file.

After performing the initial setup steps on a single system, you need to run the SYSPREP utility to prepare the sample computer for cloning.
As stated above, there is a major difference between cloning systems running Windows Server 2003/XP and those running Windows Server 2008/Vista/7.
Here are some links that will help you get started:
How to Prepare Images for Disk Duplication with SYSPREP
http://technet.microsoft.com/en-us/library/bb457067.aspx
How to use the SYSPREP tool to automate successful deployment of Windows XP
http://support.microsoft.com/kb/302577
Comparing Windows XP and Windows Vista Deployment Technologies
http://technet.microsoft.com/en-us/library/cc765993(WS.10).aspx

Create the SYSPREP.inf Answer File for Windows Server 2003/XP

The SYSPREP.inf answer file is a text file that scripts the answers for a series of graphical user interface dialog boxes.Instead of having t0 manually enter the computer's product ID, accept the license agreement, choose regional settings, enter a password, owner and computer name and so on, you can script everything inside one small text file that will provide the mini-setup wizard that runs after the computer is cloned and rebooted, with the correct answers.

To create a SYSPREP.inf answer file that is used by the SYSPREP tool, you can use a text editor or you can use the Setup Manager tool that is included on the Windows XP CD and is also included with the Microsoft Windows XP Resource Kit. The answer file must be renamed to SYSPREP.inf, and must reside in the SYSPREP folder in the root of the drive where Windows XP is installed, or these files can reside on a floppy disk. If the SYSPREP folder is named differently, the Setup program ignores it. There is no need to specify any parameter for the Mini-Setup Wizard answer file.
After preparing the answer file, run the SYSPREP tool from the C:\SYSPREP folder:

Select the “Use Mini Setup” and then click on “Reseal”:

The computer will shut down. At that  moment, if it’s a physical computer – take out the hard drive and use any cloning mechanism that you may have (i.e. Ghost etc.). If it’s a virtual machine, either use existing virtualization tools such as System Center Virtual Machine Manager (SCVMM), or simply copy the VHD file. When using virtual machines, you will need to create the settings for an X number of cloned virtual machines, and then simply connect them to the copied VHD files.
After starting up each cloned machine, if the answer file has been properly created, you will need to enter the computer name and the entire process will automatically run.

Now, you can verify your computer's SID with PsGetSID.
How to use the Sysprep tool to automate successful deployment of Windows XP
http://support.microsoft.com/kb/302577

Create the UNATTEND.xml Answer File for Windows Server 2008/Vista/7

Unlike previous versions, the unattended Windows Setup answer file in Windows Server 2008/Vista/7, is an XML file typically called Unattend.xml. This is the answer file for Windows Setup that is created by using Windows System Image Manager (Windows SIM). The answer file enables the configuration of default Windows settings, as well as the addition of drivers, software updates, and other applications. The answer file enables OEMs and corporations to customize Windows Setup tasks, for example, specifying disk configuration, changing the default values for Internet Explorer, and installing additional drivers.
Unlike previous versions, the unattended Windows Setup answer file in Windows Server 2008/Vista/7 needs to be specified during the running of SYSPREP. To do so, run the SYSPREP tool with the /unattend:filename option.
If you wish to manually configure the Windows settings after SYSPREP, run SYSPREP from the C:\Windows\System32\sysprep folder:

Make sure you do NOT FORGET to select the “Generalize” option if you need to change the computer's SID. Unlike previous versions, it seems that this version will NOT change the SID unless you pick that option!!! When done, click on “OK”:

Select the “Shutdown” and then click on “OK”:
The computer will shut down. Like in the previous section, if this is a physical computer – take out the hard drive and use any cloning mechanism that you may have. If it’s a virtual machine, either use existing virtualization tools such as System Center Virtual Machine Manager (SCVMM), or simply copy the VHD file. When using virtual machines, you will need to create the settings for an X number of cloned virtual machines, and then simply connect them to the copied VHD files.

After starting up each cloned machine, if no answer file has been created, you will be prompted to configure the computer name and some other settings. Of course, creating an answer file will greatly ease this process, and the entire process will automatically run.

Now, you can verify your computer's SID with PsGetSID and you're done!
Download: PsGetSid

Secret Commands for Emergency Maintenance from the ESXi4 Console

Anyone who has installed VMware's ESXi4 hypervisor


For setting up your ESXi4 server you don't even need to go that far in fact, since you can just point your vSphere Client at the IP assigned to the server by DHCP and do everything afterwards from there. So why should you even bother to attach a monitor and keyboard to your ESXi server?

To be honest, if everything is working as it should then you can happily leave your server humming away in its place and never need to touch it again, but when in IT has everything worked happily ever after? During my day job supporting a multitude of ESXi servers I have encountered several scenarios where the vSphere Client has failed me and more direct access was required. Power cycling the ESXi Server is usually an effective solution but in a production environment it poses additional risks and should only be treated as a last resort, so here I will look at the other options available.
Once you are at the server itself there are basically two interfaces available to you, the "official" ESXi4 console with its system configuration menu, and then the "unsupported" command line Linux console. Yes, according to the official documentation one of the fundamental differences between ESX and ESXi is the latter's lack of a console, but there is a very reduced version hidden in there. Its "unsupported" because VMware doesn't want you to go there unless directed by their support personnel, so bear that in mind and stay away from it unless you are confident you know what you are doing. Now you have been warned let's have a look at the two interfaces:

The (official) ESXi4 Console

Whether you've got a keyboard and mouse plugged in or are fortunate enough to have some sort of KVM over IP solution then the first thing you should see on your ESXi server is a screen like this:

Its most likely you have got this far because your vSphere Client isn't working, so the first thing to do is press F2 to access the System Customization menu. On here you can check the management network settings to make sure you have the correct IP address for the host, and "Test the Management Network" to see if there is connectivity to the rest of your network. Finally select the "Restart Management Agents" option, these are the services which manage the connection from the vSphere Client and control the VMs, so restarting them may enable you to connect.
Should that not work then its tempting to just go for the F12 option to restart the host, but first consider the Virtual Machines running on it, and what will happen to them. ESXi4 is remarkably resilient and often you can find that while your vSphere Client thinks the host and all its VMs are dead to the world they are in fact running fine.
Now, when everything is working properly, issuing a shutdown or restart from the console will tell ESXi to instruct the VMware Tools service on each VM to perform a safe shutdown before it does so to itself. Since you're at this console though its likely everything isn't fine, or maybe you haven't installed VMtools on every VM, which means you can't be sure of a safe shutdown so don't want to press F12 yet. Provided you know which VMs are running on this host you may be able to logon to them via RDP/SSH/etc and issue a shutdown command that way, then once you are confident they have all indeed shutdown then it will be safe to restart the host.

You may be reluctant to issue the restart command yet though, perhaps you aren't sure that all the VMs have properly shut down, or even exactly what VMs are running on that host (if you are running a fully automated vSphere cluster with several ESXi hosts this can happen quite easily!).  In this case there is another option available to you, the "unsupported" console.

The Unsupported ESXi4 Console

A word of warning again, VMware call this "unsupported" with good reason - if you make a mess of your server while in this mode they won't help you out. All the same if you follow instructions and don't get adventurous then you should be pretty safe, and after all this will usually be your last resort when all else has failed.
 Now you have a command prompt on your ESXi server let's have a look at what commands you might find useful:
services.sh restart
Its worth trying this as it will issue a restart command to all the management agents and services on the host, as it processes you should see each one stopping and then being started. However it is effectively the same as selecting the "restart management agents" option from the standard console so if that didn't help you its unlikely to be much different here.
vim-cmd vmsvc
This is the most useful command for emergency maintenance and takes several extensions, first of all you need to run vim-cmd vmsvc/getallvms . This will return a list of the virtual machines registered on the server, and most importantly, their unique ID numbers which you will require for subsequent commands.

As you can see the command returns a list of VMs, with the ID number first, followed by the VM name and other useful information such as the location of the .vmx file and any notes you have appended to the VM properties in the vSphere Client.
Following that you can try vim-cmd vmsvc/power.getstate #### , where #### is the ID number of the VM to query. This will show you the actual power status of the VM in question, in the example below I am querying my "Win2003Test" server, which is suspended:

Forcing a power off of a VM is possible using vim-cmd vmsvc/power.off #### but that rather defeats the object of what we are trying to achieve (avoiding a dirty shutdown of our virtual servers in case you have forgotten!). Instead you could try vim-cmd vmsvc/power.shutdown #### which should issue a shutdown command to the VMware Tools agent on the VM, wait a few minutes and then try another power.getstate to see if it has been successful. Should that not work then you can issue a vim-cmd vmscv/power.suspend #### which will simply suspend your VM to disk so it will not be affected by a restart of the host.
There are several other commands available with vim-cmd vmsvc/ - power.hibernate, power.reset and power.reboot for example. To see a full list of all the commands available just type vim-cmd vmsvc without any extension:

As you can see there are a lot more commands available, which replicate the actions available in the vSphere Client, and as such there is little need for you to use them.

Conclusion

Its obvious from this exercise that despite what the documentation says there is still a lot of functionality left in ESXi's stripped down console, albeit well hidden and not user-friendly. In this article we have covered those commands which could come in useful in some problem situations when you need to get direct control over a host's VMs to avoid a dirty shutdown

How to Backup Fault Tolerant VMs in vSphere 4

A major benefit of deploying VMware's vSphere 4 is the additional options it offers you for business continuity and disaster recovery, such as virtual machine level backups and high availability features. vSphere 4 Essentials Plus, Advanced, and Enterprise editions all include VMware Data Recovery

A more detailed explanation of the potential of vSphere based DR solutions is beyond the scope of this article, but all the products utilise VM snapshots to enable backing up of live VMs without affecting availability:

VMware Data Recovery Snapshot Based Backup
Another major feature in vSphere 4 Advanced and higher editions is Fault Tolerance, which is intended to eliminate VM downtime in the event of a host server failure by creating a live shadow instance of the VM on another host and keeping them in "lockstep" synchronisation. The effect is similar to clustering, but since it operates at the hypervisor level it does not require any special features at the VM software level. Again it is beyond the scope of this article to discuss the advantages and disadvantages of this technology, the important thing to note here is that FT enabled VMs cannot be snapshotted.

The Problem with Backing Up FT enabled VMs

As we have already seen, all the vSphere based VM backup solutions rely on snapshot technology to image live VMs, but Fault Tolerant VMs cannot be snapshotted which therefore precludes backing them up. Unfortunately, it seems that the first time many users discover this is when trying to run their first backup of a FT VM. Depending on their backup application, it will either not allow them to create the backup job in the first place, or the job will fail shortly after starting.
It seems that this situation was exacerbated by conflicting information from VMware when vSphere 4 was originally released, at one point they said that Fault Tolerant VMs would be allowed a single snapshot for backup purposes but this feature was not included in the released version of vSphere 4.0. Subsequently they implied that it would be enabled in a later update, but despite vSphere 4.1 including some major improvements to FT it appeared that they had given up on the single snapshot feature, at least until the next major version release.

The Solution

To be completely accurate, it is still not possible to backup FT enabled VMs. What is now the commonly accepted method is in fact a workaround, as it involves disabling FT in order to allow a snapshot to be created, and then re-enabling it once the backup has completed. This means that for the duration of the backup, the Virtual Machine will no longer have the benefit of FT protection, which may be a problem for some readers. Unfortunately if that is the case, then you will have to look at alternative backup methods, i.e., either running inside the VM or at the SAN level.
It is quite simple to test the process manually in order to establish that your backup application will be able to image an FT VM; you just need to turn off Fault Tolerance for that VM. Note that you have to "turn off" rather than "disable", otherwise it still won't allow a snapshot to be created.



Turning Off Fault Tolerance


In your vSphere Client, right-click on the FT VM and select "Fault Tolerance", then "Turn Off Fault Tolerance" from the sub-menu. The "Disable Fault Tolerance" option will only stop the lockstep synchronisation, but leaves the secondary VM in place. "Turn Off" will actually disable lockstep sync and then remove the secondary VM completely, leaving you with a normal VM that can be snapshotted as usual.
Now start your backup running and if you watch the bottom "Tasks" section in the vSphere Client, you should see the snapshot being created by the backup application. Once the backup has completed, the snapshot should be removed. Once that is done you can right-click the VM again and select the "Turn On Fault Tolerance". This might take a few minutes as it has to create a secondary VM and bring it into lockstep sync with the primary, exactly the same process as when you originally enabled FT.
This process obviously isn't a practical solution for regular scheduled backups. However, it does demonstrate the steps required to allow backups of any Fault Tolerant Virtual Machine, and it will give you an idea of the potential hazards involved. The main problem has already been mentioned; the VM will not be FT protected during the backup process so a host hardware problem would cause downtime, and an error whilst turning FT on or off could leave the VM unprotected, requiring manual intervention. On a more positive note though, in my experience such problems are very rare and are unlikely to cause downtime by themselves, and whilst the VM lacks FT protection, it will still have vSphere High Availability. Therefore, should the host fail, the VM will be started on another host. It will have undergone a "dirty shutdown" so there may be some data loss or even corruption and a short period of downtime, all of which illustrates quite neatly why Fault Tolerance was an attractive option in the first place!

Automating the Procedure

Fortunately vSphere has comprehensive scripting support, allowing for the automation of any process achievable via the vSphere Client GUI, so we can use this to turn Fault Tolerance off and on when required. vSphere scripts are written in Perl, but don't worry if you have no experience of using that - . The instructions below will show you how to implement it. However if you are using a third party application such as Veeam and have an active support agreement with the vendor, then you should contact them first to see if they have their own solution.

1. To use any scripts you first of all need to install the vSphere CLI (Command Line Interface), which you can download from the VMware website - browse to the "Downloads" section and select vSphere4, then click the "Drivers & Tools" tab. Expand the "Automation Tools & SDKs" section then download the version of the vSphere CLI that matches your vCenter installation, note that you want the standard CLI not the "PowerCLI" (the PowerCLI has similar functionality but integrates with the Windows PowerShell):
   
2. Once you have downloaded the CLI, run the file to install it. In theory, you can run the CLI on any Windows system with a network connection to your vCenter Server, but in practice it will be much easier to install it on the same system as your VM backup application. For these instructions I have assumed you will install it to the default recommended folder location, but if you choose a custom folder then just change subsequent file paths appropriately.
3. Now download the FTcli2.pl script file from http://communities.vmware.com/docs/DOC-10279 and save it to the C:\Program Files (x86)\VMware\VMware vSphere CLI\perl\bin folder.
4. If you open the Start - Programs - VMware menu you should see an entry for the "VMware vSphere CLI", with just a Command prompt icon in it. Clicking this will indeed just open a standard command prompt, but with the location changed to the vSphere CLI installation folder. At this point to simplify things in future I would recommend adding the  C:\Program Files (x86)\VMware\VMware vSphere CLI\perl\bin folder to the default folder paths list
5. With the path set, you should now find you can execute your scripts from a standard Command Prompt. Try entering FTcli2.pl /? in order to see the online help listing all the options for this script. You will see that you can specify explicit or passthrough authentication. We will assume you are running it on your vCenter Server with sufficient privileges to use passthrough.
6. Now we need to test running the script with the required options to first turn off FT, and then to turn it back on again. It would be a good idea to create a test FT VM for testing these procedures rather than using a live production VM, just in case. The command to turn off FT should be something like this: ftCLI2.pl --server vcenter.domain.local --passthroughauth --operation stop --vmname MyTestFTvm , where vcenter.domain.local is the FQDN (or IP) of your vCenter Server. Enter that command at the prompt and run it, it should return some progress information, you should see the task appear in the vSphere Client, and the Fault Tolerance will be turned off for that VM.
7. In the event that the script fails to turn off FT, then the output in the Command Prompt window, or the Task Status in the vSphere Client will usually give a good indication of the cause of the problem. You may also add the --verbose option to the command which should make it return more detailed error messages.
8. The command to turn on FT should be identical to the turn off command, except with --operation create instead, so now you should be able make a test VM Fault Tolerant and then remove FT again afterwards.
9. In order to use these new script commands effectively, they need to be coordinated with the backup application. To facilitate this, you should create two batch files; open Notepad and enter your commands, starting each separate command on a new line like this:
10. The cd C:\Program Files (x86)\.... line should not be necessary if you have already added the folder location to your default system paths list, but it won't do any harm to include it anyway. This example is for turning on FT as it contains the --operation create option.
11. Now save the file to a suitable location, making sure you change the "Save as type" to "All files" and include a .bat extension at the end of the file name. This tells Windows that it is a batch file, so the commands in it should be executed:
12. Repeat steps 9-11 but replace create with stop in order to create a batch file to turn off FT for your VM.
13. Now you have your two batch files. In the future, all you should have to do is change the --vmname MyTestFTvm option to match the name of your FT VM as shown in the vSphere Client.

Scheduling your Fault Tolerant VM Backups

The first step for every scheduled FT VM backup needs to be turning off Fault Tolerance, so you can use the Windows Task Scheduler to create a scheduled task to run your batch file at the appropriate time. The Windows Task Scheduler in Windows 2008 Server is quite different to use from the old Windows 2003 Server version

Note that when you create your task, you can specify what Windows user account it should run under. If you are using the passthrough authentication option, it is essential that you specify an account that has sufficient rights on your vCenter Server to change the Fault Tolerance settings for the VM. Configure the task to run at a suitable time and frequency for your backup schedule.
Next you need to create a backup job for your FT VM, just like any other VM backup, but you should schedule it to run at a suitable period after the "Turn Off FT" task to ensure it has time to complete that before starting the backup, otherwise it will fail. Usually I find allowing a delay of 15 minutes is ample, but you should be able to confirm what is best for your system with some testing. Setting the time for reactivating Fault Tolerance is harder because the duration of the backup job may be quite variable from day to day - set it too early and the task will fail, whilst making it too late will leave your VM unprotected for longer than necessary. The best option, which most backup applications support, is to use the option in the backup job properties to run a command after the job has completed:

Conclusion

Although it sounds like a complex and significant set of tasks to be running on a perhaps nightly basis it does in fact usually turn out to be a reliable procedure once setup and the schedules established. The main disadvantage has already been highlighted - the Fault Tolerance protection has to be turned off in order to backup the VM, which increases the risk of downtime during the backup window. Depending on the role of the VM in question this may or may not be an issue, but with some planning it should be possible to minimise the risk. For example, by combining a weekly VM level backup with a daily OS level backup.  

Using a Network Packet Analyzer on a VMware vSphere Virtual Network

Whether you are a server, network, or VMware Admin, a common tool for analyzing network issues is a protocol analyzer (also called packet analyzer or "sniffer"). These software applications analyze network traffic in real-time to allow you to view the packets traversing a network. These tools will tell you what network device is creating the most traffic on the network, what protocols are most being used on the network, who is talking to who on the LAN, and if there are network errors. If packets are being sent in clear-text, you can even decode that text to see things like passwords.

Why you need Promiscuous Mode

Network switches use a forwarding table (CAM table on a Cisco switch) to track what Ethernet devices are on what Ethernet port, and only send traffic destined for those devices out that port. By default, protocol analyzers will only see traffic sent from or to the computer they are running on. Very likely, that isn't going to help you to troubleshoot the network, so the common procedure is to perform "port mirroring" or configure "port spanning" (SPAN or RSPAN). This copies all traffic going to or from a particular port (or group of ports or list of VLANs) to a destination port. Then, you would analyze that port with your protocol analyzer.

Promiscuous Mode on the Virtual Network

But what happens when the network is virtual? Don't worry, this same process can also be performed on a virtual switch, allowing you to see all traffic traversing a virtual switch or vDS. What you would do is to run a protocol analyzer like Wireshark (free edition) inside a virtual machine and then configure the port group where the VM is connected to be in promiscuous mode, like this:

Once promiscuous mode is configured on the vSwitch, that carries down to the port groups in that vSwitch. Now, every port in the VM port group will see the traffic traversing the vSwitch (being sent to and from the VMs on the vSwitch). And suddenly, your Wireshark protocol analyzer will begin to see all traffic from all other VMs, allowing you to analyze the traffic on the vNetwork (as you see below).

Think about it, you are analyzing the virtual network at zero cost after tweaking just one vSphere virtual switch setting and installing your protocol analyzer on a VM connected to that vSwitch.

Reasons to Analyze the Virtual Network

Why would you want to analyze the virtual network? Really, the reasons to analyze the virtual network are typically the same reasons you would analyze the physical network. Here are some reasons I have analyzed the virtual network in the past:

• Identify the VM that is over utilizing network bandwidth, causing slowdowns on the virtual (or physical) network
• Find PCs that are infected with worms or viruses
• Troubleshoot malfunctioning network services (DHCP or DNS maybe) or network applications
• Prove that the network is NOT the cause of a problem
• Sniff the network for malicious or unwanted traffic
• and much more...

Tools & Resources to Help you Analyze the Virtual Network

Many of the same tools you use to analyze the physical network can be used to monitor the virtual network but there are a few additions.

• 
  

What is I/O Virtualization (IOV)?

There are various forms of virtualization with some being much more popular than others. For example:

• Server Virtualization (we all know this one) - consolidating physical servers into virtual server that run on many fewer physical servers
• Desktop Virtualization - virutalizing desktops and running them on servers
• Network Virtualization - creating virtual networks inside the software that don't require any physical network hardware (a must-have for server virtualization)

No matter the type of virtualization, the idea is that you are "decoupling the software from the hardware", making the software hardware independent. There are more benefits to virtualization than I have space to write in this article but, trust me, it's "good stuff".
Instead, let's talk about a topic that has been fascinating me lately and that is I/O Virtualization (or IOV).
What is I/O Virtualization?
Just as you decouple an operating system from the hardware with server virtualization, you decouple network and storage communications from it's typical hardware cable path, network/storage switches, and network/storage adaptors with I/O Virtualization.
In my opinion, understanding IOV can best be described with pictures and math.
Here is how the typical server datacenter "does IO" today:

With traditional I/O, EVERY server has:

• Network - between 1-4+ Ethernet network connections that require individual NICs, Ethernet cables, and switch ports
• SAN - a large majority of servers are redundantly connected to a fibre channel (FC) SAN that requires individual HBAs, FC cables, and FC switch ports

If you have 6 connections per server and 100 servers, you are talking about 600 connections - that's a lot.
The theory with IOV is to take a single cable (or two if you want redundancy) and consolidate all the network and SAN connections onto that single, high-speed cable (does it sound similar to consolidating many smaller servers onto a single high-capacity server?).
Here's what it would look like:

Notice the huge reduction in network and storage cabling. That reduction is going to save you money in network and FC switches as well as time spent managing and troubleshooting all that cabling.
I like it because it just makes things simple and that's how thing should be in a datacenter.
In this graphic, you'll find some additional benefits of IOV:

As the slide points out, physical adapters are now virtual adapters but they work just as the traditional physical adapters would.

Using Storage IO Control (SIOC) in vSphere 4.1

How Storage IO Control (SIOC) Works

Figure below illustrates how Actual Disk Resources are utilized in a previous version of vSphere vs. version 4.1, where Storage I/O Control is employed. Older versions already used what are known as “disk shares”.
In both scenarios shown in the figure, you have two ESX Servers communicating with a Storage Array. One ESX server has two VMs, while the second has only one. Also, in both scenarios, VM A is supposed to have the largest space with 1500 shares, while both VM B and VM C only have 500.

Graphic thanks to VMware.com

In the older version of vSphere, you can see that there is no clear communication between ESX Servers regarding the disk shares found in them. As a result, because VM C is the only VM in one ESX Server, it subsequently ends up occupying a larger space in the Storage Array Queue than VM A, even if VM A has a much larger amount of shares.
As shown in the scenario on the right, Storage IO Control is able to resolve that issue and each VM is given an appropriate allocation of disk resources.
Now that you know what benefit comes with Storage IO Control (SIOC), let us proceed to talk about what you need to prepare in order to use this new feature.

Preparing for SIOC

First of all, you need to have at least version 4.1 of vSphere to use SIOC. Once you have that, note that Datastores will have to be managed by a single vCenter Server. That means SIOC won’t work if you have multiple vCenter Servers talking to the same Datastores. Note also that NFS and RDM are not supported; only FC and iSCSI. Datastores with multiple extents are likewise not supported.
Make sure you have resolved these issues before configuring Storage IO Control.

How to enable SIOC in vSphere 4.1

Now that everything’s set, head on to your vSphere client. In the Inventory section, select Datastores.

Once you’re brought to the next window, look at the list of datastores in the left panel. You’ll want to prioritize datastores that are expected to experience the most contention. In all likelihood, that’s going to be the shared storage datastores; i.e., datastores being shared by multiple virtual machines. In the screenshot below, it’s the IomegaSAN which has two datastores (LUN1 and LUN2).

Next, select a datastore, click its corresponding Configuration tab, and then click the Properties link to bring forward the Properties window as shown below.

You can now enable the Storage I/O Control by simply checking the check box labeled Enabled. Click the Close button when you’re done.

You’ll know you were successful if, in the vShpere client window, you now see Enabled under the Storage I/O Control item of the Datastore Details section.

Do the same thing on your other shared storage datastores.

How to configure the shares and IOPs of the VMs on those datastores

The next step is to configure the shares and IOPs (this refers to the number of I/O per second) of the virtual machines under the datastores which have been enabled for SIOC.
Select a datastore. In the screenshot below, for example, it’s the IomegaSAN-LUN1. Click the corresponding Virtual Machines tab to reveal the VMs on that datastore.

If you scroll to the right, you’ll notice that the Shares Value and Limit - IOPs columns are empty. By default, the Share Value of each virtual machine is 1000, while the IOPs Limit is unlimited.
To configure the shares and IOPs, scroll back to the left and right-click on a VM name. When the context menu appears, select Edit Settings.

When the Virtual Machine Properties window appears, go to the Resources tab, and select Disk. In the corresponding panel on the right, you’ll notice that the settings for a disk are as follows: Shares - Normal, Shares Value - 1000, Limit - IOPs - Unlimited.

If we set the Shares to High (just click and select from the list as shown below), the Shares Value will subsequently change to 2000. Assign what you think is the appropriate amount of shares for that particular VM. This is where you can come back in the future if you want to change the Shares Values and IOPs limits.

Once you’re done, just click the OK button to go back to the Virtual Machines tab
When you’re back at the Virtual Machines tab, you can verify the changes by scrolling again to the right.

In case you haven’t noticed, not only is this datastore serving multiple virtual machines, it is also serving multiple ESX Servers. That’s because those VMs reside in different ESX Servers. This is exactly the kind of scenario illustrated in the first figure on this article.

So Storage IO Control is going to use those Share Values if contention occurs, to prevent a single VM from negatively affecting the performance of other VMs connected to the same datastore.

Where to monitor SIOC performance

You will want to monitor the disk latency and IOPs of your datastore to see if adjustments have to be made. To monitor, just click the Performance tab of the selected datastore. Next, select Performance from the drop-down list beside the View option.

The View should then change to something like this:

Here are samples of the kind of information you’ll be able to see on that page.

Performance Samples:

These graphs will help you determine whether changes have to be made in the shares per virtual machine or the maximum number of IOPs.

Summary

As soon as you have more than one ESX/ESXi host connecting to the same shared storage, the resource controls you have put into place go out the window as a storage I/O bottleneck can easily occur.