Using pyvmomi to set metadata on a VirtualMachine in vSphere

Do you need to set some kind of Key, Value info up on a VirtualMachine in vCenter that you can access from the API? I thought with 5.5 and the addition of this that there was going to be some kind of actual Matadata fields added to the vSphere API, but it seems that still hasnt happened, as this is incomplete and missing any way to actually use it. The good news is that since 4.0 there has been a way to add key value pairs to the vm.config.extraConfig. The key has to be a String, and the Value can be an object. I have used this to store all kinds of stuff from Strings to JSON and XML payloads and even base64 encoded files. I wrote a sample for the pyVmomi-community-samples project that uses the vm.config.extraConfig to store some key value pairs. You can get it here until it gets accepted by the project and merged into master.

Using pyVmomi to set a note on a VirtualMachine

Have you ever noticed the “Notes” section when looking at the VirtualMachine view either from the web client or from the vi client? Its a handy little place to leave various bits of information. I like to use it to put in who built the VirtualMachine and when, and if it had a ticket number that I could tie it to.

vCenter Notes

Right now this VirtualMachine has no note set, and I want to set one using pyVmomi, but how do you do that? Well I made a sample that demonstrates how to do it. Lets walk though how to use this sample. First lets see what the options are:

± |master ✗| → python -h
usage: [-h] -s HOST [-o PORT] -u USER [-p PASSWORD] -j UUID -m

Standard Arguments for talking to vCenter

optional arguments:
  -h, --help            show this help message and exit
  -s HOST, --host HOST  vSphere service to connect to
  -o PORT, --port PORT  Port to connect on
  -u USER, --user USER  User name to use when connecting to host
  -p PASSWORD, --password PASSWORD
                        Password to use when connecting to host
  -j UUID, --uuid UUID  UUID of the VirtualMachine you want to add a note to.
  -m MESSAGE, --message MESSAGE
                        Message to add to the notes field.

Like the last couple of samples I have covered this one takes the same basic options of Host, User, Password, and Port. This sample adds 2 new options though. The first is the -j or –uuid. What is the UUID though and why do you need it? The UUID is a unique identifier given to the VirtualMachine by the vCenter. This makes it a good choice to use for tracking the asset. If you are unsure of where to get the UUID you can use the vminfo_quick script from my previous post. The next option is the -m or –message. This option set the message that will be added to the Notes section. Lets give it a try and see how it works out.

± |master ✗| → python -s -u 'administrator@vsphere.local' -p password -j 421b3c9a-3aef-3662-c476-5b564d045c08 -m "Built by: Michael Rice"
Found: vCloud Director

Which leaves our vCenter with this:

updated vcenter

If you have any questions about this process please feel free to ask me and Ill try to help. Happy note setting!

pyVmomi on FreeBSD

I was looking around online and found someone has added pyVmomi to the FreeBSD ports. Here is how to use it:

root@fbsd-test01-pyvmomi:~ # pkg search pyvmomi

root@fbsd-test01-pyvmomi:~ # pkg install pyvmomi
Updating repository catalogue
New version of pkg detected; it needs to be installed first.
The following 1 packages will be installed:

        Upgrading pkg: 1.2.7_3 -> 1.3.8_2

The installation will require 1 MB more space

2 MB to be downloaded

Proceed with installing packages [y/N]: y
pkg-1.3.8_2.txz                                                                                                  100% 2264KB   2.2MB/s   2.2MB/s   00:01    
Checking integrity... done
[1/1] Upgrading pkg from 1.2.7_3 to 1.3.8_2... done
If you are upgrading from the old package format, first run:

  # pkg2ng
Updating FreeBSD repository catalogue...
pkg: Repo "FreeBSD" upgrade schema 2006 to 2007: Add conflicts and provides
pkg: Repo "FreeBSD" upgrade schema 2007 to 2008: Add FTS index
pkg: Repo "FreeBSD" upgrade schema 2008 to 2009: Optimize indicies
pkg: Repo "FreeBSD" upgrade schema 2009 to 2010: Add legacy digest field
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Updating database digests format: 100%
The following 3 packages will be affected (of 0 checked):

New packages to be INSTALLED:
        py27-requests: 2.3.0
        py27-six: 1.8.0

The process will require 5 MB more space.
727 KB to be downloaded.

Proceed with this action? [y/N]: y
Fetching pyvmomi- 100%  264 KB 270.1k/s    00:01
Fetching py27-requests-2.3.0.txz: 100%  448 KB 459.0k/s    00:01
Fetching py27-six-1.8.0.txz: 100%   15 KB  15.3k/s    00:01
Checking integrity... done (0 conflicting)
[1/3] Installing py27-requests-2.3.0: 100%
[2/3] Installing py27-six-1.8.0: 100%
[3/3] Installing pyvmomi- 100%

This added all the dependencies my system needed, and installed pyVmomi. Pretty simple. Now thats all done and we are ready to start running our pyVmomi code from FreeBSD.

If you arent sure where to start check out one of my other posts! Happy coding!

using pyvmomi to get a list of all virtual machines — fast

Something that often comes up from various people new to using the vSphere API is how to get information very quickly about all the VirtualMachines in the inventory. There is a sample that comes with pyvmomi called which is an obvious place to start to get this info. When its run on an inventory that only has 30 VirtualMachines it seems pretty fast. It only takes about .5 seconds to complete. Try it on a larger inventory like something with 500+ VirtualMachines and it really starts to slow down going from .5 seconds all the way up to over 6 seconds. This number just keeps growing the larger the inventory gets. Once the inventory reaches over 1000 VirtualMachines it can take over 10 seconds for this info to be returned. In other words this solution just doesnt scale, but its often the only way new comers know. The good news is that VMware provides other ways to get this info. The bad news is that its not an obvious solution, and its kind of complicated to use, but thats what I am here for 🙂

Where I work we have over 45,000 vSphere powered Virtual Machines, and its my job as a Sr. Developer there to make sure our code is stream lined, efficient, and scales the way we do. This is why I use property collectors when I need to work with objects from the vSphere inventory. To help new users I provided a sample I call vminfo_quick which as its name implies get info about a VirtualMachine, quickly. To test this lets run the from above on a vCenter with 576 VirtualMachines and time it.

time python -s -u 'administrator@vsphere.local' -p password

real 0m6.300s
user 0m2.476s
sys 0m0.123s

Almost 6 1/2 seconds. Thats not too bad right? Now lets run the vminfo_quick sample I provided against that same vCenter and see how it does. I included a counter and a timer in this sample so we dont have to run time.

python -s -u 'administrator@vsphere.local' -p password

Found 576 VirtualMachines.
Completion time: 0.368282 seconds.

As you can see using a property collector vastly improves performance. I have tested this on an inventory with 1500 VirtualMachines and it still finishes in just under 1 second. I plan to cover details around what the property collector is and how it works in future posts. Stay tuned!

pyvmomi now available in Fedora 19 20 and 21

Back in 2005 I got into making RPMs for Fedora, and by 2006 I became a sponsored packager. It was a hobby I really enjoyed, but back in 2009 or 2010 I just got too busy with work and had to pass the baton. About a month and a half ago I was looking for issues on the pyvmomi project that I could help close when I found this one asking for help making an RPM for Fedora. I got excited all over again about making RPMs so I made one and headed down the path to become sponsored again to package for Fedora. I created this bug report and became re-sponsored, and after a lengthy process I am happy to report that pyvmomi is now available in Fedora 19, 20, and 21. It is still in the testing phases for EPEL 6 and 7 but in another week or so should be available there as well. This puts us one step closer to being able to port the current OpenStack driver for vSphere to pyvmomi!

Monitor the vSphere API with PERL using nagios or icinga

I spend all day every day doing development against the vSphere API. It is crucial that we know when the API is down. If this is something you need to know about you have many choices out there to do your monitoring. Some names that come to mind for me are Nagios and Icinga. In the past icinga was something I used to take care of this task. In this post I will discuss how you can do the same thing and I will provide the PERL code required to take care of this task.

First it is important to note I will not be discussing how to setup nagios or icinga, that is beyond the scope of this post, so make sure you already have it setup and are familiar with how to do configuration tasks with it. With that out of the way lets get started.

Monitoring the API is a pretty simple task. I have found that all it takes is something as basic as logging in and checking the time the vSphere API returns. If you are unable to preform this basic task there is no chance any of the advanced features in the API will work. To get this task done you really only need to do 3 steps.

  1. Login
  2. Obtain a ServiceInstance
  3. Call the CurrentTime method on the ServiceInstance

Thats it. If you can do this your API is up and usable.

VMWare provides the vi-perl tool kit which makes the code pretty simple to write, and the code I provide at the end of this article depends on it to function. Now you just need to configure Nagios or Icinga to run that code at what ever time interval you deem fit. For me I like to know right away so I always had it run every 3 mins and alert me on any failure. As promised here is a link to the code.

How to install Python 2.7 on CentOS 6.x

I needed to install python 2.7 on a CentOS server. I did some searching and found some very broken scripts on github. I picked one of them and started hacking til I got it working. You can now find it here: centos_python_env_setup To use it, you can simply grab the raw version using wget, set the script executable with chmod, then as root run it. I tested this script about 25 times using my Rackspace Cloud server and picking the CentOS 6.4 option. Please let me know if you have any issues running it.

Simple Tomcat Hosting using Tomcat 7, Java 7, and httpd with mod_proxy_ajp

Ive been spending a lot of time in the last year and a half dealing with Apache Tomcat. The DevOps team I am on focuses on Grails for almost all development needs. We use Tomcat 6 and 7 to host a variety of applications we have written. As such I have learned a lot about Tomcat, and Im going to use this opportunity to share some of the things I have learned with you.

The server setup

I like using CentOS on my personal stuff, so for this example Ill be using CentOS 6.3 and using my Rackspace Cloud Server. Im just using the default image provided by Rackspace. From the cloud server control panel I name my server tomcat-dev1. Next I selected CentOS 6.3 for the image and 512M of RAM for the server size. Finally in the networks section I leave it defaulted to a public and private adapter, and then press the create server button to begin building my new cloud server. Keep in mind this is just a development server for me so I dont need much RAM, for a production system you will need to calculate your needs accordingly and make your server as large as you need it. Once my server is done building I will log on and install:

  • httpd
  • jdk 7 from Oracle
  • Tomcat 7
  • git (optional, but I like to keep my config files in version control)
  • jsvc (this is bundled with tomcat)

First steps

Lets begin by logging into our new cloud server as root. The first thing I do to a new cloud server is change the password. Once that is done we can begin by updating the system.
yum -y update
This is done because the base image could be several months old so we want to get the latest updates from the vendor before we begin. Once that finishes reboot if you need, if you do not know if you need a reboot or not just do it anyway to be sure.

Gathering the goods

Some of what we will be installing is either not provided by yum, or is older and we want flashy and new. We need to fetch the jdk; I will download the x64 rpm to my local desktop then upload it to my cloud server using winscp. Next we need Tomcat, and jsvc. Since these two things do not require me agreeing to some terms I will download them directly from the server.
Now on my server I should have the jdk rpm, and the Tomcat 7, and jsvc packages.(jsvc is bundled with tomcat)

Installing the jdk, Tomcat, and jsvc

We need to install the jdk first. To do that lets install the rpm:
rpm -Uvh jdk-7u10-linux-x64.rpm
If you see output like this:

Error: Could not open input file: /usr/java/jdk1.7.0_10/jre/lib/rt.pack
Error: Could not open input file: /usr/java/jdk1.7.0_10/jre/lib/jsse.pack
Error: Could not open input file: /usr/java/jdk1.7.0_10/jre/lib/charsets.pack
Error: Could not open input file: /usr/java/jdk1.7.0_10/lib/tools.pack
Error: Could not open input file: /usr/java/jdk1.7.0_10/jre/lib/ext/localedata.pack

It is safe to ignore. I didnt dig into why this output happened but it did not seem to affect my install. Now that Java is installed we need to add the java install to the alternatives system, and make it so that the JAVA_HOME environment variable is set for all our users when they log in. Lets do that by creating a file in /etc/profile.d called Using your favorite text editor create the file /etc/profile.d/ and enter the following lines into the file and then save it:

export JAVA_HOME=/usr/java/default
export JRE_HOME=/usr/java/default/jre
export JAVA_OPTS=””
export JAVA_CLASSPATH=/usr/share/java/*


Now the file needs to be set executable so issue the following command:

chmod +x /etc/profile.d/

Now to add java to the alternatives system

alternatives --install /usr/bin/java java /usr/java/latest/jre/bin/java 20000
alternatives --install /usr/bin/jar jar /usr/java/latest/bin/jar 20000
alternatives --install /usr/bin/javac javac /usr/java/latest/bin/javac 20000

Lets test that this works by typing:


Right now we should get no output. Now try logging off our server, and logging back in. Once you do that type the command again. We should now see:


Now we need to unpack Tomcat, and the jsvc daemon.

tar xzvf apache-tomcat-7.0.34.tar.gz -C /opt/

This will unpack Tomcat into the /opt directory into a folder named apache-tomcat-7.0.34 Once that is done unpack the commons-daemon source code. That code is bundled with tomcat:

cd /opt/apache-tomcat-7.0.34/bin/
tar xzvf commons-daemon-native.tar.gz -C /usr/local/src/

The source code for the jsvc daemon is now in /usr/local/src/commons-daemon-1.0.10-native-src and needs to be built. Before we can build it we still need to install a couple of things on our server. According to the docs we need a compiler, and make. In an effort to make it simple I will be using yum and the groupinstall option to install “Development tools” This will provide git, make, gcc, cpp, and many other tools.

yum -y groupinstall "Development tools"

Once that is complete we have what we need to build jsvc, so lets do that now. Since we are using CentOS we will want to:

cd /usr/local/src/commons-daemon-1.0.10-native-src/unix

From here we can issue the configure command:

./configure --with-java=/usr/java/default

This should render some output, the last thing it says if you were successful is this:

*** All done ***
Now you can issue “make”

Lets do that now.


If you got no errors you should now have a binary file in the same directory as you called jsvc. Since this Makefile does not have a target for install the next natural step of make install would fail. You have to manually place the binary somewhere on your system. I like to just put it with the tomcat binaries since thats the only thing I use jsvc for, so thats what Ill do now:

cp jsvc /opt/apache-tomcat-7.0.34/bin/

You should now have Java 7, Tomcat 7, and jsvc 1.0.10 installed on your system. We are getting close to being done. Only a few more steps left!! 🙂

Doing some Tomcat configuration, and making Tomcat start at boot

Now that we have all these things installed we need to do some configuration, and also make it so Tomcat will start when the system boots up. Lets begin with making tomcat start at boot. Im going to provide you with the init script I wrote and use on my systems. You can get the latest version from here. Download it and save the file in /etc/init.d as tomcat

wget -O /etc/init.d/tomcat

This file needs to be executable so lets issue the following command:

chmod +x /etc/init.d/tomcat

Now we need to make sure tomcat will start at boot:

chkconfig --add tomcat && chkconfig tomcat on

This will add tomcat to the chkconfig system, and enable it for start up on boot. Lets take a look at what this file does for us. On line 4 we see this:

# chkconfig: – 85 15

This is needed for chkconfig so our command above would work. For more information see man chkconfig. Scroll down in the file until you see:


This is where the user that tomcat will run as is defined. By default it wants to run as the user named tomcat. We have not created that user yet, but we will shortly. The next two things to notice are right below the TOMCAT_USER line.


These locations on the file system are not there by default so we need to add them, and make sure our tomcat user has access to them, but lets keep looking through this file for now and we will fix all the things at once. Lets scroll down to lines 20 and 21

[ -f /etc/profile.d/ ] && . /etc/profile.d/
[ -f /etc/profile.d/ ] && . /etc/profile.d/

This is checking to see if these files exist and if so source them, this will set up our environment variables for us. The tomcat file is missing still, but we will add it soon. I think one of the last things to note here in this file is the JSVC_BIN variable. If you put jsvc in some other location than where I put it then you need to adjust this to your location. Finally on line 62 we have the command we use to start tomcat. You should not need to adjust this, but if you need to thats where to do it. Now that we know what this file will do for us, lets fix all the things we found while going through it.

Finializing the Tomcat bits

The first problem we found above was our script is looking for a “tomcat” user and we dont have one yet, so lets add one:

useradd -r tomcat -m

This added a new system user named tomcat, and created a group also named tomcat. A home dir was created in /home/tomcat The next problem we found was that a couple of files need to be created, and our tomcat user needs access to them. Lets make it happen:

mkdir -p /var/lock/subsys/tomcat/
mkdir -p /var/run/tomcat/
chown -R tomcat. /var/run/tomcat
chown -R tomcat. /var/lock/subsys/tomcat

Finally the file needs to be created and make executable. Using your favorite text editor create the file /etc/profile.d/ and add the following to it, then save it:

export CATALINA_HOME=/opt/tomcat
export CATALINA_TMPDIR=/opt/tomcat
export CATALINA_OPTS=”-XX:MaxPermSize=256m”

Now we need to make it executable:

chmod +x /etc/profile.d/

You may have noticed that the location of CATALINA_HOME and CATALINA_TMPDIR dont exist on our server, good catch. I use symlinks to this. Lets get them created.

cd /opt
ln -s /opt/apache-tomcat-7.0.34/ /opt/tomcat

I do this so if I update tomcat I only have to update this symlink and do not have to edit my init script, or my profile.d script. Now all we have left for tomcat is to make sure we have the server.xml file created correctly, and if we want to allow access to the manager we need to make a tomcat-users.xml file as well. I have provided the server.xml and the tomcat-users.xml file I used here. Those files need to be saved in /opt/tomcat/conf/ as server.xml and tomcat-users.xml Once you have done that you should be able to start tomcat, so lets try that now.

service tomcat start

Lets verify its running:

ss -nap

We should see some output like the following:

[root@tomcat-demo unix]# ss -nap
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 100 ::ffff: :::* users:((“jsvc”,20103,39))
LISTEN 0 100 ::ffff: :::* users:((“jsvc”,20103,38))

This shows us that jsvc has a process listening on port 8009 and 8080, on the local loop back interface only, just like we setup in our config file. This is wonderful now we are done with tomcat for now. Lets move on to httpd and setting up the ajp.

Apache httpd and mod_proxy_ajp

We need to install httpd now.

yum -y install httpd && chkconfig httpd on

We do not need to do anything to get mod_proxy_ajp it is part of the base httpd install as of 2.2. Now that our web server is installed lets configure our first vhost, and serve up some tomcat content. Using your favorite text editor open the httpd.conf file located at /etc/httpd/conf/httpd.conf The vhost section is at the bottom of the file by default. In my case I added the following:

NameVirtualHost *:80
<VirtualHost *:80>
ProxyRequests Off
<Proxy *>
Order deny,allow
Allow from all
ProxyPass / ajp://localhost:8009/

Next I need to open port 80 on my firewall, since the default firewall will be blocking traffic on port 80. Using my text editor I will edit /etc/sysconfig/iptables so it looks like so:

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -m tcp -p tcp –dport 80 -j ACCEPT
-A INPUT -m state –state NEW -m tcp -p tcp –dport 22 -j ACCEPT
-A INPUT -j REJECT –reject-with icmp-host-prohibited
-A FORWARD -j REJECT –reject-with icmp-host-prohibited

Next I need to load this new rule set like so:

iptables-restore < /etc/sysconfig/iptables

Now I am done, and Im ready to fire up httpd and visit my site.

service httpd start

You can now visit your site in a browser and see the default Tomcat landing page, as well as add a /manager and log in using "tomcat" as the user, and "s3cret" as your password. In the coming weeks I will be adding more info about how to do vhosting using this method, and eventually covering how to do this all in a few mins using chef

Switching Falconstor IPStor to use 1 to 1 mapping for LUN assignment

Switching your Falconstor IPStor server to use 1 to 1 mapping from any of the other avaliable mapping options Like All to All or All to 1 can be an annoying task, but if you have VMWare ESX or ESXi and have VMotion you can do this with 0 down time and Ill show you how. These steps will assume you are NOT in a fail over cluster with Falconstor

In my example I am using 2 ESX 4.1 hosts. Monster01 and Monster02. Each one of my hosts is designed to support running all our virtual machines from so moving all the vms to a single host like this is not going to have a negative impact on any vm and users will never know this is going on.

First Im going to need to vmotion all the machines to a single host. I move all the virtual machines running on Monster01 to the host Monster02. Once this is done I put Monster01 in maintence mode, then shut it down. This step is only needed if you are SAN booting your ESX/i host and need to change the LUN the ESX/i OS is running on.

Once its shut down comes the hard part if no one ever documented what WWPN is used for what. If that is the case for you like it was for me then you can figure this out by looking at your physical adapters and finding all the adapters that are in target mode. Write down the WWPN of each one. You might have a bunch.. Thankfully I only had 2 adapters in terget mode that were online. Yay!! a 50/50 chance of getting it right on the first try! I took LUN 0 which has my ESX 4.1 install on it and did a 1 to 1 map. On the ESX host this was simple since we only had 1 adapter plugged in and configred. Next I had to take a shot in the dark because our cables are a mess and no one documented what WWPN was used for what. I got it wrong the first time because once I assigned the LUN and turned the host back on the box failed
to boot. I switched to my other choice and Bam it booted!

Next to quickly switch the other 40 LUNS I went to the Falconstor management console and went into the SAN Clients and selected Monster01 I right clicked on each LUN and selected properties. From here you can switch from your current mapping to 1 to 1 using a select box. You will get a warning about data transmission stopping when you do this. That is fine since there are no running virtual machines on Monster01 (remember its still in maint mode) Next you select the 2 WWPNs needed for the initiator (the ESX host) and the target (the Falconstor server). Once you have completed this for all LUNs you can bring the ESX host out of maint mode and VMotion the machines from Monster02 to Monster01 and then repeat this process on Monster02.

How to Setup a Load Balancer with Rackspace Cloud Load Balancers On-Demand

I was sitting around tonight and had an idea. I wanted to know.. How hard is it to setup a load balancer using Rackspace Clouds new offering? I logged in to the Cloud Manager interface. On the left side of the screen I select Hosting to expose the hosting options. I clicked on Load Balancers. Next I clicked on Add Load Balancer. That brings you to the configuration page. This is where you begin by giving a name to the instance. Next choose the Protocol (http, https, smtp, custom, etc..). Then you select the Virtual IP type, there is clear concise documentation that explains the various options. Next you choose the algorithm that the balancer will use. In the next section you select the region for your balancer. I selected DFW because I have several servers (nodes) in San Antonio, and several in various data centers in the DFW area. Finally you add your nodes. The awesome part is that your nodes do not have to be Rackspace servers, be that virtual or physical. You can use physical servers or virtual, and they can be virtually anywhere.. For my test I added a node that was in the Slicehost DFW data center, next I added a node from Linode, also in their DFW data center. Finally I added a node from Serverbeach, a physical server that I lease in their San Antonio data center. Thats it. Your load balancer is setup. Assuming you selected the same options as I did: connection type: http, VIP type: public, algorithm: random, and that you wanted to set this up to load balance all you need to do is go setup an A record in DNS for to point at your VIP (provided to you by Rackspace). Next go to each of your nodes and configure apache with as a vhost. Once that is complete you can visit in your web browser and you are now load balancing! It so simple and only takes a few mins to setup.