My ramblings about all things technical

1 Comment

vCAC System Exception Error

A colleague of mine at Xtravirt, Richard Renardson was experiencing an ambiguous “System Exception” error on the requests page in the vCAC 6.0.1 portal at a very high profile customer. Upon checking the server side log it was showing an error stating that “cat_request”  does not exist. We tried a few things and looked through quite a few VMware KB articles but were unable to find anything that matched our problem.

After some troubleshooting we were able to determine what the problem was and a fix so he graciously allowed me to blog it to hopefully save someone else the time especially with vCAC becoming so popular recently. The problem seems to happen when an external database has been configured and the hstore extension is missing/has not been created and this extension is required by vCAC for the creation of tables. What we had to do in the end is to create the hstore extension in the vCAC database. The steps we followed to create the hstore extension was to :

  1. Log in to the external DB using the pgAdmin tool.
  2. Within the pgAdmin console we had to run this SQL statement to connect to the vCAC DB:
    1. \connect "YOURDBNAME";
  3. Within the pgAdmin tool we needed to create the hstore extension by running the following statement:
    1. create extension hstore;
  4. Now you just need to restart the vCAC Appliance.
  5. After a bit of a wait for it to restart fully the problem was fixed Smile

Hopefully this saves someone the time we spent on the problem


Leave a comment

vCloud endpoint wont delete from vCAC 5.2

I am currently working on a project that is using vCAC 5.2 ,vFabric Application Director 5.2 and vCloud 5.1 to provide automated self service provisioning of resources for customers (super learning experience).

Whilst going through the manual steps of removing a test customer from the solution before automating the steps through VCO, I hit a very strange problem where at the point of deleting the endpoint to the vCloud Organisation that was assigned for the test customer I got an error stating “ Error has been caught,see event logs located on the vCAC server for detail” and as shown below.


If I went to the the logs within vCAC there were two errors linked to the problem. the main one stating “….  Inner Exception: the DELETE statement conflicted with the REFERENCE constraint “HostNic….”


The error is showing that a computer resource is still attached to the endpoint even though I had removed the computer resource from the vCloud Enterprise Group computer resources selection, removed the Org VDC from vCloud and run a manual data collection. It seems that there is a bug at present that doesn’t allow the removal via the UI (bug report already opened before someone asks) so what you need to do is (I make no promises or guarantees around this script so use at own discretion and backup your DB before running this):

  1. Go to the SQL server that hosts the vCAC database.
  2. Open SQL Management Studio as a user with sufficient permissions.
  3. Select the vCAC database and click the New Query button at the top left.
  4. Ensure the vCAC database is selected.
  5. Paste the following SQL script in the query box and change the ‘ORG VDC NAME’ to the name of the Organisation VDC that the endpoint was connected to and execute the query.

DECLARE @HostId uniqueidentifier

SET @HostId= (SELECT HostId FROM Host WHERE HostName = ‘ORG VDC NAME’)

DELETE FROM VirtualMachine WHERE HostID = @HostId

DELETE FROM HostNicToReservation WHERE HostNicID IN (SELECT HostNicID FROM HostNic WHERE HostID = @HostId)

DELETE FROM HostReservation WHERE HostID = @HostId


DELETE FROM HostToStorage WHERE HostID = @HostId

DELETE FROM AdminGroupToHost WHERE HostID = @HostId

DELETE FROM ResourcePool WHERE HostID = @HostId

DELETE FROM Host WHERE HostUniqueID = (SELECT HostUniqueID FROM Host WHERE HostID = @HostId) AND ClusterHostID = @HostId


  1. The results should show that some values have been changed.
  2. Now you can remove the endpoint from vCAC and the computer resource won’t show up for selection under the vCloud Enterprise Group either.

I hope this saves someone the time I spent trying to fix the problem.


    Leave a comment

    VCDX Spotlight: Garrette Grouwstra

    Name: Garrette Grouwstra

    Twitter Handle: @VirtualCanadian

    Blog URL:

    Current Employer: Long View Systems

    VCDX #: 127

    How did you get into using VMware?

    I was working for a small ISP in 2009, and began a consolidation project to go hand-in-hand with their upcoming infrastructure refresh. As soon as I started using ESXi 3.5, I thought that the whole idea of virtualization was something I needed to focus on professionally.

    What made you decide to do the VCDX?

    For the past 2.5 years, I have had the privilege of building and architecting a public cloud offering to help diversify the services that the company I work for offers to clients. VCDX-DCV had been on my mind at the time as I watched others in the organization (@vcloudmatt and @DavesRant) go through the process. Once the Cloud track was announced, I knew that would be my next goal.

    How long did it take you to complete the whole VCDX journey?

    That is a tough question, as the VCDX-Cloud certification is still brand spanking new. I had to complete all of the prerequisites first, I received the VCP-Cloud in August of 2012, followed by the Beta’s of the VCAP-CID, and VCAP-CIA, which I received in February 2013.

    I started writing my design documentation in March of 2013, putting it aside while I planned a wedding and went on a honeymoon, and picked back up in August 2013. With the help of my design Co-author, Matt Vandenbeld (@vcloudmatt), I was able to complete the application by the end of December, and was fortunately invited to defend in February 2014.

    What advice would you give to people thinking of pursuing the VCDX accreditation?

    1. Get support from your family, and work. I was lucky that both my husband and Long View Systems gave me time, and supported me on this journey. I could not have remained sane without them.

    2. Set milestones to complete sections of your design doc, and within all supporting material. Keep to the deadlines.

    3. Don’t work in a vacuum. There are many great resources out there, both blogs and people (Inside and outside of your organization). Use them

    4. Know your design inside and out. Really, I mean it.

    5. Accept that you may have submitted a design with mistakes. Own those mistakes and call them

    6. Practice, practice, practice! Find others in the community that are able to assist you, and do mock defences, design and troubleshooting scenarios. Brad Christian (@BChristian21) organized mock defences between most of the candidates defending at PEX. The group ripped my design apart more so than they did in the room, and were amazing help preparing.

    7. Breathe. Take a time out before the actual defence to compose yourself.

    If you could do the whole VCDX journey again what would you do differently?

    The only thing I would do differently is more mocks. I saw myself grow so much as a consultant and VCDX candidate during the weeks leading up to the defence, and the mocks were a LARGE part of that. I wish I would have done more troubleshooting and design mocks.

    Life after the VCDX?  How did your company respond?  Was it worth it?

    The VCDX is a certification I achieved for myself, not for my company. They have been supportive along the way, and all congratulated me, however I did not, and do not expect much to change from a company perspective.

    The biggest change personally, is that I have free time again. I’m still trying to come to terms with that and debating how to fill the void that VCDX prep has left.


    vCAC 6.0 Resources

    As most people who work with VMware will know vCloud Automation Center version 6 was released last week and with the plans for vCAC to become the route for non VSPP partners for cloud plus the added features and functionality of 6.0 the twitterverse is alive with people looking to learn more. I am one of those people and was even fortunate enough to be part of the beta for 6.o but there is loads to learn so I decided to start a list of really great resources around the product that I could use to skill up with and thought it would probably help other people as well and so this is what this blog posting is about. I am hoping to continually add more resources to this list and welcome any recommendations on good resources.

    Product Landing Page:

    Product Documentation:

    Release Notes:

    Compatibility Matrix:

    Installation and configuration blogs:

    Distributed Installation

    Sam McGeown has done a series around the distributed installation of vCAC 6.0 which isn’t as simple as you would hope.

    1. VCAC 6.0 build-out to distributed model – Part 1: Certificates
    2. vCAC 6.0 build-out to distributed model – Part 2: vPostgres
    3. vCAC 6.0 build-out to distributed model – Part 3.1: Configure Load Balancing with vCNS
    4. vCAC 6.0 build-out to distributed model – Part 3.2: Configure load balancing with NSX
    5. vCAC 6.0 build-out to distributed model – Part 4: Deploying and clustering a secondary vCAC Appliance


    Automation of the Installation

    Day 2 Automation

    Jonathan Medd as created a brilliant series of postings around vCAC tenant creation

    Automating vCAC Tenant Creation with vCO: Part 1 AD SSL
    Automating vCAC Tenant Creation with vCO: Part 2 AD Users, Groups and OUs
    Automating vCAC Tenant Creation with vCO: Part 3 Install the vCAC plugin for vCO
    Automating vCAC Tenant Creation with vCO: Part 4 Creating a Tenant
    Automating vCAC Tenant Creation with vCO: Part 5 Creating an Identity Store
    Automating vCAC Tenant Creation with vCO: Part 6 Adding Administrators


    Top orchestration blogs

    Orchestrator Videos

    Intro to vCenter Orchestrator

    vCenter Orchestrator Install and Configure

    Using vCO to manage Active Directory and Exchange

    Advanced vCO

    Converting vCO Workflows to versionable code

    FREE VMware vCenter Orchestrator Instructional Videos [Updated]

    VMware Hands On Labs


    VMwareTV have now posted 30 videos covering vCAC features

    vCAC 6.0 Extensibility Overview

    #vBrownBag Automate ALL the things – vCAC 6.0 Installation with Jon Harris (@ThevCACGuy) from ProfessionalVMware on Vimeo.

    #vBrownBag Automate ALL the things – vCAC 6.0 Getting Started with Jon Harris (@ThevCACGuy) from ProfessionalVMware on Vimeo.

    Reference Architecture Guide:

    Helpful vCAC 5.2 Resources

    Just because some people will still be asked to do vCAC 5.2 . Also the installation notes only cover installing all the components on one server whereas Arnim has detailed how to do it when you have separated the components like you would in a production environment. I’ve used these notes on a 5.2 engagement and assure you these are correct compared to the actual VMware instillation notes

    Leave a comment

    VCAP-CID Objective 2.1 – Determine Catalog Requirements for a Logical Design


    Identify what can be included in a published catalog.

    • A published catalog is one that is created in the administrative organisation with all the required components and  vApp templates published to all other organisations in the vCloud environment. Good design practice to only allow the administrative organisation to publish its catalog and deny this ability for all the standard organisations.
    • The components that can be included in a published catalog are:
      • Standardised gold master vApp that can consist of a single virtual machine all the way to 3 tiered offerings like a web service with a web front end, an application server and a database server. These are verified templates that meet regulatory and security standards which ensures consistency across the environment and provides the consumers with verified offerings that can be deployed with ease.Guest customisation changes the identity of the vApp and can be used for post-deployment steps, such as the joining of vApps to domains.
      • vApp Templates which can cannot be deployed but can be deployed (instantiated), creating a vApp that can be deployed and powered on.
      • Media like ISO files for software and applications. These are also verified and commonly customised to ensure standardisation and to provide specific capabilities.

    Identify what can be included in a private catalog.

    • A private catalog can have the exact same components but it is controlled by the user/group assigned the Catalog Author vCloud role. This catalog is limited to a specific organisation and good design practice states you should limit the ability to publish this catalog thereby making it a private catalog.
    • This can still contain standardised vApp’s and ISO’s and if you are a service provider this is where the cloud consumer will place their standardised vApp’s and ISO’s so that the organisation can use them but other organisations cannot.

    Identify permission controls for catalogs.

    • There are three Predefined roles in vCloud that have varying permissions and rights to make changes and create components in catalogs. A breakdown of the predefined roles and their rights are contained in this documentation centre link

    Explain the functionality of a catalog.

      • This should be straight forward as this is VCP-IaaS level and I think all the previous sections define it pretty well also. But just in case i have pasted the VMware definition below:
        • VMware vCloud Director uses the concept of a catalog for storing content. Organizations have their own catalog that they can populate and and share the contents with other organizations and users.

    All entities in the catalog are stored in a content repository system. The content repository, a component in the vCloud Director storage subsystem, provides an abstraction to the underlying datastores while offering features to store, search, retrieve, and remove both structured and unstructured data.

    Skills and Abilities

    Based on application requirements, determine appropriate vApp configuration.

    • As I mentioned for the published catalog and private catalog sections above you can configure vApp’s with multiple tiers to allow the organisations to provision these offerings in their vCloud organisation and maintain standardisation. If a customer asks for a web service offerings then you can provide them with a three tiered vApp with a web front end, an application server and a database server. There may even be a requirement for availability of the offering so you will created multiple front end, application servers and a clustered database back end.
    • Using the web service example this will also require different networking to ensure the security of the offering which will mean different servers connecting to different networks and vCNS endpoint devices being configured as part of the vApp. I am planning on creating a few of these as practice in visio so that I can visualise them and make sure I know what they should look like in case a visio style question comes up or i just need a good mental picture to make decisions for questions.

    Determine appropriate storage configuration for a given vApp.

    • This follows closely to what I covered above but now you need to think of the storage offering the vApp components are going to be kept on and what storage you are going to allow the vApp to be deployed onto. Using my trusty web service example you wouldn’t want the database sitting on low end storage as this would severely impact the service.
    • This is what I think they are asking for so if you think i’m wrong then please do tell me as I’m also learning and sometimes it’s difficult to gleam what they mean as this could also relate to fast provisioning.

    Given customer requirements, determine appropriate catalog design.

    • I think for this if you have created catalogs countless times and know what you can put in there and that they can be published to specific organisations from other organisations or published to all from the administrative organisation then designing it should be simple enough.

    Determine the impact of given security requirements, on a catalog structure.

    • This may be numerous things but there are times when an organisation wants only certain vApp’s and ISO’s in a catalog to be available to certain people and so you can configure the catalog to have certain portions only available to certain people.
    • There are also many organisations who have very customised and important virtual machines which they have converted to vApp templates and they want these secured so that only a certain person can access them and only that person can provision them for people.

    If you think I have totally missed something then please do tell me as I’m only learning and I’m certainly not perfect.


    Leave a comment

    VCDX Defences Dates For 2014 Announced


    Spongebob-Happy-spongebob-squarepants-154897_338_432As some people who know me may be aware I pushed for a UK VCDX defence this year as I along with numerous other people were looking to submit for the VCDX but couldn’t motivate and/or afford the costs to fly to the other defences released. Mark Brunstad was very helpful and put out a feedback form for people to submit their interest in a UK defence and sadly we fell a few people short of making it worth the expense and time for VMware. In all honesty it wasn’t a bad thing for me as with a very busy (but super exciting) work schedule and getting used to trying to study with a child under one in my house I wouldn’t have made a defence anyways.

    But at VMworld Europe this year I rekindled the idea and Mark and John Arrasjid really liked it and again voting was opened. I knew for a fact there were at least 5-6 people who were interested to submit for it from the UK and numerous more in western Europe. Now I’m certainly not saying I had anything to do with it but it seems there were so many people interested that there are not one but TWO defences going to happen in Frimley UK next year. I’m hoping I wont need a second defence date but it is really great to have the option. The other dates are below which were released by Mark on the VMware Communities here.

    February 10-14 2014: VMware Partner Exchange (PEX) – San Francisco CA US

    VCDX Candidates wishing to defend at PEX 2014 may register by following this link:

    Applications for the PEX San Francisco 2014 VCDX Design Defense will be due on or before Midnight PST (UTC -8:00)  Friday December 20,  2013

    April 2014:

    April 7-9 2014 Frimley UK

    April 7-9 2014 Sydney AU

    Applications for April 2014 Frimley and Sydney VCDX Design Defense will be due on or before Midnight PDT (UTC -8:00) Friday 02 February 2014

    July 2014:

    July 7-9 2014 Cambridge MA US

    July 7-9 2014 Frankfurt DE

    July 7-9 2014 Singapore SG

    Applications for July 2014 Cambridge, Frankfurt, and Singapore VCDX Design Defense will be due on or before Midnight PDT (UTC -8:00) Friday 09 May 2014

    October 2014:

    October 6-8 2014 Palo Alto CA USA                                   

    October 6-8 2014 Frimley UK

    October 6-8 2014 Kuala Lumpur MY

    Applications for October 2014 Palo Alto, Frimley, and Kuala Lumpur VCDX Design Defense will be due on or before Midnight PDT (UTC -8:00) Friday 08 August 2014.

    Good luck to all of those looking to submit next year and hopefully I’ll be posting this post next year this time as a VCDX Open-mouthed smile


    Leave a comment

    VCAP-CID Objective 1.3 – Determine Capacity Requirements for a Conceptual Design


    Skills and Abilities

    Determine how storage and network topologies affect capacity requirements for a vCloud conceptual design.

    • This in my opinion can be taken in a few ways so I welcome any feedback on if you think I have looked at this the wrong way but the way I am looking at this is the way it is all connected to differing portions of the environment obviously impacts the speeds that can be achieved and thereby the capacity of virtual machines that can be run over a certain link for networking or even over a specific NIC/Switch/HBA/Cable. So to use the networking topology as the example:
      • Network: For networking there are a number of constraints that can affect the capacity requirements for a vCloud conceptual design. To give an example I will use one that I am seeing a lot recently which is a 10Gb NIC connection from each blade/rack server in your proposed vCloud environment. For this 10Gb link you need to carve it up (either via native hardware methods or via NIOC) for all the varying types of traffic that needs to go over the link for your vCloud environment. Now if your network topology is inside an existing datacentre then you may have to connect to an existing top of rack switch which may only have the capability to provide two 10Gb connections per switch and the price for 2 new 10Gb switches (to obviously provide resiliency) won’t fit in the budget. So for the conceptual design if you need 10Gb of network traffic leaving each host to supply network requirements of the virtual machines on the host then you will need to either:
        • Change the hosts to have a sufficient number of NICs to provide this or
        • Go down an infiniband route or
        • Explain to the customer due to the constraint of having to use existing switches it is not possible to provide the required network bandwidth for each host so they will need to buy more hosts so that the virtual machines on each host get their required bandwidth.
      • This way of thinking applies exactly the same for storage and if you are running converged networking then it can be almost exactly the same.

    Describe VMware vCloud Director and VMware vSphere functionality and limitations related to capacity.

    • This in my opinion is all about vSphere and vCloud maximums which is always something you have to keep in mind when doing a conceptual design as for example the linked clone chain length limit is 30 and then after this a new shadow copy is created which then utilises more space on a new datastore and affects storage capacity. Actually knowing these functionality metrics and limitations is something I have been learning from going through the vCAT documentation. I did think about listing all of them but there are so many and what they could impact is so vast I think this is something where you need to know the limitations and functional capabilities of the two products and then think of it in the holistic manner of the whole design and how it impacts the conceptual design. Now remember the conceptual design is the “napkin” style design and so product names do not feature but you need to understand at a certain level what is and is not possible from the products.
    • As I mentioned in my previous point if you feel I am totally wrong then please do tell me in a friendly manner as I am certainly not perfect and am doing this to learn.

    Given current and future customer capacity requirements, determine impact to the conceptual design.

    • During your design workshops you will work out and record what the customer’s current and future capacity requirements are and then will need to plan for that 20% year on year growth they require to give an example. So if their current requirements can be met with eight hosts to be very simplistic then you will need to ensure you have sufficient capacity not just in compute but also storage, networking, cooling, power and switching.

    Given a customer datacenter topology, determine impact to the conceptual design.

    • For this I think I covered it in the first section but you now need to look at the whole topology with storage, networking, power, rack space, distances between components, distances between datacentres, cooling and weight limitations to name a few off my head that may impact your conceptual design. So say for cooling you can only put in a certain amount of hardware into each rack which then impacts your conceptual design of how many blades can fit into the datacentre/server room.

    Given cloud capacity needs, constraints, and future growth potential, create an appropriate high-level topology.

    • This is the point where you have done your design workshop and are now looking to do a high-level design of the environment that meets all the customers’ needs and shows to them you understand what they require and have planned for the future. The below diagram is a very basic version of what you would provide based on networking to show you understand their needs :


      Leave a comment

      VMware vCloud Hybrid Service Beta Impressions

      Almost two months ago I was selected as one of the very fortunate few VMware vExperts to participate in the VMware vCloud Hybrid Service beta. If you’ve not heard of vCloud Hybrid Service (vCHS) or not entirely sure what it is, then I’d recommend watching these videos before reading on:

      “An Introduction to VMware vCloud Hybrid Service”

      “A Look Inside vCloud Hybrid Service”

      We were all provided a portion (or slice?) of a virtual datacenter in a multi-tenant cloud. As a bonus I got to share mine with two VCDX’s Chris McCain and Matt Vandenbeld. clip_image001 It’s always super exciting for a nerd like me to be able to do some of the cutting edge stuff with some of the top names in the industry.


      The custom portal for vCHS looks extremely sleek and very intuitive for anyone using it for the first time or who may not have even used the vCloud GUI extensively. The front page presents you with a good overview of all your resources bundled into a Resource Snapshot section. You can easily review how much of your total resource is utilized and if you have more than one virtual datacenter you’ll observe the same utilization report per instance.


      The virtual datacenter that I shared among three other people was number 25-202. If you click on the virtual datacenter in the Virtual Datacenters section above then it will take you through to your virtual datacenter page where you can check on your Usage & Allocation, Virtual Machines, Gateways, Networks and the Users who have access to this Virtual Datacenter.

      Usage & Allocation


      Virtual Machines






      I created a number of virtual machines for a test I am planning to blog about around using vCenter Configuration Manager in vCHS. One of these virtual machines is an MS SQL server which you can see below. You can access your virtual machines from either the virtual machines tab at the top of the page or via the Virtual Datacenter tab shown previously. If you are a user with permissions to access the vCHS vCloud Director portal (VPC Administrator) you’re able to manage VMs that you have permissions to using vCloud Director by simply clicking Manage VM in vCloud Director (shown below).


      Personally I prefer working in the vCloud Director portal as this is something I’m very familiar with but the vCHS portal is more than adequate to undertake administration, it’s not too dissimilar to the standard vCloud one with an organization administrator view.


      The flagship feature of the vCHS hybrid cloud connectivity is the ability to migrate workloads using VMware’s vCloud Connector using the new Data Center Extension in vCC 2.5 between your private vCloud instance and vCHS. I’m still testing this functionality but what I’ve seen so far the stretch deploy feature is looking like an amazing use case for people looking to migrate high workload resources to vCHS Chris Colotti covered a real world case and how he utilised stretch deploy here and here.

      My initial impression of this service is really good and I’m looking forward to getting even more stuck in with real world customers and requirements. I’ll hopefully have my VCM blog posting out very soon although with all the goodness coming out of VMworld US it’s going to be hard. clip_image009