Kemp Virtual Load Balancer Review

I have had the opportunity to use a Kemp Load Balancer in my lab recently so thought it good to create a review on the product (I have a few more new toys in the lab so expect some reviews for them as well), Kemp Technologies currently have several load balancing solutions available and are a Microsoft Certified partner for both Exchange and Lync, the product I have in my lab is the Hyper V .

Over the next few months I will create a few step by steps on how to use the product (most likely against Lync 2013 as that is what is currently running in my lab), but for now I thought it would summarise the product and my first experiences with it.

Why choose it?

Kemp are highly rated in the world of load balancing and have an active partnership with Microsoft especially around Lync and Exchange, the great thing with Kemp’s load balancers is as well as providing traditional hardware based load balancing devices that we have all come to know and love they also have a virtual machine appliance for both VMWare and Microsoft Hyper-V bringing the cost of the solution down drastically so it is great for those on a tight budget.

I really like the idea that I can have it on my virtual hosts with my Lync servers as it is one less device in the data centre, the only issue I did found is that the licensing system being used is very inflexible as you are unable to move the virtual machine without contacting Kemp to relicense the device (I tried both a manual export and using System Centre 2012 VMM). This restriction might be a shortcoming for the product especially for those in very fluid data centres where you need to be able to move virtual machines quickly and at a whim, as your virtual infrastructure expands and contracts so it is worth bearing this in mind as it might be a sticking point.

Documentation

Kemp have an excellent 40+ page step by step setup guide for getting the product going with Lync 2010 and is very comprehensive and straightforward, in my lab I trialled it between my internet connection and my edge servers as a test and it worked great. In the real world remember that only having a single instance of the product means that in theory it becomes a single point of failure BUT as this is a lab it is not a problem just something to keep in mind when taking the product out in to the real world.

Ease of Use

The product was very easy to use, I got mine up and running in about an hour and using the step by step documentation made this an even easier job, whilst the Kemp web GUI itself is not the prettiest of GUI’s it gets the job done and does not feel clunky which is great, lets face it eye candy is not exactly on the priority list when it comes to configuring and using load balancers (we are engineers here after all and not Apple users)!

Summary

From the time I have spent with the product in my test lab the virtual load balancer has been great, no issues with it and it has done everything I have thrown at it and carried on working flawlessly. I would highly recommend this product for any UC implementation where you require a load balancer.

More Information

To see the setup guide for the product click here

For more information on the product I used click here

Lync & Active Directory Time when in a Virtual Environment (Hyper-V or VMWare)

I had a problem recently whereby the domains AD time was out of sync across various domain controllers which wreaked havoc across the Lync installation, the biggest issue it caused was that response group calls were getting stuck in the queue and were not routing to agents. I saw plenty of blog posts about time in a virtual environment but none of them seemed to definitively fix the problem offering a myriad of registry key changes so I have documented how I fixed it below.

The problem

AD servers synchronise with there PDC time server (nothing new there) but once they get what it believes is an accurate time from it this moves to being checked every 8 hours. This is fine in a traditional hardware based environment where your domain controllers are running on dedicated pieces of kit but in a virtualisation environment this becomes an issue as you are effectively time sharing the CPU, once your AD server has finished with its rush hour traffic of logons and people opening up file shares at 9 o clock in the morning it will most likely require a lot less resource so your virtualisation platform will start taking CPU cycles from it and your clock will start to drift.

On a normal virtual server drifting time doesn’t normally occur because you will use either Hyper-V or VMWare’s built in tools to synchronise the clock with the physical machine, but on a domain controller this feature MUST be turned off to prevent anomalies in the clock.

Stopping time synchronisation between your guest virtual machine and the host machine.

Hyper V

Select the Hyper V guest server from the Hyper-V management console, select integration services and ensure your “Time Synchronisation” flag is set to unchecked. Press OK and you should be good to go.

image

VMWare

VMWare controls the time synchronisation from within the virtual machine, to change the setting find the Vmware tools icon normally located in the system tray and double click it as per below.

image

Once VMWare tools opens un-tick “Time Synchronisation between the virtual machine and the ESX Server” which will disable the synchronisation.

image

Correcting the first Domain Controller (the PDC Emulator)

Now that the time synchronisation between guest and host has been stopped it is time to sort out the first domain controller, this needs to be your PDC emulator.

The easiest way to find which of your domain controllers is the PDC is to open up Active Directory Users and Computers, right click your domain name i.e. mydomain.com and then select “Operations Masters” from the drop down box.

Once the operations master window appears as per below, select PDC from the tabs and you will see the current operations masters as well as the option to move it to a new server which we don’t want to do so click cancel .

image

Now that we know the operations master server (in my case myserver1.mydomain.com) it is time to take control over its desktop and change some registry settings.

Warning – You are about to change the settings of your domain controllers registry, although these settings should not harm your server operationally it is recommended that you backup your registry prior to making any changes to it.

Open registry editor and navigate to

HKEY_LOCAL_MACHINE | SYSTEM | CurrentControlSet | services | W32Time | Parameters

You should see a window with similar options as below.

The important registry settings are as follows:

Key Name Type Purpose
NTPServer String Value Specify the NTP server you wish the PDC to use, I use the closest public NTP server to my site (in this case Manchester University in the UK), this can be a default one such as time.windows.com. Ensure you follow up the NTP server with the prefix 0x9.

Your NTP server should look something like, “time.windows.com,0x9”

Period DWORD This is the period of time in seconds that the server checks with its time source (in the case of the PDC Emulator it is the public NTP server).
ReliableTimeSource DWORD This needs to be set to the decimal value “1”, this forces your other servers to implicitly trust this server as a reliable source for the time across your domain.

If you see a registry key missing from the above list on your PDC Emulator simply create it with the above type and name.

Once complete your registry should look something like this.

clip_image001

Now that your PDC Emulator is set up and running your will need to restart the “Windows Time” service on your server to ensure the new settings take effect.

Configure Non PDC Domain Controllers

Next we need to go on each domain controller and make a registry change to each of the servers, I recommend looking in “Active Directory Sites and Services” to ensure you don’t miss off a domain controller off as in a large site this can lead to continuing issues with the time.

On each server you will need to create and set the period value in the registry

Again navigate to the registry key

HKEY_LOCAL_MACHINE | SYSTEM | CurrentControlSet | services | W32Time | Parameters

The registry key for this will most likely not exist so you will need to create it as per the table below.

Key Name Type Purpose
Period DWORD This is the period of time in seconds that the server checks with its time source (in the case of the PDC Emulator it is the public NTP server).

Set the value of the period registry key to 300 again so that this server requests a time update every 5 minutes.

image

Once you have completed these changes simply restart the “Windows Time” service as you did on the PDC server and your work on this server is complete. Repeat the above on each non PDC server.

Please note, although there is an NTP Server set on these servers this value is simply ignored as your server is joined to a domain so your PDC overrides this.

Troubleshooting

If your time is out of sync and you wish to force a synchronisation with the PDC emulator, open the command prompt and type the following command

net time \\Server1 /set

Replace \\server1 with the name of your PDC server as found above, this will force it to synchronise with your PDC emulator and from then on it should synchronise

Unable to Transfer Calls in A Response Group with a PSTN connected through Call Manager Express/UC500

Another small integration issue when connecting Call Manager Express with Lync 2010, users in a response group were getting thrown an error when the call arrived from the PSTN via our CME SIP trunk.

The Lync error this time was slightly less informative “Cannot complete the transfer, when contacting your support team, reference error ID 503 (source ID 239)” or see below for the graphical version.

image

After a bit of digging this was related to our trunk configuration being misconfigured when it came to how SIP refer messages were being handled. This was caused because in essence the Lync server had picked up the call to play our messages, music whilst waiting etc. before transferring it to a real human which is where the problem starts, the real human cannot now transfer the call to another person.

After doing some digging I found a TechNet article describing my exact symptoms (http://support.microsoft.com/kb/2506083), the problem occurs when you have “enable refer support” switched on.

To resolve you need to go in to your trunk configuration and remove the “enable refer support” check box as below, the GUI will also fail to save you configuration unless you also remove “enable media bypass” so un-tick that as well.

This will leave you with a trunk configured similar to the below.

image

Once completed press OK and then select commit all in the next screen to save and activate your changes within Lync. Again as with any changes grab a cup of coffee and leave Lync for 5 minutes so that it processes your changes.

That is it, your response group transfer issues should now be sorted.

Lync 2010 hold issue with Cisco Call Manager Express/UC500

Well we went live with Lync 2010 on our customer support desk and as per any go live there were a couple of teething issues so I thought I would blog one of the biggies I got that didn’t show up in testing, but was a nasty on go live.

This seems to apply to CME as well as the UC500 platform but from what I have seen it will most likely apply to Cisco Call Manager as well (I don’t have a CM to play with and test though).

The issue is that when you place somebody on hold after what seemed like a random amount of times the call would get cut off (it normally seemed to cut you off between 30 and 40 seconds or 150-180 seconds from test calls). Investigation showed in the monitoring server that the call failed with a diagnostic code of “32 – Call terminated on mid-call media failure where both endpoints are internal” as shown below.

image

image

So after seeing this we can see that the mediation server has sent a “BYE” SIP message back to our CME for some reason.

I did a lot of digging through the Lync traces as well as Cisco debug (Debug CCSIP messages is your friend) and determined that the Lync server seemed to be cutting the call off as it was not receiving RTCP messages from the Cisco CME when the call was placed on hold, this made sense as it would be sensible that Lync cuts a call off if it stops receiving messages from the other end of the trunk to prevent large bills being run up if something has gone wrong!

I did some digging on Google and found that a similar issues happens between OCS 2007R2 and Cisco CM’s Unity voicemail so this confirmed what I was seeing although this was not voicemail the issue was similar so a good jumping off point. The issue then came that the fix for OCS2007 R2 doesn’t work with Lync 2010 and so it was back to the drawing board (for those interested the TechNet article for OCS2007 R2 is here http://support.microsoft.com/kb/981218/) .

So knowing what the problem was I decided to take matters in to my own hands as Google held no results that were close to my issue. Time to dig in PowerShell and have a look at the trunk to see some of the options not being exposed by the GUI.

Under the PowerShell you will find several new options that you can’t define in the GUI, Get-CSTrunkConfiguration is the key to seeing your current configuration. After having a look at the options available I decided to change 2 of them from their defaults there are 3 interesting results worth looking at as shown below:

EnableSessionTimer                  : False
RTCPActiveCalls                     : True
RTCPCallsOnHold                     : True

After having a read on TechNet it looks to me like these were the options previously defined in the XML file in OCS2007 R2 which as Lync depends on a centralised configuration it made sense they were now stored in Lync’s PowerShell instead.

According to TechNet RTCPCallsOnHold is ignored if you are using hold music so there is no point in playing with this if you have enabled Lync’s music on hold like I have (although if your not you might want to set this to false). So according to TechNet if we want to ignore the missing RTCP messages that Cisco have decided not to send back to our mediation server we need to set RTCPActiveCalls to true.

This can be accomplished as follows:

PS C:\>Set-CsTrunkConfiguration –Identity {Name here} -RTCPActiveCalls $False

WARNING: When RTCP active calls or RTCP calls on hold is false, it is recommended that you enable session timer to periodically verify that the call is still active.

 

As you can see when you type the above command in Lync will tell you this is a bad idea unless you enable the session timer. I tested anyway after doing this to see if the above had any effect on my hold issue and the answer is yes! the problem went away, I now proceeded to make several calls and put myself on hold for 10-15 minutes and received no issues. Time to move on to the last piece, enable the session timer. To do this you need to do the following:

Set-CsTrunkConfiguration –Identity {Name here} -EnableSessionTimer $True

now reading some of the OCS 2007 R2 articles once this is enabled you will start getting your voicemail/hold calls cut off again after 30 minutes, I did a couple of final tests of this issue in Lync 2010 and was unable to reproduce I left calls on hold for well over 30 minutes and they were fine.

Hopefully you will find the above useful if you experience the same issue!! I shall update my connecting Lync 2010 to CME and add a part 3 with this and a couple of other “adjustments” I have found since putting our Lync solution live.

Step by Step Lync 2010 Enterprise Voice with Cisco Call Manager Express (or UC500) Part 2

Words of warning!!

Be careful if you use these configurations on a live system and do not simply copy and paste this configuration in to a running CME or UC500 system, check your dial-peers and translation rule numbers (“show run” is your friend) otherwise you might overwrite something you later come to regret!

Call Manager Express Inbound Call Configuration

This first key to enterprise voice is to allow the users on Lync to dial our Cisco phone users as well as external numbers on the PSTN. The following example below is from my live running CME router:

dial-peer voice 552 voip
 description **Incoming Call from LYNC**
 b2bua
 session protocol sipv2
 session target ipv4:192.168.100.64:5068
 incoming called-number .%
 voice-class codec 1
 voice-class sip dtmf-relay force rtp-nte
 dtmf-relay rtp-nte
 no vad
!

Nothing clever above, the IP address above is that of your Lync mediation server and the standard mediation server port (5068), keep an eye on the port number and ensure it matches up with your mediation server port if you changed it and do not simply enter 5060 because it is your UC’s SIP port. This port number is the number the UC talks BACK to your Lync server on.

Although the above doesn’t appear to be mandatory as our UC still dialled a number coming out of Lync without this, we found that it was intermittent at best and we could no longer control things such as class of restrictions without it.
Call Manager Express Outbound Call Configuration

The next step is to allow a user to call a user who is on the Lync platform, to do this is a little bit more complicated. To make the experience a little easier on the user (and you can’t easily dial a + on a Cisco phone) we are going to create a translation rule and link this to our dial peer.

By creating this translation rule it will allow a user to dial simply 5xxx and the CME/UC500 router will automatically add the + to the extension as it exits the CME/UC500 system, remember Lync requires e.164 style numbers so this is going to give Lync what it wants.

Be aware single number reach will not work with this configuration, my next blog post will be on how to co-exist with Lync using Cisco Signal Number Reach on the CME/UC platform.

Translation Rules

A translation rule consists of 2 parts, the rule itself and a profile that the rule is linked to, below is the translation rule and translation profile created on our system to make this work. Be aware you need to create the rule first before creating a profile.

The rule below simply adds a + in front of anything dialled that starts with a 5 and is 4 digits long.

voice translation-rule 4000
 rule 1 /\(5...\)/ /+\1/
!

Translation Profile Creation

This profile simply calls the above translation rule.

voice translation-profile LYNC_ADD_PLUS
 translate called 4000
!

Now that we have create the translation profile and translation rule it is time to create a dial peer that will call the Lync server when a user dials 5xxx.

Below is the live running configuration from our CME router, again be wary of the port as it needs to be the port of the mediation server.

dial-peer voice 551 voip
 description ** SIP Trunk to Lync Core **
 translation-profile outgoing LYNC_ADD_PLUS
 destination-pattern 5...
 notify redirect ip2pots
 session protocol sipv2
 session target ipv4:192.168.100.64:5068
 session transport tcp
 dtmf-relay rtp-nte
 codec g711ulaw
 fax rate disable
 fax protocol pass-through g711ulaw
 no vad
!

Once you have done the above is you should now have a fully functional enterprise voice installation, ensure that you enable a user with Enterprise voice.

Enabling a User for Enterprise Voice

An example Lync user configuration is below, enabling Enterprise voice is simple just select the option. It is important to set the Line URI.

In the example below I have configured the “tel:” to be my Lync phone number so in my example below it is 5346 and have also included my Cisco desk phone which is “6346” it is important if you use the extension that you do not add the + to the front of it.

Adding your desk extension improves the Lync experience as Lync will recognise you from your desk phone when you dial in to things such as the conference centre.

image

Until next time have fun with Lync and Enterprise Voice!!

I would like to give special thanks to my colleague Matt Johnson who assisted me greatly in this configuration on our environment!

Changing The Lync 2010 Conference Attendant messages

One of my latest requests was to add a company banner introductory message to Lync’s conference attendant instead of the default. For this we re-recorded all of the system messages and not simply the banner to ensure that the messages sound as professional as possible, although if you want to just change the welcome banner it is called “JOIN_WELCOME.wma”.

Changing the messages is not “officially supported” by Microsoft although a Technet article does exist to do this in OCS 2007 R2 at http://support.microsoft.com/kb/961177. This does not put your system in an “unsupported” state with Microsoft according to the article, although if you have issues with you conference attendant I would recommend putting the files back whilst you call Microsoft to ensure that the issue is not traced back to this.

After spending an hour listening to every message the EN-GB attendant woman says and making a spread sheet I thought it would be useful to share my findings.

To download the UK English messages click here, I presume the US messages are similar except that when we say “hash” in our instructions the US messages will probably say “pound” but I have not had chance to listen to them to confirm this.

Microsoft advise that all messages are in the following format although Lync seems happy with any WMA but it is best to stick to the Microsoft recommendation where possible:

  • Windows Media Audio (WMA) file format
  • 16-bit mono
  • 48 kbps 2-pass CBR (constant bit rate)
  • Speech level at –24DB

Personally I use Audacity to do most of my audio conversions but in this case these messages were being recorded by a professional audio company so this was not a concern for this project, just ensure you let whoever is recording them that they are in the above format.

Once you have your recordings back and are happy with them they need to be placed in to the following folder on your Lync server (mine test server is Standard Edition, for Enterprise installations it will be on the server containing the conferencing services on them).

C:\Program Files\Microsoft Lync Server 2010\Application Host\Applications\Conferencing Attendant\Media\en-GB

Ensure that before replacing the files you taken a backup of them in case you need to revert back to the original attendant!!

One you are done give your conferencing attendant a quick call and test to see if it now has your new messages.

Step by Step Lync 2010 Enterprise Voice with Cisco Call Manager Express (or UC500) Part 1

This article will shows how to integrate Lync 2010 and the Cisco Call Manager Express to offer Enterprise Voice capabilities to your Lync installation.

Lab Configuration
The installation has a 4 digit dial plan, all of our Cisco phones are in the 3… range and our Lync users are in the 5… range.

We currently run a UC560 running CME 8.1 so have no access to E.164 support although 8.5 will have support and is coming to the UC500 soon. This document will allow the configuration without using this support so will probably be updated once I can play with 8.5’s E.164 support.

Lync Configuration

Topology Updates
From topology builder we are going to create a new PSTN gateway, to do this expand your site Media Servers and then select your mediation server and select properties to open the following window.

image

Select the option “New” to create a PSTN gateway, type in the FQDN if you have an A record setup for your phone system OR type the IP address in. Override the port from the default port to 5060 which is the standard port for SIP and CME’s default configuration, finally ensure TCP is set (TLS is out of scope of this document).

image

Click OK and ensure you publish your topology to your environment.

Trunk Configuration

Now its time to create your PSTN routing

image

Create the first of our 2 rules, this rule will remove the +44 (UK dialling prefix, replace with your own if outside of the UK) from the beginning of the number dialled from Lync and replace it with a 0 so that the CME can understand the number we are trying to dial.

image

Our final translation rules is designed to remove the + sign from the from our extensions as they are being dialled, again this is so that the CME can understand what we are sending it.

image

Normalisation Rules

Normalisation rules are used to try and form an E.164 number from the digits dialled by an end user, for example if you were to dial 01234 567890 from your Lync client the normalisation rule will turn this in to an E.164 of +441234567890 .

image

Again we are going to create 2 normalisation rules, to create the first select “New” under the “Associated Normalization Rules”.

This rule is going to allow us to dial a PSTN call through the CME device, we are looking for any number that starts with a 0 and is at least 2 characters long, once we have this we are going to remove the 0 and add +44 to form a valid E.164 number.

image

Rule number 2 allows us to dial an extension on the CME, this rule finds and number beginning with 3 and is 4 digits long and appends + to it.

image

Now that your configuration is complete ensure that you select Commit All to upload your dial plans back in to Lync.

Route Configuration

The final piece of configuration on Lync is to create a route from Lync to the CME, below is a working configuration that allows all numbers starting +44 and +3 to be sent to the CME. Ensure that you select the previously configured PSTN gateway as well as a PSTN policy.

image

Now that the Lync configuration is completed, ensure you go to each section and ensure everything is committed. Once you have committed all of the changes leave your Lync installation, get a cup of coffee and let Lync simmer for around 10 minutes to ensure that everything has replicated around.

In Part 2 we will configure Cisco Call Manager Express to work with the above configuration. Click here to go to part 2

Help! My Exchange 2010 Outlook Anywhere Proxy Address is being set incorrectly.

After spending a couple of hours trying to get to the bottom of why our Outlook Anywhere proxy settings were incorrect set I finally solved the issue and thought it best to document on the web as this information seemed scarce to say the least and there didn’t seem to be a definitive answer to this issue.

Outlook Anywhere has 2 sets of parameters within the auto discovery service that require modifying for Outlook Anywhere to function correctly in our solution, before I start to delve in to my solution I will give you some background information on our Client Access Role server architecture. The solution I architected for our company consists of 2 x Exchange 2010 CAS servers (excas1 and excas2) which are load balanced and 1 pre-existing Exchange 2007 CAS server (atlantis).

For obfuscation our old Exchange 2007 infrastructure was sat behind a different set of DNS records than our live domain (as well as our internal network) the name of which was relatively unrelated to the company’s real name. I thought it was a great idea at the time when I was designing the solution but over the last couple of years it has proven to be more of a burden than a help, so as part of our migration we are decommissioning this domain.

For the rest of this article please assume the following domain information …

Our live domain is ContosoA.com
Our obfuscation domain is ContosoB.com

For some reason our Exchange Server had randomly picked an Outlook Anywhere proxy and SSL principal name of mail.contosob.com which was completely wrong as our previous external address was webmail.contosob.com so I have no idea where this address came from! We now needed to get this changed to mail.contosoa.com so that auto discovery would deploy the correct settings to our Outlook Clients.

Solving the SSL Principal Certificate error.

After a bit of digging through our Outlook Web Services as well as running a couple of diagnostics I could see the issue, part of the fix for this issue can be found on Technet.

http://technet.microsoft.com/en-us/library/aa998424(EXCHG.80).aspx

After running get-outlookprovider I could see that our EXCH variable was set incorrectly, as shown below

[PS] C:\Windows\system32>get-outlookprovider

Name                          Server                        CertPrincipalName                   TTL
EXCH                                                                                                                                        1
EXPR                                                                     msstd:mail.contosob.com              1
WEB                                                                                                                                           1

Resolving this issue was the easy bit, simply running the command below will set our Certificates Principal Name to the correct address mail.contosoa.com.t

Set-OutlookProvider EXPR -CertPrincipalName:”msstd:mail.contosoa.com”

So problem A was now fixed and a quick forced update of my Outlook Settings now showed that the Certificate Principal is now set to mail.contosoa.com so on to problem B.

Setting the Outlook Anywhere Proxy Address

The fix to this was relatively simple but it took some tracking down to find where this was being set and information on this was scarce and hard to come by!

To diagnose the problem I ran the command

test-outlookwebservices -clientaccessserver “excas1.contosoa.com” which came back with a lot of information about what the auto discovery service was sending out to my clients. In particular a few things appeared in this as strange and were all related to our rogue URL mail.contosob.com.

I then inspected the RPC web service configuration using the command:

get-outlookanywhere |fl externalhostname

This returned the following very useful information

ExternalHostname : webmail.contosob.com
ExternalHostname : mail.contosob.com
ExternalHostname : mail.contosoa.com

After doing a bit more digging I found that the first entry was our 2007 server and so was correctly set, the 2nd rogue entry appeared to be the first of our CAS servers and the 2nd correct entry appeared to be the second CAS server that we have. Having 1 server set correctly and the other not probably explains some of the erratic behaviour we have seen with the configuration of our clients, as depending on which CAS server Outlook picked up would have influenced the configuration returned by the auto discovery service.

Fixing the issue is now a relatively small change as follows:

set-outlookanywhere –identity “excas1\rpc (Default Web Site) “ -externalhostname “mail.contosoa.com”

Note that the server’s identity must be in the form of <server name>\<web site> (i.e. rpc (Default Web Site)) for this command to work correctly.

Forcing a refresh of the Outlook Client

Normally Outlook refreshes client settings every 60 minutes, obviously when you are testing a fix this would be a long time to wait! To test your fix a little bit quicker go to Tools | Accounts | Select your account | Repair. This will force a refresh from the auto discovery web services and allow you to see if a fix has worked.

Follow

Get every new post delivered to your Inbox.