Defending the Castle – Protect your Endpoints

As technology becomes an increasing presence in our daily lives. The risks have increased dramatically to on both an personal and organizational level. If we step back 6 or 7 years a managed and up to date Windows XP machine with Anti-Virus protection was considered secure. Today this is no longer the case. As threats and malware becomes more sophisticated it is easily side stepping traditional Anti-Virus/Anti-Malware. Even top of the line Next gen Firewalls and Intrusion Detection Systems are not capable of stopping some threats. The risk is moving beyond malware spreading across the network or creating a giant botnet.  The new target is the information that is contained within the organization.

The Castle

Let’s for a moment consider the castle analogy. Not too long ago, the typical approach to Information Security (infosec) was to build the walls high and strong. This meant having a good application aware firewall, multiple intrusion detection systems and depending on the local laws a data loss prevention solution. The prevailing thought of the day was that if the walls were strong enough and if the guards at the top were armed that the castle would be protected. Unfortunately, for the kingdom to be successful and to operate fully the soldiers needed to go in and out of the gate.  The soldiers needed armour while outside the castle walls so we gave them some body armor in the form of  “endpoint protection” and this worked for a little while until the enemy discovered new weapons.

The Problem

The first step in moving forward is realizing that you are the target. You are the new attack surface and the easiest way to get the information that the attacker wants is if you give them the proverbial keys to the kingdom.  Think about your current situation. If you were an attacker what would you go after in your organization? Would you go after the executives?  How about the trusted users? Users with access to protected information. If you are an education institution. How about your teachers? Whatever information is valuable to you is certainly valuable to someone else . Information Security is no longer just the job of the IT security folks but each citizen in the kingdom must understand what is at stake.

Small system.. large consequences

In recent news the giant US retailer Target was breached. While the full forensics of the breach have not been disclosed what we do know about the breach is that it began with a 3rd party contractor who was phished and internal systems were compromised with malware. A seemingly small risk within the Target organization ended up with 110 million people having their personal information compromised.

Retailers with access to credit card info are not the only target. Numerous other types of organizations have been targets. If you have private information you are a target. The University of Chicago was recently breached and both employee and student information was stolen.

The new weapons in the enemy’s arsenal cannot be thwarted by building a stronger castle wall or giving the soldiers better armor. The king and his armies must regroup and come up with a new defense strategy. Defense in depth must be the central theme of this strategy.

Creating a solution

 1. Understand your risk

Your new defense strategy must begin with a solid understanding of your risk and what information or services that you need to protect. What information is your organization storing? What information might an attacker find valuable? Is it your employee database? Is it your CRM? How about your unencrypted customer credit card database?

2. Create a plan

What high level changes must your organization make? Do your key leaders have a understanding of the risk and the depth of changes that must occur to mitigate that risk? Once your key leaders understand what is at stake, together you can formulate a new defense plan.

3. Arm your soldiers
Educate your team

Giving your soldiers a set of body armor is fine but an even better strategy would be if you teach them how to fight. Educating your team should be a key part of your defense plan. Consider engaging an outside team that specializing in Information Security training. Make the training fun and engaging so that your employees retain the information long term and refresh the training often.  If your team can see the attack coming they will be a lot more likely to resist the attacker and protect themselves and your organization

Arm your endpoints

The shield of ‘Endpoint Security’ is only so strong. Strongly consider implementing a Privilege Management and Application White listing solution. Some good examples of solutions that do this well are Bit9 and Avecto.

  • Avecto starts with the concept of least privilege and trusted applications are given permissions to run. Avecto also adds the feature of sandboxing so if you have vulnerable legacy applications you can run them in a contained, separate user space which is transparent to the end user.
  • Bit9 offers a different approach to malware protection – you can’t always know what is bad and therefore you don’t look for “known bad” software, (a losing battle); you only allow what you know and trust to execute. Bit9 is a security solution that uses their proprietary dynamic whitelisting service to identify risk and only allow what you have specifically allowed to execute.

Removing admin privilege from the end user is a giant leap forward in securing your organization. It will dramatically reduce the risk of external attacks and threats that detonate from within.

Summary

The first step is admitting that you have a problem and that you are new target. Most organizational attacks are targeted and compromised systems occur because we let the attacker through the front gate. The increased presence of connected technology in our daily lives means that the attack surface has greatly increased. A proper defense strategy for yourself and your organization begins when you have an understanding of what needs to be protected and what is at risk. Make sure that your leadership is on board and supports the new strategy. Help your leaders to see the risks and what is at stake. Lastly, make sure that your people have a fighting chance when attacked. Make infosec training fun and exciting and not a boring lecture provided by poor communicators. Give your team devices that are secured and reduce the risk in case they forget.

Why Process and Management Matters

One of the challenges for IT folks is we often want to solve organizational problems with technological solutions. We try and approach all kinds of problems with our technological proverbial hammer when often a hammer is not even necessary.

An example problem would be “Our users are pirating software or using free software that loaded with malware”. The simple technological “solution” is that we remove the ability for end users to install software. However, this is just a band aid to much larger organizational issue. Why are the users installing non-approved software? What functional problem are they attempting to solve?

The end result of removing admin access to install software is that it increases the complexity and workload of your environment. All of a sudden the support staff are now inundated with requests to install software. Removing install access may have solved one problem but created a bunch of others.

The problem is not one with the technology but with the process and management and until a proper documented process has been communicated to the end users and the appropriate management functions applied. This will continue to a problem plaguing the organization. Although, the symptoms may have changed.

Over the years, the Microsoft Operations Framework has developed and matured. I recently came across an TechNet article from 2012 that goes over the different phases of the framework.  I have attached a link to the article and a copy of the presentation. I would highly encourage you if you work in the IT or Management sectors to take a look. It changed my thinking on a few concepts..

http://blogs.technet.com/b/ptsblog/archive/2012/10/02/microsoft-operations-framework-mof-4-0-fundamentals.aspx

Microsoft Operations Framework 4.0 Foundations

 

Token Bloat

Here is a short story about Token Bloat and what we did to resolve it (Hint: it had nothing to do with Tums)

The scenario:

 It’s 4:30pm on what was a relatively calm afternoon and the phone rings. It’s the CFO and she cannot access her email, shared drives or her personal files and of course it’s the night that a whole bunch of financial year end reports are due.

The problem:

The problem only occurs when she logs into her Citrix Xen Desktop VDI. If she logs into a domain joined notebook or a XenApp 4.5 Shared Desktop then she is able to get into her email, shared drives and her personal files.

On the surface it looks like a Xen Desktop problem. However, the same Desktop is shared by 300+ other users and none of them are experiencing these issues.

We saw events similar to this in the event log:

screenshot1

5327_clip_image006_08171E4A

When the CFO logs into the XenDesktop VDI she receives a couple of errors in the eventlog similar to the ones above. Her token size was 12,400 bytes. which it 400 bytes bigger than the default Windows Token size.

A quick google search brings up Shane Cothran’s post on the Technet forums:

http://blogs.technet.com/b/shanecothran/archive/2010/07/16/maxtokensize-and-kerberos-token-bloat.aspx

 

Solution

We attempted to create a Group Policy Object and apply it to the VDI Organizational Unit in Active Directory. However, due to the static nature of the VDI image this had a difficult time applying. If you are looking a set of instructions on how to apply this to a group of servers or desktops. Here is the KB article http://support.microsoft.com/kb/938118/EN-US

The change that had to be made in our case was to create registry entry on the Static Xen Desktop image.

We followed Shane’s instructions on the Registry Entry that needs to be created. The instructions are below.

To use this parameter:

  1. Start Registry Editor (Regedt32.exe).
  1. Locate and click the following key in the registry:
    SystemCurrentControlSetControlLsaKerberosParameters
  1. If this key is not present, create the key. To do so:
    1. Click the following key in the registry:
      SystemCurrentControlSetControlLsaKerberos
    2. On the Edit menu, click Add Key.
    3. Create a Parameters key.
    4. Click the new Parameters key.
  1. On the Edit menu, click Add Value, and then add the following registry value:
    Value name: MaxTokenSize
    Data type: REG_DWORD
    Radix: Decimal
    Value data: 65636
  2. Quit Registry Editor.
  3. Reboot

Once these changes are applied to the image. The user will be able to access the network resources again.

Conclusion

We discovered after some investigation that someone had added the CFO’s Active Directory User account to 46 additional groups causing the issue in the VDI environment.

 

 

Corporate Culture and why it matters

Laura and I recently watched The Internship starring Vince Vaughn and Owen Wilson. The portrayed culture intrigued me. I remember reading several articles that described similar things about the Google culture.

The questions that I am currently mulling over is: What kind of company would I like to work for? What culture would I most engage with and produce my best work? If I had carte blanche to make whatever changes I wanted in my current organization what kind of changes would I make? and why?

A classmate of mine (Kevan Gilbert) in university once said about the firm that he works for:

It’s like Domain7 actually wants me to be me, instead of trying to squash me into an employee-shaped role. And I think that means clients get much better work from all of us, than from a typical agency. (http://domain7.com/us/kevan-gilbert)

Kevan describes a culture where he feels he can be himself and produce his best work. A boilerplate HR policy obviously does not create this kind of culture but is this only possible in firms under 50 employees?

Free food and a roller rink are not the answer.. Or are they? I think the answer is the fact that google has put considerable research into what employees are looking for. They have created a place where employees want to be. Is your organization a place where people want to be/work? If yes what makes it so? If not, what changes would you make to make it a place that people want to be/work?
Or does it matter?

 

Troubleshooting Lync Mobility

If your users are anything like my users the release of the Lync Client for mobile devices had them super excited and the pressure was on to “make it work”

Unfortunately there are a significant number of ‘gotchas’ and with our roll out we seemed to hit them all.

Here are the steps we took, the problems we ran into and how we fixed them.

 

1. First Step to deploying Lync Mobility is to install the CU4 update.

Before you install the updates ensure you have a backup of the server or a snapshot. The last thing you want to do is rebuild your front end server.

Once you have your backup/snapshot.

a. Install the “Dynamic Content Compression” feature in the IIS role in Server Manager

 

b. The next thing you need to do is stop the services before installation. In Lync Server Management Shell – Type “Stop-CSWindowsService”

c. From a Elevated Command Prompt: type Net Stop W3SVC

d. Run the CU4 Updates Available Here: http://www.microsoft.com/download/en/details.aspx?id=11551

e. Once the CU4 updates are installed. Reboot the Front End Server.

2. Create your DNS Records.

On your internal domain server:

a. Create a CNAME record for your front end server. The CNAME record should be “Lyncdiscoverinternal.internaldomain.com” which points to the FQDN of your FrontEnd Server. e.g. “Frontend.internalDomain.com

b. Create an Public A Record for your Reverse Proxy

nbd. The assumption here is that you have created a Reverse Proxy for your WebComponents already. If you haven’t created a reverse proxy with TMG this is a pre-requisite. For documentation on how to setup the Reverse Proxy I recommend Daryl Hunter’s Blog. He does a really good job of going step by step.. http://www.darylhunter.me/blog/2011/11/lync-2010-reverse-proxy-part-1.html

The A Record should point to the IP address of your TMG FW proxy.

The A record should be : “Lyncdiscover” pointing to 212.111.111.101

3. Install the Autodiscover/Mobility Update

a. Shut down the “CS-WindowsService” via the Lync Server Management Shell again

b. Stop the Web server “net stop w3svc”

c. Download the Update: http://www.microsoft.com/download/en/details.aspx?id=28356 and install it.

Adam Jacobs on his blog (link) recommends installing this way.

First you’ll need to copy the McxStandalone.msi to C:ProgramDataMicrosoftLync ServerDeploymentcache4.0.7577.0setup, then execute C:Program FilesMicrosoft Lync Server
2010DeploymentBootstrapper.exe

However, double clicking the msi worked for me.

d. Reboot the Front End server.

4. Powershell Commands

We need to run some Powershell commands:

The first one enables listening on the Internal Side:  Set-CsWebServer –Identity frontend.internaldomain.com -McxSipPrimaryListeningPort 5086

The next command is for the External Site: Set-CsWebServer –Identity frontend.internaldomain.com -McxSipExternalListeningPort 5087

The Next Command is : Enable-CsTopology –verbose

This next set of commands is to enable push notifications:

Set-CsPushNotificationConfiguration

New-CsHostingProvider –Identity “LyncOnline” –Enabled $True –ProxyFqdn “sipfed.online.lync.com” –VerificationLevel UseSourceVerification

New-CsAllowedDomain –Identity “push.lync.com”

 

The last Powershell Command is to update the Database (I totally forgot about this step thanks to @itommyclarke for reminding me

If you are running Standard server you need to run this command: Install-CsDatabase –Update –LocalDatabases

If your Enterprise SQL Backend is on another server: Install-CsDatabase –Update –ConfiguredDatabases –SqlServerFqdn <SQL Server FQDN>

Lastly if you have the Monitoring and Archiving roles co-located on the same server as your other databases you will need to run this command:

Install-CsDatabase –Update –ConfiguredDatabases –SqlServerFqdn <SQL Server FQDN> –ExcludeCollocatedStores

 

5. File Edits

This is SUPER important.

a. We will start with the ApplicationHost.config file found here :C:WindowsSystem32inetsrvconfig

This I took from Microsoft (link)

  • Use a text editor such as Notepad to open the applicationHost.config file, located at C:WindowsSystem32inetsrvconfigapplicationHost.config.
  • Search for the following:
    &lt;Add name="CSExtMcxAppPool"
  • At the end of the line, before the ending angle bracket (>), type the following:
    CLRConfigFile="C:Program FilesMicrosoft Lync Server 2010Web ComponentsMcxExtAspnet_mcx.config"
  • Search for the following:
    &lt;Add name="CSIntMcxAppPool"
  • At the end of the line, before the ending angle bracket (>), type the following:
    CLRConfigFile="C:Program FilesMicrosoft Lync Server 2010Web ComponentsMcxIntAspnet_mcx.config"

 

b. The next two files we need to check are:

C:Program FilesMicrosoft Lync Server 2010Web ComponentsExternal Websiteweb.config (open with Notepad)

Add this at the end of the file before the <rules> tag

<rule name=”autodiscover rule 1″ enabled=”true” stopProcessing=”true”>

<match url=”(.*)” />

<conditions logicalGrouping=”MatchAll”>

<add input=”{HTTP_HOST}” pattern=”.*lyncdiscover.*” />

<add input=”{REQUEST_URI}” pattern=”Autodiscover/AutodiscoverService.svc/root” negate=”true” />

</conditions>

<action type=”Rewrite” url=”Autodiscover/AutodiscoverService.svc/root” />

</rule>

<rule name=”Client access policy Rule” enabled=”true” stopProcessing=”true”>

<match url=”clientaccesspolicy.xml” />

<action type=”Rewrite” url=”meet/clientaccesspolicy.aspx” />

</rule>

Make sure that you do not have two Client Access Policy rules in the file or it will create Internal Server Errors on your Autodiscover Service.

The Next file is for the internal site: C:Program FilesMicrosoft Lync Server 2010Web ComponentsInternal Website

The code is entered in the exact same place except you need this code:

<rule name=”autodiscover rule 1″ enabled=”true” stopProcessing=”true”>

<match url=”(.*)” />

<conditions logicalGrouping=”MatchAll”>

<add input=”{HTTP_HOST}” pattern=”.*lyncdiscoverinternal.*” />

<add input=”{REQUEST_URI}” pattern=”Autodiscover/AutodiscoverService.svc/root” negate=”true” />

</conditions>

<action type=”Rewrite” url=”Autodiscover/AutodiscoverService.svc/root” />

</rule>

<rule name=”Client access policy Rule” enabled=”true” stopProcessing=”true”>

<match url=”clientaccesspolicy.xml” />

<action type=”Rewrite” url=”meet/clientaccesspolicy.aspx” />

</rule>

Again make sure tha you dont  have two: “<rule name=”Client access policy Rule”….> rules.

If you don’t have these lines in your web.config files it can cause some big problems. I chased .net errors for a good week before I figured out what the problem was.

 

6. Certificates

 

We will start with the Internal Certs

On your FE server. Run the Lync Server Deployment Wizard again.

Choose to Install Lync Server

Run “Request, Install or Assign Certificates

 

Request new Certificates from your internal CA (Ensure that all three certificates are selected)

 

The Mobility and CU4 updates will fill in all the proper Subject Alternative Names (SAN).

Assign the certificates.

The External Certificates

Now Microsoft does not support Wildcard Certificates for UC purposes. We had a Wildcard Cert on our TMG Front End prior to the CU4 updates and it worked fine.

However, with the Mobility updates it did not work. You will need a UCC certificate from your Public Certificate Vendor. The SAN’s that need to be included on the Cert are:

  • meet.publicdomain.com
  • dialin.publicdomain.com
  • lyncdiscover.publicdomain.com

 

7. The Firewall Rules

 

We will need to create a new FW rule for the Lync Discover Service:

On your TMG Firewall Create a Web Site Publishing Rule:

 

Create a rule

 

It’s an Allow Rule

 

Publish a Single Web Site

 

Use SSL

 

Enter the name of your Lync Front End Server

 

Set the Path as /* and Forward the Original Host Header

 

The public DNS name of the autodiscover service: LyncDiscover.PublicDomain.com

 

Use the same Web Listener that you are using for your other Web Components (Meet,Dialin and Addressbook)

image

Set Delegation as:  No delegation, but client may authenticate directly

 

Complete The Rule with the default settings.

Once the rule is created go back and edit it.

On the Bridging Tab. Redirect the ports to 8080 and 4443

 

Click on the Listener Tab and choose properties to Edit the listener

On the Certificates Tab replace the Certificate with the new public certificate you created in Step 5

 

 

Conclusion

Hopefully this helps in your installation of the Lync Mobility features. Cheers.

Organization as domination

I wrote this post for a class I’m taking on organizational dynamics. I know that normally I write about technical things here but this directly applies to the IT world and is food for thought for our role as IT within the greater organization.

One of the things that I struggle with in my current role is the tension between freedom/creativity and bureaucracy/structure. I have seen the devastation that comes from organizations that have a ‘Wild West’ philosophy where everyone is able to do as they please. The end result is mayhem and un-productivity. However, I have seen the exact opposite where an organization is so structured and bureaucratic that working in that organization feels often like a prison. Gareth Morgan (2006) addresses this in his book Images of Organization. He writes:

(Max) Weber is famous among organization theorists for his work on the nature of bureaucracy. However, his main concern was to understand how different societies and epochs are characterized by different forms of social domination. He viewed bureaucracy as a special mode of social domination and was interested in the role of bureaucratic organizations in creating and sustaining structures of domination. (pg 294)

Like my previous post, I agree with Morgan (2006) and Weber in the concept of bureaucratic organizations can be structures of domination. Weber came up with 3 types of domination that he gathered from historical research. (see attached jpg taken from pg 295). I struggle because I see myself sometimes in the ‘rational-legal’ category. I wonder about the impact I’m having on my organization. Am I dominating our people with rules and procedures? Am I stifling creativity with bureaucracy? Obviously there needs to be a tension but where does that tension rest? Any ideas?

Doing an Authoritative Restore on an Active Directory Domain

Sometimes organizations no matter the size allow people to have way more Administrative Access than they should. Have you ever had someone who didn’t know what they were doing mess up your Active Directory Infrastructure?

If you are an architect or IT manager in your organization take a second to think about how many Domain Admins that you have? Do you really need many admins? Are you admins abiding by your change management process? Far too often even with a change management process in place, admins try to sneak changes in under the wire hoping no one will notice. If you work for an enterprise you will see this on a much larger scale than perhaps a small to medium business.

A good practice is to have object auditing enabled on your AD infrastructure. There are number of tools available that do a really good job at this. Some good examples are the tools provided by Quest and Scriptlogic both of these products do a really good job of helping you keep solid track of what is occurring in your infrastructure. If you are using tools like Arcsight or Tripwire to audit your entire network these tools will give you generic information about AD but not the granular info that you need e.g. what changed and who changed it.

However, unless a government regulation requires them to . Most organizations do not have these types of tools in place and sometimes things get deleted or even worse yet entire portions of your tree gets corrupted.

This is when you need to do authoritative restore. This should be a last resort after you have tried everything else to reverse the changes.

1. You need a backup. If you do not have a vaild Systems State backup then I am afraid you are out of luck.

2. Reboot your DC Press F8 until the Advanced Options Menu shows up

3. Choose DSRM (Directory Services Restore Mode) and press enter

4. Logon using the DSRM password. (You created this when you promoted the Domain Controller.. If you didn’t promote the DC find the person who did or check your organizations AD documentation)

5. Click Start –> Run – > Ntbackup.exe

6. Click Restore – and select System State

7. Reboot server and go back into the DSRM

8. Once you have rebooted start a command prompt

9. Type NTDSUTIL

10. Type authoritative restore

11. Restore the OU  – type “restore subtree (e.g. OU=Users,OU=Employees,DC=chrismadge,DC=com

12. A popup will occur asking if you are sure you want to perform the Authoritative Restore.. Click Yes

13. The restore will occur. Hooray!

nbd…. Should you need to restore your entire active directory cause this person caused wide spread damage. Instead of typing “restore subtree” and specifying the OU. type “restore database”

The wonder that is LastPass

All it takes is you to be compromised once for you to realize the importance of strong passwords. If your enterprise is anything like mine you will know that password complexity is the ultimate conundrum for IT administrators. If you make the password policy too complex people will simply write it down and attach it to their computer via post it note.  If the password policy is too simple ir leaves your organization vulnerable to attack.

I myself have struggled with making my passwords complex enough but simple enough that I can remember. I have numerous systems and sites that I must log into on a daily basis and creating a individual complex password for each one seemed impossible until I discovered password managers.

Password managers have been around for quite a while but most have lacked user friendliness and most have lived on the desktop. Meaning if your hard drive dies or you forget the master password then you are hooped. I’ve tried several over the years from iKeePass to 1Password on the OSX platfrom and finally ewallet. None of these solutions really did it for me. I wanted a solution that worked cross platform and would sync with my mobile devices. This hasn’t existed until now.

A company out of Virginia called LastPass has created a hosted solution for your desktop, notebook and mobile devices that works with your choice of webbrowser. That’s right friends, you are not limited to using strictly Internet Explorer. LastPass allows you to use passwords up to 20 characters with any degree of complexity including special characters. It stores your passwords using 256 bit AES encryption on their host proof servers. What this means is that the passwords are encrypted locally before being transported across the network. Especially important if you are worried about things like Man in the middle attacks. One of the things I like most about last pass is that it is only $1/month. Even on a lowly Systems Admin salary I can afford a dollar a month.

I highly recommend this product. This product can be ordered directly from the the vendor at http://www.lastpass.com

-chris

Protecting your Organization – Acceptable Use Policy #1

As an IT professional one of the first steps you need to take to protect your organization is to draft and implement an Acceptable Use Policy (AUP).  An AUP is pretty standard in most organizations and should be in place whether you have 5 or 50,000 users. If you do not have an Acceptable Use Policy you do not have to reinvent the wheel. You can find samples online that you can tailor to your uses. A few samples can be found here. Once you have drafted your policy you may need to contact your legal council for your organization before it is distributed. Next you will need to work with your HR department to ensure that this policy is communicated and each employee signs a document saying that they not only have read the policy but that they understand the policy. This is a key component of the employee signoff. Many a wrongful termination case has been proved valid because the employee simply said “they just told me to sign it, I didn’t understand.”

Now that you have your AUP in place, you must ensure that it has bite. When you have an AUP that does not have consequences it  is like an alligator with no teeth. The alligator may intimidate some but in most cases will be ignored. When you investigate AUP violations ensure that your HR team is involved, ensure that your evidence is bulletproof e.g make sure your policy states that the employee is responsible for all use with their assigned asset. This removes the “my son must have been using my company computer. etc” excuse.

Lastly, as an IT manager you need to start looking for it.  Too many managers and administrators turn a blind eye and think that this does not affect their organization. The truth of the matter is that if your organization is larger than 10 people it probably will affect your organization and no industry is immune.   One of the most recent cases that has made the news here in Vancouver has been the case with the Vancouver School Board employees at the Maintenance Shed. Several employees were caught viewing Pornography on company systems and on company time.

No matter how you personally  feel about Pornography. It is for sure offensive to some and is a legal risk for your organization. Not only do organizations need to consider lost time in productivity, but a lot of Adult websites are riddled with trojans and viruses just waiting to infect your corporate workstations. The cost to remediate these infections is costing organizations billions of dollars.

While Pornography is the predominant AUP violation it is not the only one. More and more organizations are letting their end users have local administrative rights on the workstation. This has led to everything from pirated software to freeware and shareware being installed. Taking the legal licensing risks aside, there are numerous corruption and infection risks associated.

Having a strong AUP is by no means a complete solution. It will not solve all of your user related problems. However, it is the first step in ensuring that your organization is well protected and a mandatory part of ensuring that your organization is doing it’s “due diligence”.