Tuesday, September 17, 2013

Windows 8.1 Internet Explorer Hang & Crash

Windows 8.1.. :-)  Flattering, isn't it? Indeed it is.. As soon as i got to know that Microsoft released Windows 8.1 to Developers, I jumped on and downloaded the much awaited Operating System. I upgraded my machine from Windows 8 and the upgrade was smooth and easy..

But.. when i tried running IE, it hung and crashed... and there was no log or event for it? Confusing..
Reset the browser, removed add-ons but still the same.. But fortunately i was able to run it with "Run as Administrator", now that was interesting..

On the contrast, when i logged in with Local User account, it worked ..Hmm, but always failed to start with Domain account..

After working thru different options and I found disabling "Enable Enhanced Protected Mode" under Advanced | Security fixes the issue.


I will post my investigation in the next article but till then if your IE 11 doesn't work on Windows 8.1 RTM image try disabling the Enhanced Protected Mode :)

Tuesday, August 6, 2013

Unable to run exe's from the network drives on the Citrix Servers.



One of my colleague came to me with a weird issue..wherein he was not able to run an application over the network on one of the Citrix servers.

Here's the error

It looked like a security permission issue after reading the error message, so we started with the file ACLs and made sure the user account had required privilege. Hmm, the user had the required access & windows was not blocking it.. so we copied the file over to the server and ran it, oops it worked.. 

So, there is something blocking the files to be accessed over the network.. We then decided to dig in a bit more and figure out why its throwing the error..

- Tried opening other files on the same network share - Success
- Tried opening other network share and access other exe's - Failed
- Created a new text file on the share - Success
- Renamed the text file to ".exe" and ran it - Hmm, Failed

By now, we were quite sure that its only impacting the "EXE's".. we therefore quickly pulled up RSOP to check if there is any GPO that is blocking the access but there was nothing..

We consulted our very good friend "Google" and found the fix :)

By design you cannot install apps from a networked drive on the Citrix server. To enable this we changed ExecuteFromMappedDrive to 1 and rebooted the server..

HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/Cdm/ Parameters/ExecuteFromMappedDrive

3. To grant users execute permission on mapped drives, set  ExecuteFromMappedDrive to 1. 


Awesome, that fixed the issue.. Thanks Google.. :-)


Thursday, June 27, 2013

Administrator Password for Windows Server R2 Preview VHD

I downloaded the Windows Server VHD as a quick way of getting started – and to test Windows 2012 R2 but I spent hours looking for administrator password... After much hunting I found http://www.aidanfinn.com/?p=15058 where the password is revealed. And it is

R2Preview!

Enjoy

Tuesday, May 14, 2013

How to install\uninstall application in Safe Mode\Safe Mode with Networking?


How to install\uninstall application in Safe Mode\Safe Mode with Networking?

By default Microsoft doesn't allow Windows Installer service to run in Safe Mode\Safe Mode with Networking to make sure no virus\spyware get installed on the machine while you are working in Safe Mode.


The name "Safe Mode" itself gives you an idea that you machine is running in a more secure enviroment eliminating all the 3rd Party drives, services and other utilites. An operating system in safe mode will have reduced functionality, but the task of isolating problems is easier because many non-core components are disabled (turned off) and if we add the basic network components to the safe mode to get the connectivity over the LAN we call it "Safe mode with Networking".

Lets see the registry keys responsible for the same:

1. Safe Mode 
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SafeBoot\Minimal

2. Safe Mode with Networking
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SafeBoot\Network


Now lets do the real magic and make you install\uninstall applications in safe mode:

1Boot the system in safe mode by pressing F8 while booting up the machine and select safe mode.

2. Login with administrator account.

3. Start | Run | Regedit (To open registry)

4. Browse to registry key for safe mode or safe mode with networking. In this scenario we will choose safe mode option but don't you worry the following steps are same if you want to run windows installer service in Safe mode\Safe mode with networking.

Safe Mode 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SafeBoot\Minimal

Safe Mode with Networking
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SafeBoot\Network

5. Right click on Minimal key and  create new key.



6. Change the new key name to "MSIServer".



7. Now go to Start | Run | Cmd and run the command Net start Msiserver.

8. Now you can install or uninstall any application.


This way you can run any service in safe mode unless and untill its dependency service are running.

Important This section, method, or task contains steps that tell you how to modify the registry. However, serious problems might occur if you modify the registry incorrectly. Therefore, make sure that you follow these steps carefully. For added protection, back up the registry before you modify it. Then, you can restore the registry if a problem occurs. For more information about how to back up and restore the registry, click the following article number to view the article in the Microsoft Knowledge Base:

How to back up and restore the registry in Windows, KB 322756

Saturday, May 11, 2013

Active Directory Database Corruption - Investigate & Fix it

Suddenly, our script master reported that we may have a replication issue so I started looking into it and to give a brief background of the environment.. we have almost 48 Windows 2008 R2 domain controllers globally, so we needed to find out where and how the replication is broken.. 

Now, i needed a tool that can go and check all domain controllers to summarize the replication inbound and outbound replication status.. so I pulled up "REPADMIN" to find out the inbound and outbound replication status of my domain.. 
I ran "repadmin /replsummary"  and i started counting dots on the command screen which represent the progress. 

So after few minutes of processing, I had a summary report of the servers and unfortunately i found one of our DCs hasn't replicated in last 16 hrs (quite worrying, huh!! ). But just next to it had a reason of the failure which said "The replication operation encountered a database error" Oopps, this is getting interesting now..

So, i logged in to the Domain Controller reporting database issue to investigate further and fix it. The directory service Event log showed me Database index corruption errors.. hmm interesting.. 

Log Name:      Directory Service
Source:        NTDS ISAM
Date:          10.5.2013 10:03:21
Event ID:      467
Task Category: Database Corruption
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Test.domain.local
Description:
NTDS (492) NTDSA: Database C:\Windows\NTDS\ntds.dit: Index DRA_USN_index of table datatable is corrupted (0).

Corrupt database? This will definitely skip a heartbeat of most of the AD administrators.. :(

so we ran little PowerShell script to quickly check all domain controllers for Event ID 467 and make sure we are not spreading the corruption over to other servers.  Thankfully no other DC is experiencing the corruption..

Generally, the corruption can be caused by numerous reasons but i had few in my mind that requires a check there and then...

  • Hardware
  • Outdated Drivers/firmware especially disk controller & controller cache.
  • Sudden power loss
  • Lingering objects
Time to fix it then.. most of the time the Domain Administrators prefer to go ahead and rebuild the domain controller and sync everything back, but the real concern is how many changes does this box hold and what would be the impact if we go ahead with demote and re promote of the server.. hmm, so in our case we decided to go a bit further and look for clues to fix the issue instead of going for a demotion....

So, the question was how can we find more details about the error.. and like always the answer was enable more logging..To increase NTDS diagnostic logging, change the following REG_DWORD values in the registry of the destination domain controller under the following registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics

Set the value of the following subkeys to 5:
5 Replication Events
9 Internal Processing

Make sure, you are careful while editing registry and once the diagnostic logging is enabled it will start writing hell lot of information in the event log, so in case you want to save old information save it before you enable diagnostic logging.

Review the event logs for the new events that were generate from the increased logging for error values that will give a definitive view of the original 8451 error. For example, an Internal Processing event ID 1173 with error value of -1526 would indicate that we have a corruption in long-value tree.

Based on the additional information from the increased logging consult the table below for a potential resolution.
Error (decimal)
Error (hex)
Symbolic name
Error message
Potential resolution
-1018
0xfffffc06
JET_errReadVerifyFailure
Checksum error on a database page
Hardware + firmware + driver check. Restore from backup. Demote/promote
-1047
0xfffffbe9
JET_errInvalidBufferSize
Data buffer doesn't match column size
832851  Inbound Replication Fails on Domain Controllers with Event ID: 1699, Error 8451 or jet error -1601
-1075
0xfffffbcd
JET_errOutOfLongValueIDs
Long-value ID counter has reached maximum value. (perform offline defrag to reclaim free/unused LongValueIDs)
Offline Defrag
-1206
0xfffffb4a
JET_errDatabaseCorrupted
Non database file or corrupted db
Hardware + firmware + driver check.
ESENTUTIL /K + NTDSUTIL FILE INTEGRITY + UTDSUTIL Semantic Database Analysis + Offline Defrag.
Otherwise restore from backup or demote/promote
-1414
0xfffffa7a
JET_errSecondaryIndexCorrupted
Secondary index is corrupt. The database must be defragmented
Offline Defrag
-1526
0xfffffa0a
JET_errLVCorrupted
Corruption encountered in long-value tree
Hardware + firmware + driver check.
ESENTUTIL /K + NTDSUTIL FILE INTEGRITY + UTDSUTIL Semantic Database Analysis + Offline Defrag.
Otherwise restore from backup or demote/promote
-1601
0xfffff9bf
JET_errRecordNotFound
The key was not found
Hardware + firmware + driver check.
ESENTUTIL /K + NTDSUTIL FILE INTEGRITY + UTDSUTIL Semantic Database Analysis + Offline Defrag.
Otherwise restore from backup or demote/promote
-1603
0xfffff9bd
JET_errNoCurrentRecord
Currency not on a record
Hardware + firmware + driver check.
ESENTUTIL /K + NTDSUTIL FILE INTEGRITY + UTDSUTIL Semantic Database Analysis + Offline Defrag.
Otherwise restore from backup or demote/promote
8451
0x2103
ERROR_DS_DRA_DB_ERROR
The replication operation encountered a database error
Hardware + firmware + driver check.
ESENTUTIL /K + NTDSUTIL FILE INTEGRITY + UTDSUTIL Semantic Database Analysis + Offline Defrag.
Otherwise restore from backup or demote/promote
In our event viewer we found error id 1404 which is quite close to 1414 mentioned on the above table, so we decided to go ahead with:
NTDSUTIL ->Semantic database analysis
+
NTDSUTIL -> Offline Defrag 
The beauty of Windows 2008 R2 domain controller is that you can stop NTDS service and perform defrag unlike in earlier version where in you need to boot the system in "Directory Service Restore Mode" to anything with the DB.. 
I know some of you guys know the command by heart but i always prefer to open article /steps just to be sure i don't make any mistakes.. 
Offline Defrag  Article (http://support.microsoft.com/kb/232122 )
Semantic database analysis (http://support.microsoft.com/kb/315136)
Hmm, everything went smooth surprisingly (you usually don't see that working smoothly specially on Friday evenings..lol) . Anyways, the good news was that we were ready to go ahead and pull the trigger and that's what we did ..
To my surprise, the errors went away and i could see server replicating stuff now. Just to make sure everything is back up and running, we planned to bring back our friend REPADMIN ;-) .. We ran Repadmin /replsummary and it showed successful delta replication :)
Wooohhhooo... i can go home now and enjoy my weekend :-)
But, If in your case the above steps doesn't fix the issue, you may always demote and promote the server (worst case AD restore)...  

Thursday, April 25, 2013

A Case Of Mysterious NETBIOS Traffic from Domain Controller

Few days back our Security team started reporting NETBIOS traffic going out to Internet from Domain Controllers in one of our sites... Weird huh?? Were the servers compromised??? Oh, that wont be good for a Domain Controller .. :(

Anyways, we all started looking into Tipping Point logs provided by the security team to understand the reason why our DCs were sending out NBT packets outside our network. We saw good number of packets flowing out from our DCs to some unknown public subnets, which wasn't good for a server which we thought or knew doesn't and shouldn't have exposure to External world.

Anyways, we planned to gather data for few days and investigate further... We picked up one of the Domain Controllers and started investigating...Awesome, our logic did gave us a big hint.. We found out the server were sending packets thrice a day.. quite consistent of time.. 5AM, 6PM  & 11PM.. and it was doing the same everyday.. So we started looking for any suspicious process running on the box or any specific activity its doing that could be generating that traffic but unfortunately we found nothing..  We went through Eventlogs as per the trend we found but didnt see anything unusual.. and nothing we could find in the Task Scheduler that could trigger something ..

Hmm, now we were left to run Network Monitor on the Domain Controller and see what process is sending those packets.. So, we downloaded Network Monitor and captured data .. As you know, running Netmon on a production server isn't good option as it can quickly consume server resources and may bring down the server. So, we decided to filter the traffic and keep a close eye on Netmon ;-)

Here;s the filter we used(SSS.SSS.SSS.SSS is the Source IP address classs and DDD.DDD.DDD.DDD is the destination IP address class)


((ipv4.SourceAddress & 255.255.255.0) ==SSS.SSS.SSS.SSS)

 ||

((ipv4.DestinationAddress & 255.255.255.0) == DDD.DDD.DDD.DDD)


Great, now we started looking thru Network packet to find the trace and we found SYSTEM process sending out that traffic..


Network Packet Frame
  Frame: Number = 11, Captured Frame Length = 92, MediaType = ETHERNET

- Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[DD-DD-DD-DD-DD-DD],SourceAddress:[SS-SS-SS-SS-SS-SS]
  + DestinationAddress: [DD-DD-DD-DD-DD-DD]
  + SourceAddress: [SS-SS-SS-SS-SS-SS]
    EthernetType: Internet IP (IPv4), 2048(0x800)
+ Ipv4: Src = SS..SS.SS.SS, Dest = DD.DD.DD.DD, Next Protocol = UDP, Packet ID = 20651, Total IP Length = 78
+ Udp: SrcPort = NETBIOS Name Service(137), DstPort = NETBIOS Name Service(137), Length = 58
- Nbtns: Query Request for *<00><00><00><00><00><00><00><00><00><00><00><00><00><00><00> <0x00>Workstation Service
    TransactionId: 49998 (0xC34E)
  + Flag: 0 (0x0)
    QuestionCount: 1 (0x1)
    AnswerCount: 0 (0x0)
    NameServiceCount: 0 (0x0)
    AdditionalCount: 0 (0x0)
  + NbtNsQuestionSectionData: 

Now, we needed to find out what's running under System or what is System process doing. The only way for us to find it out was run Process Monitor and capture more logs. We knew the trend of the activity and we made sure we run Network Monitor and Process Monitor well before the server starts sending out the packets again..

My colleague enabled the logging and save the logs for me to review as he thinks UK shift is on a much more leisurely schedule.. :)

So, i pulled up the logs from the server and started going thru it.. it was like finding a needle in the haze.. lol
I started Process Monitor and opened the saved log file.. and just to let you all know that by default Process Monitor has a filter to hide SYSTEM process activity.


So, please make sure you remove that filter as i spent first 10 mins looking for system process in there.. :)

Then i searched one of the destination IPs and found few UDP requests sent by the system process and just above it i found UDP receive request made to DNS, interesting.. huh!!



Now, i had to go back and look in the Network Monitor logs and find out the packets.. So, i started searching under system process and found the Network Packet sent out.. then i looked under DNS and found a machine sending a request to DNS server to resolve one of the suspicious  subnet IP addresses.

Here's the Packet frame showing a DNS request from a machine

SS.SS.SS.SS = Source Machine IP address
DC.DC.DC.DC = Domain Controller IP address
SIP.SIP.SIP.SIP = Suspicious UP address



Frame: Number = 221100, Captured Frame Length = 85, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[DD-DD-DD-DD-DD-DD],SourceAddress:[SS-SS-SS-SS-SS-SS]
+ Ipv4: Src = SS.SS.SS.SS, Dest = DC.DC.DC.DC, Next Protocol = UDP, Packet ID = 0, Total IP Length = 71
- Udp: SrcPort = 60665, DstPort = DNS(53), Length = 51
    SrcPort: 60665
    DstPort: DNS(53)
    TotalLength: 51 (0x33)
    Checksum: 23415 (0x5B77)
    UDPPayload: SourcePort = 60665, DestinationPort = 53
- Dns: QueryId = 0x7DFF, QUERY (Standard query), Query  for SIP.SIP.SIP.SIP in-addr.arpa of type PTR on class Internet
    QueryIdentifier: 32255 (0x7DFF)
  + Flags:  Query, Opcode - QUERY (Standard query), RD, Rcode - Success
    QuestionCount: 1 (0x1)
    AnswerCount: 0 (0x0)
    NameServerCount: 0 (0x0)
    AdditionalCount: 0 (0x0)
  - QRecord: SIP.SIP.SIP.SIP.in-addr.arpa of type PTR on class Internet
     QuestionName: 41.79.71.206.in-addr.arpa
     QuestionType: PTR, Domain name pointer, 12(0xc)
     QuestionClass: Internet, 1(0x1)

Here's the packet sent out by System Process looking for the Suspicious IP address

  Frame: Number = 221101, Captured Frame Length = 92, MediaType = ETHERNET
+ Ethernet: Etype = Internet IP (IPv4),DestinationAddress:[DD-DD-DD-DD-DD-DD],SourceAddress:[SS-SS-SS-SS-SS-SS]
+ Ipv4: Src = DC.DC.DC.DC, Dest = SIP.SIP.SIP.SIP, Next Protocol = UDP, Packet ID = 9293, Total IP Length = 78
- Udp: SrcPort = NETBIOS Name Service(137), DstPort = NETBIOS Name Service(137), Length = 58
    SrcPort: NETBIOS Name Service(137)
    DstPort: NETBIOS Name Service(137)
    TotalLength: 58 (0x3A)
    Checksum: 47118 (0xB80E)
    UDPPayload: SourcePort = 137, DestinationPort = 137
+ Nbtns: Query Request for *<00><00><00><00><00><00><00><00><00><00><00><00><00><00><00> <0x00> Workstation Service

So, at last we were able to track the machine that was forcing our DCs to search for these subnets. The next question was why is this machine doing it? We found out its one of the IP address manager which also scans subnets. Hmm.. that's was interesting so why was this machine looking for external subnet.. apparently the subnet used to be part of our network and was sold off later...Oooppss..

A bit of clean up of the tool resolved the mystery.. Thank God!!




Monday, March 11, 2013

Basics of Server Cluster

One of my dearest friend asked me to write a blog on cluster and the hidden technology beneath. So, let's talk about cluster and understand it..

First of all "What is Microsoft Server Cluster"?
Server cluster is a group of independent servers running Windows Server Operating System and working together to provide high availability of services for the client. In case of any failure the services are redirected to other node in the cluster and share the work load. You will find more detailed info on TechNet

OK, now we know the terminology of what cluster means and what it does. So, how does it really works? It is quite interesting how the cluster technology evolved and improved since Windows 2000. Let's us understand the basic components on which cluster works and provide services:
  • Cluster IP Address.
  • Cluster Name
  • Cluster Disk
  • Cluster Node
  • Cluster Application

Cluster IP Address:
As we all know, if you want your machine to communicate with other machines on the network, it has to have an IP address. Cluster IP address resource is the prime component and helps cluster to communicate with clients as well as other nodes/servers part of the cluster. 

Cluster Name:
Like any other machine on your domain you need a name for your cluster to register itself in AD to have an identify in the your domain and provide the services. Cluster Name resource has dependency on cluster IP address resource and act as bridge for communication. So, its quite logical if Cluster IP address resources goes down, cluster Name resource will go down as well.

Cluster Disk:
Ok, so now you have a Cluster IP & Name resource for network communication but what can it communicate? End user data? Print Data? Or your customer application code? Or Cluster configuration?

What's Quorum disk?
The quorum resource plays a crucial role in the operation of the cluster. In every cluster, a single resource is designated as the quorum resource. A quorum resource can be any resource with the following functionality:
·         It offers a means of persistent arbitration. Persistent arbitration means that the quorum resource must allow a single node to gain physical control of the node and defend its control. For example, Small Computer System Interface (SCSI) disks can use Reserve and Release commands for persistent arbitration.
·         It provides physical storage that can be accessed by any node in the cluster. The quorum resource stores data that is critical to recovery after there is a communication failure between cluster nodes

What's Data Disk?
Now, you got data for you cluster but you surely need data for other applications that's configured to provide services through Cluster. Forex: Disks for File Services or Print Services.


Cluster Node:

Now, we have an IP, a Computer Name & Data so how can we process this? I hope it's not difficult to find ? Is it? Let me help you then, we need  Processor, Memory and other hardware components for data computation. So, we give cluster virtual resources to act like a computer and provide the services.

Like every computer cluster does require hardware resources to compute stuff and that requirement are fulfilled by servers part of cluster or so called nodes. These nodes virtually take over cluster resources and provide services to clients. The nodes run in the background using Virtual Cluster Name & IP address.

Cluster Application:

You have an IP, a Computer Name, Disks and a Node which means now you fulfill the basic requirement to run an application. In an cluster environment the application instance is installed on all the nodes part of the cluster or the nodes which will host the application. The application can be as simple as Notepad or DHCP Exchange or SQL.

If you have all the above components running, you have your Cluster working :)

Hopefully, the blog provides you a basic idea about pillars of cluster technology and their roles. I will write a detailed blog soon on Cluster explianing the archetecture.










Friday, March 8, 2013

PowerShell - Calculate size for a folder/drive/item

Guys,

One of the tasks an administrator or help-desk has been doing for ages is to calculate the size of a folder or drives or an item. Recently i had to work on a script that require space calculation, so i thought i should share the information as it could be handy for the new learners :-)

Here's the small code you can use:

 (Get-ChildItem -Path 'Folder/Drive Path' -Recurse | Measure-Object -Sum Length).sum /1GB

The above command will provide you the output in GB (gigabytes).

Let's understand how it works

 (Get-ChildItem -Path 'Folder/Drive Path' -Recurse | Measure-Object -Sum Length).sum /1GB

The () used above is to contain the command that will calculate the size of the object for you.

So what's running in the ()

(Get-ChildItem -Path 'Folder/Drive Path' -Recurse | Measure-Object -Sum Length).sum /1GB

Get-ChildItem 
As the name suggest, the command will get the child items in the Path specified.

Check TechNet Article to find more :
http://technet.microsoft.com/library/hh849800.aspx

-Path 
Specifies a path to one or more locations. Wildcards are permitted. The default location is the current directory. You can specify folder on the local disk or a network share where you have the access to read the data. For ex: D:\ , c:\foldertoread or \\calculate\foldersize or c:\yourfile.docx

-Recurse
Gets the items in the specified locations and in all child items of the locations.

In Windows PowerShell 2.0 or earlier versions of Windows PowerShell, the Recurse parameter works only when the value of the Path parameter is a container that has child items, such as c:\windows and not when it is an item does not have child item, such as C:\Windows\*.exe

Ok, now we have the path and PowerShell now knows what and where to look. 

This is used to pipe the output to the next command.

Measure-Object
Calculates the numeric properties of objects, and the characters, words, and lines in string objects, such as files of text
Check TechNet article for more details

-Sum
Displays the sum of the values of the specified properties 

Length
Is the file size of the item.

.Sum
Will give you the sum total of the all the items accessed by PowerShell in the ().

/1Gb or /1Mb or /1Kb 
This will divide the size into GB, MB or KB.

OK, now we have understood the command so let's use it as a script 

$Path = Read-host 'Provide the path to calculate the size'

$TotalSize = (Get-ChildItem -Path $Path -Recurse | Measure-Object -Sum Length).sum

$SizeinGB = $TotalSize /1GB
$SizeinMB = $TotalSize /1MB
$SizeinKB = $TotalSize /1KB

Write-Host "$SizeinGB GB , $SizeinMB MB, $SizeinKB KB"

Copy the code and save the file as .ps1 ;-)

Thanks for reading it, i hope you find it useful.