Content database is in compatibility range and upgrade is recommended

Incident

This warning message has turned up in the Health Analyzer on a SharePoint 2013 OnPrem Server.

“Product / patch installation or server upgrade required”

productpatch

Finding the root cause

I opened the “Version history” to see when this message started to show up:

verhistorymes

I also checked the “Database status” in “Upgrade and Migration” and found that the same message was displayed on all web applications in the server farm:

dbupgradestatus

According to some articles out there, this can happen if a CU / patch is installed. Now this server is maintained by a host provider so it could of course have happened and they don’t have in their routines that they must run the SharePoint Configuration Wizard after a CU or any other SharePoint related patch has been installed. Normally the security updates should not affect this, but this time it seems like these patches (Security Updates for Microsoft SharePoint Foundation 2013…) has caused the incompatibility as the warnings started coming after these were installed:

winupdateslist

Solution

The solution was to complete the upgrade by either:

  • run the SharePoint Configuration Wizard
  • run the PowerShell command as stated in the above message

I decided to run the SharePoint Configuration Wizard and after quite a long time running the upgrade step, it gave me an error message first saying it could not start the User Profile service and then I got the Configuration Failed message. But no worries, it is because it could not start the ups service.

messconffailed

First I went back into the Health Analyzer and clicked the button “Reanalyze now” on the message and then it was gone. Also checked the “Upgrade and Migration” link, “Review database status” to see that the normal “No action needed” message was back on the databases. And it was, so the upgrade was indeed successful.

dbupgradeviewstatus

Went into “Services on server” and started the UPS service again.

Done.

Delete sites with “Internal server error” messages

I inherited permissions on a site the other day and boom! Got an Error message and then…. the site got the annoying “500 Internal server error” and was no longer accessible. Great. No immediate cause in the log files, though scanned them up and down. Enabled custom errors in web.config and got this error:
“The DataSourceID of ‘TopNavigationMenuV4’ must be of the ID of a control of type IHierarchicalDataSource. A control with ID ‘topSiteMap’ could not be found.”

Wow. What the hell is that. Googled it and found a SQL query you could run in your db to fix this. But that only ended up in “0 rows affected” so no luck. I guess there were more errors than just this…

 

Tried all possible ways to access the sites like adding _layouts/settings.aspx, adding ?contents=1 after welcome.aspx, opened Manage content and structure but the site was not listed there, opened Site Settings and Site hierarchy and clicked on Manage (strangely enough the link to the site and subsites was still in there) but all I got was the “Internal server error” message.
I googled to death to find a solution but there seemed to be none that worked for me. But found this article on MS that this is a known issue: http://support.microsoft.com/kb/937038. But that is for MOSS and we are on SP2010 so did not dare apply that.

Seemed like the only solution was to restore the site.
But I could not restore it back to its original place since the sites were still there… even if I tried the force command I could not overwrite the sites (using the Import-SPWeb and -force). Error in Powershell was cannot find an spweb object with ID or URL…
Could not delete the site through Powershell command delete either (Remove-SPWeb). Same error in powershell as above.

Finally found a command that worked though and that is run through STSADM in Powershell:

1) First enum all sites stsadm -o enumallwebs -databasename WSS_YourDBname
2) Then you need to find out what GUID the db’s has (I usually export them to a txt file for easy copy)
Get-SPWebApplication “http://spserver/” | Get-SPSite -Limit All | Get-SPWeb -Limit All | Select Title, ID, URL | out-file d:webs.txt | format-table -auto
3) In Powershell enter this command
stsadm -o deleteweb -force -webid “GUID” -databaseserver yourdbserverinstancename -databasename WSS_YourDBname

This will delete the sites.
Some of these deletions can take a looong time. But just leave it running, do not interrupt it.

 

Script that delete old backup folders

Yey! If you want to save the server administrator (speaking!) some hell by deleting old backup folders and save disk space, then this is a solution for you!

Create a new file with Notepad and name it like deletebackupfiles.vbs and paste the following code (I cannot write the code here of course, but you can copy the code from the site http://www.mssqltips.com/tip.asp?tip=1324):

Edit the number of days on the first line, that is how old files you want to delete from the backup share. Then on the next line, enter the path to your share where the backup files are.

Remove the ‘ sign before “objFile.Delete” so that the script executes the deletion 🙂
Put a ‘ sign before the MsgBox instead, or else you must press OK to every file it wants to delete.
Then, create a new file with Notepad and name that deletebackup.bat (or what ever) and enter one line to it, and that is the path to your vbs file:

c:backupscriptsdeletebackupfiles.vbs

Open Task Scheduler on your server and create a new task where you want your .bat file to start at a scheduled time. I run it the night before my fullbackup creates new folders, so that the backup share is empty.

Part 2: Search issues

This is the rest of the story regarding Search issues

Problem: the customer has a server farm (the platform is one SQL, two WFE with MOSS 2007) and NLB running. When accessing one of the servers, the Search Box is missing. This is what the customer reported to me, and where my troubleshooting started.

Summary: Resolving a database schema mismatch after applying Security Update KB 956716

So, I had tried to reset the content index on the WFE in the server farm. But that crashed and I got into a real mess where I restarted the search services, reset the passwords on all search accounts, etc. Nothing helped me so that I could perform a Full crawl again. I got a lot of errors within Search Settings and finally what happened was that when I clicked on Full Crawl it only jumped back to “Idle”, so nothing could trigger the Full crawl.

Then I discovered the real issue: on the WFE where Search box was disabled, a patch for SharePoint had been installed. No one knows why that patch was installed, probably it has been downloaded and installed along with some other updates. So, there was a mismatch in patches since the KB956716 only was installed on one server and not on the other (as you can see, it is divided into three instances – strange):

So, I ran the Sharepoint Configuration Wizard. That was successful, and the search box came back! Success I thought and ran a Full Crawl. Everything was working perfectly. The morning after I tested the search again… No results!! Hm, what had happened during the night?? Looked through Event Viewer and there were some error messages saying that there was a database error:

Event Type: ErrorEvent Source: Office Server SearchEvent Category: Gatherer Event ID: 10036Date: 18-03-2009Time: 14:00:07User: N/AComputer: Dxxxx001
Description:A database error occurred.
Source: Microsoft OLE DB Provider for SQL
ServerCode: 4060 occurred 1 time(s)
Description: Cannot open database “SharedServices2_Search_DB” requested by the login. The login failed.

So I installed the kb on the search server as well, but the first time the server hung and we had to restart it. Then I ran the kb installation again and success this time. Restarted the server. Ran the configuration wizard. and it was successful. Started CA and Search settings and tada, there was a new link, Search Administration which is the same link that you get when you install the infrastructure upgrade.Performed a full crawl…. and yes! It started!

Conclusion: always make sure your server farm has a “patch match” 🙂 And always run the Config Wizard after a patch has been installed, then all will work fine.

Attempted to read or write protected memory

I have installed WSS 3 and Search Server Express on a Windows Server 2008 machine. No problems with that, all is fine. But today I found that three errors were logged in Event Viewer. They are:

Event ID 6398
Event ID 6482
Event ID 7076

Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchAdminSharedWebServiceInstance (02203890-d48f-4db7-9704-017c33fe14ff).
Reason: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Techinal Support Details:

System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt. at System.DirectoryServices.Interop.UnsafeNativeMethods.IntADsOpenObject(String path, String userName, String password, Int32 flags, Guid& iid, Object& ppObject) at System.DirectoryServices.Interop.UnsafeNativeMethods.ADsOpenObject(String path, String userName, String password, Int32 flags, Guid& iid, Object& ppObject) at System.DirectoryServices.DirectoryEntry.Bind(Boolean throwIfFail) at System.DirectoryServices.DirectoryEntry.Bind() at System.DirectoryServices.DirectoryEntry.get_IsContainer() at System.DirectoryServices.DirectoryEntries.CheckIsContainer() at System.DirectoryServices.DirectoryEntries.Find(String name, String schemaClassName) at Microsoft.SharePoint.Metabase.MetabaseObjectCollection`1.Find(String name) at Microsoft.SharePoint.Metabase.MetabaseObjectCollection`1.get_Item(String name) at Microsoft.SharePoint.Administration.SPProvisioningAssistant.ProvisionIisApplicationPool(String name, ApplicationPoolIdentityType identityType, String userName, SecureString password, TimeSpan idleTimeout, TimeSpan periodicRestartTime) at Microsoft.SharePoint.Administration.SPMetabaseManager.ProvisionIisApplicationPool(String name, Int32 identityType, String userName, SecureString password, TimeSpan idleTimeout, TimeSpan periodicRestartTime) at Microsoft.Office.Server.Administration.SharedWebServiceInstance.Synchronize() at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)

I have not noticed any errors inside of SharePoint or that any functions are corrupt. The only issue I found was that my scheduled task for the Backup has not run as it should.

Found this article today, and I will probably not install the hotfix until other issues inside SharePoint are reported. The customer runs a Sharepoint agent within Backup Exec which should take care of the backup task issue.

Search Server Express 2008 on a WSS farm NO NO

That will NOT work! Even if it says so when you read about it… or does it? Quite unclear information on MS pages I would say. But after having installing, trying to make it work, uninstalling, and install again I finally got an answer to why it didn’t work:

So, do NOT install the advanced option of Search Server Express 2008 if you have a WSS 3 server farm. It will NOT work! So bad for me it had to take 2 days and nights to find that out… I thought that the issue with Search was that we are using Basic Authentication on our sites, since that is a problem as well. My work right now only consists of troubleshooting, errors, ugly sessionstate errors in web.config files etc. Read article on issues with using Basic Auth and Search here:
Now I will try to reinstall SSEX using Basic installation and not the Advanced….

Installation of Microsoft Search Server 2008 Express

Yes that succeeded today! But not without some issues and error messages, hello what do ya think? I think those who writes the installation scripts are terrorists and they want you to fear heart failure each time you must install something new on the server. :)… To be continued, I will show you the errors and what must be done to get rid of those… but now I must sleep… working for 16 hours now… zzz

Forms authentication using the AD

I am setting up an extranet site where the customer wants the user to be logged out after a certain idle time. Since there is no way to make the user logged out using windows authentication, for logical reasons, then I must activate forms authentication. But I dont want to have yet another user register like in a SQL db, so therefore I must activate it against the AD. So this is what I did:

Locate the web.config file that belongs to your site in the virtual directory (you find the path in your IIS site):

Make sure you backup your web.config file first! Locate the configSections tag. You can remark the authentication mode for windows, and instead enter this code:

These steps must be performed both in your sites web.config, and also in the web.config that belongs to the Central Admin site. Locate the tag system.web and add the following LDAP connection string to your Active Directory:

Also add the following section under the system.web tag:

Actually, I removed the connectionUsername and connectionPassword lines from my web.config since that file is quite unprotected on the server and you don’t want the Administrator password out!! So if you remove those lines, the application pool user account will be used instead. So you must make sure that the account has rights on your AD.

Then in CA you go to the Authentication Provider link, select your site, click on the zone you would like to activate it on, and select “Forms” instead of “Windows”.

Do an iisreset and you should have a login page on your site now:

Sharepoint Services Timer hangs

This is really irritating. The service Windows Sharepoint Services Timer regularly hangs on the server, which leads to that no alerts are sent out. And you cannot just restart the service, that wont help, you must stop and start the service. Weird. Yet another issue to add to the list…

Victory over SBS, YES!

Yes, today I have been fighting with a customers small business server – but I won!! What I did? The most dreaded – uninstalled WSS 2 on a SBS server and installed a fresh WSS 3 version. Everything went perfect until I wanted to give the new portal a hostname. Did not work. Every hostname or IP just redirected to the ugly Companyweb (even if I updated AAM in CA). Crazy. I ended up adding a new IP-address to the server and dedicate the new portal to that address instead. Removed all “old” shit in the IIS like old application STS-pools and stuff. In this order: added a new IP, added a new record in DNS pointing to that IP, changed the IIS web site to the new IP, added a new AAM-record in Central Administration. IISreset. Worked like a charm 🙂 You just have to do things in the correct order, or everything will freak out. Now I just need a SSL cert and all is done. And me too…