Second Life of a Hungarian SharePoint Geek

July 2, 2015

Sending mails from a test SharePoint system, Updated for SharePoint 2013

Filed under: Mails, Reflection, SP 2013 — Tags: , , — Peter Holpar @ 23:48

Last year I published a post about how to redirect mails sent from your SharePoint application to a list, storing the mails as attachments for the list items. Unfortunately, the Reflection-based approach described there seems to no longer work in SharePoint 2013. My goal with the current post is to provide an alternative solution to the problem.

The first issue is with the signature of the private Send method of the MailMessage class we used to save the content of the mail into a MemoryStream: it has been extended with an extra bool parameter (allowUnicode) since the .NET  4.5 version. This problem could be easily fixed by altering the method invocation with the extra parameter, as:

sendMethod.Invoke(message, BindingFlags.Instance | BindingFlags.NonPublic, null, new[] { mailWriter, true, true }, null);

If you test the code in a simple console application, it works, the mail message will be saved into the MemoryStream. However, if you try to use the same code from a SharePoint application (an application referencing the server side SharePoint assemblies: either a console application or web part), you become a nasty exception:

An unhandled exception of type ‘System.AccessViolationException’ occurred
Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

After a short research, it turned out that the reason is the x64 process the SharePoint server side assemblies require. Since we can not change that requirement, I had to find an alternative solution. It was easy, as the stackoverflow thread the original solution originate from already contains an other option that plays with the DeliveryMethod of the SmtpClient class, saving the mail into a file in a temporary folder in the file system. Based on that answer I’ve altered the code of my DummyMailSender.

First, I’ve defined the helper class TemporaryDirectory to clean up the rest after the work:

  1. internal class TemporaryDirectory : IDisposable
  2. {
  3.     public TemporaryDirectory()
  4.     {
  5.         DirectoryPath = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName());
  6.         Directory.CreateDirectory(DirectoryPath);
  7.     }
  9.     public string DirectoryPath { get; private set; }
  11.     public void Dispose()
  12.     {
  13.         if (Directory.Exists(DirectoryPath))
  14.         {
  15.             Directory.Delete(DirectoryPath, true);
  16.         }
  17.     }
  18. }

and another class Extensions that implements an extension method to save the mail into a byte array:

  1. internal static class Extensions
  2. {
  3.     public static byte[] GetAsByteArray(this MailMessage m)
  4.     {
  5.         byte[] messageData = null;
  7.         var smtpClient = new SmtpClient { DeliveryMethod = SmtpDeliveryMethod.SpecifiedPickupDirectory };
  9.         using (var tempDir = new TemporaryDirectory())
  10.         {
  11.             smtpClient.PickupDirectoryLocation = tempDir.DirectoryPath;
  12.             smtpClient.Send(m);
  13.             var emlFile = Directory.GetFiles(smtpClient.PickupDirectoryLocation).FirstOrDefault();
  14.             if (emlFile != null)
  15.             {
  16.                 // read all file contents and trim the carriage return / new line at the end
  17.                 var messageAsText = File.ReadAllText(emlFile).Trim();
  18.                 System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
  19.                 messageData = enc.GetBytes(messageAsText);
  20.             }
  21.         }
  22.         return messageData;
  23.     }
  24. }

Having these classes that perform the bulk of the functionality, the DummyMailSender can be simplified as:

  1. public void Send(MailMessage message)
  2. {
  3.     Trace.TraceInformation("Mail (subject: '{0}') sending started via DummyMailSender", message.Subject);
  5.     byte[] messageData = message.GetAsByteArray();
  7.     // RootWeb mustn't be disposed, see:
  8.     //
  9.     SPWeb rootWeb = _site.RootWeb;
  10.     bool allowUnsafeOriginal = rootWeb.AllowUnsafeUpdates;
  11.     try
  12.     {
  13.         rootWeb.AllowUnsafeUpdates = true;
  14.         SPList mailList = rootWeb.Lists[Constants.DummyMailList];
  15.         SPListItem mailItem = mailList.AddItem();
  16.         mailItem[SPBuiltInFieldId.Title] = message.Subject;
  17.         mailItem.Attachments.Add("Mail.eml", messageData);
  18.         mailItem.Update();
  20.         string mailUrl = string.Format("{0}{1}?ID={2}", _site.Url, mailList.DefaultDisplayFormUrl, mailItem.ID);
  21.         Trace.TraceInformation("Dummy mail with subject '{0}' \"sent\" to '{1}' (cc: '{2}'), saved to '{3}'", message.Subject, message.To, message.CC, mailUrl);
  22.     }
  23.     catch (Exception ex)
  24.     {
  25.         Trace.TraceError("Error sending dummy mail: {0}\r\n{1}", ex.Message, ex.StackTrace);
  26.     }
  27.     finally
  28.     {
  29.         rootWeb.AllowUnsafeUpdates = allowUnsafeOriginal;
  30.     }
  31. }

This new version of DummyMailSender might perform not so well, as the original, Reflection-based one, but it should not be a major issue in a test system it was planned for, and at least it is a supported solution. At least, you don’t have to worry about that it won’t work after a newer .NET Framework version.

April 13, 2015

Breaking Changes in Project Server Update?

Filed under: Managed Client OM, Project Server — Tags: , — Peter Holpar @ 21:02

Recently I have extended the Visual Studio solution, that includes the code sample for a former blog post, illustrating how to register Project Server event handlers via the managed client object model.

When I wanted to build the solution, the build was broken because of a compile time error in the code I have not change since last November:

‘Microsoft.ProjectServer.Client.EventHandlerCreationInformation’ does not contain a definition for ‘CancelOnError’


I’ve opened the corresponding assembly (C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\ISAPI\Microsoft.ProjectServer.Client.dll) in Reflector,  and found that there is really no CancelOnError property defined on the class Microsoft.ProjectServer.Client.EventHandlerCreationInformation.


Based on the official documentation, this property should exist, and I’m sure I was able to compile my code, so it existed at that time.

Our developer environment was recently patched from the SP1 patch level to the February 2015 PU patch level, so it must have been “lost” that time.

I found another VM that is at the RTM level, checked the same class in the same assembly, and found the property defined:


Note: the problem affects not only the managed client object model, but the JavaScript object model and the server-side object model as well. It should affect the out-of-the box feature receiver described in my former post, and all of the packages that rely on this functionality.

We have the same issue on the server side as well. Both of the the classes Microsoft.ProjectServer.EventHandlerCreationInformation and Microsoft.ProjectServer.EventHandler (C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\15\CONFIG\BIN\Microsoft.ProjectServer.dll) had the CancelOnError property in the previous version, but it is simply no more defined without any official warning on the change, causing both server side and client side code referring to this property to fail.

April 8, 2015

Automating the Deployment of a Customized Project Web Site Template via PowerShell and the Managed Client Object Model

Filed under: ALM, Managed Client OM, PowerShell, Project Server — Tags: , , , — Peter Holpar @ 21:45

Assume, you have created a customized web site template for your enterprise project type in the development environment as described here, and now you would like to deploy it into the test farm. Of course, you can manually delete the former site template, upload the new one, re-configure it to be the associated web site template for your enterprise project type, and finally re-create your test project (that means, checking in and deleting the existing one, and create it again using the new template), but this procedure is boring, cumbersome and – as any human-based process – rather error-prone.

Why do not automate this step as well?

I’ve created a PowerShell script that performs the steps outlined above. The first steps (deleting the former version of the site template and uploading the new one) can be done by native PowerShell Cmdlets, but for the remaining, Project Server related tasks require the Managed Client Object Model, so we import the necessary assemblies into the process.

First we get a list of all projects and a list of all enterprise project types, then query for the right ones on the “client side”.

Note: Although PowerShell does not support .NET extension methods (like the Where and Include methods of the client object model) natively, we could restrict the items returned by these queries to include really only the item we need (see solution here), and include only the properties we need (as described here). As the item count of the projects and enterprise project types is not significant, and we should use the script on the server itself due to the SharePoint Cmdlets, it has no sense in this case to limit the network traffic via these tricks.

Next, we update the web site template setting (WorkspaceTemplateName  property) of the enterprise project type. We need this step as the original vale was reset to the default value as we deleted the original site template on re-upload.

If the test project is found, we delete it (after we checked it in, if it was checked out), and create it using the updated template.

Since these last steps (project check-in, deletion, and creation) are all queue-based operations, we should use the WaitForQueue method to be sure the former operation is completed before we start the next step.

$pwaUrl = "http://YourProjectServer/PWA/"
$solutionName = "YourSiteTemplate"
$wspFileName = $solutionName + ".wsp"
$timeoutSeconds = 1000
$projName = "TestProj"

# English
$projType = "Enterprise Project"
$pwaLcid = 1033
# German
#$projType = "Enterprise-Projekt"
#$pwaLcid = 1031

# path of the folder containing the .wsp
$localRootPath = "D:\SiteTemplates\"
$wspLocalPath = $localRootPath + $wspFileName

# uninstall / remove the site template if activated / found
$solution = Get-SPUserSolution -Identity $wspFileName -Site $pwaUrl -ErrorAction SilentlyContinue
If ($solution -ne $Null) {
  If ($solution.Status -eq "Activated") {
    Write-Host Uninstalling web site template
    Uninstall-SPUserSolution -Identity $solutionName -Site $pwaUrl -Confirm:$False
  Write-Host Removing web site template
  Remove-SPUserSolution -Identity $wspFileName -Site $pwaUrl -Confirm:$False

# upload and activate the new version
Write-Host Uploading new web site template
Add-SPUserSolution -LiteralPath $wspLocalPath -Site $pwaUrl
Write-Host Installing new web site template
$dummy = Install-SPUserSolution -Identity $solutionName -Site $pwaUrl
# set the path according the location of the assemblies
Add-Type -Path "c:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.ProjectServer.Client.dll"
Add-Type -Path "c:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"

$projectContext = New-Object Microsoft.ProjectServer.Client.ProjectContext($pwaUrl)

# get lists of enterprise project types and projects
$projectTypes = $projectContext.LoadQuery($projectContext.EnterpriseProjectTypes)
$projects = $projectContext.Projects
$projectList = $projectContext.LoadQuery($projectContext.Projects)


$entProjType = $projectTypes | ? { $_.Name -eq $projType }
$project = $projectList | ? { $_.Name -eq $projName }

Write-Host Updating web site template for the enterprise project type
$web = Get-SPWeb $pwaUrl
$template = $web.GetAvailableWebTemplates($pwaLcid) | ? { $_.Title -eq $solutionName }

$entProjType.WorkspaceTemplateName = $template.Name

If ($project -ne $Null) {
  If ($project.IsCheckedOut) {
    Write-Host Project $projName is checked out, checking it in before deletion
    $checkInJob = $project.Draft.CheckIn($True)
    $checkInJobState = $projectContext.WaitForQueue($checkInJob, $timeoutSeconds)
    Write-Host Check-in project job status: $checkInJobState
  Write-Host Deleting existing project $projName
  # we can delete the project either this way
  #$removeProjResult = $projects.Remove($project)
  #$removeJob = $projects.Update()
  # or
  $removeJob = $project.DeleteObject()
  $removeJobState = $projectContext.WaitForQueue($removeJob, $timeoutSeconds)
  Write-Host Remove project job status: $removeJobState

I found the set of Project Server PowerShell Cmdlets is limited, and rather operation-based. You can use it, as long as your single task is to administer Project Server instances and databases. However, when it comes to the interaction with Project Server entities, you have to involve the Managed Client Object Model. Hopefully this example provides not only a reusable tool, but also helps you understand, how to extend your own PowerShell library with the methods borrowed from the client side .NET libraries.

Automating Project Server development tasks via PowerShell and the Client Object Model – Customizing Project Web Site templates

Filed under: ALM, Managed Client OM, PowerShell, Project Server — Tags: , , , — Peter Holpar @ 21:35

I start with a note this time: Even though you were not interested in Project Server itself at all, I suggest you to read the post further, while most of the issues discussed below are not Project Server specific, they apply to SharePoint as well.

Recently I work mostly on a Project Server customization project. As I’ve learned on my former development projects, I try to automate so much repetitive tasks as possible (like automating the PWA provisioning), thus remains more time for the really interesting stuff. I plan to post my results on this blog to share the scripts and document the experiences for myself as well.

One of the very first tasks (and probably a never-ending one) was to create a customized Project Web Site (PWS) site template. New Enterprise Projects created in the Project Web Access (PWA) should have their PWS created based on the custom site template.

The process of customizing a PWS site template is described in this post, however, there are a few issues if we apply this approach alone, just to name a few of them:

– PWS master pages cannot be edited using SharePoint Designer by default. There is a workaround for this issue.

– If I create a custom master page for the PWA and would like a PWS to refer the same master page, I can set it for example using PowerShell. However, if I create a site template from this PWS, this configuration seems to be lost in the template, and template refers to the default seattle.master. I have similar experience with the site image / logo, I can set one, but this setting seems to be not saved in the template.

– The standard navigation structure of a project site (and all site template created based on it) contains project-specific navigation nodes, like Project Details that contains the Guid of the current project as a query string parameter. If you create a site template from this site, any project sites that will be created based on this template will contain this node twice: one of the is created based on the site template (wrong Guid, referring to the project the original site belongs to, thus wrong URL), and another one is created dynamically as the project web site gets provisioned.

The workflow of my web site template creation and customization process includes four main parts, and two of them – step 2 and step 4 – are automated by our script.

The first part of the process (including step 1 and step 2) is optional. If you have changed nothing in your web site prototype, you can immediately start with the manual manipulation of the extracted web site template content (step 3), otherwise, we have to get a fresh version of the template into our local system for the further customizations.

Step 1: Creation and customization a SharePoint web site, that serves as a prototype for the web site template.

A SharePoint web site is customized based on the requirements using the SharePoint web UI, SharePoint Designer (for PWA see this post), or via other tools, like PowerShell scripts (for example, JSLink settings). This is a “manual” task.

Step 2: Creation of the web site template based on the prototype, downloading and extracting the site template.

A site template is created (including content) based on the customized web site. If a former site template having the same name already exists, if will be deleted first.

The site template is downloaded to the local file system (former file having the same name is deleted first).

The content of the .wsp file (CAB format) is extracted into a local folder (folder having the same name is deleted first, if it exists).

Step 3: Customization of the extracted web site template artifacts.

The script is paused. In this step you have the chance to manual customization of the solution files, like ONet.xml.

Step 4: Compressing the customized files into a new site template, and uploading it to SharePoint.

After a key press the script runs further.

Files having the same name as our site template and extension of .cab or .wsp will be deleted. The content of the folder is compressed as .cab and the renamed to .wsp.

In the final step the original web site template is removed and the new version is installed.

Next, a few words about the CAB extraction and compression tools I chose for the automation. Minimal requirements were that the tool must have a command line interface and it should recognize the folder structure to be compressed automatically, without any helper files (like the DDF directive file in case of makecab).

After reading a few comparisons (like this and this one) about the alternative options, I first found IZArc and its command line add-on (including IZARCC for compression and IZARCE for extraction, see their user’s manual for details) to be the best choice. However after a short test I experienced issues with the depth of the folder path and file name length in case of IZARCE, so I fell back to extrac32 for the extraction.

Finally, the script itself:

$pwaUrl = "http://YourProjectServer/PWA/"
$pwsSiteTemplateSourceUrl = $pwaUrl + "YourPrototypeWeb"
$solutionName = "YourSiteTemplate"
$wspFileName = $solutionName + ".wsp"
$cabFileName = $solutionName + ".cab"
$siteTemplateTitle = $solutionName
$siteTemplateName = $solutionName
$siteTemplateDescription = "PWS Website Template"

$localRootPath = "D:\SiteTemplates\"
$wspExtractFolderName = $solutionName
$wspExtractFolder = $localRootPath + $wspExtractFolderName
$wspFilePath = $localRootPath + $wspFileName
$wspLocalPath = $localRootPath + $wspFileName
$wspUrl = $pwaUrl + "_catalogs/solutions/" + $wspFileName

$cabFilePath = $localRootPath + $cabFileName

function Using-Culture (
   [System.Globalization.CultureInfo]   $culture = (throw "USAGE: Using-Culture -Culture culture -Script {…}"),
   $script = (throw "USAGE: Using-Culture -Culture culture -Script {…}"))
     $OldCulture = [Threading.Thread]::CurrentThread.CurrentCulture
     $OldUICulture = [Threading.Thread]::CurrentThread.CurrentUICulture
         try {
                 [Threading.Thread]::CurrentThread.CurrentCulture = $culture
                 [Threading.Thread]::CurrentThread.CurrentUICulture = $culture
                 Invoke-Command $script
         finally {
                 [Threading.Thread]::CurrentThread.CurrentCulture = $OldCulture
                 [Threading.Thread]::CurrentThread.CurrentUICulture = $OldUICulture

function Remove-SiteTemplate-IfExists($solutionName, $wspFileName, $pwaUrl) 
  $us = Get-SPUserSolution -Identity $solutionName -Site $pwaUrl -ErrorAction SilentlyContinue
  if ($us -ne $Null)
    Write-Host Former version of site template found on the server. It will be removed…
    Uninstall-SPUserSolution -Identity $solutionName -Site $pwaUrl -Confirm:$False
    Remove-SPUserSolution -Identity $wspFileName -Site $pwaUrl -Confirm:$False

function Remove-File-IfExists($path)
  If (Test-Path $path)
    If (Test-Path $path -PathType Container)
      Write-Host Deleting folder: $path
      Remove-Item $path -Force -Recurse
      Write-Host Deleting file: $path
      Remove-Item $path -Force

Do { $downloadNewTemplate = Read-Host "Would you like to get a new local version of the site template to edit? (y/n)" }
Until ("y","n" -contains $downloadNewTemplate )

If ($downloadNewTemplate -eq "y")

    Remove-SiteTemplate-IfExists $solutionName $wspFileName $pwaUrl

    Using-Culture de-DE { 
     Write-Host Saving site as site template including content
     $web = Get-SPWeb $pwsSiteTemplateSourceUrl
     $web.SaveAsTemplate($siteTemplateName, $siteTemplateTitle, $siteTemplateDescription, 1)

  Remove-File-IfExists $cabFilePath

  Write-Host Downloading site template
  $webClient = New-Object System.Net.WebClient
  $webClient.UseDefaultCredentials  = $True 
  $webClient.DownloadFile($wspUrl, $cabFilePath)

  # clean up former version before downloading the new one
  # be sure you do not lock the deletion, for example, by having one of the subfolders opened in File Explorer,
  # or via any file opened in an application
  Remove-File-IfExists $wspExtractFolder

  Write-Host Extracting site template into folder $wspExtractFolder
  # limited file lenght / folder structure depth! :-(
  #& "C:\Program Files (x86)\IZArc\IZARCE.exe" -d $cabFilePath $wspExtractFolder

  #expand $cabFilePath $wspExtractFolder -F:*.*
  extrac32 /Y /E $cabFilePath /L $wspExtractFolder

Write-Host "Alter the extracted content of the site template, then press any key to upload the template…"
# wait any key press without any output to the console
$dummy = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")

# clean up former version before creating the new one
# TODO rename it using a date time pattern instead of deletion!
Remove-File-IfExists $cabFilePath
Remove-File-IfExists $wspFilePath

# makecab: we cannot include multiple files directly. To do that, we have to create a directive file called a Diamond Directive File(DDF) and include instructions in it
& "C:\Program Files (x86)\IZArc\IZARCC.exe" -a -r -p $cabFilePath $wspExtractFolder

Rename-Item $cabFilePath $wspFileName

# remove former solution before uploading and activating the new one
Remove-SiteTemplate-IfExists $solutionName $wspFileName $pwaUrl

Write-Host Installing the new version of the site template
Add-SPUserSolution -LiteralPath $wspFilePath -Site $pwaUrl
$dummy = Install-SPUserSolution -Identity $solutionName -Site $pwaUrl

Note: If you are working with the English version of the PWA and have an English operating system on the server, you don’t need the Using-Culture function. To learn more about it see this post.

April 6, 2015

Make the Status Bar Visible on Project Detail Pages for Users with Read-Only Permissions on PWA with Publishing Feature

Filed under: JavaScript, Project Server, Security — Tags: , , — Peter Holpar @ 22:03

Assume the following situation: You have a Project Web Access site (PWA) on your Project Server that has the SharePoint Server Publishing feature activated both on the site collection and on the site level. This detail might seem to be irrelevant now, but gets an importance pretty soon. The permissions on the PWA are customized and rather restricted, even project managers have only Read permissions on the root site level. The PMs complain often, that they can not open the project from the Project Detail Pages (PDP) for editing, the corresponding buttons are inactive on the ribbon.


The administrators check the project and see, that it is checked out to someone else.

Problem: the administrators see the status information on the PDPs, however other user do not. For example, if an administrator opens the project for editing, the following screen is displayed:


The users with standard rights see however only this, without the status bar:


I found, that the content of the status bar is contained in a DIV with id="pageStatusBar" on the web page. There are several JavaScript methods, that manipulate this content, like addStatus, appendStatus, removeAllStatus, removeStatus, setStatusPriColor (all of these in init.debug.js). After setting breakpoints in these methods, and reloading the site, I found, that the status bar is displayed temporally in each cases, but it is hidden then for standard users, via a JavaScript method that is included in the PDP itself (like Schedule.aspx):

document.onreadystatechange=fnRemoveAllStatus; function fnRemoveAllStatus(){removeAllStatus(true)};

For the administrators, this script was not included in the page, and so the status bar remained visible.

After a search using Reflector, I found the very same script block being inserted to the page by the HideStatusBar method of the PublishingRibbon class (namespace: Microsoft.SharePoint.Publishing.Internal.WebControls, assembly: Microsoft.SharePoint.Publishing, just as any other classes below). This method was called by the HideRibbonAndStatusBarAndSiteActionsMenu method, which is called by the OnLoad method of the same PublishingRibbon class, if the CanShowSiteActionsMenuItems method of the ConsoleVisibleUtilities class returns false:

public static bool CanShowSiteActionsMenuItems
    SPBasePermissions permissions = SPBasePermissions.BrowseUserInfo | SPBasePermissions.CreateSSCSite | SPBasePermissions.CreateAlerts | SPBasePermissions.UseRemoteAPIs | SPBasePermissions.UseClientIntegration |  SPBasePermissions.ViewVersions | SPBasePermissions.OpenItems | SPBasePermissions.ViewListItems | SPBasePermissions.ViewPages | SPBasePermissions.Open | SPBasePermissions.ViewFormPages;
    CombinedBasePermissions permissions2 = new CombinedBasePermissions();
    If ((((permissions | permissions2.ListItemPermissions) == permissions) && ((permissions | permissions2.ListPermissions) == permissions)) && ((permissions | permissions2.RootSitePermissions) == permissions))
      return ((permissions | permissions2.SitePermissions) != permissions);
    return true;

The CombinedBasePermissions is a helper class that aggregates the site and list permissions of the current user from the current SharePoint context.

The value of the permissions variable is a combination of several list- and site-level base permissions. If you compare it with the standard Read permission level (see the next two screenshots below), you can see, that it is exactly the same combination of permissions:

List permissions for the Read permission level


Site permissions for the Read permission level


We compare the site and list permission returned by CombinedBasePermissions with these predefined set of base permissions. In the case of the PDPs we have no list context, that means, only the site permissions will be compared. If the current user has no permission beyond the ones included in the standard Read permission level, the remove script will be injected into the page, and the status bar won’t be displayed for the user.

The following screenshot illustrates the site permissions for the Team members (Microsoft Project Web App) permission level:


It includes the Browse Directories and the Edit Personal User Information base permissions. If the user had the Team members (Microsoft Project Web App) permission level either with or without the Read permission level, they would like to see the status bar. However, granting extra permissions to the users might be not desired on one hand, on the other hand, its effect would not be limited to the PDPs, the relevant status bar would be displayed on all other pages as well.

You can consider, I you really need the SharePoint Server Publishing feature. If not, deactivating it on the site level might solve the issue as well.

If you are OK with a quick and dirty solution, include the following script in you PDPs, for example, via a Script Editor Web Part:

<script type="text/ecmascript">
  function removeAllStatus(b) {

The script overrides the default implementation of the removeAllStatus method in the scope of the page it is included into, and makes it impossible to hide the status bar.

As the result of the page customization, that status bar left displayed for standard users as well:


March 29, 2015

May Merge-SPLogFile Flood the Content Database of the Central Administration Web Application?

Filed under: Content database, PowerShell, SP 2013 — Tags: , , — Peter Holpar @ 00:20

In the recent weeks we searched for the cause of a specific error in one of our SharePoint 2013 farms. To get detailed trace information, we switched the log level often to the VerboseEx mode. A few days later the admins alerted us that the size of the Central Administration content database has been increased enormously (at that time it was about 80 GB!).

Looking for the source of this unexpected amount of data I found a document library called Diagnostics Log Files (Description: View and analyze merged log files from the all servers in the farm) that contained 1824 items.



Although I consider myself primarily a SharePoint developer, I always try to remain up-to-date in infrastructural themes as well, but to tell the truth I’ve never seen this document library before. Searching the web didn’t help as well.

Having a look into the document library I found a set of folders with GUID names.


Each of the folders contained a lot of files: a numbered set for each of the servers in the farm.


The files within the folders are archived logs in .gz file format, each around 26 MB.


Based on the description of the library I guessed that it has a relation to the Merge-SPLogFile cmdlet, that performs collection of ULS log files from all of the servers in the farm an saves the aggregation in the specified file on the local system, although I have not found any documentation how it performs this action and if it has anything to do with the content DB of the central admin site.


After a few hours of “reflectioning”, it was obvious, how this situation was achieved. If you are not interested in call chains, feel free to jump through the following part.

All of the classes and methods below are defined in the Microsoft.SharePoint assembly, if not specified otherwise.

The InternalProcessRecord method of the Microsoft.SharePoint.PowerShell.SPCmdletMergeLogFile class (Microsoft.SharePoint.PowerShell assembly) creates a new instance of the SPMergeLogFileUtil class based on the path of the aggregated log folder and the filter expression, and calls its Run method:

SPMergeLogFileUtil util = new SPMergeLogFileUtil(this.Path, filter);
ThreadPool.QueueUserWorkItem(new WaitCallback(util.Run));

In the Run method of the Microsoft.SharePoint.Diagnostics.SPMergeLogFileUtil class:

public void Run(object stateInfo)
    this.Progress = 0;
    // Creates the diagnostic log files document library in central admin (if does not yet exist) via the GetLogFilesList method
    // Add a new subfolder (having GUID name) to the library. If the folder already exists, deletes its content.
    // Executes child jobs (SPMergeLogFilesJobDefinition) on each farm member, see more details about it later below
    List<string> jobs = this.DispatchCollectingJobs();
    // Waits for all child jobs to be finished on the farm members
    // Merges the content of the collected files from the central admin document library into the specified local file system folder by calling the MergeFiles method
    // Finally deletes the temporary files one by one from the central admin document library and at the very end deletes their folder as well
  catch (Exception exception)
    this.Error = exception;
    this.Progress = 100;

Yes, as you can see, if there is any error in the file collection process on any of the farm members, or the merging process fails, the files won’t be deleted from the Diagnostics Log Files document library.

Instead of the RunMerge method, the deletion process would have probably a better place in the finally block, or at least, in the catch block one should check if the files were removed successfully.


A few words about the Microsoft.SharePoint.Diagnostics.SPMergeLogFilesJobDefinition, as promised earlier. Its Execute method calls the CollectLogFiles method that creates an ULSLogFileProcessor instance based on the requested log filter and gets the corresponding ULS entries from the farm member the job running on, stores the entries in a temporary file in the file system, uploads them to the actual subfolder (GUID name) of the Diagnostics Log Files document library (file name pattern: [ServerLocalName] (1).log.gz or [ServerLocalName].log.gz if a single file is enough to store the log entries from the server), and delete the local temporary file.


A few related text values can we read via PowerShell as well:

In the getter of the DisplayName property in the SPMergeLogFilesJobDefinition class returns:

SPResource.GetString("CollectLogsJobTitle", new object[] { base.Server.DisplayName });

reading the value via PowerShell:



Collection of log files from server |0

In the GetLogFilesList method of the SPMergeLogFileUtil class we find the title and description of the central admin document library used for log merging

string title = SPResource.GetString(SPGlobal.ServerCulture, "DiagnosticsLogsTitle", new object[0]);
string description = SPResource.GetString(SPGlobal.ServerCulture, "DiagnosticsLogsDescription", new object[0]);

Reading the values via the PowerShell:



Diagnostics Log Files



View and analyze merged log files from the all servers in the farm

These values supports our theory, that the library was created and filled by these methods.


Next, I’ve used the following PowerShell script to look for the failed merge jobs in the time range when the files in the Central Administration have been created:

$farm = Get-SPFarm
$from = "3/21/2015 12:00:00 AM"
$to = "3/21/2015 6:00:00 AM"
$farm.TimerService.JobHistoryEntries | ? {($_.StartTime -gt $from) -and ($_.StartTime -lt $to) -and ($_.Status -eq "Failed") -and ($_.JobDefinitionTitle -like "Collection of log files from server *")}

The result:


As you can see, in our case there were errors on both farm member servers, for example, the storage space was not enough on one of them. After the jobs have failed, the aggregated files were not deleted from the Diagnostics Log Files document library of the Central Administration.

Since even a successful execution of the Merge-SPLogFile cmdlet can temporarily increase the size of the Central Administration content database considerably, and the effect of the failed executions is not only temporary (and particularly large, if it happens several times and is combined with verbose logging), SharePoint administrators should be aware of these facts, consider them when planning database sizes and / or take an extra maintenance step to remove the rest of failed merge processes from the Diagnostics Log Files document library regularly. As far as I see, the issue hits SharePoint 2010 and 2013 as well.

March 26, 2015

Strange Localization Issue When Working with List and Site Templates

Filed under: MUI, PowerShell, SP 2013 — Tags: , — Peter Holpar @ 23:44

One of our clients runs a localized version of SharePoint 2013. The operating system is a Windows 2012 R2 Server (English), the SharePoint Server itself is English as well. The German language pack is installed and sites and site collections were created in German. We are working with various custom site templates. Recently one of these templates had to be extended with a Task-list based custom lists (called ToDos). The users prepared the list in a test site, and we saved the list as a template. We created a new site using the site template (we will refer to this site later as prototype), and next we created a new list based on the custom list template. Finally, we saved the altered web site as site template, including content using the following PowerShell commands:

$web = Get-SPWeb $siteTemplateSourceUrl
$web.SaveAsTemplate($siteTemplateName, $siteTemplateTitle, $siteTemplateDescription, 1)

We created a test site using the new site template, and everything seemed to be OK. However, after a while, the users started to complain, that a menu for the new list contains some English text as well. As it turned out, some of the views for the new list were created with English title:



First, we verified the manifest.xml of the list template, by downloading the .stp file (that has a CAB file format) and opening it using IZArc. We found, that the DisplayName property of the default view (“Alle Vorgänge” meaning “All Tasks”) and a custom datasheet view (called “db”, stands for “Datenblatt”) contains the title as text, the DisplayName property of the other views contains a resource reference (like “$Resources:core,Late_Tasks;”).


Next, we downloaded the site template (the .wsp file has also a CAB file format, and can be opened by IZArc), and verified the schema.xml for the ToDos list. We found, that original, German texts (“Alle Vorgänge” and “db”) were kept, however, all other view names were “localized” to English.


At this point I guessed already, that problem was caused by the local of the thread the site template exporting code was run in. To verify my assumption, I saved the prototype site from the site settings via the SharePoint web UI (that is German in our case). This time the resulting schema.xml in the new site template .wsp contained the German titles:


We got the same result (I mean German view titles) if we called our former PowerShell code by specifying German as the culture for the running thread. See more info about the Using-Culture helper method, SharePoint Multilingual User Interface (MUI) and PowerShell here:

Using-Culture de-DE 
  $web = Get-SPWeb $pwsSiteTemplateSourceUrl
  $web.SaveAsTemplate($siteTemplateName, $siteTemplateTitle, $siteTemplateDescription, 1)

We’ve fixed the existing views via the following PowerShell code (Note: Using the Using-Culture helper method is important in this case as well. We have only a single level of site hierarchy in this case, so there is no recursion in code!):

$web = Get-SPWeb http://SharePointServer

function Rename-View-IfExists($list, $viewNameOld, $viewNameNew)
  $view =  $list.Views[$viewNameOld]
  If ($view -ne $Null) {
      Write-Host Renaming view $viewNameOld to $viewNameNew
      $view.Title = $viewNameNew
  Else {
    Write-Host View $viewNameOld not found

Using-Culture de-DE {
  $web.Webs | % {
    $list = $_.Lists["ToDos"]
    If ($list -ne $Null) {
      Write-Host ToDo list found in $_.Title
      Rename-View-IfExists $list "Late Tasks" "Verspätete Vorgänge"
      Rename-View-IfExists $list "Upcoming" "Anstehend"
      Rename-View-IfExists $list "Completed" "Abgeschlossen"
      Rename-View-IfExists $list "My Tasks" "Meine Aufgaben"
      Rename-View-IfExists $list "Gantt Chart" "Gantt-Diagramm"
      Rename-View-IfExists $list "Late Tasks" "Verspätete Vorgänge" 

Strange, that we had no problem with field names or other localizable texts when worked with the English culture.

March 24, 2015

How to Read the Values of Fields bound to Lookup Tables via REST

Filed under: JavaScript, Project Server, REST — Tags: , , — Peter Holpar @ 22:51

In my recent post I’ve illustrated how to read the values of Enterprise Custom Fields (ECT) that are bound to Lookup Tables. I suggest you to read that post first, as it can help you to better understand the relations between the custom field values and the internal names of the lookup table entries.

In this post I show you how to read such values using the REST interface. Instead of C# I use JavaScript in this example. The sample code is using the version 3.0.3-Beta4 of the LINQ for JavaScript library (version is important, as this version contains lower case function names in contrast to the former stable relases!) and the version 1.8.3 of jQuery.

Assuming that these scripts are all located in the PSRESTTest/js subfolder in the Layouts folder, we can inject them via a Script Editor Web Part using this HTML code:

<script type="text/ecmascript" src="/_layouts/15/PSRESTTest/js/jquery-1.8.3.min.js"></script>
<script type="text/ecmascript" src="/_layouts/15/PSRESTTest/js/linq.min.js"></script>
<script type="text/ecmascript" src="/_layouts/15/PSRESTTest/js/GetCustFieldREST.js"></script>

In our GetCustFieldREST.js script we define the String.format helper function first:

String.format = (function () {
    // The string containing the format items (e.g. "{0}")
    // will and always has to be the first argument.
    var theString = arguments[0];

    // start with the second argument (i = 1)
    for (var i = 1; i < arguments.length; i++) {
        // "gm" = RegEx options for Global search (more than one instance)
        // and for Multiline search
        var regEx = new RegExp("\\{" + (i – 1) + "\\}", "gm");
        theString = theString.replace(regEx, arguments[i]);

    return theString;

Another helper function supports sending fault-tolerant REST-queries:

function sendRESTQuery(queryUrl, onSuccess, retryCount) {

    var retryWaitTime = 1000; // 1 sec.
    var retryCountMax = 3;
    // use a default value of 0 if no value for retryCount passed
    var retryCount = (retryCount != undefined) ? retryCount : 0;

    //$.support.cors = true; // enable cross-domain query
        //beforeSend: function (request) {
        //    request.withCredentials = false;
        type: ‘GET’,
        //xhrFields: { withCredentials: false },
        contentType: ‘application/json;odata=verbose’,
        url: baseUrl + queryUrl,
        headers: {
            ‘X-RequestDigest': $(‘#__REQUESTDIGEST’).val(),
            "Accept": "application/json; odata=verbose"
        dataType: "json",
        complete: function (result) {
            var response = JSON.parse(result.responseText);
            if (response.error) {
                if (retryCount <= retryCountMax) {
                    window.setTimeout(function () { sendRESTQuery(queryUrl, onSuccess, retryCount++) }, retryWaitTime);
                else {
                    alert("Error: " + response.error.code + "\n" + response.error.message.value);
            else {
                bgDetails = response.d;


The baseUrl variable holds the root of the REST endpoint for Project Server. I assume your PWA site is provisioned to the PWA managed path.

var baseUrl = String.format("{0}//{1}/PWA/_api/ProjectServer/", window.location.protocol,;

We call the getResCustProp method when the page is loaded:


In the getResCustProp method I first query the InternalName of the custom field, as well as the InternalName and the FullValue properties of the lookup entries of the corresponding lookup table. In a second query I read the custom field value for the specified enterprise resource, and compare the value (or values) stored in the field with the InternalName property of the lookup table entries from the first query. Note, that we should escape the underscore in the InternalName property, and should use the ‘eval’ JavaScript function, as we don’t know the name of the custom field (that is the name of the property) at design time.

function getResCustProp() {
    var resourceName = "Brian Cox";
    var fieldName = "ResField";
    var fieldQueryUrl = String.format("CustomFields?$expand=LookupTable/Entries&$filter=Name eq ‘{0}’&$select=InternalName,LookupTable/Entries/InternalName,LookupTable/Entries/FullValue", fieldName);
    sendRESTQuery(fieldQueryUrl, function (fieldResponseData) {
        var field = fieldResponseData.results[0];
        var lookupTableEntries = Enumerable.from(field.LookupTable.Entries.results);
        var resourceQuery = String.format("EnterpriseResources?$filter=Name eq ‘{0}’&$select={1}", resourceName, field.InternalName)
        sendRESTQuery(resourceQuery, function (resourceResponseData) {
            var resource = resourceResponseData.results[0];
            var encodedInternalName = field.InternalName.replace(‘_’, ‘_x005f_’);
            var fieldValue = eval("resource." + encodedInternalName);
            Enumerable.from(fieldValue.results).forEach(function (fv) {
                var entry = lookupTableEntries.first(String.format(‘$="{0}"’, fv));
                alert(String.format("Value: ‘{0}’, Entry: ‘{1}’", entry.FullValue, fv));

Of course, if you would like to use the code in production, you should add further data validation (if there is any resource and custom field returned by the queries, etc.) and error handling to the method.

March 23, 2015

How to Read the Values of Fields bound to Lookup Tables via the Client Object Model

Filed under: Managed Client OM, Project Server — Tags: , — Peter Holpar @ 05:16

Assume you have an Enterprise Custom Field (let’s call this ECFResField’) defined for project resources, that is bound to a Lookup Table.

How can we read the textural values of the field as it is assigned to you resources? After all, what makes reading such values makes any different than getting field values without lookup tables?

If we have a look at a resource with a lookup table based custom field via an OData / REST query (for example, by http://YourProjectServer/pwa/_api/ProjectServer/EnterpriseResources), you can see, that the value is stored as a reference, like ‘’Entry_4d65d905cac9e411940700505634b541‘.


If we access the value via the managed client OM, we get it as a string array, even if only a single value can be selected from the lookup table. The reference value in the array corresponds to the value in the InternalName property of the lookup table entry. If we know the ID of the resource (we want to read the value from), the enterprise custom field (that means we know its internal name as well) and the related lookup table, we can get the result in a single request as shown below:

  1. using (var projectContext = new ProjectContext(pwaUrl))
  2. {
  3.     var lookupTableId = "4c65d905-cac9-e411-9407-00505634b541";
  4.     var resourceId = "f9497d1d-9145-e411-9407-00505634b541";
  6.     var res = projectContext.EnterpriseResources.GetById(resourceId);
  7.     var lt = projectContext.LookupTables.GetById(lookupTableId);
  8.     var cfInternalName = "Custom_80bd269ecbc9e411940700505634b541";
  9.     projectContext.Load(res, r => r[cfInternalName]);
  10.     projectContext.Load(lt.Entries);
  11.     projectContext.ExecuteQuery();
  13.     var valueEntries = res[cfInternalName] as string[];
  14.     if (valueEntries != null)
  15.     {
  16.         foreach (var valueEntry in valueEntries)
  17.         {
  18.             var lookupText = lt.Entries.FirstOrDefault(e => e.InternalName == valueEntry) as LookupText;
  19.             var ltValue = (lookupText != null) ? lookupText.Value : null;
  20.             Console.WriteLine("Value: '{0}' (Entry was '{1}')", ltValue, valueEntry);
  21.         }
  22.     }
  23. }

However, if these values are unknown, and we know only the name of the resource and the field, we need to submit an extra request to get the IDs for the second step:

  1. using (var projectContext = new ProjectContext(pwaUrl))
  2. {
  3.     var resourceName = "Brian Cox";
  4.     var fieldName = "ResField";
  6.     projectContext.Load(projectContext.EnterpriseResources, ers => ers.Include(r => r.Id, r => r.Name));
  7.     projectContext.Load(projectContext.CustomFields, cfs => cfs.Include(f => f.Name, f => f.InternalName, f => f.LookupTable.Id, f => f.LookupEntries));
  9.     projectContext.ExecuteQuery();
  11.     var resourceId = projectContext.EnterpriseResources.First(er => er.Name == resourceName).Id.ToString();
  12.     var cf = projectContext.CustomFields.First(f => f.Name == fieldName);
  13.     var cfInternalName = cf.InternalName;
  14.     var lookupTableId = cf.LookupTable.Id.ToString();
  16.     var res = projectContext.EnterpriseResources.GetById(resourceId);
  17.     var lt = projectContext.LookupTables.GetById(lookupTableId);
  19.     projectContext.Load(res, r => r[cfInternalName]);
  20.     projectContext.Load(lt.Entries);
  21.     projectContext.ExecuteQuery();
  23.     var valueEntries = res[cfInternalName] as string[];
  24.     if (valueEntries != null)
  25.     {
  26.         foreach (var valueEntry in valueEntries)
  27.         {
  28.             var lookupText = lt.Entries.FirstOrDefault(e => e.InternalName == valueEntry) as LookupText;
  29.             var ltValue = (lookupText != null) ? lookupText.Value : null;
  30.             Console.WriteLine("Value: '{0}' (Entry was '{1}')", ltValue, valueEntry);
  31.         }
  32.     }
  33. }

Note: although this post was about a custom field defined for the resource entity, you can apply the same technique for project and task fields as well.

March 13, 2015

Strange Access Denied Error in Project Server Client OM

Filed under: Managed Client OM, Project Server — Tags: , — Peter Holpar @ 22:42

Recently I worked with the Managed Client Object Model of Project Server 2013. Although being Farm Administrator, Site Collection Administrator (Site Owner) of every site collections and member of the Administrators group in Project Server, I received an Access Denied error (UnauthorizedAccessException) when executing even the simplest queries like this one:

  1. using (var projectContext = new ProjectContext(pwaUrl))
  2. {
  3.     var projects = projectContext.Projects;
  4.     projectContext.Load(projects);
  5.     projectContext.ExecuteQuery();
  6. }

The error message was:

Access denied. You do not have permission to perform this action or access this resource.

Error code: –2147024891


No entries corresponding the error or the correlation ID in the response was found in the ULS logs.

The real problem was the pwaUrl parameter in the ProjectContext constructor: by mistake I used the URL of the root site (like http://projectserver) instead of the URL of the PWA site (http://projectserver/PWA).

Older Posts »

The Shocking Blue Green Theme. Create a free website or blog at


Get every new post delivered to your Inbox.

Join 55 other followers