Second Life of a Hungarian SharePoint Geek

April 30, 2018

Faking feature activation properties

Filed under: Features, Reflection, SP 2013, Tips & Tricks — Tags: , , , — Peter Holpar @ 21:19

Assume you have a feature receiver in your SharePoint project. You would like to perform multiple actions if the feature is activated, so you organize your code according to the actions into static methods of a helpers class, like shown in the code snippet below:

  1. [Guid("8cb098ae-2017-4fff-8a53-b315abb85d79")]
  2. public class YourFeatureReceiver : SPFeatureReceiver
  3. {
  4.     public override void FeatureActivated(SPFeatureReceiverProperties properties)
  5.     {
  6.         SomeHelperClass.DoSomethingOnTheSite(properties);
  7.         SomeHelperClass.DoSomethingOtherOnTheSite(properties);
  8.         SomeHelperClass.AndFinallyDoThat(properties);
  9.     }
  10. }

Note: In this case, it is a site level feature, and we handle the FeatureActivated event, but you can apply the same technique for other kinds of feature receivers and events as well.

A helper method has the following signature, and contains code like this one:

  1. public static void DoSomethingOnTheSite(SPFeatureReceiverProperties properties)
  2. {
  3.     SPSite site = properties.Feature.Parent as SPSite;
  4.     if (site != null)
  5.     {
  6.         // do something here
  7.     }
  8. }

You would like to test the functionality of your method one-by-one from a console application, without actually having to deploy your SharePoint solution an activate your feature each time again (of course, you need at least an initial deployment of the whole solution). In my case, the helper class is included in a separate assembly, but even if it is the assembly of your SharePoint project, you should only deploy the new version of the assembly into the Global Assembly Cache (GAC).

As you can see in the method above, we use the Feature property of the SPFeatureReceiverProperties class we received in the properties parameter. Of course, the Feature property is read-only, so we need some tricks, to be able to pass the SPFeatureReceiverProperties parameter populated with the correct SPFeature in its Feature property to the method. We create first a new instance of the SPFeatureReceiverProperties class, than query the site for the feature based on its ID (se my important notice about this ID after the code block!). We can inject this SPFeature instance into our formerly crated SPFeatureReceiverProperties instance using Reflection, via the internal SetFeature method of the SPFeatureReceiverProperties class. Finally, we can invoke our helper method with the already-populated properties variable.

  1. using (SPSite site = new SPSite(yourSiteUrl))
  2. {
  3.     var properties = new SPFeatureReceiverProperties();
  4.  
  5.     var yourFeatureId = new Guid("51bf2a39-b527-46cf-abd6-39aaf1dcd19b");
  6.  
  7.     var feature = site.Features.FirstOrDefault(f => f.DefinitionId == yourFeatureId);
  8.     if (feature != null)
  9.     {
  10.         MethodInfo mi_setFeature = typeof(SPFeatureReceiverProperties).GetMethod("SetFeature",
  11.             BindingFlags.NonPublic | BindingFlags.Instance);
  12.  
  13.         if (mi_setFeature != null)
  14.         {
  15.             mi_setFeature.Invoke(properties, new object[] { feature });
  16.             SomeHelperClass.DoSomethingOnTheSite(properties);
  17.         }
  18.     }
  19. }

Note: It is important to notice, that the Guid we used in the feature-lookup above, comparing it to the DefinitionId property, is not the same, as the Guid we saw earlier in the feature receiver code snippet. The latter one is used only by Visual Studio during the deployment process to find up the correct ReceiverClass for the event receiver. The Guid we need, the ID of the feature is available on the Manifest tab of the feature as illustrated below:

image

Applying this technique made our development and testing process a lot faster.

Creating statistics about web part usage from the SharePoint content database

Filed under: PowerShell, Reflection, SP 2013, SQL, Web part — Tags: , , , — Peter Holpar @ 21:08

Recently I had to create some statistics about SharePoint web site customizations, like on which pages are there Script Editor Web Parts, or Content Editor Web Parts, etc. I knew I could and probably should have done it by iterating through all web sites, all pages and then looking up the web parts on each page using SPLimitedWebPartManager class, but I was aware, the same information should be available via the content database as well, making it possible to query the info much easier and faster, although unsupported. In this post I describe, how you can do it, but use the solution at your own risk.

The web part information is stored in the AllWebParts table, the information about the pages in the AllDocs table. I joined these tables together for the first report about the Script Editor Web Parts.

SELECT AD.DirName + ‘/’ + AD.LeafName as PageUrl, AWP.tp_ZoneID as ZoneId, AWP.tp_PartOrder as WebPartOrder, AWP.tp_Class AS WebPartClass
FROM
AllWebParts AWP (nolock)
INNER JOIN AllDocs (nolock) AD ON AWP.tp_SiteId = AD.SiteId AND AWP.tp_PageUrlID = AD.Id
WHERE tp_Class LIKE ‘%ScriptEditorWebPart’

Next, I was to create a report about the Content Editor Web Parts, using a filter like:

WHERE tp_Class = ‘%ContentEditorWebPart’

However, no result found, although I was pretty sure, there are a lot of them in our web site. How is it possible?

To test it further, I’ve included a Script Editor Web Part and a Content Editor Web Part on the AllItems.aspx page of the Tasks list in one of our sub-site, and created a new query with the filter below:

WHERE DirName LIKE ‘%site/subsite/Lists/Tasks%’
AND LeafName LIKE ‘%AllItem%’

This was the result:

image

As you see, the Script Editor Web Part is there, and you see two further web parts (they should be the Content Editor Web Part and the XsltListViewWeb part, that was originally on the page and is responsible to display the task items in the list), however both of them with a NULL value in the WebPartClass column. What should it mean?

I have studied the structure of the AllWebParts table and the relations of its fields further, and found that there are two fields (tp_Class and tp_Assembly) that are always populated for the records, where the WebPartClass is not NULL, and there is a field called tp_WebPartTypeId – populated for each entries, even for those, where the WebPartClass , tp_Class and tp_Assembly fields are empty – that we could eventually use to find the matching web parts. But how? I made a search for ‘WebPartTypeId’ using .NET Reflector, and found the internal class Microsoft.SharePoint.WebPartPages.WebPartTypeInfo, having a private static method called GetWellKnownTypeIdDictionary that returns a Dictionary<Guid, Type> mapping Guids (WebPartTypeIds) to the actual web part type. Remark: The Guids in the WebPartTypeId are actually created from the MD5 hash of the bytes of the joined full assembly name and web part class name, see the internal static  GetTypeIdUnsafe(MD5HashProvider md5Provider, string typeFullName, string assemblyName) method of the internal sealed class Microsoft.SharePoint.ApplicationRuntime.SafeControls.

image

To support those so called well-known types in my former SQL-query, I wrote a short PowerShell script that invokes the private static GetWellKnownTypeIdDictionary method of the internal WebPartTypeInfo class, and emits the resulting Dictionary to a text file I can use to extend my query:

  1. $webPartTypeInfoType = [System.Type]::GetType('Microsoft.SharePoint.WebPartPages.WebPartTypeInfo, Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c')
  2. $mi_GetWellKnownTypeIdDictionary = $webPartTypeInfoType.GetMethod('GetWellKnownTypeIdDictionary', [Reflection.BindingFlags]'NonPublic, Static')
  3. $wellKnownTypeIdDictionary = $mi_GetWellKnownTypeIdDictionary.Invoke($null, $null)
  4.  
  5. $wpTypes = $wellKnownTypeIdDictionary.Keys | % { "INSERT INTO @WPTypes VALUES ('$_', '$($wellKnownTypeIdDictionary[$_].Assembly.FullName)', '$($wellKnownTypeIdDictionary[$_].FullName)')" }
  6. Set-Content -Path 'C:\Data\WPTypes.txt' -Value $wpTypes

And that is already the extended version of the SQL query:

  1. DECLARE @WPTypes TABLE
  2.    (
  3.      Id uniqueidentifier NOT NULL,
  4.      AssemblyName varchar(500),
  5.      ClassName varchar(100)
  6.    )
  7.  
  8. INSERT INTO @WPTypes VALUES ('8e20cf70-0fd5-1e08-9972-38f63a6bd59a', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ImageWebPart')
  9. INSERT INTO @WPTypes VALUES ('ba009853-eac3-16c8-9094-a8834485ad33', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.DataFormWebPart')
  10. INSERT INTO @WPTypes VALUES ('83216ab2-cd0e-e9fc-fc5e-6a8f3b21c37b', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.DataViewWebPart')
  11. INSERT INTO @WPTypes VALUES ('42fddde2-e0cf-c8ab-48b7-db1fcac0a917', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ListFormWebPart')
  12. INSERT INTO @WPTypes VALUES ('05d0fd94-372a-5ee7-b480-ccb8f9cd2c23', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ListViewWebPart')
  13. INSERT INTO @WPTypes VALUES ('aef28218-44f8-0538-9805-4842c0e62811', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.XsltListFormWebPart')
  14. INSERT INTO @WPTypes VALUES ('a6524906-3fd2-ee4e-23ee-252d3c6e0dc9', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.XsltListViewWebPart')
  15. INSERT INTO @WPTypes VALUES ('0c6143a7-d68b-bade-e0ef-2c4d01182b0c', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.BlogAdminWebPart')
  16. INSERT INTO @WPTypes VALUES ('afef48e1-8f94-eb71-03a6-ffceb685306a', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.BlogMonthQuickLaunch')
  17. INSERT INTO @WPTypes VALUES ('4c06cea2-364f-47e3-e1d7-08d53f441157', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ContentEditorWebPart')
  18. INSERT INTO @WPTypes VALUES ('e6047383-438e-ed87-1a93-f1ff71729044', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.TitleBarWebPart')
  19. INSERT INTO @WPTypes VALUES ('707c1e73-0b3d-898b-c755-01621802ab8c', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.SilverlightWebPart')
  20. INSERT INTO @WPTypes VALUES ('28c23aec-2537-68b3-43b6-845b13cea19f', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  21. INSERT INTO @WPTypes VALUES ('8d6034c4-a416-e535-281a-6b714894e1aa', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  22. INSERT INTO @WPTypes VALUES ('8e814083-396a-e7d1-148b-316e3a7283f7', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  23. INSERT INTO @WPTypes VALUES ('e6377261-6920-bbfe-501f-fda7a61db10f', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  24. INSERT INTO @WPTypes VALUES ('8efd140d-eae9-5feb-06e3-f771842d2e43', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  25. INSERT INTO @WPTypes VALUES ('b3294a07-46bf-e661-d036-10670590bbd3', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.SPUserCodeWebPart')
  26.  
  27. SELECT AD.DirName + '/' + AD.LeafName as PageUrl, AWP.tp_ZoneID as ZoneId, AWP.tp_PartOrder as WebPartOrder, ISNULL(AWP.tp_Class, WPT.ClassName) AS WebPartClass
  28. FROM AllWebParts AWP (nolock)
  29. INNER JOIN AllDocs AD (nolock) ON AWP.tp_SiteId = AD.SiteId AND AWP.tp_PageUrlID = AD.Id
  30. LEFT JOIN @WPTypes WPT ON AWP.tp_WebPartTypeId = WPT.Id
  31. WHERE ISNULL(AWP.tp_Class, WPT.ClassName) LIKE '%ContentEditorWebPart'

Of course, you can change the conditions of the query as you like, for example, you can restrict it to two web part type, like:

WHERE ISNULL(AWP.tp_Class, WPT.ClassName) IN (‘Microsoft.SharePoint.WebPartPages.ScriptEditorWebPart’, ‘Microsoft.SharePoint.WebPartPages.ContentEditorWebPart’)

There are a few more columns in the AllWebParts table, that you eventually would include either in the SELECT statement or in its WHERE clause, these are:

  • tp_IsIncluded: The web part is displayed on the page, if the value is 1 (default). If you close (not delete!) a web part, the value is 0. Deleted web parts are removed from the table.
  • tp_Deleted: Assume you have a list with some pages that includes web parts, like view pages including XsltListViewWebPart instances. The web part entries in the AllWebParts table have a value of 0 at this stage. If you delete the list, these values change to 1. The web part entries will be kept in the table even after deleting the list from the first (user) level Recycle Bin, and removed only after the list is deleted from the second (site collection) level Recycle Bin.
  • tp_ListId: This is a field that is populated for list-related built-in web parts, like XsltListViewWebPart. You can look up the related list and web instances by joining the Lists and Webs views in your query respectively, as shown below (this time I omit the declaration of the @WPTypes variable and its population with value for the sake of brevity, but of course, you need it this time either):

SELECT L.tp_Title as ListTitle, W.FullUrl AS WebUrl, AD.DirName + ‘/’ + AD.LeafName as PageUrl, AWP.tp_ZoneID as ZoneId, AWP.tp_PartOrder as WebPartOrder, ISNULL(AWP.tp_Class, WPT.ClassName) AS WebPartClass, tp_ListId
FROM AllWebParts AWP (nolock)
INNER JOIN AllDocs AD (nolock) ON AWP.tp_SiteId = AD.SiteId AND AWP.tp_PageUrlID = AD.Id
LEFT JOIN @WPTypes WPT ON AWP.tp_WebPartTypeId = WPT.Id
LEFT JOIN Lists L (nolock) ON AWP.tp_SiteId = L.tp_SiteId AND AWP.tp_ListId = L.tp_ID
LEFT JOIN Webs W (nolock) ON AWP.tp_SiteId = W.SiteId AND L.tp_WebId = W.Id
WHERE ISNULL(AWP.tp_Class, WPT.ClassName) LIKE ‘%ListViewWebPart’

By including the list title or the web URL in the WHERE clause (or the ID of the list or the web if you wish) you can further limit the items returned by the query.

If there are records returned with NULL in the ListTitle and WebUrl columns it means typically that the list was deleted, but yet available in the Recycle Bin. See my comments regarding the tp_Deleted field above. Note, that despite the name of the FullUrl column in the Web view, it is actually a server relative URL.

I hope this overview has helped you to better understand what and how is stored in these tables of the SharePoint content database, as well, how the “magical” IDs of the well-known web part types do fit into the whole picture.

March 23, 2018

Changing SharePoint Search Preferences from Code

Filed under: PowerShell, Reflection, Search, SP 2013 — Tags: , , , — Peter Holpar @ 22:22

In my recent post I’ve illustrated with C# and PowerShell examples, how to read search preferences info from code, both for the current user as well as for other users. In this post we will see, how to change that preference from code.

We will use the same object, the Microsoft.Office.Server.Search.Administration.UserPreference class. Unlike its static GetUserPreference method, the either static SetUserPreference method has no overload that accepts the SharePoint context (SPContext) as parameter. The single overload of this method accepts a UserPreference instance. That makes our life not easier when it comes later to changing the preferences for another user. Don’t lose the hope, it is not impossible. Bur first things first.

Once we received a UserPreference instance via the GetUserPreference method, you should change certain preference properties, like OpenDocumentsInClient. How to do it? There are two methods, EnableSettings and DisableSetting (both having a parameter of  the nested enumeration type Settings) defined in the UserPreference class. If you would like to activate a setting, you should call the EnableSettings method, if you need to deactivate it, call the DisableSetting method, then finally invoke the SetUserPreference method to persist the changes. For example:

  1. var userPref = UserPreference.GetUserPreference();
  2. // if you would like to open documents in Office client, like Word or Excel
  3. userPref.UpdateSetting(UserPreference.Settings.OpenDocumentsInClient, true);
  4. // if you would like to open documents in Browser (Office Web Apps)
  5. userPref.UpdateSetting(UserPreference.Settings.OpenDocumentsInClient, false);
  6. UserPreference.SetUserPreference(userPref);

Note: This code works only if  you try it within a SharePoint context, like on an application page or in web part. In a console application you will receive an exception when you invoke the overload of the GetUserPreference method without the SPContext parameter:

ArgumentNullException
The value must not be null.
Parameter name: SPContext.Current

As we’ve already seen, there is an overload of the GetUserPreference method that accepts a SPContext parameter, so you could use that to get the preferences, but as there is no such overload for the SetUserPreference method, at least at this points will be the same type of exception thrown again. We will revisit the question shortly, how to set your own preferences from a console application, but we make a quick detour first.

To tell the truth, I don’t like the above pattern at all. Instead of these two methods I created an extension method with a Boolean parameter that encapsulates the functionality:

  1. public static void UpdateSetting(this UserPreference userPreference, UserPreference.Settings setting, bool value)
  2. {
  3.     if (value)
  4.     {
  5.         userPreference.EnableSetting(setting);
  6.     }
  7.     else
  8.     {
  9.         userPreference.DisableSetting(setting);
  10.     }
  11. }

Using this new method one can enable / disable preference settings like:

  1. var userPref = UserPreference.GetUserPreference();
  2. // if you would like to open documents in Office client, like Word or Excel
  3. userPref.EnableSetting(UserPreference.Settings.OpenDocumentsInClient);
  4. // if you would like to open documents in Browser (Office Web Apps)
  5. userPref.DisableSetting(UserPreference.Settings.OpenDocumentsInClient);
  6. UserPreference.SetUserPreference(userPref);

Back to the question, how to set your own preferences when the code runs without SharePoint context, like from a console application?

The “trivial” way is to fake a SharePoint context, using the method described here:

  1. using (SPSite site = new SPSite(url))
  2. {
  3.     using (SPWeb web = site.OpenWeb())
  4.     {
  5.         HttpRequest request = new HttpRequest(string.Empty, url, string.Empty);
  6.  
  7.         HttpResponse response = new HttpResponse(new System.IO.StreamWriter(new System.IO.MemoryStream()));
  8.  
  9.         HttpContext ctx = new HttpContext(request, response);
  10.         ctx.Items["HttpHandlerSPWeb"] = web;
  11.         HttpContext.Current = ctx;
  12.  
  13.         Console.WriteLine(SPContext.Current.Web.CurrentUser.LoginName);
  14.  
  15.         var userPref = UserPreference.GetUserPreference();
  16.         userPref.UpdateSetting(UserPreference.Settings.OpenDocumentsInClient, true);
  17.         UserPreference.SetUserPreference(userPref);
  18.  
  19.         //set back the original context (e.g. null)
  20.         HttpContext.Current = null;
  21.     }
  22. }

Another option is, to try to understand, how the SetUserPreference method internally works. It turns out, that it call the internal static UpdatePreference method:

UpdatePreference(preference, false, SPContext.Current);

So I’ve created just another extension method that wraps invoking the UpdatePreference method using Reflection:

  1. public static void Update(this UserPreference userPreference, bool fClearClickHistory, SPContext context)
  2. {
  3.     Type[] paramTypes = { typeof(UserPreference), typeof(bool), typeof(SPContext) };
  4.     MethodInfo updatePreference = userPreference.GetType().GetMethod("UpdatePreference", BindingFlags.Static | BindingFlags.NonPublic, null, paramTypes, null);
  5.     object[] parameters = { userPreference, fClearClickHistory, context };
  6.     updatePreference.Invoke(null, parameters);
  7. }

And a further helper method that accepts a SPContext object as parameter, writes out, preferences of which user we are to change and performs the change itself via the methods we have already:

  1. private void UpdatePreference(SPContext ctx)
  2. {
  3.     var user = ctx.Web.CurrentUser;
  4.     Console.WriteLine("Setting preferences for '{0}'", user.LoginName);
  5.  
  6.     var userPref = UserPreference.GetUserPreference(false, ctx);
  7.     userPref.UpdateSetting(UserPreference.Settings.OpenDocumentsInClient, false);
  8.     userPref.Update(false, ctx);
  9. }

I think the code we achieved using this extension method is much more readable as the former one with the dummy context:

  1. using (SPSite site = new SPSite(url))
  2. {
  3.     using (SPWeb web = site.OpenWeb())
  4.     {
  5.         var ctx = SPContext.GetContext(web);
  6.         UpdatePreference(ctx);
  7.     }
  8. }

The code snippets until this point have effect only on the current user. How to change the settings for other user? That is possible either, as soon we combine the methods we already have with impersonation.

First, the version that uses the dummy context:

  1. using (SPSite site = new SPSite(url))
  2. {
  3.     using (SPWeb web = site.OpenWeb())
  4.     {
  5.         var user = web.EnsureUser(@"i:0#.w|domain\user");
  6.  
  7.         SPSecurity.RunWithElevatedPrivileges(
  8.           () =>
  9.           {
  10.               using (SPSite impSite = new SPSite(url, user.UserToken))
  11.               using (SPWeb impWeb = impSite.OpenWeb())
  12.               {
  13.                   HttpRequest request = new HttpRequest(string.Empty, url, string.Empty);
  14.  
  15.                   HttpResponse response = new HttpResponse(new System.IO.StreamWriter(new System.IO.MemoryStream()));
  16.  
  17.                   HttpContext impersonatedContext = new HttpContext(request, response);
  18.  
  19.                   impersonatedContext.Items["HttpHandlerSPWeb"] = impWeb;
  20.  
  21.                   HttpContext.Current = impersonatedContext;
  22.  
  23.                   Console.WriteLine(SPContext.Current.Web.CurrentUser.LoginName);
  24.  
  25.                   var userPref = UserPreference.GetUserPreference();
  26.                   userPref.UpdateSetting(UserPreference.Settings.OpenDocumentsInClient, true);
  27.                   UserPreference.SetUserPreference(userPref);
  28.  
  29.                   //set back the original context (e.g. null)
  30.                   HttpContext.Current = null;
  31.               }
  32.           });
  33.     }
  34. }

Next, the other version using Reflection:

  1. using (SPSite site = new SPSite(url))
  2. {
  3.     using (SPWeb web = site.OpenWeb())
  4.     {
  5.         var user = web.EnsureUser(@"i:0#.w|domain\user");
  6.  
  7.         SPSecurity.RunWithElevatedPrivileges(
  8.           () =>
  9.           {
  10.               using (SPSite impSite = new SPSite(url, user.UserToken))
  11.               using (SPWeb impWeb = impSite.OpenWeb())
  12.               {
  13.                   var impCtx = SPContext.GetContext(impWeb);
  14.                   UpdatePreference(impCtx);
  15.               }
  16.           });
  17.     }
  18. }

Mission completed.

For those of you who would like to have the same functionality from PowerShell (of course, there are no SharePoint context inherited from the process at all), I include the equivalent methods below.

These are the helper methods we rely on:

  1. function UpdateSetting($userPreference, $setting, $value) {
  2.     If ($value)
  3.     {
  4.         $userPreference.EnableSetting($setting)
  5.     }
  6.     Else
  7.     {
  8.         $userPreference.DisableSetting($setting)
  9.     }
  10. }
  11.  
  12. function Update($userPreference, $fClearClickHistory, $context) {
  13.     $paramTypes = ($up, [bool], [Microsoft.SharePoint.SPContext])
  14.     $updatePreference = $up.GetMethod("UpdatePreference", [System.Reflection.BindingFlags]"Static, NonPublic" , $null, $paramTypes, $null)
  15.     $parameters = ($userPreference, $fClearClickHistory, $context)
  16.     $updatePreference.Invoke($null, $parameters)
  17. }

Furthermore, we declared the following shortcuts:

  1. # shortcut for UserPreference
  2. $up = [Microsoft.Office.Server.Search.Administration.UserPreference]
  3. # shortcut for the nested class Settings in UserPreference
  4. $ups = [Microsoft.Office.Server.Search.Administration.UserPreference+Settings]

Set preferences for the current user via Reflection:

  1. $web = Get-SPWeb $url
  2. $ctx = [Microsoft.SharePoint.SPContext]::GetContext($web)
  3.  
  4. $pref = $up::GetUserPreference($false, $ctx)
  5. UpdateSetting $pref $ups::OpenDocumentsInClient $true
  6. Update $pref $false $ctx

Set preferences for another user via Reflection:

  1. $userName = 'i:0#.w|domain\user'
  2.  
  3. $web = Get-SPWeb $url
  4. $user = $web.EnsureUser($userName)
  5. $userToken = $user.UserToken
  6.  
  7. $impersonatedSite = New-Object Microsoft.SharePoint.SPSite($url, $userToken)
  8. $ctx = [Microsoft.SharePoint.SPContext]::GetContext($impersonatedSite.RootWeb)
  9.  
  10. $pref = $up::GetUserPreference($false, $ctx)
  11. UpdateSetting $pref $ups::OpenDocumentsInClient $true
  12. Update $pref $false $ctx

Set preferences for the current user using a dummy context (see this post about injecting a fake SharePoint context into PowerShell):

  1. $web = Get-SPWeb $url
  2. $ctx = [Microsoft.SharePoint.SPContext]::GetContext($web)
  3.  
  4. $sw = New-Object System.IO.StringWriter
  5. $request = New-Object System.Web.HttpRequest "", $url, ""
  6. $response = New-Object System.Web.HttpResponse $sw
  7. $dummyContext = New-Object System.Web.HttpContext $request, $response
  8. [System.Web.HttpContext]::Current = $dummyContext
  9. $dummyContext.Items["HttpHandlerSPWeb"] = $ctx.Web;
  10.  
  11. $pref = $up::GetUserPreference($false, $ctx)
  12. #$pref.EnableSetting($ups::OpenDocumentsInClient)
  13. #or
  14. #$pref.DisableSetting($ups::OpenDocumentsInClient)
  15. UpdateSetting $pref $ups::OpenDocumentsInClient $true
  16. $up::SetUserPreference($pref)
  17.  
  18. [System.Web.HttpContext]::Current = $null

Set preferences for another user using a dummy context:

  1. $userName = 'i:0#.w|domain\user'
  2.  
  3. $web = Get-SPWeb $url
  4. $user = $web.EnsureUser($userName)
  5. $userToken = $user.UserToken
  6.  
  7. $impersonatedSite = New-Object Microsoft.SharePoint.SPSite($url, $userToken)
  8. $ctx = [Microsoft.SharePoint.SPContext]::GetContext($impersonatedSite.RootWeb)
  9.  
  10. $sw = New-Object System.IO.StringWriter
  11. $request = New-Object System.Web.HttpRequest "", $url, ""
  12. $response = New-Object System.Web.HttpResponse $sw
  13. $dummyContext = New-Object System.Web.HttpContext $request, $response
  14. [System.Web.HttpContext]::Current = $dummyContext
  15. $dummyContext.Items["HttpHandlerSPWeb"] = $ctx.Web;
  16.  
  17. $pref = $up::GetUserPreference($false, $ctx)
  18. #$pref.EnableSetting($ups::OpenDocumentsInClient)
  19. #or
  20. #$pref.DisableSetting($ups::OpenDocumentsInClient)
  21. UpdateSetting $pref $ups::OpenDocumentsInClient $true
  22. $up::SetUserPreference($pref)
  23.  
  24. [System.Web.HttpContext]::Current = $null

July 26, 2016

How to Invoke PSI Methods using PSContext, Demonstrated by an Example of Reading Delegation Information

Filed under: PS 2013, PSI, Reflection — Tags: , , — Peter Holpar @ 21:21

The Challange

So you may ask now, why would we like to access PSI functionality from PSContext at all? Let me explain our situation.

Recently we had to extend an existing server side code in our custom Project Server solution to enable delegate users (that means ones acting for another users at the given time) to access the same custom functionality as the user they acting for.

Note: You can manage delegation via PWA Settings / Manage Delegates (either in the Personal Setting or in the Security section), see the page http://YourProjectServer/PWA/_layouts/15/pwa/userdelegation/ManageDelegations.aspx?mgm=true.

To make the things event worse, we had to check the delegation information in code running in the SharePoint web context, and in code that runs as a simple server side process without web context. It is actually not a timer job, but an automated server process scheduled by Windows Task Scheduler for the sake of simplicity, that perform some kind of housekeeping and reporting tasks for the custom application.However, in the case of a time job, an asynchronous event receiver, or a workflow step running without web context, you would have the same issue. Why these two cases (with or without SharePoint web context) differ from each other, will be discussed later in more details.

You probably know, that although the server side object model of the Project Server (available via the Microsoft.ProjectServer.PSContext) covers quite a lot of the functionality of the PSI, there is still a plenty of features that are simply inaccessible via the objects available through PSContext. For example, there is no way to access the delegation information. That means for us, that we should read this information via PSI, as its Resource class provides a ReadDelegations method just for this purpose. If you are not new in Project Server development, you should have already your own experience with PSI. If you have not any experience with that yet, I can say you, that it is rather complex comparing to the server side OM provided by PSContext. You should work with various DataSet and DataTable objects, but the greatest problem in our case that it requires a lot of configuration, either by config files (see the Configuring the Services with app.config section of the Walkthrough: Developing PSI Applications Using WCF article for example) , or via code (see section 11 of  the Walkthrough: Developing PSI Applications Using WCF article, or the Configuring the Services Programmatically section of the Walkthrough: Developing PSI Applications Using WCF article for example), just to be able to start using the PSI web services. If you have more applications (in our case one with, and another one without SharePoint web context), multiple environments (development, test, and production, one with HTTP, another one with HTTPS), and multiple servers in the environments, creating and maintaining the configuration information is typically not a trivial task.

Why could not we simply access the PSI endpoints via the PSContext? It would be great, wouldn’t it? Are you surprised if I say, it is possible? Well, actually it is. It requires an amount of hacking, and probably not a supported solution, but it is technically possible, and it made our life easier. If you take the risk on your own, you can try it as well, however there is no guarantee, that it works for you, or that once it works, a new Project Server patch could not break the functionality. So be warned.

After this introduction you should have already an overview of the problem and hopefully you are ready to read about the technical stuff. If you are not interested in such details, and need only the result of the analysis, you can skip the next two sections.

Connection between the Project Server client side object model and the server side object model

If one dig into the assemblies implementing the server side object model (Microsoft.Project.dll), and the client side object model (Microsoft.Project.Client.dll) of Project Server, then it turns out, that Microsoft has not re-implemented the functionality of PSI, these libraries still utilize the good-old PSI infrastructure either directly (in case of the server side OM) or indirectly (in case of the client side OM, that calls PSI via the server side OM).

How the client side OM invokes the server side OM is not Project Server specific, the infrastructure is inherited from SharePoint. Although there were changes between the SharePoint versions 2010 and 2013, the main concepts remain the same. Both the server side and client side of that bridge were deeply researched and analyzed in my posts five years ago. Those posts should give you enough insight into the internal functionality of the client side OM, this topic is outside of scope of the current post.

It is more exciting (at least, for now), how the Project Server server side OM invokes the PSI infrastructure. Exactly that will I describe in the next section.

Connection between the Project Server server side object model and PSI

The Microsoft.Office.Project.PWA namespace contains a public class called PSI (implemented in assembly Microsoft.Office.Project.Server.PWA). This class exposes more than 20 public properties with postfix “WebService”, each of them returns the WCF proxy of the corresponding PSI channel. For example, the ResourceWebService property returns a reference for a Resource object (namespace: Microsoft.Office.Project.Server.WebServiceProxy, assembly: Microsoft.Office.Project.Server.PWA), inherited from WcfProxyBase<IResource>. Once created in the “WebService” property getters, these WCF proxy objects are stored in the private field _proxyContainer (of  type ProxyContainer, implemented as a nested class) of the PSI class for any further access.

For the purpose of our solution, it is irrelevant how the WCF proxy objects (like the Resource object returned by the ResourceWebService property of the PSI class) know, what configuration they should use and how they can access the WCF endpoint. Probably I write about sometimes later in an other post.

The internal PJClientCallableContext class (Microsoft.ProjectServer namespace, Microsoft.ProjectServer assembly) contains a static read-only property called PJContext. It returns an instance of the PJContext class (Microsoft.Office.Project.PWA namespace, Microsoft.Office.Project.Server.PWA assembly) using the generic private static RetrieveValue method. If the process runs in the HTTP context (HttpContext.Current is not null) the RetrieveValue method invokes the GetContext(bool isWebServiceCall, bool ignoreCachedContext) method of the PJContext class. Otherwise (without HTTP context), the lazy-initialized value of the _pjcontext field will be returned by RetrieveValue method. The lazy-initialization of this field can be found in the Initialize method of the PJClientCallableContext class, the static GetObjectModelContext method of the PJContext class is invoked using the SPWeb object instance passed to the Initialize method as parameter. This is the same SPWeb object, that you passed to the constructor of the PSContext object (or it is the RootWeb of the site if you used the PSContext constructor with the SPSite object; or the RootWeb of the site corresponding to the URL if you used the PSContext constructor with the Uri object), as in the PSContext(SPWeb web) constructor an instance of the PJClientCallableContext class is created using the web instance as parameter.

Side note: On the other hand, using the PSContext constructors with parameters in a HTTP context may result you do no get the expected outcome, but it is beyond our scope again, more on that in a later post.

After this theoretical explanation let’s see some functional code.

The Code

My goal was to access the static PJContext property of the internal PJClientCallableContext class via Reflection, and expose its PSI property to my code as an extension method.

The solution described below requires adding the following assembly references:

  • Microsoft.Project.dll
  • Microsoft.Office.Project.Server.Library.dll
  • Microsoft.Office.Project.Server.PWA.dll
  • Microsoft.Office.Project.Schema.dll
  • Microsoft.Office.Project.Server.Administration.dll

The extension method is rather simple, and requires this using directive to work:

using Microsoft.Office.Project.PWA;

  1. public static PSI GetPSI(this PSContext dummyContext)
  2. {
  3.     PSI result = null;
  4.     Assembly psAssembly = typeof(PSContext).Assembly;
  5.  
  6.     // get internal type
  7.     Type type_PJClientCallableContext = psAssembly.GetType("Microsoft.ProjectServer.PJClientCallableContext");
  8.     PropertyInfo pi_PJContext = type_PJClientCallableContext.GetProperty("PJContext");
  9.     PJContext pjContext = pi_PJContext.GetValue(null) as PJContext;
  10.  
  11.     if (pjContext != null)
  12.     {
  13.         result = pjContext.PSI;
  14.     }
  15.  
  16.     return result;
  17. }

Note: As you can see, in the code above we don’t use the PSContext object passed as parameter to the extension method at all. It is only to enable attaching the functionality to a PSContext instance via the extension method, and so enforcing the creation of a PSContext instance first. If you have an application without HTTP context (like a console application), and you call this code without creating a PSContext instance first, like:

PSI psi = Extensions.GetPSI(null);

an InvalidOperationException will be thrown by the RetrieveValue method of the PJClientCallableContext class:

You must wrap your server OM code in a PJClientCallableContext when outside of an HttpContext.

In the case of a web application you don’t have such problem, as the PJContext instance is created using the current HTTP context.

Below I illustrate the usage of the extension method, by querying the delegation information of an enterprise resource in Project Server. To achieve that, I invoke the ReadDelegations method of the Resource class. See the available values of the DelegationFilter enumeration in the documentation.

The code requires the following using directives:

using Microsoft.Project;
using Microsoft.Office.Project.PWA;
using schema = Microsoft.Office.Project.Server.Schema;
using lib = Microsoft.Office.Project.Server.Library;

  1. using (PSContext projectContext = PSContext.GetContext(new Uri(pwaUrl)))
  2. {
  3.     PSI psi = projectContext.GetPSI();
  4.                     
  5.     lib.UserDelegationConsts.DelegationFilter filter = lib.UserDelegationConsts.DelegationFilter.All;
  6.  
  7.     // replace the GUID with an ID of a resource in your environment
  8.     // whose delegation information you would like to display
  9.     Guid resId = Guid.Parse("087ade95-281e-e411-9568-005056b45654");
  10.  
  11.     using (schema.UserDelegationDataSet delegationDS = psi.ResourceWebService.ReadDelegations(filter, resId))
  12.     {
  13.         foreach (schema.UserDelegationDataSet.ResourceDelegationsRow resDelegation in delegationDS.ResourceDelegations)
  14.         {
  15.             Console.WriteLine("Resource is substituted by '{0}' from '{1}' to '{2}'", resDelegation.DELEGATE_NAME, resDelegation.DELEGATION_START, resDelegation.DELEGATION_FINISH);
  16.         }
  17.     }
  18. }

Using the approach above you don’t have to bother with the configuration of the PSI proxy you need in your application, and still access such features in your server side code, that are available in PSI.

Note: There is a major restriction using this method. Although the PSI web services are available from a client computer, or from a remote server as well, you can not access remote servers using the code above. You can access only the PSI resources exposed by the local farm.

April 18, 2016

Re-creating the Missing Out-of-the-Box Search Locations using PowerShell

Filed under: PowerShell, Reflection, Search, SP 2013 — Tags: , , , — Peter Holpar @ 22:40

Recently we migrated a SharePoint 2010 web application into an existing SharePoint 2013 farm to free up the old SharePoint 2010 farm for “recycling” (it was the last application running in the old farm). The web application has a single site collection, and contains a single business application with views and list forms having a lot of customizations via “jQuery magic”. Since a replacement of the business application is planed for the near future, we decided not to upgrade the site collection to the SharePoint 2013 user interface (version 15). Leaving it in the SharePoint 2010 mode (version 14) ensures the views and forms are working further without any modifications in the JavaScript codes. After a few days a user complained, that when searching the local web site an error is displayed instead of the search results on the _layouts/OSSSearchResults.aspx page:

Unable to display this Web Part. To troubleshoot the problem, open this Web page in a Microsoft SharePoint Foundation-compatible HTML editor such as Microsoft SharePoint Designer. If the problem persists, contact your Web server administrator

In the ULS logs we found these entries:

CoreResultsWebpart: Couldnt find location with internal name LocalSearchIndex
CoreResultsDatasourceView: Couldnt find location with internal name LocalSearchIndex

On the web we found a post from Sushant Dukhande with the description of the issue, and a suggestion for the solution.

Using the script on that site it turned out, that the LocationConfigurations property of the search proxy is really empty. In an other environment, where we tested the migration we had no such issue.

Sushant Dukhande suggests to re-provision the search application. It might really solve the problem, however in our case I felt it to be an intense change, and searched for an alternative solution. Having a look into what happens under the cover of a provisioning process, I found the method responsible for provisioning the missing search locations. It is the internal static CreateOOBLocations method of the Microsoft.Office.Server.Search.Administration.LocationFactory class (in the Microsoft.Office.Server.Search assembly).

First, we need a reference to the search service application. You can get it like this (assuming it is named "Search Service Application"):

$ssa = Get-SPEnterpriseSearchServiceApplication "Search Service Application"

or via this script (as long as you are sure, you have a single instance of this service application type in your farm):

[Microsoft.Office.Server.Search.Administration.SearchServiceApplication]$ssa = Get-SPServiceApplication | ? { $_.TypeName -eq "Search Service Application" }

To display the names of the existing search locations:

$locConfigs = $ssa.LocationConfigurations
$locConfigs | % { $_.InternalName }

The following PowerShell script shows how to invoke the CreateOOBLocations method passing the search service application as parameter using PowerShell and Reflection:

$searchAssembly = [Microsoft.Office.Server.Search.Administration.SearchServiceApplication].Assembly
$locationFactory_Type = $searchAssembly.GetType("Microsoft.Office.Server.Search.Administration.LocationFactory")

$bindingFlags = [Reflection.BindingFlags]::NonPublic -bor [Reflection.BindingFlags]::Static
$mi_CreateOOBLocations = $locationFactory_Type.GetMethod("CreateOOBLocations", $bindingFlags)
$mi_CreateOOBLocations.Invoke($null, @([Microsoft.Office.Server.Search.Administration.SearchServiceApplication]$ssa))

Invoking the CreateOOBLocations method might be not always the solution for you. The same is true for the re-provisioning process suggested by Sushant Dukhande, since it invokes the same method as well. The problem, that this method has a condition, before provisioning all of the default search locations:

if (searchApp.LocationConfigurations.Count < 1)
{
    LocationConfigurationCollection locationConfigurationsInternal = searchApp.GetLocationConfigurationsInternal(true);
    CreateLiveLocation(locationConfigurationsInternal);
    CreateLiveSuggestionsLocation(locationConfigurationsInternal);
    CreateLocalSharepointLocation(locationConfigurationsInternal);
    CreateLocalPeopleLocation(locationConfigurationsInternal);
    CreateLocalFSSharePointLocation(locationConfigurationsInternal);
}

I don’t see, how our farm “lost” its search locations, but if it is possible to “lose” only a subset of the search locations (for example, only the one called LocalSearchIndex), it won’t be re-created by the CreateOOBLocations method, as the count of search location is still not zero.

In this case, the solution may be to re-create only the missing search location via the corresponding method. In the case of the LocalSearchIndex search location it is the CreateLocalSharepointLocation method of the LocationFactory class:

$locConfigs = $ssa.LocationConfigurations
$mi_CreateLocalSharepointLocation = $locationFactory_Type.GetMethod("CreateLocalSharepointLocation", $bindingFlags)
$mi_CreateLocalSharepointLocation.Invoke($null, @([Microsoft.Office.Server.Search.Administration.LocationConfigurationCollection]$locConfigs))

After fixing the issue in the farm, I’ve tested our other farms as well to find out, whether they are affected by the same problem or not. In one of the farm, the script provided in the post I mentioned earlier detected the issue, although I was sure, there is no problem with the search. It turned out to be a false positive test. This farm has its search service as a shared service from another farm, and the user account the script was run with had no permission on the search service in that remote farm. The script simply hid away the access denied error.

However, if we create a LocationConfigurationCollection instance via its internal constructor (either with a parameter of type SearchServiceApplication or of type SearchServiceApplicationProxy), the access denied error is displayed in the case the user has no permissions, and the items of the collection can be accessed if there is no problem with the permissions.

Let’s see first the script using the SearchServiceApplication:

$bindingFlags = [Reflection.BindingFlags]::NonPublic -bor [Reflection.BindingFlags]::Instance
$ci_LocationConfigurationCollection = [Microsoft.Office.Server.Search.Administration.LocationConfigurationCollection].GetConstructor($bindingFlags, $null, @([Microsoft.Office.Server.Search.Administration.SearchServiceApplication]), $null)
$locConfigs = $ci_LocationConfigurationCollection.Invoke(@([Microsoft.Office.Server.Search.Administration.SearchServiceApplication]$ssa))
$locConfigs | % { $_.InternalName }

As I wrote, you can achieve the same via a service proxy. It is useful for example, if the application is connected to a shared search service of another farm. First, we get the proxy as:

$url = "http://YourSharePointApp/&quot;
$site = Get-SPSite $url
$serviceContext = [Microsoft.SharePoint.SPServiceContext]::GetContext($site)
$ssaAppProxy = $serviceContext.GetDefaultProxy([Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy])

Next, we can use the same script as earlier, but in this case we invoke the internal constructor having the SearchServiceApplicationProxy parameter type:

$bindingFlags = [Reflection.BindingFlags]::NonPublic -bor [Reflection.BindingFlags]::Instance
$ci2_LocationConfigurationCollection = [Microsoft.Office.Server.Search.Administration.LocationConfigurationCollection].GetConstructor($bindingFlags, $null, @([Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy]), $null)
$locConfigs = $ci2_LocationConfigurationCollection.Invoke(@([Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy]$ssaAppProxy))
$locConfigs | % { $_.InternalName }

November 24, 2015

Recovering Passwords for SharePoint Managed Accounts

Filed under: PowerShell, Reflection, Security, SP 2013 — Tags: , , , — Peter Holpar @ 23:47

We have a SharePoint 2013 farm on a Windows 2008 R2 server. Recently we found this error in the Windows event logs in relation with the web application pool account:

Event ID 1511 – Windows cannot find the local profile and is logging you on with a temporary profile. Changes you make to this profile will be lost when you log off.

We tried to solve the issue based on the information we found in this post, but at the step below we faced the problem, that the password stored for the web application pool account (in this case we assumed domain\wa_pool_account) in our password repository does not work any more.

runas /u:domain\wa_pool_account /profile cmd

The web application pool account is registered as a managed account in SharePoint, at the original password has been already automatically changed by the system.

We could reset the password for the managed account as described in this article, but before changing the password I wanted to be sure there is no way to recover the current password from the system. I found a blog post and the related PowerShell code in TechNet Gallery, but I found the method described there (creating a new web application, and using an external tool, appcmd.exe) a bit overkill.

Instead of this I came up with an alternative solution that query the password directly from the SPManagedAccount object, via its private m_Password field (of type SPEncryptedString) that we can access by using Reflection. The public SecureStringValue property of the SPEncryptedString class returns an instance of the System.Security.SecureString class, and as illustrated here, we can “decode” its value to a simple string via Marshaling.

Using this approach, recovering the managed account password is so simple:

$ma = Get-SPManagedAccount domain\wa_pool_account
$maType = $ma.GetType()

$bindingFlags = [Reflection.BindingFlags]::NonPublic -bor [Reflection.BindingFlags]::Instance

$m_Password = $maType.GetField("m_Password", $bindingFlags)
$pwdEnc = $m_Password.GetValue($ma)

$ssv = $pwdEnc.SecureStringValue
$ptr = [System.Runtime.InteropServices.Marshal]::SecureStringToGlobalAllocUnicode($ssv)
[System.Runtime.InteropServices.Marshal]::PtrToStringUni($ptr)

July 2, 2015

Sending mails from a test SharePoint system, Updated for SharePoint 2013

Filed under: Mails, Reflection, SP 2013 — Tags: , , — Peter Holpar @ 23:48

Last year I published a post about how to redirect mails sent from your SharePoint application to a list, storing the mails as attachments for the list items. Unfortunately, the Reflection-based approach described there seems to no longer work in SharePoint 2013. My goal with the current post is to provide an alternative solution to the problem.

The first issue is with the signature of the private Send method of the MailMessage class we used to save the content of the mail into a MemoryStream: it has been extended with an extra bool parameter (allowUnicode) since the .NET  4.5 version. This problem could be easily fixed by altering the method invocation with the extra parameter, as:

sendMethod.Invoke(message, BindingFlags.Instance | BindingFlags.NonPublic, null, new[] { mailWriter, true, true }, null);

If you test the code in a simple console application, it works, the mail message will be saved into the MemoryStream. However, if you try to use the same code from a SharePoint application (an application referencing the server side SharePoint assemblies: either a console application or web part), you become a nasty exception:

An unhandled exception of type ‘System.AccessViolationException’ occurred
Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

After a short research, it turned out that the reason is the x64 process the SharePoint server side assemblies require. Since we can not change that requirement, I had to find an alternative solution. It was easy, as the stackoverflow thread the original solution originate from already contains an other option that plays with the DeliveryMethod of the SmtpClient class, saving the mail into a file in a temporary folder in the file system. Based on that answer I’ve altered the code of my DummyMailSender.

First, I’ve defined the helper class TemporaryDirectory to clean up the rest after the work:

  1. internal class TemporaryDirectory : IDisposable
  2. {
  3.     public TemporaryDirectory()
  4.     {
  5.         DirectoryPath = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName());
  6.         Directory.CreateDirectory(DirectoryPath);
  7.     }
  8.  
  9.     public string DirectoryPath { get; private set; }
  10.  
  11.     public void Dispose()
  12.     {
  13.         if (Directory.Exists(DirectoryPath))
  14.         {
  15.             Directory.Delete(DirectoryPath, true);
  16.         }
  17.     }
  18. }

and another class Extensions that implements an extension method to save the mail into a byte array:

  1. internal static class Extensions
  2. {
  3.     public static byte[] GetAsByteArray(this MailMessage m)
  4.     {
  5.         byte[] messageData = null;
  6.  
  7.         var smtpClient = new SmtpClient { DeliveryMethod = SmtpDeliveryMethod.SpecifiedPickupDirectory };
  8.  
  9.         using (var tempDir = new TemporaryDirectory())
  10.         {
  11.             smtpClient.PickupDirectoryLocation = tempDir.DirectoryPath;
  12.             smtpClient.Send(m);
  13.             var emlFile = Directory.GetFiles(smtpClient.PickupDirectoryLocation).FirstOrDefault();
  14.             if (emlFile != null)
  15.             {
  16.                 // read all file contents and trim the carriage return / new line at the end
  17.                 var messageAsText = File.ReadAllText(emlFile).Trim();
  18.                 System.Text.ASCIIEncoding enc = new System.Text.ASCIIEncoding();
  19.                 messageData = enc.GetBytes(messageAsText);
  20.             }
  21.         }
  22.         return messageData;
  23.     }
  24. }

Having these classes that perform the bulk of the functionality, the DummyMailSender can be simplified as:

  1. public void Send(MailMessage message)
  2. {
  3.     Trace.TraceInformation("Mail (subject: '{0}') sending started via DummyMailSender", message.Subject);
  4.  
  5.     byte[] messageData = message.GetAsByteArray();
  6.  
  7.     // RootWeb mustn't be disposed, see:
  8.     // http://blogs.msdn.com/b/rogerla/archive/2008/10/04/updated-spsite-rootweb-dispose-guidance.aspx
  9.     SPWeb rootWeb = _site.RootWeb;
  10.     bool allowUnsafeOriginal = rootWeb.AllowUnsafeUpdates;
  11.     try
  12.     {
  13.         rootWeb.AllowUnsafeUpdates = true;
  14.         SPList mailList = rootWeb.Lists[Constants.DummyMailList];
  15.         SPListItem mailItem = mailList.AddItem();
  16.         mailItem[SPBuiltInFieldId.Title] = message.Subject;
  17.         mailItem.Attachments.Add("Mail.eml", messageData);
  18.         mailItem.Update();
  19.  
  20.         string mailUrl = string.Format("{0}{1}?ID={2}", _site.Url, mailList.DefaultDisplayFormUrl, mailItem.ID);
  21.         Trace.TraceInformation("Dummy mail with subject '{0}' \"sent\" to '{1}' (cc: '{2}'), saved to '{3}'", message.Subject, message.To, message.CC, mailUrl);
  22.     }
  23.     catch (Exception ex)
  24.     {
  25.         Trace.TraceError("Error sending dummy mail: {0}\r\n{1}", ex.Message, ex.StackTrace);
  26.     }
  27.     finally
  28.     {
  29.         rootWeb.AllowUnsafeUpdates = allowUnsafeOriginal;
  30.     }
  31. }

This new version of DummyMailSender might perform not so well, as the original, Reflection-based one, but it should not be a major issue in a test system it was planned for, and at least it is a supported solution. At least, you don’t have to worry about that it won’t work after a newer .NET Framework version.

June 25, 2014

How to restrict the properties of a single object in the results of Client Object Model requests sent from PowerShell on the server side

Filed under: Managed Client OM, PowerShell, Reflection, SP 2013 — Tags: , , , — Peter Holpar @ 21:49

In my recent posts I illustrated how can we restrict (either using expression trees or via dynamically compiled assemblies) on the server side the properties of entity collections returned by the client object model requests when using PowerShell. Although it has less effect on the network traffic, it may be interesting to see, how can we achieve a similar result when working with single objects (like a single list instance) instead of entity collections (for example, fields of a list).

A query without property restrictions might look like this in C#:

ClientContext ctx = new ClientContext("http://sp2013&quot;);
var list = ctx.Web.Lists.GetByTitle("Images");
ctx.Load(list);
ctx.ExecuteQuery();

The request contains no property restriction (see SelectAllProperties=true below),

image

and the response contains indeed dozens of properties.

image

In C# it is quite simple to specify using lambda expressions, which properties of the list we need:

ClientContext ctx = new ClientContext("http://sp2013&quot;);
var list = ctx.Web.Lists.GetByTitle("Images");
ctx.Load(list, l => l.Title, l => l.ItemCount);
ctx.ExecuteQuery();

The request will include the limitation,

image

and only the requested properties are returned in the response:

image

But how to achieve the same result from PowerShell?

Again, we can easily refactor the code with the lambda expressions into a static helper class:

public static class QueryHelper
{
    public static void LoadListWithLimtedFields(ClientContext ctx, List list)
    {
        ctx.Load(list, l => l.Title, l => l.ItemCount);
    }
}

In this case, the original method contains only the code below that is easy to translate to PowerShell:

ClientContext ctx = new ClientContext("http://sp2013&quot;);
var list = ctx.Web.Lists.GetByTitle("Images");
QueryHelper.LoadListWithLimtedFields(ctx, list);
ctx.ExecuteQuery();

The same functionality transformed to PowerShell is shown below:

  1. $url = "http://sp2013&quot;
  2.  
  3. $referencedAssemblies = (
  4.     "Microsoft.SharePoint.Client, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c",
  5.     "Microsoft.SharePoint.Client.Runtime, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c",
  6.     "System.Core, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089")
  7. $sourceCode = @"
  8. using Microsoft.SharePoint.Client;
  9. using System.Collections.Generic;
  10. using System.Linq;
  11.  
  12. public static class QueryHelper
  13. {
  14.     public static void LoadListWithLimtedFields(ClientContext ctx, List list)
  15.     {
  16.         ctx.Load(list, l => l.Title, l => l.ItemCount);
  17.     }
  18. }
  19. "@
  20.  
  21. Add-Type -ReferencedAssemblies $referencedAssemblies -TypeDefinition $sourceCode -Language CSharp;
  22. Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.dll"
  23. Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
  24.  
  25. $ctx = New-Object Microsoft.SharePoint.Client.ClientContext($url)
  26. $list = $ctx.Web.Lists.GetByTitle("Images")
  27. [QueryHelper]::LoadListWithLimtedFields($ctx, $list)
  28. $ctx.ExecuteQuery()
  29.  
  30. Write-Host $list.Title, $list.ItemCount

Although more complex as the former solution, and recommended only for the hard-core developers, we can achieve the same result using expression trees built using Reflection. Let’s see how Visual Studio compiles the lambda expressions in our C# code into plain .NET objects and method calls. We have to build the code and open the assembly with a decompiler tool.

The original code as decompiled by JetBrains dotPeek:

ClientContext clientContext = new ClientContext("http://sp2013&quot;);
List byTitle = clientContext.Web.Lists.GetByTitle("Images");
clientContext.Load<List>(byTitle, new Expression<Func<List, object>>[2]
{
  (Expression<Func<List, object>>) (l => l.Title),
  (Expression<Func<List, object>>) (l => (object) l.ItemCount)
});
clientContext.ExecuteQuery();

The original code as decompiled by Reflector:

ParameterExpression CS$0$0001;
ClientContext ctx = new ClientContext("http://sp2013&quot;);
List list = ctx.Web.Lists.GetByTitle("Images");
ctx.Load<List>(list, new Expression<Func<List, object>>[] { Expression.Lambda<Func<List, object>>(Expression.Property(CS$0$0001 = Expression.Parameter(typeof(List), "l"), (MethodInfo) methodof(List.get_Title)), new ParameterExpression[] { CS$0$0001 }), Expression.Lambda<Func<List, object>>(Expression.Convert(Expression.Property(CS$0$0001 = Expression.Parameter(typeof(List), "l"), (MethodInfo) methodof(List.get_ItemCount)), typeof(object)), new ParameterExpression[] { CS$0$0001 }) });
ctx.ExecuteQuery();

The decompiled code helps us to create the same expression tree from PowerShell using Reflection:

  1. $url = "http://sp2013&quot;
  2.  
  3. # load the required client object model assemblies
  4. Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.dll"
  5. Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
  6.  
  7. $ctx = New-Object Microsoft.SharePoint.Client.ClientContext($url)
  8. $list = $ctx.Web.Lists.GetByTitle("Images")
  9.  
  10. $expressionType = [System.Linq.Expressions.Expression]
  11. $parameterExpressionArrayType = [System.Linq.Expressions.ParameterExpression].MakeArrayType()
  12.  
  13. $lambdaMethod = $expressionType.GetMethods() | ? { $_.Name -eq "Lambda" -and $_.IsGenericMethod -and $_.GetParameters().Length -eq 2  -and $_.GetParameters()[1].ParameterType -eq $parameterExpressionArrayType }
  14. $lambdaMethodGeneric = $lambdaMethod.MakeGenericMethod([System.Func“2[Microsoft.SharePoint.Client.List,System.Object]])
  15.  
  16. $listType = [Microsoft.SharePoint.Client.List]
  17.  
  18. # query the Title propery of type System.String
  19. $param1 = [System.Linq.Expressions.Expression]::Parameter($listType, "l")
  20. $name1 = [System.Linq.Expressions.Expression]::Property($param1, "Title")
  21. $expression1 = $lambdaMethodGeneric.Invoke($Null, [System.Object[]] @($name1, [System.Linq.Expressions.ParameterExpression[]] @($param1)))
  22.  
  23. # query the ItemCount of type System.Int32
  24. $param2 = [System.Linq.Expressions.Expression]::Parameter($listType, "l")
  25. $name2 = [System.Linq.Expressions.Expression]::Property($param2, "ItemCount")
  26.  
  27. # convert the Int32 type to object
  28. $body2 = [System.Linq.Expressions.Expression]::Convert($name2, [System.Object])
  29.  
  30. $expression2 = $lambdaMethodGeneric.Invoke($Null, [System.Object[]] @($body2, [System.Linq.Expressions.ParameterExpression[]] @($param2)))
  31.  
  32. $loadMethod = [Microsoft.SharePoint.Client.ClientRuntimeContext].GetMethod("Load")
  33. $loadMethodGeneric = $loadMethod.MakeGenericMethod([Microsoft.SharePoint.Client.List])
  34.  
  35. # call Load with both of the expressions
  36. $loadMethodGeneric.Invoke($ctx, [System.Object[]] @($list, [System.Linq.Expressions.Expression`1[System.Func“2[Microsoft.SharePoint.Client.List,System.Object]][]] @($expression1, $expression2)))
  37.  
  38. $ctx.ExecuteQuery()
  39.  
  40. Write-Host $list.Title, $list.ItemCount

Using the methods described in this and in the former posts we can achieve the same network traffic limiting results from PowerShell as we got used to in the case of the C# code.

June 15, 2014

How to restrict the properties in the results of Client Object Model requests sent from PowerShell on the server side, implementing the Include method

Filed under: Managed Client OM, PowerShell, Reflection, SP 2013 — Tags: , , , — Peter Holpar @ 22:51

In my recent post I illustrated how one can limit the entities returned by the server when responding a Client Object Model request sent from PowerShell. Although that method restricts the entities to the ones we really need, it still returns all of the properties for those items.

See the SelectAllProperties = true attribute of the ChildItemQuery in the request captured by Fiddler:

image

And the response includes really all of the properties…

image

However, we need typically only a few properties, the others only increase the size of the response package sent by the server unnecessarily.

In the C# syntax we can limit the fields using the Include method (see Retrieving only specified properties of lists):

  1. ClientContext ctx = new ClientContext("http://sp2013&quot;);
  2. var list = ctx.Web.Lists.GetByTitle("Images");
  3. var fieldsQuery = list.Fields.Include(f => f.InternalName, f => f.Id);
  4. // var fieldsQuery = list.Fields;
  5. var fields = ctx.LoadQuery(fieldsQuery);
  6. ctx.ExecuteQuery();

Remark: The JavaScript version supports a kind of Include as well (see Retrieving Only Specified Properties of Lists Using JavaScript).

Following the same strategy as in the case of the Where method, we compile our code to an assembly, and decompile it using JetBrains dotPeek:

  1. ClientContext clientContext = new ClientContext("http://sp2013&quot;);
  2. IQueryable<Field> clientObjects = ClientObjectQueryableExtension.Include<Field>((IQueryable<Field>)clientContext.Web.Lists.GetByTitle("Images").Fields, new Expression<Func<Field, object>>[2]
  3.   {
  4.     (Expression<Func<Field, object>>) (f => f.InternalName),
  5.     (Expression<Func<Field, object>>) (f => (object) f.Id)
  6.   });
  7. clientContext.LoadQuery<Field>(clientObjects);
  8. clientContext.ExecuteQuery();

Decompiling the code with Reflector reveals the internals of the lambda expression query:

IQueryable<Field> fieldsQuery = ctx.Web.Lists.GetByTitle("Images").Fields.Include<Field>(new Expression<Func<Field, object>>[] { Expression.Lambda<Func<Field, object>>(Expression.Property(CS$0$0001 = Expression.Parameter(typeof(Field), "f"), (MethodInfo) methodof(Field.get_InternalName)), new ParameterExpression[] { CS$0$0001 }), Expression.Lambda<Func<Field, object>>(Expression.Convert(Expression.Property(CS$0$0001 = Expression.Parameter(typeof(Field), "f"), (MethodInfo) methodof(Field.get_Id)), typeof(object)), new ParameterExpression[] { CS$0$0001 }) });

Transferring this quite complicate expression to a bit more readable format our method looks like this:

  1. ClientContext ctx = new ClientContext("http://sp2013&quot;);
  2. var list = ctx.Web.Lists.GetByTitle("Images");
  3.  
  4. // query the InternalName propery of type System.String
  5. var param1 = Expression.Parameter(typeof(Field), "f");
  6. var name1 = Expression.Property(param1, "InternalName");
  7.  
  8. var expression1 = Expression.Lambda<Func<Field, object>>(name1, param1);
  9.  
  10. // query the Id of type System.Guid
  11. var param2 = Expression.Parameter(typeof(Field), "f");
  12. var name2 = Expression.Property(param2, "Id");
  13. // convert the Guid type to object
  14. var body2 = Expression.Convert(name2, typeof(object));
  15.  
  16. var expression2 = Expression.Lambda<Func<Field, object>>(body2, param2);
  17.  
  18. // call Include with both of the expressions
  19. var fieldsQuery = ClientObjectQueryableExtension.Include<Field>(list.Fields, expression1, expression2);
  20.  
  21. var fields = ctx.LoadQuery(fieldsQuery);
  22. ctx.ExecuteQuery();

The PowerShell version of the same functionality:

  1. $url = "http://sp2013&quot;
  2.  
  3. # load the required client object model assemblies
  4. Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.dll"
  5. Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
  6.  
  7. $ctx = New-Object Microsoft.SharePoint.Client.ClientContext($url)
  8. $list = $ctx.Web.Lists.GetByTitle("Images")
  9.  
  10. $expressionType = [System.Linq.Expressions.Expression]
  11. $parameterExpressionArrayType = [System.Linq.Expressions.ParameterExpression].MakeArrayType()
  12.  
  13. $lambdaMethod = $expressionType.GetMethods() | ? { $_.Name -eq "Lambda" -and $_.IsGenericMethod -and $_.GetParameters().Length -eq 2  -and $_.GetParameters()[1].ParameterType -eq $parameterExpressionArrayType }
  14. $lambdaMethodGeneric = $lambdaMethod.MakeGenericMethod([System.Func“2[Microsoft.SharePoint.Client.Field,System.Object]])
  15.  
  16. $fieldType = [Microsoft.SharePoint.Client.Field]
  17.  
  18. # query the InternalName propery of type System.String
  19. $param1 = [System.Linq.Expressions.Expression]::Parameter($fieldType, "f")
  20. $name1 = [System.Linq.Expressions.Expression]::Property($param1, "InternalName")
  21. $expression1 = $lambdaMethodGeneric.Invoke($Null, [System.Object[]] @($name1, [System.Linq.Expressions.ParameterExpression[]] @($param1)))
  22.  
  23. # query the Id of type System.Guid
  24. $param2 = [System.Linq.Expressions.Expression]::Parameter($fieldType, "f")
  25. $name2 = [System.Linq.Expressions.Expression]::Property($param2, "Id")
  26.  
  27. # convert the Guid type to object
  28. $body2 = [System.Linq.Expressions.Expression]::Convert($name2, [System.Object])
  29.  
  30. $expression2 = $lambdaMethodGeneric.Invoke($Null, [System.Object[]] @($body2, [System.Linq.Expressions.ParameterExpression[]] @($param2)))
  31.  
  32. $includeMethod = [Microsoft.SharePoint.Client.ClientObjectQueryableExtension].GetMethod("Include")
  33. $includeMethodGeneric = $includeMethod.MakeGenericMethod([Microsoft.SharePoint.Client.Field])
  34.  
  35. # call Include with both of the expressions
  36. $fieldsQuery = $includeMethodGeneric.Invoke($Null, [System.Object[]] @($list.Fields, [System.Linq.Expressions.Expression`1[System.Func“2[Microsoft.SharePoint.Client.Field,System.Object]][]] @($expression1, $expression2)))
  37.  
  38. $fields = $ctx.LoadQuery($fieldsQuery)
  39. $ctx.ExecuteQuery()
  40.  
  41. $fields | % { Write-Host $_.InternalName: $_.Id }

Testing the script we found in Fiddler, that the request is limited to the properties we need:

image

and the response contains only the requested properties as well:

image

The output of the script should be similar to this one:

image

Note: In the case above, the child items are not restricted via a QueryableExpression, however you can combine the Where method described in the former post and the Include method to restrict both the child entities and their properties returned by the server:

fields.Where(f => f.TypeAsString == "Guid").Include(f => f.Id, f => f.InternalName)

In the PowerShell equivalent we chain our expression to achieve the same result:

  1. $url = "http://sp2013&quot;
  2.  
  3. # load the required client object model assemblies
  4. Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.dll"
  5. Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
  6.  
  7. $ctx = New-Object Microsoft.SharePoint.Client.ClientContext($url)
  8. $list = $ctx.Web.Lists.GetByTitle("Images")
  9.  
  10. $expressionType = [System.Linq.Expressions.Expression]
  11. $parameterExpressionArrayType = [System.Linq.Expressions.ParameterExpression].MakeArrayType()
  12.  
  13. $lambdaMethod = $expressionType.GetMethods() | ? { $_.Name -eq "Lambda" -and $_.IsGenericMethod -and $_.GetParameters().Length -eq 2  -and $_.GetParameters()[1].ParameterType -eq $parameterExpressionArrayType }
  14.  
  15. $fieldType = [Microsoft.SharePoint.Client.Field]
  16.  
  17. # call the Where method first
  18. $lambdaMethodGenericBool = $lambdaMethod.MakeGenericMethod([System.Func“2[Microsoft.SharePoint.Client.Field,System.Boolean]])
  19.  
  20. $param = [System.Linq.Expressions.Expression]::Parameter($fieldType, "f")
  21. $name = [System.Linq.Expressions.Expression]::PropertyOrField($param, "TypeAsString")
  22. $body = [System.Linq.Expressions.Expression]::Equal($name, [System.Linq.Expressions.Expression]::Constant("Guid"))
  23. $expression = $lambdaMethodGenericBool.Invoke($Null, [System.Object[]] @($body, [System.Linq.Expressions.ParameterExpression[]] @($param)))
  24.  
  25. $whereMethod = [System.Linq.Queryable].GetMethods() | ? { $_.Name -eq "Where" -and $_.ToString() -like "*TSource,System.Boolean*" }
  26. $whereMethodGeneric = $whereMethod.MakeGenericMethod([Microsoft.SharePoint.Client.Field])
  27.  
  28. $fieldsQuery = $whereMethodGeneric.Invoke($Null, [System.Object[]] @($list.Fields, $expression))
  29.  
  30. # call the Include method next on the result of the Where method
  31. $lambdaMethodGenericObj = $lambdaMethod.MakeGenericMethod([System.Func“2[Microsoft.SharePoint.Client.Field,System.Object]])
  32.  
  33. # query the InternalName propery of type System.String
  34. $param1 = [System.Linq.Expressions.Expression]::Parameter($fieldType, "f")
  35. $name1 = [System.Linq.Expressions.Expression]::Property($param1, "InternalName")
  36. $expression1 = $lambdaMethodGenericObj.Invoke($Null, [System.Object[]] @($name1, [System.Linq.Expressions.ParameterExpression[]] @($param1)))
  37.  
  38. # query the Id of type System.Guid
  39. $param2 = [System.Linq.Expressions.Expression]::Parameter($fieldType, "f")
  40. $name2 = [System.Linq.Expressions.Expression]::Property($param2, "Id")
  41.  
  42. # convert the Guid type to object
  43. $body2 = [System.Linq.Expressions.Expression]::Convert($name2, [System.Object])
  44.  
  45. $expression2 = $lambdaMethodGenericObj.Invoke($Null, [System.Object[]] @($body2, [System.Linq.Expressions.ParameterExpression[]] @($param2)))
  46.  
  47. $includeMethod = [Microsoft.SharePoint.Client.ClientObjectQueryableExtension].GetMethod("Include")
  48. $includeMethodGeneric = $includeMethod.MakeGenericMethod([Microsoft.SharePoint.Client.Field])
  49.  
  50. # call Include with both of the expressions
  51. $fieldsQuery2 = $includeMethodGeneric.Invoke($Null, [System.Object[]] @($fieldsQuery, [System.Linq.Expressions.Expression`1[System.Func“2[Microsoft.SharePoint.Client.Field,System.Object]][]] @($expression1, $expression2)))
  52.  
  53. $fields = $ctx.LoadQuery($fieldsQuery2)
  54. $ctx.ExecuteQuery()
  55.  
  56. $fields | % { Write-Host $_.InternalName: $_.Id }

When checking the network traffic using Fiddler we found that both the request

image

and the response fulfills our expectations.

The output of the script contains only the two returned fields due to the filtering on the server side:

image

How to filter the results of Client Object Model requests sent from PowerShell on the server side, implementing the Where method

Filed under: Managed Client OM, PowerShell, Reflection, SP 2013 — Tags: , , , — Peter Holpar @ 00:31

There are cases when we have to use the Managed Client Object Model from PowerShell. One of the most well-known cases is when we would like to manage our Office 365 environment (or access data stored in it) from our PowerShell scripts.

The Managed Client Object Model itself supports the server side filtering of data requested from the client using the Where method (see Applying filters to list retrieval), in contrary to the ECMAScript Client Object Model, where it is possible only via an unsupported workaround.

What it makes easy to use from advanced .NET programming languages like C# are the concepts of static methods, LINQ and lambda expression, all of them is quite far away from the standard PowerShell samples. No surprise, that the examples we can find on the web do not apply this filtering technique at all. If you have a look at this example, you can see, that it first downloads the whole list of site columns over the wire, then filters them on the client side just to get the two columns we need. If we have hundreds of columns, it can be pretty an overkill.

Note: Yes, I know that REST makes it possible (via the $filter query option) to filter the items returned, however it has its own limitation, like the lack of the batch requests – e.g. aggregating requests on the client side and sending them in batches via executeQuery – that is (at least, IMHO) one of the best features of the Client Object Model. Furthermore, I feel REST does not fit so good to the concept of PowerShell as the usage of the Managed Client Object Model.

Since we are able to use the same .NET libraries from PowerShell as from C#, it should be possible to achieve the same filtering result as well. But how to start?

First, let’s see a simple C# sample of a Managed Client Object Model call. In this sample we get the list of those fields of the Images list, that are of type Guid.

  1. ClientContext ctx = new ClientContext("http://sp2013&quot;);
  2. var list = ctx.Web.Lists.GetByTitle("Images");
  3. var fieldsQuery = list.Fields.Where(f => f.TypeAsString == "Guid");
  4. var fields = ctx.LoadQuery(fieldsQuery);
  5. ctx.ExecuteQuery();

How could we translate this one to a version that (although maybe less readable for us, humans) better reflects the object model and method calls behind the syntax sugar of C# (like static methods, and so on)? If we compile our code into an assembly, and open that assembly with a decompiler tool, like JetBrains dotPeek, the result may help us a step further:

  1. ClientContext clientContext = new ClientContext("http://sp2013&quot;);
  2. IQueryable<Field> clientObjects = Queryable.Where<Field>((IQueryable<Field>)clientContext.Web.Lists.GetByTitle("Images").Fields, (Expression<Func<Field, bool>>)(f => f.TypeAsString == "Guid"));
  3. clientContext.LoadQuery<Field>(clientObjects);
  4. clientContext.ExecuteQuery();

That seems already far better, but what can we do with the lambda expression?

Opening the assembly with another decompiler tool, Reflector reveals further details about the internal working of this functionality:

IQueryable<Field> fieldsQuery = Queryable.Where<Field>(ctx.Web.Lists.GetByTitle("Images").Fields, Expression.Lambda<Func<Field, bool>>(Expression.Equal(Expression.Property(CS$0$0000 = Expression.Parameter(typeof(Field), "f"), (MethodInfo) methodof(Field.get_TypeAsString)), Expression.Constant("Guid", typeof(string)), false, (MethodInfo) methodof(string.op_Equality)), new ParameterExpression[] { CS$0$0000 }));

As this decompiled code suggest, and as learned from this forum answer, we can substitute the lambda expression with an equivalent expression tree built dynamically.

The version we receive after this transformation:

  1. ClientContext ctx = new ClientContext("http://sp2013&quot;);
  2. var list = ctx.Web.Lists.GetByTitle("Images");
  3.  
  4. var param = Expression.Parameter(typeof(Field), "f");
  5. var name = Expression.PropertyOrField(param, "TypeAsString");
  6. var body = Expression.Equal(name, Expression.Constant("Guid"));
  7.  
  8. var expression = Expression.Lambda<Func<Field, bool>>(body, param);
  9. var fieldsQuery = Queryable.Where<Field>(list.Fields, expression);
  10.  
  11. var fields = ctx.LoadQuery(fieldsQuery);
  12. ctx.ExecuteQuery();

All we now have yet to do is to translate the C# code to PowerShell. The translation is not extremely complicated, it is mainly a mechanical process, although includes a lot of Type objects and Reflection calls, and we need a few tricks as well to achieve our goal.

For example, in the code above we use the static generic Lambda method of the Expression class with parameter types Expression and ParameterExpression[]. If we liked to get this method by name and parameter types directly using the GetMethod method of the Type class, we would receive an “Ambiguous match found” exception. The cause is that the Expression class has two methods with the same name: one generic one, and a non-generic version, and the GetMethod method does not support to restrict the filter to only one of these. Probably we can get the right one using the FindMembers method as well, but I found it easier to use the GetMethods method to get all of the methods of the Expression class and filter the results in my PowerShell code. In the code below we filter the methods, first by name, next we restrict the results to generic methods, and take the override that has two parameters, and the second parameter is of type ParameterExpression[]:

$lambdaMethod = $expressionType.GetMethods() | ? { $_.Name -eq "Lambda" -and $_.IsGenericMethod -and $_.GetParameters().Length -eq 2  -and $_.GetParameters()[1].ParameterType -eq $parameterExpressionArrayType }

I had similar problems with the Where method of the Query class. Although this method has only two overrides, and these ones have different parameter signatures, the parameters are rather complicated generic parameters, like Expression<Func<TSource, int, bool>>, so I decided to filter the methods again from my code, taking the methods called Where, and selecting the one that has a textural representation with the right parameter types:

$whereMethod = [System.Linq.Queryable].GetMethods() | ? { $_.Name -eq "Where" -and $_.ToString() -like "*TSource,System.Boolean*" }

The result of the translation is:

  1. $url = "http://sp2013&quot;
  2.  
  3. $ctx = New-Object Microsoft.SharePoint.Client.ClientContext($url)
  4. $list = $ctx.Web.Lists.GetByTitle("Images")
  5.  
  6. $expressionType = [System.Linq.Expressions.Expression]
  7. $parameterExpressionArrayType = [System.Linq.Expressions.ParameterExpression].MakeArrayType()
  8.  
  9. $lambdaMethod = $expressionType.GetMethods() | ? { $_.Name -eq "Lambda" -and $_.IsGenericMethod -and $_.GetParameters().Length -eq 2  -and $_.GetParameters()[1].ParameterType -eq $parameterExpressionArrayType }
  10. $lambdaMethodGeneric = $lambdaMethod.MakeGenericMethod([System.Func“2[Microsoft.SharePoint.Client.Field,System.Boolean]])
  11.  
  12. $fieldType = [Microsoft.SharePoint.Client.Field]
  13. $param = [System.Linq.Expressions.Expression]::Parameter($fieldType, "f")
  14. $name = [System.Linq.Expressions.Expression]::PropertyOrField($param, "TypeAsString")
  15. $body = [System.Linq.Expressions.Expression]::Equal($name, [System.Linq.Expressions.Expression]::Constant("Guid"))
  16. $expression = $lambdaMethodGeneric.Invoke($Null, [System.Object[]] @($body, [System.Linq.Expressions.ParameterExpression[]] @($param)))
  17.  
  18. $whereMethod = [System.Linq.Queryable].GetMethods() | ? { $_.Name -eq "Where" -and $_.ToString() -like "*TSource,System.Boolean*" }
  19. $whereMethodGeneric = $whereMethod.MakeGenericMethod([Microsoft.SharePoint.Client.Field])
  20.  
  21. $fieldsQuery = $whereMethodGeneric.Invoke($Null, [System.Object[]] @($list.Fields, $expression))
  22.  
  23. #$whereMethod = [System.Linq.Queryable].GetMethod("Where", [System.Type[]] @([System.Linq.IQueryable[TSource]], [System.Linq.Expressions.Expression[System.Func[TSource,bool]]]))
  24.  
  25. $fields = $ctx.LoadQuery($fieldsQuery)
  26. $ctx.ExecuteQuery()

Note: Before using this code, you should load the necessary client object model assemblies via:

Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"

When validating by Fiddler, we can see that the request contains the filter we need:

image

And when checking the result, only the matching fields are returned, and all without filtering the results on the client side.

image

Conclusion: I don’t want to suggest at all that you should always use this kind of filtering when requesting data from the server. It may be unnecessary or even inefficient if you have only a few or a few dozens of entities to return from the server. However, if it is really a large number of items, it may be useful to know how to get only the entities you really need.

Note: I’ve tested my solution with SharePoint 2013 on-premise, but it should work with SharePoint 2010 and Office 365 as well. You can however apply this technique not only to the SharePoint object model as well. For example, I used it first with the Project Server 2013 Client Object Model.

Older Posts »

Create a free website or blog at WordPress.com.