Second Life of a Hungarian SharePoint Geek

April 30, 2018

Faking feature activation properties

Filed under: Features, Reflection, SP 2013, Tips & Tricks — Tags: , , , — Peter Holpar @ 21:19

Assume you have a feature receiver in your SharePoint project. You would like to perform multiple actions if the feature is activated, so you organize your code according to the actions into static methods of a helpers class, like shown in the code snippet below:

  1. [Guid("8cb098ae-2017-4fff-8a53-b315abb85d79")]
  2. public class YourFeatureReceiver : SPFeatureReceiver
  3. {
  4.     public override void FeatureActivated(SPFeatureReceiverProperties properties)
  5.     {
  6.         SomeHelperClass.DoSomethingOnTheSite(properties);
  7.         SomeHelperClass.DoSomethingOtherOnTheSite(properties);
  8.         SomeHelperClass.AndFinallyDoThat(properties);
  9.     }
  10. }

Note: In this case, it is a site level feature, and we handle the FeatureActivated event, but you can apply the same technique for other kinds of feature receivers and events as well.

A helper method has the following signature, and contains code like this one:

  1. public static void DoSomethingOnTheSite(SPFeatureReceiverProperties properties)
  2. {
  3.     SPSite site = properties.Feature.Parent as SPSite;
  4.     if (site != null)
  5.     {
  6.         // do something here
  7.     }
  8. }

You would like to test the functionality of your method one-by-one from a console application, without actually having to deploy your SharePoint solution an activate your feature each time again (of course, you need at least an initial deployment of the whole solution). In my case, the helper class is included in a separate assembly, but even if it is the assembly of your SharePoint project, you should only deploy the new version of the assembly into the Global Assembly Cache (GAC).

As you can see in the method above, we use the Feature property of the SPFeatureReceiverProperties class we received in the properties parameter. Of course, the Feature property is read-only, so we need some tricks, to be able to pass the SPFeatureReceiverProperties parameter populated with the correct SPFeature in its Feature property to the method. We create first a new instance of the SPFeatureReceiverProperties class, than query the site for the feature based on its ID (se my important notice about this ID after the code block!). We can inject this SPFeature instance into our formerly crated SPFeatureReceiverProperties instance using Reflection, via the internal SetFeature method of the SPFeatureReceiverProperties class. Finally, we can invoke our helper method with the already-populated properties variable.

  1. using (SPSite site = new SPSite(yourSiteUrl))
  2. {
  3.     var properties = new SPFeatureReceiverProperties();
  4.  
  5.     var yourFeatureId = new Guid("51bf2a39-b527-46cf-abd6-39aaf1dcd19b");
  6.  
  7.     var feature = site.Features.FirstOrDefault(f => f.DefinitionId == yourFeatureId);
  8.     if (feature != null)
  9.     {
  10.         MethodInfo mi_setFeature = typeof(SPFeatureReceiverProperties).GetMethod("SetFeature",
  11.             BindingFlags.NonPublic | BindingFlags.Instance);
  12.  
  13.         if (mi_setFeature != null)
  14.         {
  15.             mi_setFeature.Invoke(properties, new object[] { feature });
  16.             SomeHelperClass.DoSomethingOnTheSite(properties);
  17.         }
  18.     }
  19. }

Note: It is important to notice, that the Guid we used in the feature-lookup above, comparing it to the DefinitionId property, is not the same, as the Guid we saw earlier in the feature receiver code snippet. The latter one is used only by Visual Studio during the deployment process to find up the correct ReceiverClass for the event receiver. The Guid we need, the ID of the feature is available on the Manifest tab of the feature as illustrated below:

image

Applying this technique made our development and testing process a lot faster.

Advertisements

Creating statistics about web part usage from the SharePoint content database

Filed under: PowerShell, Reflection, SP 2013, SQL, Web part — Tags: , , , — Peter Holpar @ 21:08

Recently I had to create some statistics about SharePoint web site customizations, like on which pages are there Script Editor Web Parts, or Content Editor Web Parts, etc. I knew I could and probably should have done it by iterating through all web sites, all pages and then looking up the web parts on each page using SPLimitedWebPartManager class, but I was aware, the same information should be available via the content database as well, making it possible to query the info much easier and faster, although unsupported. In this post I describe, how you can do it, but use the solution at your own risk.

The web part information is stored in the AllWebParts table, the information about the pages in the AllDocs table. I joined these tables together for the first report about the Script Editor Web Parts.

SELECT AD.DirName + ‘/’ + AD.LeafName as PageUrl, AWP.tp_ZoneID as ZoneId, AWP.tp_PartOrder as WebPartOrder, AWP.tp_Class AS WebPartClass
FROM
AllWebParts AWP (nolock)
INNER JOIN AllDocs (nolock) AD ON AWP.tp_SiteId = AD.SiteId AND AWP.tp_PageUrlID = AD.Id
WHERE tp_Class LIKE ‘%ScriptEditorWebPart’

Next, I was to create a report about the Content Editor Web Parts, using a filter like:

WHERE tp_Class = ‘%ContentEditorWebPart’

However, no result found, although I was pretty sure, there are a lot of them in our web site. How is it possible?

To test it further, I’ve included a Script Editor Web Part and a Content Editor Web Part on the AllItems.aspx page of the Tasks list in one of our sub-site, and created a new query with the filter below:

WHERE DirName LIKE ‘%site/subsite/Lists/Tasks%’
AND LeafName LIKE ‘%AllItem%’

This was the result:

image

As you see, the Script Editor Web Part is there, and you see two further web parts (they should be the Content Editor Web Part and the XsltListViewWeb part, that was originally on the page and is responsible to display the task items in the list), however both of them with a NULL value in the WebPartClass column. What should it mean?

I have studied the structure of the AllWebParts table and the relations of its fields further, and found that there are two fields (tp_Class and tp_Assembly) that are always populated for the records, where the WebPartClass is not NULL, and there is a field called tp_WebPartTypeId – populated for each entries, even for those, where the WebPartClass , tp_Class and tp_Assembly fields are empty – that we could eventually use to find the matching web parts. But how? I made a search for ‘WebPartTypeId’ using .NET Reflector, and found the internal class Microsoft.SharePoint.WebPartPages.WebPartTypeInfo, having a private static method called GetWellKnownTypeIdDictionary that returns a Dictionary<Guid, Type> mapping Guids (WebPartTypeIds) to the actual web part type. Remark: The Guids in the WebPartTypeId are actually created from the MD5 hash of the bytes of the joined full assembly name and web part class name, see the internal static  GetTypeIdUnsafe(MD5HashProvider md5Provider, string typeFullName, string assemblyName) method of the internal sealed class Microsoft.SharePoint.ApplicationRuntime.SafeControls.

image

To support those so called well-known types in my former SQL-query, I wrote a short PowerShell script that invokes the private static GetWellKnownTypeIdDictionary method of the internal WebPartTypeInfo class, and emits the resulting Dictionary to a text file I can use to extend my query:

  1. $webPartTypeInfoType = [System.Type]::GetType('Microsoft.SharePoint.WebPartPages.WebPartTypeInfo, Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c')
  2. $mi_GetWellKnownTypeIdDictionary = $webPartTypeInfoType.GetMethod('GetWellKnownTypeIdDictionary', [Reflection.BindingFlags]'NonPublic, Static')
  3. $wellKnownTypeIdDictionary = $mi_GetWellKnownTypeIdDictionary.Invoke($null, $null)
  4.  
  5. $wpTypes = $wellKnownTypeIdDictionary.Keys | % { "INSERT INTO @WPTypes VALUES ('$_', '$($wellKnownTypeIdDictionary[$_].Assembly.FullName)', '$($wellKnownTypeIdDictionary[$_].FullName)')" }
  6. Set-Content -Path 'C:\Data\WPTypes.txt' -Value $wpTypes

And that is already the extended version of the SQL query:

  1. DECLARE @WPTypes TABLE
  2.    (
  3.      Id uniqueidentifier NOT NULL,
  4.      AssemblyName varchar(500),
  5.      ClassName varchar(100)
  6.    )
  7.  
  8. INSERT INTO @WPTypes VALUES ('8e20cf70-0fd5-1e08-9972-38f63a6bd59a', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ImageWebPart')
  9. INSERT INTO @WPTypes VALUES ('ba009853-eac3-16c8-9094-a8834485ad33', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.DataFormWebPart')
  10. INSERT INTO @WPTypes VALUES ('83216ab2-cd0e-e9fc-fc5e-6a8f3b21c37b', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.DataViewWebPart')
  11. INSERT INTO @WPTypes VALUES ('42fddde2-e0cf-c8ab-48b7-db1fcac0a917', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ListFormWebPart')
  12. INSERT INTO @WPTypes VALUES ('05d0fd94-372a-5ee7-b480-ccb8f9cd2c23', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ListViewWebPart')
  13. INSERT INTO @WPTypes VALUES ('aef28218-44f8-0538-9805-4842c0e62811', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.XsltListFormWebPart')
  14. INSERT INTO @WPTypes VALUES ('a6524906-3fd2-ee4e-23ee-252d3c6e0dc9', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.XsltListViewWebPart')
  15. INSERT INTO @WPTypes VALUES ('0c6143a7-d68b-bade-e0ef-2c4d01182b0c', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.BlogAdminWebPart')
  16. INSERT INTO @WPTypes VALUES ('afef48e1-8f94-eb71-03a6-ffceb685306a', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.BlogMonthQuickLaunch')
  17. INSERT INTO @WPTypes VALUES ('4c06cea2-364f-47e3-e1d7-08d53f441157', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ContentEditorWebPart')
  18. INSERT INTO @WPTypes VALUES ('e6047383-438e-ed87-1a93-f1ff71729044', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.TitleBarWebPart')
  19. INSERT INTO @WPTypes VALUES ('707c1e73-0b3d-898b-c755-01621802ab8c', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.SilverlightWebPart')
  20. INSERT INTO @WPTypes VALUES ('28c23aec-2537-68b3-43b6-845b13cea19f', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  21. INSERT INTO @WPTypes VALUES ('8d6034c4-a416-e535-281a-6b714894e1aa', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  22. INSERT INTO @WPTypes VALUES ('8e814083-396a-e7d1-148b-316e3a7283f7', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  23. INSERT INTO @WPTypes VALUES ('e6377261-6920-bbfe-501f-fda7a61db10f', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  24. INSERT INTO @WPTypes VALUES ('8efd140d-eae9-5feb-06e3-f771842d2e43', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  25. INSERT INTO @WPTypes VALUES ('b3294a07-46bf-e661-d036-10670590bbd3', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.SPUserCodeWebPart')
  26.  
  27. SELECT AD.DirName + '/' + AD.LeafName as PageUrl, AWP.tp_ZoneID as ZoneId, AWP.tp_PartOrder as WebPartOrder, ISNULL(AWP.tp_Class, WPT.ClassName) AS WebPartClass
  28. FROM AllWebParts AWP (nolock)
  29. INNER JOIN AllDocs AD (nolock) ON AWP.tp_SiteId = AD.SiteId AND AWP.tp_PageUrlID = AD.Id
  30. LEFT JOIN @WPTypes WPT ON AWP.tp_WebPartTypeId = WPT.Id
  31. WHERE ISNULL(AWP.tp_Class, WPT.ClassName) LIKE '%ContentEditorWebPart'

Of course, you can change the conditions of the query as you like, for example, you can restrict it to two web part type, like:

WHERE ISNULL(AWP.tp_Class, WPT.ClassName) IN (‘Microsoft.SharePoint.WebPartPages.ScriptEditorWebPart’, ‘Microsoft.SharePoint.WebPartPages.ContentEditorWebPart’)

There are a few more columns in the AllWebParts table, that you eventually would include either in the SELECT statement or in its WHERE clause, these are:

  • tp_IsIncluded: The web part is displayed on the page, if the value is 1 (default). If you close (not delete!) a web part, the value is 0. Deleted web parts are removed from the table.
  • tp_Deleted: Assume you have a list with some pages that includes web parts, like view pages including XsltListViewWebPart instances. The web part entries in the AllWebParts table have a value of 0 at this stage. If you delete the list, these values change to 1. The web part entries will be kept in the table even after deleting the list from the first (user) level Recycle Bin, and removed only after the list is deleted from the second (site collection) level Recycle Bin.
  • tp_ListId: This is a field that is populated for list-related built-in web parts, like XsltListViewWebPart. You can look up the related list and web instances by joining the Lists and Webs views in your query respectively, as shown below (this time I omit the declaration of the @WPTypes variable and its population with value for the sake of brevity, but of course, you need it this time either):

SELECT L.tp_Title as ListTitle, W.FullUrl AS WebUrl, AD.DirName + ‘/’ + AD.LeafName as PageUrl, AWP.tp_ZoneID as ZoneId, AWP.tp_PartOrder as WebPartOrder, ISNULL(AWP.tp_Class, WPT.ClassName) AS WebPartClass, tp_ListId
FROM AllWebParts AWP (nolock)
INNER JOIN AllDocs AD (nolock) ON AWP.tp_SiteId = AD.SiteId AND AWP.tp_PageUrlID = AD.Id
LEFT JOIN @WPTypes WPT ON AWP.tp_WebPartTypeId = WPT.Id
LEFT JOIN Lists L (nolock) ON AWP.tp_SiteId = L.tp_SiteId AND AWP.tp_ListId = L.tp_ID
LEFT JOIN Webs W (nolock) ON AWP.tp_SiteId = W.SiteId AND L.tp_WebId = W.Id
WHERE ISNULL(AWP.tp_Class, WPT.ClassName) LIKE ‘%ListViewWebPart’

By including the list title or the web URL in the WHERE clause (or the ID of the list or the web if you wish) you can further limit the items returned by the query.

If there are records returned with NULL in the ListTitle and WebUrl columns it means typically that the list was deleted, but yet available in the Recycle Bin. See my comments regarding the tp_Deleted field above. Note, that despite the name of the FullUrl column in the Web view, it is actually a server relative URL.

I hope this overview has helped you to better understand what and how is stored in these tables of the SharePoint content database, as well, how the “magical” IDs of the well-known web part types do fit into the whole picture.

March 26, 2018

Querying SharePoint list items including their attachment in a single request from the JavaScript Client Object Model

Filed under: CAML, ECMAScript Client Object Model, JSCOM, SP 2013 — Tags: , , , — Peter Holpar @ 23:31

Assume there is a SharePoint list including a few hundreds of items, each item might have a few attachments, but there might be items without attachments as well. We need to display only some of the items (filtered dynamically based on specific conditions) for the users on a web page. We use already the JavaScript client object model on that page, so it would be ideal to keep this technology, instead of involving another one (like REST or SharePoint web services) just because this task.

I found several useful pages while looking for a solution for my problem, like this one about using CAML from the client object model, although some of the statements in the sections “Include attachment URLs” and “Limitations” turned to be a bit misleading later.

Few posts and forum answers suggested to query the items first using a CAML query, then read the attachments for each matching item in subsequent requests individually. That would not perform very well if there are a lot of matching item and is generally not a nice solution. Another suggestion I often found is to query the items with a CAML query (just like in the previous case), then query all of the attachments (they are stored in a folder called Attachments n the list with a separate subfolder, like 1, 2, etc. for each list items) in a second query, then match the attachments files to items based on the relative path of the attachments (attachment for the list item with ID 1 are in the folder /Attachments/1, etc.). If you have a lot of items having a few attachments for each item, that might have performance problem as well, and the implementation itself is challenging.

Vadim Gremyachev posted a solution using the managed client object model with a single request only on SharePoint Stack Exchange. In his solution, he uses the following line to create an execute the CAML query:

var items = list.GetItems(CamlQuery.CreateAllItemsQuery());

The static CreateAllItemsQuery method of the CamlQuery object is only an easy way to create a CAML query having the RecursiveAll view scope, it means, content of the subfolders (if there is any) would be searched through either.

public static CamlQuery CreateAllItemsQuery()
{
    return new CamlQuery { ViewXml = "<View Scope=\"RecursiveAll\">\r\n    <Query>\r\n    </Query>\r\n</View>" };
}

As we have a flat list, without any folders, I’ve slightly modified Vadim’s code first:

  1. var siteUrl = "http://YourSharePoint/site/subsite&quot;;
  2. using (var ctx = new ClientContext(siteUrl))
  3. {
  4.     var web = ctx.Web;
  5.     var list = web.Lists.GetByTitle("TitleOfYourList");
  6.     // Vadim's version fo a list with a folder struture
  7.     //var items = list.GetItems(CamlQuery.CreateAllItemsQuery());
  8.     // my version for a flat list without folders
  9.     var items = list.GetItems(new CamlQuery());
  10.     ctx.Load(items, icol => icol.Include(i => i["Title"], i => i.AttachmentFiles));
  11.     ctx.ExecuteQuery();
  12.  
  13.     //Print attachment Urls for list items
  14.     foreach (var item in items)
  15.     {
  16.         if (item.AttachmentFiles.Count > 0)
  17.         {
  18.             Console.WriteLine("{0} attachments:", item["Title"]);
  19.             foreach (var attachment in item.AttachmentFiles)
  20.             {
  21.                 Console.WriteLine(attachment.ServerRelativeUrl);
  22.             }
  23.         }
  24.         else
  25.         {
  26.             Console.WriteLine("No attachments were found for list item '{0}' ", item["Title"]);
  27.         }
  28.     }
  29. }

It resulted in the following request sent to the server, captured by Fiddler:

image

Note, that only the properties Title and AttachmentFiles were requested in the Include statement, and the ViewXml is empty.

We wanted to reproduce the same request using the JavaScript object model, but haven’t found any solution immediately ready to use, so first made a quick check, which alternative solutions are available in JavaScript, if we were not able to transform the managed code sample to JavaScript. Let’s see what we found.

I found a jQuery-based approach here, that utilizes the GetListItems method of the Lists SharePoint web service:

  1. $("table.ms-listviewtable tr:has(td.ms-vb2)").find("td:first").filter(function() {
  2.     return $(this).text().toLowerCase() == ID;
  3. }).html("<img src='" + url[1] + "' width=150 height=100 />");
  4.  
  5. $(document).ready(function() {
  6.     querySPWebServices();
  7. });
  8. function querySPWebServices() {
  9.     var soapEnv = "<soapenv:Envelope xmlns:soapenv='http://schemas.xmlsoap.org/soap/envelope/'&gt; \\par         <soapenv:Body> \\par         <GetListItems xmlns='http://schemas.microsoft.com/sharepoint/soap/'&gt; \\par         <listName>Projects</listName> \\par         <viewFields> \\par         <ViewFields> \\par         <FieldRef Name='Title' /> \\par         <FieldRef Name='Body' /> \\par         <FieldRef Name='ID' /> \\par         <FieldRef Name='Attachments' /> \\par         </ViewFields> \\par         </viewFields> \\par         <query> \\par         <Query /> \\par         </query> \\par         <queryOptions> \\par         <QueryOptions> \\par         <IncludeAttachmentUrls>TRUE</IncludeAttachmentUrls> \\par         </QueryOptions> \\par         </queryOptions> \\par         </GetListItems> \\par         </soapenv:Body> \\par         </soapenv:Envelope>";
  10.     $.ajax({
  11.         async: false,
  12.         url: "http://server.name.com/site/_vti_bin/lists.asmx&quot;,
  13.         type: "POST",
  14.         dataType: "xml",
  15.         data: soapEnv,
  16.         complete: processResult,
  17.         contentType: "text/xml; charset=\"utf-8\""
  18.     });
  19. };
  20.  
  21. function processResult(xData, status) {
  22.     $(xData.responseXML).find("z\\:row").each(function() {
  23.         if ($(this).attr("ows_Attachments") != 0) {
  24.             var url = $(this).attr("ows_Attachments").replace(/#/g, "").split(';');
  25.             var ID = $(this).attr("ows_ID");
  26.             $("table.ms-listviewtable tr:has(td.ms-vb2)").find("td:first").filter(function() {
  27.                 return $(this).text().toLowerCase() == ID;
  28.             }).html("<img src='" + url[1] + "' width=150 height=100 />");
  29.         };
  30.     });
  31. };

Another solution invoking the GetListItems method as well, but this time using SPServices can be found here:

  1. var p = $().SPServices({
  2.     operation: "GetListItems",
  3.     listName: "listname",
  4.     CAMLQueryOptions: "<QueryOptions>" +
  5.             "<IncludeAttachmentUrls>TRUE</IncludeAttachmentUrls>" +
  6.         "</QueryOptions>"
  7. });
  8.   
  9. p.done(function() {
  10.   
  11.     $(p.responseXML).).SPFilterNode("z:row").each(function() {
  12.       
  13.         var attachments = [];
  14.   
  15.         var att = $(this).attr("ows_Attachments");
  16.         if(att !== "0") {
  17.             attachments = att.split(";#"); // Now you'll have an array of attachment URLs
  18.         } else {
  19.             // att will be "0", indicating that there are no attachments
  20.         }
  21.       
  22.     });
  23.   
  24. });

You could query list items with their attachments via the REST interface as well:

http://YourSharePoint/site/subsite/_api/Web/Lists/GetByTitle('TitleOfYourList')/Items/?$expand=AttachmentFiles

One can use CAML in conjunction with REST by using POST requests and the GetItems method as described here, but as far as I see, the $expand=AttachmentFiles plays not well with the GetItems method. Instead of CAML, you can add the condition, ordering and row limit attributes of your query via $filter, $orderby and $top to the REST query as described here.

After this short overview, let’s try to rewrite our C# code above in JavaScript.

First, I came up with a query like the one below. I requested the very same properties, that we had in the managed client object version: the Title and AttachmentFiles. Note, that I omitted all of the filters I needed in the case of real life list from the CAML query due to the readability as this part is not relevant now in solving the main problem. In contrast to the managed client object code above, I restricted the fields in the CAML query as well via the ViewFields element. That was a mistake, as it turned out later, but I’m a strong believer, that mistakes are there to teach us something new.

  1. var list = web.get_lists().getByTitle("TitleOfYourList");
  2.  
  3. var camlQuery = new SP.CamlQuery();
  4. var query = '<View><ViewFields><FieldRef Name="Title"/><FieldRef Name="AttachmentFiles"/></ViewFields></View>';
  5. camlQuery.set_viewXml(query);
  6.  
  7. var items = list.getItems(camlQuery);
  8. ctx.load(items, 'Include(Title, AttachmentFiles)');

The result was not satisfying, the request failed (screenshot from Fiddler):

image

ErrorMessage: Value does not fall within the expected range.

ErrorTypeName: System.ArgumentException

ErrorStackTrace:

Microsoft.SharePoint.SPFieldMap.GetColumnNumber(String strFieldName, Boolean bThrow)
   at Microsoft.SharePoint.SPListItemCollection.GetColumnNumber(String groupName, Boolean bThrowException)
   at Microsoft.SharePoint.SPListItem.GetValue(SPField fld, Int32 columnNumber, Boolean bRaw, Boolean bThrowException)
   at Microsoft.SharePoint.SPListItem.get_Attachments()
   at Microsoft.SharePoint.SPAttachmentCollection_Client.<GetEnumerator>d__0.MoveNext()
   at Microsoft.SharePoint.Client.ServerStub.<EnumerateChildItems>d__36.MoveNext()
   at Microsoft.SharePoint.Client.ServerStub.WriteChildItems(JsonWriter writer, Object obj, ClientObjectQuery objectQuery, ProxyContext proxyContext)
   at Microsoft.SharePoint.Client.ServerStub.WriteAsJson(JsonWriter writer, Object obj, ClientObjectQuery objectQuery, ProxyContext proxyContext)
   at Microsoft.SharePoint.Client.ServerStub.WriteAsJsonWithMonitoredScope(JsonWriter writer, Object value, ClientObjectQuery objectQuery, ProxyContext proxyContext)
   at Microsoft.SharePoint.ServerStub.SPListItemServerStub.WriteOnePropertyValueAsJson(JsonWriter writer, Object target, ClientQueryProperty field, ProxyContext proxyContext)
   at Microsoft.SharePoint.Client.ServerStub.WriteAsJson(JsonWriter writer, Object obj, ClientObjectQuery objectQuery, ProxyContext proxyContext)
   at Microsoft.SharePoint.Client.ServerStub.WriteChildItems(JsonWriter writer, Object obj, ClientObjectQuery objectQuery, ProxyContext proxyContext)
   at Microsoft.SharePoint.Client.ServerStub.WriteAsJson(JsonWriter writer, Object obj, ClientObjectQuery objectQuery, ProxyContext proxyContext)
   at Microsoft.SharePoint.Client.ServerStub.WriteAsJsonWithMonitoredScope(JsonWriter writer, Object value, ClientObjectQuery objectQuery, ProxyContext proxyContext)
   at Microsoft.SharePoint.Client.ClientMethodsProcessor.WriteQueryResults(Object obj, ClientObjectQuery objQuery)
   at Microsoft.SharePoint.Client.ClientMethodsProcessor.ProcessQuery(XmlElement xe)
   at Microsoft.SharePoint.Client.ClientMethodsProcessor.ProcessOne(XmlElement xe)
   at Microsoft.SharePoint.Client.ClientMethodsProcessor.ProcessStatements(XmlNode xe)
   at Microsoft.SharePoint.Client.ClientMethodsProcessor.Process()

You can see the getter method of the Attachments property of the SPListItem object in the fourth line of the stack trace. Let’s have a quick view, how this method works:

  1. public SPAttachmentCollection get_Attachments()
  2. {
  3.    if (this.m_Attachments == null)
  4.    {
  5.        string pbstrUrlPrefix = null;
  6.        object[,] objArrAttachments = null;
  7.        uint pdwRowCount = 0;
  8.        bool bDoQuery = false;
  9.        if (!this.HasExternalDataSource && (this.ID != 0))
  10.        {
  11.            object obj3;
  12.            object obj2 = this.GetValue("Attachments");
  13.            if (obj2 != null)
  14.            {
  15.                bDoQuery = obj2.ToString() != "0";
  16.            }
  17.            SPSecurity.SetListInHttpContext(HttpContext.Current, this.m_Items.List.InternalName);
  18.            SPWeb web = this.Web;
  19.            web.Request.GetAttachmentsInfo(web.Url, this.m_Items.List.InternalName, Convert.ToInt32(this.ID), bDoQuery, out pbstrUrlPrefix, out pdwRowCount, out obj3);
  20.            if (bDoQuery)
  21.            {
  22.                objArrAttachments = (object[,]) obj3;
  23.            }
  24.        }
  25.        this.m_Attachments = new SPAttachmentCollection(this, pbstrUrlPrefix, pdwRowCount, objArrAttachments);
  26.    }
  27.    return this.m_Attachments;
  28. }

As we’ve learnt from Karine’s post, the Attachments field is a flag indicating if a list item has any attachments or not. Obviously, the getter method depends on this property (see the line object obj2 = this.GetValue("Attachments"); in code above), and we should request this property explicitly in our code.

That is the new version of my query:

  1. var camlQuery = new SP.CamlQuery();
  2. var query = '<View><ViewFields><FieldRef Name="Title"/><FieldRef Name="Attachments"/><FieldRef Name="AttachmentFiles"/></ViewFields></View>';
  3. camlQuery.set_viewXml(query);
  4.  
  5. var items = list.getItems(camlQuery);
  6. ctx.load(items, 'Include(Title, Attachments, AttachmentFiles)');

The request was captured by Fiddler as shown bellow. See the properties included in the Include statement, as well the ViewXml:

image

This version works already as expected, but comparing the request with the one captured at C#-based version (see above) I found, that one can omit the Attachments field from the Include statement without any problem.

  1. var camlQuery = new SP.CamlQuery();
  2. var query = '<View><ViewFields><FieldRef Name="Title"/><FieldRef Name="Attachments"/><FieldRef Name="AttachmentFiles"/></ViewFields></View>';
  3. camlQuery.set_viewXml(query);
  4.  
  5. var items = list.getItems(camlQuery);
  6. ctx.load(items, 'Include(Title, AttachmentFiles)');

Furthermore, there is no problem, if you omit the ViewFields element entirely from the CAML query (in our case, we needed later other parts of the CAML query, like the Where element to build up the filters, but it is irrelevant now). That is actually the same query we had with the managed client object model sample at the beginning of this post.

  1. var list = web.get_lists().getByTitle("TitleOfYourList");
  2.  
  3. var camlQuery = new SP.CamlQuery();
  4.  
  5. var items = list.getItems(camlQuery);
  6. ctx.load(items, 'Include(Title, AttachmentFiles)');

Note: Using of the ViewFields element might have an effect on the performance of the query on the server side (especially if you have a lot of columns in your list and you need only a few of them) and so on the response time as well, but after the query has been completed on the server, it seems to be irrelevant regarding to the network bandwidth usage.

That is the full version of the code (to be exact, one of the working ones), including displaying the information (this time in the console only) returned by the server:

  1. var ctx = SP.ClientContext.get_current();
  2. var web = ctx.get_web();
  3.  
  4. var list = web.get_lists().getByTitle("TitleOfYourList");
  5.  
  6. var camlQuery = new SP.CamlQuery();
  7.  
  8. var items = list.getItems(camlQuery);
  9. ctx.load(items, 'Include(Title, Attachments, AttachmentFiles)');
  10.  
  11. ctx.executeQueryAsync(
  12.     function () {
  13.         var enumerator = items.getEnumerator();
  14.         while (enumerator.moveNext()) {
  15.             var item = enumerator.get_current();
  16.             console.log(item.get_item('Title'));
  17.             if (item.get_item('Attachments')) {
  18.                 var attEnumerator = item.get_attachmentFiles().getEnumerator();
  19.                 while (attEnumerator.moveNext()) {
  20.                     var attachment = attEnumerator.get_current();
  21.                     console.log(attachment.get_serverRelativeUrl()); // or get_fileName()
  22.                 }
  23.             }
  24.         }
  25.     },
  26.     function (sender, args) {
  27.         console.log('Request failed. ' + args.get_message() + '\n' + args.get_stackTrace());
  28.     }
  29. );

That means, we fulfilled our goal by returning the items and their attachments in a single request/response via the JavaScript client object model.

Note, that after the response returned, we first check if there are any attachments for the item by reading the value of item.get_item(‘Attachments’), then accessing the attachments via the item.get_attachmentFiles() method. You could, however, check the existence of the attachments simply by using the get_attachmentFiles() method, without the Attachments field, like this:

var af = item.get_attachmentFiles();
if (af) {
    var attEnumerator = af.getEnumerator();

After extending the CAML query with the filters I need in the Where element, the query returned only the filtered items without any negative effect of the existing functionality, so that part of the task was completed either.

Lessons learned: If you need to read the attachments of your items, you must include the AttachmentFiles field in your Include statement. Including the Attachments field in the Include statement is optional, you need it only if you would like to use it explicitly (see the condition if (item.get_item(‘Attachments’)) in the last code snippet), otherwise you can omit it. If you would like to restrict the fields via the ViewFields element of your CAML query, you should include both of the Attachments and AttachmentFiles fields, but if you don’t restrict the columns by the usage of a ViewFields element, it works as well.

One final note: As so many times before, Fiddler was an invaluable tool to understand how the various versions work, or actually how they don’t work, by comparing the requests sent by the codes down to the wire.

March 23, 2018

How to check if a specific file exists in a folder structure of a SharePoint document library using the client object model

Filed under: CAML, Managed Client OM, SP 2013 — Tags: , , — Peter Holpar @ 22:25

Recently we had to create a utility function that makes it us possible to check if a file having a specific name exists anywhere within a folder structure of a SharePoint document library.

As long as you know not only the title of the document library, but its server relative URL as well, it requires only a single round-trip to the server:

  1. private bool FileExists(ClientContext clientContext, String docLibTitle, String fileName, String rootFolderServerRelativeUrl, String folderPath)
  2. {
  3.     var folderServerRelativeUrl = string.Format("{0}/{1}", rootFolderServerRelativeUrl, folderPath);
  4.     // or use a helper method to combine URL parts
  5.     //var folderServerRelativeUrl = JoinUrlParts(rootFolderServerRelativeUrl, folderPath);
  6.  
  7.     List DocumentsList = clientContext.Web.Lists.GetByTitle(docLibTitle);
  8.  
  9.     CamlQuery camlQuery = new CamlQuery();
  10.     camlQuery.ViewXml = @"<View Scope='Recursive'>
  11.                         <Query>
  12.                             <Where>
  13.                                 <Eq>
  14.                                     <FieldRef Name='FileLeafRef'></FieldRef>
  15.                                     <Value Type='Text'>" + fileName + @"</Value>
  16.                                 </Eq>
  17.                             </Where>
  18.                         </Query>
  19.                 </View>";
  20.     camlQuery.FolderServerRelativeUrl = folderServerRelativeUrl;
  21.     ListItemCollection listItems = DocumentsList.GetItems(camlQuery);
  22.     clientContext.Load(listItems);
  23.     clientContext.ExecuteQuery();
  24.  
  25.     return listItems.Count > 0;
  26. }

Usage:

  1. var webUrl = "http://YourSharePoint/site/subsite&quot;;
  2. string docLibTitle = "Documents";
  3. var rootFolderServerRelativeUrl = "/site/subsite/Shared Documents";
  4. var folderPath = "folder/subfolder";
  5.  
  6. ClientContext clientContext = new ClientContext(webUrl);
  7. var fileName = "document.docx";
  8.  
  9. bool fileFound = FileExists(clientContext, docLibTitle, fileName, rootFolderServerRelativeUrl, folderPath);

Note: If the folderServerRelativeUrl points to a location not within the document library (rootFolderServerRelativeUrl is wrong), the CAML query will ignore the FolderServerRelativeUrl and the entire library will be searched for a matching file. If  the folderPath part is wrong (not existing folder) then no matching item will be found, the query will return always false. Although the SPFolder server-side object model provides an Exists property to check if the folder at the given oath exists, there is no such property for the Folder object in the client object model. As a workaround, you can detect such mistakes by including these two lines of code in the FileExists method before invoking the ExecuteQuery method:

Folder folder = clientContext.Web.GetFolderByServerRelativeUrl(folderServerRelativeUrl);
clientContext.Load(folder);

If either part of the folderServerRelativeUrl is wrong, a File not found exception will be thrown on calling the ExecuteQuery method.

The helper method mentioned in the code is useful if you would not like to bother with leading and trailing slashes in the URL and in the folder path:

  1. public static string JoinUrlParts(params string[] urlParts)
  2. {
  3.     return string.Join("/", urlParts.Where(up => !string.IsNullOrEmpty(up)).ToList().Select(up => up.Trim('/')).ToArray());            
  4. }

If you, however, know only the title of the document library but not its server relative URL you need two round-trips:

  1. private bool FileExists(ClientContext clientContext, String docLibTitle, String fileName, String folderPath)
  2. {
  3.     List docLib = clientContext.Web.Lists.GetByTitle(docLibTitle);
  4.     Folder rootFolder = docLib.RootFolder;
  5.     clientContext.Load(rootFolder, f => f.ServerRelativeUrl);
  6.     clientContext.ExecuteQuery();
  7.  
  8.     string rootFolderServerRelativeUrl = rootFolder.ServerRelativeUrl;
  9.     var folderServerRelativeUrl = string.Format("{0}/{1}", rootFolderServerRelativeUrl, folderPath);
  10.     // or use a helper method to combine URL parts
  11.     //var folderServerRelativeUrl = JoinUrlParts(rootFolderServerRelativeUrl, folderPath);
  12.  
  13.     Folder folder = clientContext.Web.GetFolderByServerRelativeUrl(folderServerRelativeUrl);
  14.     clientContext.Load(folder);
  15.  
  16.     CamlQuery camlQuery = new CamlQuery();
  17.     camlQuery.ViewXml = @"<View Scope='Recursive'>
  18.                         <Query>
  19.                             <Where>
  20.                                 <Eq>
  21.                                     <FieldRef Name='FileLeafRef'></FieldRef>
  22.                                     <Value Type='Text'>" + fileName + @"</Value>
  23.                                 </Eq>
  24.                             </Where>
  25.                         </Query>
  26.                 </View>";
  27.     camlQuery.FolderServerRelativeUrl = folderServerRelativeUrl;
  28.     ListItemCollection listItems = docLib.GetItems(camlQuery);
  29.     clientContext.Load(listItems);
  30.     clientContext.ExecuteQuery();
  31.  
  32.     return listItems.Count > 0;
  33. }

Note: This version already includes the two-liner to check the existence of the folder path. If you don’t need that, remove it.

Usage of this version:

  1. var webUrl = "http://YourSharePoint/site/subsite&quot;;
  2. var docLibTitle = "Documents";
  3. var folderPath = "folder/subfolder";
  4. ClientContext clientContext = new ClientContext(webUrl);
  5. var fileName = "document.docx";
  6.  
  7. bool fileFound = FileExists(clientContext, docLibTitle, fileName, folderPath);

Changing SharePoint Search Preferences from Code

Filed under: PowerShell, Reflection, Search, SP 2013 — Tags: , , , — Peter Holpar @ 22:22

In my recent post I’ve illustrated with C# and PowerShell examples, how to read search preferences info from code, both for the current user as well as for other users. In this post we will see, how to change that preference from code.

We will use the same object, the Microsoft.Office.Server.Search.Administration.UserPreference class. Unlike its static GetUserPreference method, the either static SetUserPreference method has no overload that accepts the SharePoint context (SPContext) as parameter. The single overload of this method accepts a UserPreference instance. That makes our life not easier when it comes later to changing the preferences for another user. Don’t lose the hope, it is not impossible. Bur first things first.

Once we received a UserPreference instance via the GetUserPreference method, you should change certain preference properties, like OpenDocumentsInClient. How to do it? There are two methods, EnableSettings and DisableSetting (both having a parameter of  the nested enumeration type Settings) defined in the UserPreference class. If you would like to activate a setting, you should call the EnableSettings method, if you need to deactivate it, call the DisableSetting method, then finally invoke the SetUserPreference method to persist the changes. For example:

  1. var userPref = UserPreference.GetUserPreference();
  2. // if you would like to open documents in Office client, like Word or Excel
  3. userPref.UpdateSetting(UserPreference.Settings.OpenDocumentsInClient, true);
  4. // if you would like to open documents in Browser (Office Web Apps)
  5. userPref.UpdateSetting(UserPreference.Settings.OpenDocumentsInClient, false);
  6. UserPreference.SetUserPreference(userPref);

Note: This code works only if  you try it within a SharePoint context, like on an application page or in web part. In a console application you will receive an exception when you invoke the overload of the GetUserPreference method without the SPContext parameter:

ArgumentNullException
The value must not be null.
Parameter name: SPContext.Current

As we’ve already seen, there is an overload of the GetUserPreference method that accepts a SPContext parameter, so you could use that to get the preferences, but as there is no such overload for the SetUserPreference method, at least at this points will be the same type of exception thrown again. We will revisit the question shortly, how to set your own preferences from a console application, but we make a quick detour first.

To tell the truth, I don’t like the above pattern at all. Instead of these two methods I created an extension method with a Boolean parameter that encapsulates the functionality:

  1. public static void UpdateSetting(this UserPreference userPreference, UserPreference.Settings setting, bool value)
  2. {
  3.     if (value)
  4.     {
  5.         userPreference.EnableSetting(setting);
  6.     }
  7.     else
  8.     {
  9.         userPreference.DisableSetting(setting);
  10.     }
  11. }

Using this new method one can enable / disable preference settings like:

  1. var userPref = UserPreference.GetUserPreference();
  2. // if you would like to open documents in Office client, like Word or Excel
  3. userPref.EnableSetting(UserPreference.Settings.OpenDocumentsInClient);
  4. // if you would like to open documents in Browser (Office Web Apps)
  5. userPref.DisableSetting(UserPreference.Settings.OpenDocumentsInClient);
  6. UserPreference.SetUserPreference(userPref);

Back to the question, how to set your own preferences when the code runs without SharePoint context, like from a console application?

The “trivial” way is to fake a SharePoint context, using the method described here:

  1. using (SPSite site = new SPSite(url))
  2. {
  3.     using (SPWeb web = site.OpenWeb())
  4.     {
  5.         HttpRequest request = new HttpRequest(string.Empty, url, string.Empty);
  6.  
  7.         HttpResponse response = new HttpResponse(new System.IO.StreamWriter(new System.IO.MemoryStream()));
  8.  
  9.         HttpContext ctx = new HttpContext(request, response);
  10.         ctx.Items["HttpHandlerSPWeb"] = web;
  11.         HttpContext.Current = ctx;
  12.  
  13.         Console.WriteLine(SPContext.Current.Web.CurrentUser.LoginName);
  14.  
  15.         var userPref = UserPreference.GetUserPreference();
  16.         userPref.UpdateSetting(UserPreference.Settings.OpenDocumentsInClient, true);
  17.         UserPreference.SetUserPreference(userPref);
  18.  
  19.         //set back the original context (e.g. null)
  20.         HttpContext.Current = null;
  21.     }
  22. }

Another option is, to try to understand, how the SetUserPreference method internally works. It turns out, that it call the internal static UpdatePreference method:

UpdatePreference(preference, false, SPContext.Current);

So I’ve created just another extension method that wraps invoking the UpdatePreference method using Reflection:

  1. public static void Update(this UserPreference userPreference, bool fClearClickHistory, SPContext context)
  2. {
  3.     Type[] paramTypes = { typeof(UserPreference), typeof(bool), typeof(SPContext) };
  4.     MethodInfo updatePreference = userPreference.GetType().GetMethod("UpdatePreference", BindingFlags.Static | BindingFlags.NonPublic, null, paramTypes, null);
  5.     object[] parameters = { userPreference, fClearClickHistory, context };
  6.     updatePreference.Invoke(null, parameters);
  7. }

And a further helper method that accepts a SPContext object as parameter, writes out, preferences of which user we are to change and performs the change itself via the methods we have already:

  1. private void UpdatePreference(SPContext ctx)
  2. {
  3.     var user = ctx.Web.CurrentUser;
  4.     Console.WriteLine("Setting preferences for '{0}'", user.LoginName);
  5.  
  6.     var userPref = UserPreference.GetUserPreference(false, ctx);
  7.     userPref.UpdateSetting(UserPreference.Settings.OpenDocumentsInClient, false);
  8.     userPref.Update(false, ctx);
  9. }

I think the code we achieved using this extension method is much more readable as the former one with the dummy context:

  1. using (SPSite site = new SPSite(url))
  2. {
  3.     using (SPWeb web = site.OpenWeb())
  4.     {
  5.         var ctx = SPContext.GetContext(web);
  6.         UpdatePreference(ctx);
  7.     }
  8. }

The code snippets until this point have effect only on the current user. How to change the settings for other user? That is possible either, as soon we combine the methods we already have with impersonation.

First, the version that uses the dummy context:

  1. using (SPSite site = new SPSite(url))
  2. {
  3.     using (SPWeb web = site.OpenWeb())
  4.     {
  5.         var user = web.EnsureUser(@"i:0#.w|domain\user");
  6.  
  7.         SPSecurity.RunWithElevatedPrivileges(
  8.           () =>
  9.           {
  10.               using (SPSite impSite = new SPSite(url, user.UserToken))
  11.               using (SPWeb impWeb = impSite.OpenWeb())
  12.               {
  13.                   HttpRequest request = new HttpRequest(string.Empty, url, string.Empty);
  14.  
  15.                   HttpResponse response = new HttpResponse(new System.IO.StreamWriter(new System.IO.MemoryStream()));
  16.  
  17.                   HttpContext impersonatedContext = new HttpContext(request, response);
  18.  
  19.                   impersonatedContext.Items["HttpHandlerSPWeb"] = impWeb;
  20.  
  21.                   HttpContext.Current = impersonatedContext;
  22.  
  23.                   Console.WriteLine(SPContext.Current.Web.CurrentUser.LoginName);
  24.  
  25.                   var userPref = UserPreference.GetUserPreference();
  26.                   userPref.UpdateSetting(UserPreference.Settings.OpenDocumentsInClient, true);
  27.                   UserPreference.SetUserPreference(userPref);
  28.  
  29.                   //set back the original context (e.g. null)
  30.                   HttpContext.Current = null;
  31.               }
  32.           });
  33.     }
  34. }

Next, the other version using Reflection:

  1. using (SPSite site = new SPSite(url))
  2. {
  3.     using (SPWeb web = site.OpenWeb())
  4.     {
  5.         var user = web.EnsureUser(@"i:0#.w|domain\user");
  6.  
  7.         SPSecurity.RunWithElevatedPrivileges(
  8.           () =>
  9.           {
  10.               using (SPSite impSite = new SPSite(url, user.UserToken))
  11.               using (SPWeb impWeb = impSite.OpenWeb())
  12.               {
  13.                   var impCtx = SPContext.GetContext(impWeb);
  14.                   UpdatePreference(impCtx);
  15.               }
  16.           });
  17.     }
  18. }

Mission completed.

For those of you who would like to have the same functionality from PowerShell (of course, there are no SharePoint context inherited from the process at all), I include the equivalent methods below.

These are the helper methods we rely on:

  1. function UpdateSetting($userPreference, $setting, $value) {
  2.     If ($value)
  3.     {
  4.         $userPreference.EnableSetting($setting)
  5.     }
  6.     Else
  7.     {
  8.         $userPreference.DisableSetting($setting)
  9.     }
  10. }
  11.  
  12. function Update($userPreference, $fClearClickHistory, $context) {
  13.     $paramTypes = ($up, [bool], [Microsoft.SharePoint.SPContext])
  14.     $updatePreference = $up.GetMethod("UpdatePreference", [System.Reflection.BindingFlags]"Static, NonPublic" , $null, $paramTypes, $null)
  15.     $parameters = ($userPreference, $fClearClickHistory, $context)
  16.     $updatePreference.Invoke($null, $parameters)
  17. }

Furthermore, we declared the following shortcuts:

  1. # shortcut for UserPreference
  2. $up = [Microsoft.Office.Server.Search.Administration.UserPreference]
  3. # shortcut for the nested class Settings in UserPreference
  4. $ups = [Microsoft.Office.Server.Search.Administration.UserPreference+Settings]

Set preferences for the current user via Reflection:

  1. $web = Get-SPWeb $url
  2. $ctx = [Microsoft.SharePoint.SPContext]::GetContext($web)
  3.  
  4. $pref = $up::GetUserPreference($false, $ctx)
  5. UpdateSetting $pref $ups::OpenDocumentsInClient $true
  6. Update $pref $false $ctx

Set preferences for another user via Reflection:

  1. $userName = 'i:0#.w|domain\user'
  2.  
  3. $web = Get-SPWeb $url
  4. $user = $web.EnsureUser($userName)
  5. $userToken = $user.UserToken
  6.  
  7. $impersonatedSite = New-Object Microsoft.SharePoint.SPSite($url, $userToken)
  8. $ctx = [Microsoft.SharePoint.SPContext]::GetContext($impersonatedSite.RootWeb)
  9.  
  10. $pref = $up::GetUserPreference($false, $ctx)
  11. UpdateSetting $pref $ups::OpenDocumentsInClient $true
  12. Update $pref $false $ctx

Set preferences for the current user using a dummy context (see this post about injecting a fake SharePoint context into PowerShell):

  1. $web = Get-SPWeb $url
  2. $ctx = [Microsoft.SharePoint.SPContext]::GetContext($web)
  3.  
  4. $sw = New-Object System.IO.StringWriter
  5. $request = New-Object System.Web.HttpRequest "", $url, ""
  6. $response = New-Object System.Web.HttpResponse $sw
  7. $dummyContext = New-Object System.Web.HttpContext $request, $response
  8. [System.Web.HttpContext]::Current = $dummyContext
  9. $dummyContext.Items["HttpHandlerSPWeb"] = $ctx.Web;
  10.  
  11. $pref = $up::GetUserPreference($false, $ctx)
  12. #$pref.EnableSetting($ups::OpenDocumentsInClient)
  13. #or
  14. #$pref.DisableSetting($ups::OpenDocumentsInClient)
  15. UpdateSetting $pref $ups::OpenDocumentsInClient $true
  16. $up::SetUserPreference($pref)
  17.  
  18. [System.Web.HttpContext]::Current = $null

Set preferences for another user using a dummy context:

  1. $userName = 'i:0#.w|domain\user'
  2.  
  3. $web = Get-SPWeb $url
  4. $user = $web.EnsureUser($userName)
  5. $userToken = $user.UserToken
  6.  
  7. $impersonatedSite = New-Object Microsoft.SharePoint.SPSite($url, $userToken)
  8. $ctx = [Microsoft.SharePoint.SPContext]::GetContext($impersonatedSite.RootWeb)
  9.  
  10. $sw = New-Object System.IO.StringWriter
  11. $request = New-Object System.Web.HttpRequest "", $url, ""
  12. $response = New-Object System.Web.HttpResponse $sw
  13. $dummyContext = New-Object System.Web.HttpContext $request, $response
  14. [System.Web.HttpContext]::Current = $dummyContext
  15. $dummyContext.Items["HttpHandlerSPWeb"] = $ctx.Web;
  16.  
  17. $pref = $up::GetUserPreference($false, $ctx)
  18. #$pref.EnableSetting($ups::OpenDocumentsInClient)
  19. #or
  20. #$pref.DisableSetting($ups::OpenDocumentsInClient)
  21. UpdateSetting $pref $ups::OpenDocumentsInClient $true
  22. $up::SetUserPreference($pref)
  23.  
  24. [System.Web.HttpContext]::Current = $null

Querying SharePoint Search Preferences from Code

Filed under: PowerShell, Search, SP 2013 — Tags: , , — Peter Holpar @ 22:20

Recently a user complained, that he is not able to open office documents from the SharePoint portal of the company when working at home. He attached a screenshot of the error message in the browser to his mail. From this screenshot was it obvious, that there is an issue with the accessibility of the Office Web Applications server (OWA, also known as WAC), see the word wac in the address on the browser screen or WopiFrame.aspx in the address bar.

image

As it turned out, the WAC-server of the company has not been published externally via the firewall, but it was on purpose. Users should have been able to work with documents using their locally installed Office applications.

As you might know, you can configure the behavior, if document would be opened in browser or in the Office client application instead on either the site collection or on the document library level (see details here). The documents the user complained about were located in a library with the setting “Use the server default (Open in the client application)”, but it has not helped, when we changed it to “Open in the client application” explicitly.

It was really curious, but after a little while it turned out, that he wanted to open the document not from the library, but from a search result. At least a step further to the solution, have we thought.

You should know, that the behavior, if the Office documents get opened in the client application or in the browser is independent from the site collection level settings as well as from the document library settings. There is a (in my personal opinion pretty hidden) Preference link at the bottom of the search results page:

image

On this page the users can configure their own preferences, among others, if they would like to open the Office documents in the client application or in the browser:

image

It’s a cool option to enable users to decide which way they prefer, although it is pretty inconsistent with the other options (available for the administrators) we mentioned earlier. But there is an even bigger issue (at least, for me) with that. There is (as far as I know) no option / UI for administrators to query the value configured for a user, not to mention, how to change it remotely, without end user interaction.

Although it might have been the easiest choice to ask the user, which value he has configured for himself , I’m not the man of easy options if there might be a programmatic approach as well and a chance to learn something new. So let’s see, what I’ve learned.

The user preferences regarding the search are available via the Microsoft.Office.Server.Search.Administration.UserPreference class. If you need the user preferences from the current SPContext (e.g. for the current user), you can use either the static GetUserPreference() method or the other static overload GetUserPreference(bool lookupFromCache). If, however, you need the preference for another user, you can inject it via the static GetUserPreference(bool lookupFromCache, SPContext context) method.

For example, the DisplayPreference method below displays a few of the available preferences from a context it receives as parameter:

  1. private void DisplayPreference(SPContext ctx)
  2. {
  3.     var user = ctx.Web.CurrentUser;
  4.     Console.WriteLine("Reading preferences for '{0}'", user.LoginName);
  5.  
  6.     var userPref = UserPreference.GetUserPreference(false, ctx);
  7.  
  8.     Console.WriteLine("ShowPrequerySuggestion: {0}", userPref.IsSettingEnabled(UserPreference.Settings.ShowPrequerySuggestion));
  9.     Console.WriteLine("ShowPersonalSuggestions: {0}", userPref.IsSettingEnabled(UserPreference.Settings.ShowPersonalSuggestions));
  10.     Console.WriteLine("OpenDocumentsInClient: {0}", userPref.IsSettingEnabled(UserPreference.Settings.OpenDocumentsInClient));
  11. }

The following code snippet (taken from a console application) invokes the DisplayPreference method first to display the preferences of the current user, then again to display the preferences of an impersonated user:

  1. using (SPSite site = new SPSite(url))
  2. {
  3.     using (SPWeb web = site.OpenWeb())
  4.     {
  5.         var ctx = SPContext.GetContext(web);
  6.         DisplayPreference(ctx);
  7.  
  8.         var user = web.EnsureUser(@"i:0#.w|domain\user");
  9.  
  10.         SPSecurity.RunWithElevatedPrivileges(
  11.           () =>
  12.           {
  13.               using (SPSite impSite = new SPSite(url, user.UserToken))
  14.               using (SPWeb impWeb = impSite.OpenWeb())
  15.               {
  16.                   var impCtx = SPContext.GetContext(impWeb);
  17.  
  18.                   DisplayPreference(impCtx);
  19.               }
  20.           });
  21.     }
  22. }

Of course, if your code runs in a SharePoint process, you can get the context as SPContext.Current as well for the current user.

The same information is available via PowerShell either. For example, displaying preferences for the current user:

  1. $web = Get-SPWeb $url
  2. $ctx = [Microsoft.SharePoint.SPContext]::GetContext($web)
  3.  
  4. # shortcut for UserPreference
  5. $up = [Microsoft.Office.Server.Search.Administration.UserPreference]
  6. # shortcut for the nested class Settings in UserPreference
  7. $ups = [Microsoft.Office.Server.Search.Administration.UserPreference+Settings]
  8.  
  9. $pref = $up::GetUserPreference($false, $ctx)
  10. $pref.IsSettingEnabled($ups::ShowPrequerySuggestion)
  11. $pref.IsSettingEnabled($ups::ShowPersonalSuggestions)
  12. $pref.IsSettingEnabled($ups::OpenDocumentsInClient)

If you need the preferences of another user, you should impersonate it first as described here. After the impersonation, the code is pretty the same as earlier:

  1. $userName = 'i:0#.w|domain\user'
  2.  
  3. $web = Get-SPWeb $url
  4. $user = $web.EnsureUser($userName)
  5. $userToken = $user.UserToken
  6.  
  7. $impersonatedSite = New-Object Microsoft.SharePoint.SPSite($url, $userToken)
  8. $ctx = [Microsoft.SharePoint.SPContext]::GetContext($impersonatedSite.RootWeb)
  9.  
  10. # shortcut for UserPreference
  11. $up = [Microsoft.Office.Server.Search.Administration.UserPreference]
  12. # shortcut for the nested class Settings in UserPreference
  13. $ups = [Microsoft.Office.Server.Search.Administration.UserPreference+Settings]
  14.  
  15. $pref = $up::GetUserPreference($false, $ctx)
  16. $pref.IsSettingEnabled($ups::ShowPrequerySuggestion)
  17. $pref.IsSettingEnabled($ups::ShowPersonalSuggestions)
  18. $pref.IsSettingEnabled($ups::OpenDocumentsInClient)

Using this code we were able to detected that the complaining user has really the wrong preference (OpenDocumentsInClient was false). Now we had two choices: either to call the user, and ask him to change the preference, or to find a solution, how it would be possible to change it from code on behalf of the user remotely. Of course, this time we didn’t want to change the preferences without the explicit permission of the user, so took option 1, but I show you in my next post, how you could do it from code.

February 27, 2018

Copy an XsltListViewWebPart from another SharePoint Site via PowerShell – The client-side solution

Filed under: Managed Client OM, PowerShell, SP 2013 — Tags: , , — Peter Holpar @ 21:14

In my recent post I’ve illustrated, how to display SharePoint lists from other sites via PowerShell. As I told you, that solution does work only if you have direct access to the SharePoint server. Based on my experience that is not always the case. In the current post I introduce you a solution that should work even in such cases. We built this solution on the Managed Client-Object Modell of SharePoint.

Since I knew, that the CreateWebPartFromList method of the Microsoft.SharePoint.WebPartPages.SPWebPartManager is not accessible via the client object model, I first planned to apply the another approach: export the source web part via a LimitedWebPartManager instance (the client-side equivalent of SPLimitedWebPartManager), then use another LimitedWebPartManager instance to import it onto the target page. BUT (there is almost always a but…) it turned out, that although LimitedWebPartManager supports the ImportWebPart method, the ExportWebPart method is not available on the client-side (Note: as Waldek Mastykarz reported, the ExportWebPart method should be available since the March 2016 SharePoint Online CSOM update). So I came up with a fall-back plan and exported the web part by calling the exportwp.aspx page as described here by Anatoly Mironov.

As we export and import the web part from / to another sites, we create to different context objects to access them.

We read the response from the exportwp.aspx page as XML, and set the WebId property according to the ID of the source web site. There is apparently an issue with the ViewGuid property (more about them here), so we have to append it, for example, by cloning an existing XML node, like the one for the WebId property. A bit dirty workaround, but seems to work at me…

Finally, we import the web part to the target page and add it to the web part zone / position we wish.

  1. $sourceWebUrl = "http://YourSharePoint/Site1/Site2&quot;
  2. $listTitle = "YourList"
  3. $viewTitle = "YourView" # name of the view, like "All Items"
  4.  
  5. # set the path according the location of the assemblies
  6. Add-Type -Path "c:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.dll"
  7. Add-Type -Path "c:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
  8.  
  9. $clientContext = New-Object Microsoft.SharePoint.Client.ClientContext($sourceWebUrl)
  10.  
  11. $sourceWeb = $clientContext.Web
  12. $list = $sourceWeb.Lists.GetByTitle($listTitle)
  13. $views = $list.Views
  14.  
  15. $clientContext.Load($sourceWeb)
  16. $clientContext.Load($list)
  17. $clientContext.Load($views)
  18. $clientContext.ExecuteQuery()
  19.  
  20. $sourceWebId = $sourceWeb.Id
  21.  
  22. $view = $views | ? { $_.Title -eq $viewTitle }
  23.  
  24. $targetWebUrl = "http://YourSharePoint/Site1&quot;
  25. # This path should be the site relative URL of the page. If you have the page sub webs, include them in the path
  26. $targetPageSiteRelUrl = "/Site1/SitePages/SubSiteLisTest.aspx"
  27. $targetWebPartZoneId = "Bottom" # change the ID of the web part zone to match your needs
  28. $targetWebPartIndex = 0 # the intended position of the web part in the zone
  29.  
  30.  
  31. if (!$view.ServerObjectIsNull)
  32. {
  33.     Write-Host View found, exporting WebPart…
  34.  
  35.     $file = $list.RootFolder.Files.GetByUrl($view.ServerRelativeUrl)
  36.     $clientContext.Load($file)
  37.  
  38.     $webPartManager = $file.GetLimitedWebPartManager([Microsoft.SharePoint.Client.WebParts.PersonalizationScope]::Shared)
  39.     $webParts = $webPartManager.WebParts
  40.     $clientContext.Load($webParts)
  41.     $clientContext.ExecuteQuery()
  42.  
  43.     # I assume there is a single web part on the view page
  44.     # if this assumption is false, you should filter the web parts first
  45.     $webPart = $webParts[0]
  46.     # https://www.red-gate.com/simple-talk/blogs/getting-the-absolute-url-of-a-file-in-csom/
  47.     $viewAbsoluteUrl = (New-Object System.Uri($clientContext.Url)).GetLeftPart([System.UriPartial]::Authority) + $view.ServerRelativeUrl
  48.  
  49.     $exportWPUrl = $sourceWebUrl + "/_vti_bin/exportwp.aspx?pageurl=" + [System.Web.HttpUtility]::UrlEncode($viewAbsoluteUrl) + "&guidstring=" + $webPart.Id
  50.  
  51.     $request = [System.Net.WebRequest]::Create($exportWPUrl)
  52.     $request.UseDefaultCredentials = $true
  53.  
  54.     $response = $request.GetResponse()
  55.     $reader = New-Object System.IO.StreamReader $response.GetResponseStream()
  56.  
  57.     $wpXml = [Xml]$reader.ReadToEnd()
  58.     $properties = $wpXml.webParts.webPart.data.properties
  59.     $webIdProp = $properties.property | ? { $_.name -eq "WebId" }
  60.     $webIdProp.InnerText = $sourceWebId
  61.  
  62.     # "For example, setting the ViewGuid or the Toolbar properties doesn't do anything!" see:
  63.     # http://blog.bonzai-intranet.com/analysthq/2014/10/adding-an-xsltlistviewwebpart-with-a-custom-view-using-javascript/
  64.     # as a workaround for the missing ViewGuid property, clone the WebId property and change its name / type / value
  65.     $viewIdProp = $webIdProp.Clone()
  66.     $viewIdProp.name = "ViewGuid"
  67.     $viewIdProp.type = "string"    
  68.     $viewIdProp.InnerText = $view.ID.ToString("B").ToUpper() # "convert to a format like {8C1D2A1A-5BE8-469D-806E-2112965D2C1C}"
  69.  
  70.     [Void]$properties.AppendChild($viewIdProp)  
  71.  
  72.     $wpText = $wpXml.OuterXml
  73.  
  74.     Write-Host Export completed
  75.  
  76.     Write-Host Importing WebPart…
  77.     
  78.     $targetClientContext = New-Object Microsoft.SharePoint.Client.ClientContext($targetWebUrl)
  79.     $targetFile = $targetClientContext.Web.GetFileByServerRelativeUrl($targetPageSiteRelUrl)
  80.  
  81.     $targetWebPartManager = $targetFile.GetLimitedWebPartManager([Microsoft.SharePoint.Client.WebParts.PersonalizationScope]::Shared)
  82.     # note the difference:
  83.     # the server-side version of ImportWebPart returns a WebPart
  84.     # the client-side equvalent of ImportWebPart returns a WebPartDefinition
  85.     $targetWebPartDef = $targetWebPartManager.ImportWebPart($wpText)
  86.     # you could set optionally the WebId and ViewGuid properties at this point as well, but be aware of the issue with the ViewGuid property I mentioned above..
  87.     #$targetWebPartDef.WebPart.Properties["WebId"] = $sourceWebId
  88.     #$targetWebPartDef.WebPart.Properties["ViewGuid"] = $view.ID
  89.     # note the difference:
  90.     # the server-side version of AddWebPart returns void
  91.     # the client-side equvalent of AddWebPart returns a WebPartDefinition
  92.     [Void]$targetWebPartManager.AddWebPart($targetWebPartDef.WebPart, $targetWebPartZoneId, $targetWebPartIndex)
  93.     # or if you need the WebPartDefinition later, you can use these lines
  94.     #$targetWebPartDef = $targetWebPartManager.AddWebPart($targetWebPartDef.WebPart, $targetWebPartZoneId, $targetWebPartIndex)
  95.     #$targetClientContext.Load($targetFile)
  96.     $targetClientContext.ExecuteQuery()
  97.     
  98.     # write code to check in / publish the page here as required
  99.  
  100.     Write-Host Import completed
  101.  
  102. }
  103. else
  104. {
  105.     Write-Host View $viewTitle not found
  106. }
  107.  
  108.  
  109. # http://wvg-epm01e.sv-services.at/Test-Site2/_vti_bin/exportwp.aspx?pageurl=/Test-Site2/Lists/TestList/test.aspx&guidstring=8c1d2a1a-5be8-469d-806e-2112965d2c1c

That’s it, you should now be able to copy list views (XsltListViewWebPart web parts) from one SharePoint site to another from client-side via PowerShell. Of course, that is limited to a site collection scope, and there are still known issues with copying list views (generally, not limited to the PowerShell solutions), for example, copying views for a document library with a folder structure seems not to work if  you copy it from a parent site to a sub site. More about that eventually later, as soon as I collect a bit more background information about the problem.

Paging through the Entities returned by the ProjectData OData Interface of Project Server using PowerShell

Filed under: OData, PowerShell, Project Server, REST — Tags: , , , — Peter Holpar @ 21:13

About a year ago I wrote about how you can use PowerShell and Project Server REST interfaces (ProjectServer and ProjectData) to generate reports. However, at the end of last year we had a problem with the script that query the projects via the ProjectData interface:

  1. $url = 'http://YourProjectServer/PWA/_api/ProjectData/%5Ben-US%5D/Projects?$select=ProjectId,ProjectName,ProjectCreatedDate&#039;
  2. $reportPath = "C:\Data\ProjServerReports\"
  3.  
  4. $request = [System.Net.WebRequest]::Create($url)
  5. $request.UseDefaultCredentials = $true
  6. $request.Accept = "application/json;odata=verbose"
  7.  
  8. $response = $request.GetResponse()
  9. $reader = New-Object System.IO.StreamReader $response.GetResponseStream()
  10. $data = $reader.ReadToEnd()
  11.  
  12. $result = ConvertFrom-Json -InputObject $data
  13. $result.d.results | select ProjectId, ProjectName, ProjectCreatedDate | Export-Csv -Path ($reportPath + 'ProjectsFromProjectDataNoBatching.csv') -Delimiter ";" -Encoding UTF8 -NoTypeInformation

Important! You should use single quotes around the URL in this code snippet and in the other ones below to avoid PowerShell to remove the string $select in the REST query. More about that here.

A user reported an error that some of the projects were not included in the results. We found that the count of projects included in the report via ProjectServer was greater than the count of the projects we queried via  ProjectData. It is the script version that use the ProjectServer interface:

  1. $url = 'http://YourProjectServer/PWA/_api/ProjectServer/Projects?$select=Id,Name,CreatedDate&#039;
  2. $reportPath = "C:\Data\ProjServerReports\"
  3.  
  4. $request = [System.Net.WebRequest]::Create($url)
  5. $request.UseDefaultCredentials = $true
  6. $request.Accept = "application/json;odata=verbose"
  7.  
  8. $response = $request.GetResponse()
  9. $reader = New-Object System.IO.StreamReader $response.GetResponseStream()
  10. $data = $reader.ReadToEnd()
  11.  
  12. $result = ConvertFrom-Json -InputObject $data
  13. $result.d.results | select Id, Name, CreatedDate | Export-Csv -Path ($reportPath + 'ProjectsFromProjectServer.csv') -Delimiter ";" -Encoding UTF8 -NoTypeInformation

First I thought it is an issue with some permissions, but it was suspicious, that all of the missing projects were at the end of the project list (when ordered alphabetically), and the number of included project was exactly 100 in the ProjectData-based report.

Then I found this statement in the ProjectData – Project OData service reference:

“There are limits to the number of entities that can be returned in one query of the ProjectData service.”

What a surprise, the default limit for projects in the on-premise version we have is exactly 100!

You can use the Set-SPProjectOdataConfiguration PowerShell cmdlet for on-premise instances of Project Server to override the default query page size limits for any specified entity set.

Set-SPProjectOdataConfiguration -EntitySetName Projects -PageSizeOverride 200

However, it is a global change on the server, that affects all queries of the specific entity type, that can adversely the server performance, and as such, is not recommended. Instead of this, the suggested solution to query all instance of a given entity type is to implement paging by sending multiple queries using the $top and $skip operator.

Instead of a fix URL we define only a formatting pattern with placeholders for the $skip and $top parameters for the subsequent queries. More about this format is available here.

By applying the $inlinecount=allpages in the query we can assure that the result is always returned in $result.d.results as described in this post.

In the script we assume that the default limit of 100 project / query has not been changed. Note, that we use the $firstBatch variable to decide, if we should append the data to the existing report file in the Export-Csv cmdlet, or to create a new one.

In the first version of the script we process the results (output them into a CSV) immediately after receiving the response.

  1. $urlPattern = 'http://YourProjectServer/PWA/_api/ProjectData/%5Ben-US%5D/Projects?$select=ProjectId,ProjectName,ProjectCreatedDate&$skip={0}&$top={1}&$inlinecount=allpages'
  2. $reportPath = "C:\Data\ProjServerReports\"
  3.  
  4. $firstItem = 0
  5. $batchSize = 100
  6. $firstBatch = $true
  7.  
  8. Do
  9. {
  10.   Write-Host Requesting next batch of $batchSize items, starting at $firstItem
  11.   $url = $urlPattern -f $firstItem, $batchSize
  12.   Write-Host $url
  13.   $request = [System.Net.WebRequest]::Create($url)
  14.   $request.UseDefaultCredentials = $true
  15.   $request.Accept = "application/json;odata=verbose"
  16.  
  17.   $response = $request.GetResponse()
  18.   $reader = New-Object System.IO.StreamReader $response.GetResponseStream()
  19.   $data = $reader.ReadToEnd()
  20.  
  21.  
  22.   $result = ConvertFrom-Json -InputObject $data
  23.   $count = $result.d.results.Count
  24.   Write-Host Item count is $count
  25.   $result.d.results | select ProjectId, ProjectName, ProjectCreatedDate |
  26.       Export-Csv -Path ($reportPath + 'ProjectsFromProjectData_20171130_03.csv') -Delimiter ";" -Encoding UTF8 -NoTypeInformation -Append:(-not $firstBatch)
  27.   $firstBatch = $false
  28.   $firstItem += $batchSize
  29. }
  30. While ($count -eq $batchSize)

In the second we aggregate the results into an array, and process the results only after receiving the last response.

  1. $urlPattern = 'http://YourProjectServer/PWA/_api/ProjectData/%5Ben-US%5D/Projects?$select=ProjectId,ProjectName,ProjectCreatedDate&$skip={0}&$top={1}&$inlinecount=allpages'
  2. $reportPath = "C:\Data\ProjServerReports\"
  3.  
  4. $results = @()
  5. $firstItem = 0
  6. $batchSize = 100
  7.  
  8. Do
  9. {
  10.   Write-Host Requesting next batch of $batchSize items, starting at $firstItem
  11.   $url = $urlPattern -f $firstItem, $batchSize
  12.   Write-Host $url
  13.   $request = [System.Net.WebRequest]::Create($url)
  14.   $request.UseDefaultCredentials = $true
  15.   $request.Accept = "application/json;odata=verbose"
  16.  
  17.   $response = $request.GetResponse()
  18.   $reader = New-Object System.IO.StreamReader $response.GetResponseStream()
  19.   $data = $reader.ReadToEnd()
  20.  
  21.   $result = ConvertFrom-Json -InputObject $data
  22.   $count = $result.d.results.Count
  23.   Write-Host Item count is $count
  24.   $results += $result.d.results
  25.   $firstItem += $batchSize
  26. }
  27. While ($count -eq $batchSize)
  28.  
  29. $results | select ProjectId, ProjectName, ProjectCreatedDate |
  30.       Export-Csv -Path ($reportPath + 'ProjectsFromProjectDataWithBatching.csv') -Delimiter ";" -Encoding UTF8 -NoTypeInformation

Although in this case both of the scripts have the same result, you can use the second version if you need the complete list of the projects at a later time, for example, you would like to extend it with data not available at the time of the query.

Copy an XsltListViewWebPart from another SharePoint Site via PowerShell

Filed under: PowerShell, SP 2013 — Tags: , — Peter Holpar @ 21:02

It’s a common request, that the user would like to display a SharePoint List / Document library from another site. Unfortunately, that is not so easy. Users see only the lists exist on the current site when they want to add a web part to a page via the browser. Although the XsltListViewWebPart (the web part that is responsible for rendering most of the lists views with a very few exceptions, like Calendar views that still use the good-old ListViewWebPart web part you might know from SharePoint 2003) has a property WebId, you can not change it in the browser by editing the web part, just like you can not change its ListId property. If you would like to display a list on a page that belongs to another site as the one where the list exist, you can either use SharePoint Designer to copy the web part from the page of the list view to the target page, and extend it with the WebId property (ID of the source site), or export the web part from the list view, edit it, and upload to the target page as described in this post. Both of these methods require a certain amount of manual actions and as such are error-prone.

Instead of the above methods, I prefer to use PowerShell scripts to perform the job. There are multiple options to achieve that. The trivial one is to export the source web part via a SPLimitedWebPartManager instance, then use another SPLimitedWebPartManager instance to import it onto the target page, and set its WebId property:

  1. $sourceWebUrl = "http://YourSharePoint/Site1/Site2&quot;
  2. $listTitle = "YourList"
  3. $viewTitle = "YourView" # name of the view, like "All Items"
  4.  
  5. $sourceWeb = Get-SPWeb $sourceWebUrl
  6. $sourceWebId = $sourceWeb.Id
  7. $list = $sourceWeb.Lists[$listTitle]
  8. $view = $list.Views | ? { $_.Title -eq $viewTitle }
  9.  
  10. $targetWebUrl = "http://YourSharePoint/Site1&quot;
  11. # This path should be the site relative URL of the page. If you have the page sub webs, include them in the path
  12. $targetPageSiteRelUrl = "/Site1/SitePages/SubSiteLisTest.aspx"
  13. $targetWebPartZoneId = "FullPage" # change the ID of the web part zone to match your needs
  14. $targetWebPartIndex = 0 # the intended position of the web part in the zone
  15. $targetWeb = Get-SPWeb $targetWebUrl
  16.  
  17. if ($view)
  18. {
  19.     $viewPageUrl = $sourceWeb.Url + '/' + $view.Url
  20.     $webPartManager = $web.GetLimitedWebPartManager($viewPageUrl, [System.Web.UI.WebControls.WebParts.PersonalizationScope]::Shared)
  21.     $webPart = $webPartManager.WebParts[0]
  22.     # I assumed there is a single web part on the view page,
  23.     # otherwise you should find it first, like:
  24.     # $webPart = $webPartManager.WebParts | ? { $_; $_.Title -eq $listName }
  25.     if ($webPart -ne $Null) {
  26.         Write-Host WebPart found, exporting…
  27.         $webPart.ExportMode = "All"
  28.  
  29.         # use XmlTextWriter, not XmlWriter!
  30.         # https://social.msdn.microsoft.com/Forums/office/en-US/16391ce2-0454-44bc-b0db-c184898c588e/splimitedwebpartmanagerexportwebpartwebpart-xmlwriter-throws-xmlexception
  31.         $sw = New-Object System.IO.StringWriter
  32.         $xw = New-Object System.Xml.XmlTextWriter($sw)
  33.         $webPartManager.ExportWebPart($webPart, $xw);
  34.         $xw.Flush()
  35.         $xw.Close()
  36.         $wpText = $sw.ToString()
  37.  
  38.         # optionally, we could replace the WebId="00000000-0000-0000-0000-000000000000" in the text itself, like
  39.         #$wpText = $wpText.Replace('WebId="' + [Guid]::Empty + '"', 'WebId="' + $sourceWebId.Guid + '"')
  40.         # but I think setting the WebId property of the WebPart (see below) is more reliable
  41.         Write-Host Export completed
  42.  
  43.         Write-Host Importing WebPart…
  44.         $targetWebPartManager = $targetWeb.GetLimitedWebPartManager($targetPageSiteRelUrl, [System.Web.UI.WebControls.WebParts.PersonalizationScope]::Shared)
  45.         # https://stackoverflow.com/questions/4518544/xmlreader-from-a-string-content
  46.         $sr = New-Object System.IO.StringReader($wpText)
  47.         $xr = [System.Xml.XmlReader]::Create($sr)
  48.         # http://www.sites.se/2013/04/webpart-powershell-sharepoint-2013/
  49.         $errorMessage = $null
  50.         $targetWebPart = $targetWebPartManager.ImportWebPart($xr, [ref] $errorMessage)
  51.         $targetWebPart.WebId = $sourceWebId
  52.         $targetWebPart.ViewGuid = $view.ID
  53.         $targetWebPartManager.AddWebPart($targetWebPart, $targetWebPartZoneId, $targetWebPartIndex)
  54.         # check in / publish as required
  55.         Write-Host Import completed
  56.     }
  57.     else
  58.     {
  59.         Write-Host WebPart not found
  60.     }
  61. }
  62. else
  63. {
  64.     Write-Host View $viewTitle not found
  65. }

Using the same method (of course without setting the WebId property) you can copy another type of web parts as well between sites.

Probably a bit less know, and XsltListViewWebPart / ListViewWebPart specific option is to use the static CreateWebPartFromList method of the Microsoft.SharePoint.WebPartPages.SPWebPartManager class to create an instance of the source web part. After setting its WebId property, you can import the web part using a SPLimitedWebPartManager instance as earlier. The code below assumes you need the web part that belongs to the default list view:

  1. $sourceWebUrl = "http://YourSharePoint/Site1/Site2&quot;
  2. $listTitle = "YourList"
  3.  
  4. $sourceWeb = Get-SPWeb $sourceWebUrl
  5. $sourceWebId = $sourceWeb.Id
  6. $list = $sourceWeb.Lists[$listTitle]
  7.  
  8. $targetWebUrl = "http://YourSharePoint/Site1&quot;
  9. # This path should be the site relative URL of the page. If you have the page sub webs, include them in the path
  10. $targetPageSiteRelUrl = "/Site1/SitePages/SubSiteLisTest.aspx"
  11. $targetWebPartZoneId = "Bottom" # change the ID of the web part zone to match your needs
  12. $targetWebPartIndex = 0 # the intended position of the web part in the zone
  13. $targetWeb = Get-SPWeb $targetWebUrl
  14.  
  15. if ($list)
  16. {
  17.         Write-Host Source list found, importing WebPart…
  18.  
  19.         $targetWebPart = [Microsoft.SharePoint.WebPartPages.SPWebPartManager]::CreateWebPartFromList($list, $false)
  20.         $targetWebPart.WebId = $sourceWebId
  21.         $targetWebPartManager = $targetWeb.GetLimitedWebPartManager($targetPageSiteRelUrl, [System.Web.UI.WebControls.WebParts.PersonalizationScope]::Shared)
  22.         $targetWebPartManager.AddWebPart($targetWebPart, $targetWebPartZoneId, $targetWebPartIndex)
  23.         # check in / publish as required
  24.         Write-Host Import completed
  25.  
  26. }
  27. else
  28. {
  29.     Write-Host List $listTitle not found
  30. }

If you would like to copy another view, not the default one, you should get the ID of the view, and set the ViewGuid property of the web part as well:

  1. $sourceWebUrl = "http://YourSharePoint/Site1/Site2&quot;
  2. $listTitle = "YourList"
  3. $viewTitle = "YourView"
  4.  
  5. $sourceWeb = Get-SPWeb $sourceWebUrl
  6. $sourceWebId = $sourceWeb.Id
  7. $list = $sourceWeb.Lists[$listTitle]
  8. $view = $list.Views | ? { $_.Title -eq $viewTitle }
  9.  
  10. $targetWebUrl = "http://YourSharePoint/Site1&quot;
  11. # This path should be the site relative URL of the page. If you have the page sub webs, include them in the path
  12. $targetPageSiteRelUrl = "/Site1/SitePages/SubSiteLisTest.aspx"
  13. $targetWebPartZoneId = "Bottom" # change the ID of the web part zone to match your needs
  14. $targetWebPartIndex = 0 # the intended position of the web part in the zone
  15. $targetWeb = Get-SPWeb $targetWebUrl
  16.  
  17. if ($view)
  18. {
  19.         Write-Host Source list and view found, importing WebPart…
  20.  
  21.         $targetWebPart = [Microsoft.SharePoint.WebPartPages.SPWebPartManager]::CreateWebPartFromList($list, $false)
  22.         $targetWebPart.WebId = $sourceWebId
  23.         $targetWebPart.ViewGuid = $view.ID
  24.         $targetWebPartManager = $targetWeb.GetLimitedWebPartManager($targetPageSiteRelUrl, [System.Web.UI.WebControls.WebParts.PersonalizationScope]::Shared)
  25.         $targetWebPartManager.AddWebPart($targetWebPart, $targetWebPartZoneId, $targetWebPartIndex)
  26.         # check in / publish as required
  27.         Write-Host Import completed
  28.  
  29. }
  30. else
  31. {
  32.     Write-Host View $viewTitle not found
  33. }

At this point you might say: Wait a moment, that is nice, but using the original methods, like SharePoint Designer or via the browser, I was able to perform this task even if I have no direct access to the server. That is definitely not applies to the PowerShell scripts above. You are right, we identified this handicap as well, and came up with a version that works even from a client computer. I show you this version in a forthcoming post, so stay tuned!

November 29, 2017

How to detect if your DbConext is already disposed

Filed under: Entity Framework, Tips & Tricks — Tags: , — Peter Holpar @ 23:34

A little background (or let’s say context) to the story. Recently I had to create a tool (of course it was SharePoint-related) that needs to import a quite large number of entries into a SQL database for further analysis. I chose Entity Framework (DB-first) as a wrapper to our database entities. Having the knowledge that I already have now, I should admit, it was not the perfect choice, but rather a great opportunity to learn from our failures. In the development environment, where we have a smaller amount of data, the tool was quite quick, so I anticipated, it would be quick enough in the test and productive environment either. Just to have an idea about the order of magnitude, in these environments we have around 0,5 Mio. entries to export. It is a large number of entries, but I don’t think it is extreme large. After the first run in the test environment I knew, I was completely wrong with my assumption.

After letting the tool run on the weekend, and even a few day after, we stopped it, because it simply have not finished, and we haven’t seen the end of the tunnel. I analyzed the log data and found, that at the beginning the tool saved a batch of 100 entries in approximately 50 milliseconds. But later it was slower and slower, and after having saved already about 15.000 entries, the saving of the same batch of 100 entries took more than 7 seconds. What was the problem, and how have we solved it? You can read more about the issue here and here, and about the solution we applied here.

As part of the solution, we disposed and re-created our data context after saving every batch of items. However, this solution has introduced a new problem.

In the original version we have called the SaveChanges method at the end of our processing in an outer code block to ensure saving any remaining items that do not completed an entire batch. But in the new version we got this exception at this point:

InvalidOperationException: The operation cannot be completed because the DbContext has been disposed.
at System.Data.Entity.Internal.LazyInternalContext.InitializeContext()
at System.Data.Entity.Internal.LazyInternalContext.get_ObjectContext()
at System.Data.Entity.Internal.InternalContext.DetectChanges(Boolean force)
at System.Data.Entity.Internal.InternalContext.GetStateEntries(Func`2 predicate)
at System.Data.Entity.Infrastructure.DbChangeTracker.Entries()
at System.Data.Entity.DbContext.GetValidationErrors()
at System.Data.Entity.Internal.InternalContext.SaveChanges()

Note: you have this error on a disposed DbContext object only if there were something to save. If no object has been changed, no exception is thrown on calling SaveChanges on a disposed DbContext object.

This exception was due to disposing of the data context we created in a using block. We passed the data context to a method, that disposed (and re-created it) as part of the pattern suggested for performance tuning for importing of large number of items via the Entity Framework. See the sample code with Exporter class and it static DoSomething method further down. The data context was disposed in the DoSomething method. In the original version we tried to invoke dbContext.SaveChanges after returning from the DoSomething method, although the data context was already disposed in the DoSomething method, and it resulted the exception above.

Although some disposable object types might provide a kind of check if the object has been already disposed (see the IsDisposed property of the System.Windows.Forms.Control class, for example), it is definitely not part of the IDisposable interface. If the object belongs to your solution, you can implement a similar property yourself. If you can’t change the source code of the class (it is a class in the .NET libraries or any 3rd party class), you can derive a subclass from it, and extend it by the functionality as shown in this answer. This method works of course only as long as the base class is not marked as sealed.

In the case of DbContext I found a private field of type System.Data.Entity.Internal.InternalContext called _internalContext. InternalContext is an abstract internal class, you can find references for it and for its derived class LazyInternalContext in the error stack trace above as well. The InternalContext class has a public property called IsDisposed. The LazyInternalContext class invokes in its InitializeContext method the CheckContextNotDisposed method of the base class, that throws the InvalidOperationException above if the context has been already disposed.

I created the extension method below using Reflection to read the value of the IsDisposed property of the _internalContext field.

  1. public static bool IsDisposed(this DbContext context)
  2. {
  3.     var result = true;
  4.  
  5.     var typeDbContext = typeof(DbContext);
  6.     var typeInternalContext = typeDbContext.Assembly.GetType("System.Data.Entity.Internal.InternalContext");
  7.  
  8.     var fi_InternalContext = typeDbContext.GetField("_internalContext", BindingFlags.NonPublic | BindingFlags.Instance);
  9.     var pi_IsDisposed = typeInternalContext.GetProperty("IsDisposed");
  10.  
  11.     var ic = fi_InternalContext.GetValue(context);
  12.  
  13.     if (ic != null)
  14.     {
  15.         result = (bool)pi_IsDisposed.GetValue(ic);
  16.     }
  17.  
  18.     return result;
  19. }

You can test the functionality in a code block like this one:

  1. using (dbContext = new YourDbContext())
  2. {
  3.     Console.WriteLine("Disposed: {0}", dbContext.IsDisposed());
  4.     Exporter.DoSomething(dbContext);
  5.     Console.WriteLine("Disposed: {0}", dbContext.IsDisposed());
  6. }
  7. Console.WriteLine("Disposed: {0}", dbContext.IsDisposed());

The Exporter class and its static DoSomething method in the code above are just some arbitrary functionality you would like to perform.

Having the IsDisposed method defined, you can check if the context is disposed before calling SaveChanges (or any other method that are dangerous to be invoked on a disposed context):

  1. if ((dbContext != null) && (!dbContext.IsDisposed()))
  2. {
  3.     dbContext.SaveChanges();
  4. }

Note: The code in this post was tested with Entity Framework 6.1.3. Other versions might behave in another way, so there is no guarantee that the same code works with those versions too.

Older Posts »

Create a free website or blog at WordPress.com.