Second Life of a Hungarian SharePoint Geek

April 30, 2018

Creating statistics about web part usage from the SharePoint content database

Filed under: PowerShell, Reflection, SP 2013, SQL, Web part — Tags: , , , — Peter Holpar @ 21:08

Recently I had to create some statistics about SharePoint web site customizations, like on which pages are there Script Editor Web Parts, or Content Editor Web Parts, etc. I knew I could and probably should have done it by iterating through all web sites, all pages and then looking up the web parts on each page using SPLimitedWebPartManager class, but I was aware, the same information should be available via the content database as well, making it possible to query the info much easier and faster, although unsupported. In this post I describe, how you can do it, but use the solution at your own risk.

The web part information is stored in the AllWebParts table, the information about the pages in the AllDocs table. I joined these tables together for the first report about the Script Editor Web Parts.

SELECT AD.DirName + ‘/’ + AD.LeafName as PageUrl, AWP.tp_ZoneID as ZoneId, AWP.tp_PartOrder as WebPartOrder, AWP.tp_Class AS WebPartClass
FROM
AllWebParts AWP (nolock)
INNER JOIN AllDocs (nolock) AD ON AWP.tp_SiteId = AD.SiteId AND AWP.tp_PageUrlID = AD.Id
WHERE tp_Class LIKE ‘%ScriptEditorWebPart’

Next, I was to create a report about the Content Editor Web Parts, using a filter like:

WHERE tp_Class = ‘%ContentEditorWebPart’

However, no result found, although I was pretty sure, there are a lot of them in our web site. How is it possible?

To test it further, I’ve included a Script Editor Web Part and a Content Editor Web Part on the AllItems.aspx page of the Tasks list in one of our sub-site, and created a new query with the filter below:

WHERE DirName LIKE ‘%site/subsite/Lists/Tasks%’
AND LeafName LIKE ‘%AllItem%’

This was the result:

image

As you see, the Script Editor Web Part is there, and you see two further web parts (they should be the Content Editor Web Part and the XsltListViewWeb part, that was originally on the page and is responsible to display the task items in the list), however both of them with a NULL value in the WebPartClass column. What should it mean?

I have studied the structure of the AllWebParts table and the relations of its fields further, and found that there are two fields (tp_Class and tp_Assembly) that are always populated for the records, where the WebPartClass is not NULL, and there is a field called tp_WebPartTypeId – populated for each entries, even for those, where the WebPartClass , tp_Class and tp_Assembly fields are empty – that we could eventually use to find the matching web parts. But how? I made a search for ‘WebPartTypeId’ using .NET Reflector, and found the internal class Microsoft.SharePoint.WebPartPages.WebPartTypeInfo, having a private static method called GetWellKnownTypeIdDictionary that returns a Dictionary<Guid, Type> mapping Guids (WebPartTypeIds) to the actual web part type. Remark: The Guids in the WebPartTypeId are actually created from the MD5 hash of the bytes of the joined full assembly name and web part class name, see the internal static  GetTypeIdUnsafe(MD5HashProvider md5Provider, string typeFullName, string assemblyName) method of the internal sealed class Microsoft.SharePoint.ApplicationRuntime.SafeControls.

image

To support those so called well-known types in my former SQL-query, I wrote a short PowerShell script that invokes the private static GetWellKnownTypeIdDictionary method of the internal WebPartTypeInfo class, and emits the resulting Dictionary to a text file I can use to extend my query:

  1. $webPartTypeInfoType = [System.Type]::GetType('Microsoft.SharePoint.WebPartPages.WebPartTypeInfo, Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c')
  2. $mi_GetWellKnownTypeIdDictionary = $webPartTypeInfoType.GetMethod('GetWellKnownTypeIdDictionary', [Reflection.BindingFlags]'NonPublic, Static')
  3. $wellKnownTypeIdDictionary = $mi_GetWellKnownTypeIdDictionary.Invoke($null, $null)
  4.  
  5. $wpTypes = $wellKnownTypeIdDictionary.Keys | % { "INSERT INTO @WPTypes VALUES ('$_', '$($wellKnownTypeIdDictionary[$_].Assembly.FullName)', '$($wellKnownTypeIdDictionary[$_].FullName)')" }
  6. Set-Content -Path 'C:\Data\WPTypes.txt' -Value $wpTypes

And that is already the extended version of the SQL query:

  1. DECLARE @WPTypes TABLE
  2.    (
  3.      Id uniqueidentifier NOT NULL,
  4.      AssemblyName varchar(500),
  5.      ClassName varchar(100)
  6.    )
  7.  
  8. INSERT INTO @WPTypes VALUES ('8e20cf70-0fd5-1e08-9972-38f63a6bd59a', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ImageWebPart')
  9. INSERT INTO @WPTypes VALUES ('ba009853-eac3-16c8-9094-a8834485ad33', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.DataFormWebPart')
  10. INSERT INTO @WPTypes VALUES ('83216ab2-cd0e-e9fc-fc5e-6a8f3b21c37b', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.DataViewWebPart')
  11. INSERT INTO @WPTypes VALUES ('42fddde2-e0cf-c8ab-48b7-db1fcac0a917', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ListFormWebPart')
  12. INSERT INTO @WPTypes VALUES ('05d0fd94-372a-5ee7-b480-ccb8f9cd2c23', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ListViewWebPart')
  13. INSERT INTO @WPTypes VALUES ('aef28218-44f8-0538-9805-4842c0e62811', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.XsltListFormWebPart')
  14. INSERT INTO @WPTypes VALUES ('a6524906-3fd2-ee4e-23ee-252d3c6e0dc9', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.XsltListViewWebPart')
  15. INSERT INTO @WPTypes VALUES ('0c6143a7-d68b-bade-e0ef-2c4d01182b0c', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.BlogAdminWebPart')
  16. INSERT INTO @WPTypes VALUES ('afef48e1-8f94-eb71-03a6-ffceb685306a', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.BlogMonthQuickLaunch')
  17. INSERT INTO @WPTypes VALUES ('4c06cea2-364f-47e3-e1d7-08d53f441157', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ContentEditorWebPart')
  18. INSERT INTO @WPTypes VALUES ('e6047383-438e-ed87-1a93-f1ff71729044', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.TitleBarWebPart')
  19. INSERT INTO @WPTypes VALUES ('707c1e73-0b3d-898b-c755-01621802ab8c', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.SilverlightWebPart')
  20. INSERT INTO @WPTypes VALUES ('28c23aec-2537-68b3-43b6-845b13cea19f', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  21. INSERT INTO @WPTypes VALUES ('8d6034c4-a416-e535-281a-6b714894e1aa', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  22. INSERT INTO @WPTypes VALUES ('8e814083-396a-e7d1-148b-316e3a7283f7', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  23. INSERT INTO @WPTypes VALUES ('e6377261-6920-bbfe-501f-fda7a61db10f', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  24. INSERT INTO @WPTypes VALUES ('8efd140d-eae9-5feb-06e3-f771842d2e43', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.ErrorWebPart')
  25. INSERT INTO @WPTypes VALUES ('b3294a07-46bf-e661-d036-10670590bbd3', 'Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c', 'Microsoft.SharePoint.WebPartPages.SPUserCodeWebPart')
  26.  
  27. SELECT AD.DirName + '/' + AD.LeafName as PageUrl, AWP.tp_ZoneID as ZoneId, AWP.tp_PartOrder as WebPartOrder, ISNULL(AWP.tp_Class, WPT.ClassName) AS WebPartClass
  28. FROM AllWebParts AWP (nolock)
  29. INNER JOIN AllDocs AD (nolock) ON AWP.tp_SiteId = AD.SiteId AND AWP.tp_PageUrlID = AD.Id
  30. LEFT JOIN @WPTypes WPT ON AWP.tp_WebPartTypeId = WPT.Id
  31. WHERE ISNULL(AWP.tp_Class, WPT.ClassName) LIKE '%ContentEditorWebPart'

Of course, you can change the conditions of the query as you like, for example, you can restrict it to two web part type, like:

WHERE ISNULL(AWP.tp_Class, WPT.ClassName) IN (‘Microsoft.SharePoint.WebPartPages.ScriptEditorWebPart’, ‘Microsoft.SharePoint.WebPartPages.ContentEditorWebPart’)

There are a few more columns in the AllWebParts table, that you eventually would include either in the SELECT statement or in its WHERE clause, these are:

  • tp_IsIncluded: The web part is displayed on the page, if the value is 1 (default). If you close (not delete!) a web part, the value is 0. Deleted web parts are removed from the table.
  • tp_Deleted: Assume you have a list with some pages that includes web parts, like view pages including XsltListViewWebPart instances. The web part entries in the AllWebParts table have a value of 0 at this stage. If you delete the list, these values change to 1. The web part entries will be kept in the table even after deleting the list from the first (user) level Recycle Bin, and removed only after the list is deleted from the second (site collection) level Recycle Bin.
  • tp_ListId: This is a field that is populated for list-related built-in web parts, like XsltListViewWebPart. You can look up the related list and web instances by joining the Lists and Webs views in your query respectively, as shown below (this time I omit the declaration of the @WPTypes variable and its population with value for the sake of brevity, but of course, you need it this time either):

SELECT L.tp_Title as ListTitle, W.FullUrl AS WebUrl, AD.DirName + ‘/’ + AD.LeafName as PageUrl, AWP.tp_ZoneID as ZoneId, AWP.tp_PartOrder as WebPartOrder, ISNULL(AWP.tp_Class, WPT.ClassName) AS WebPartClass, tp_ListId
FROM AllWebParts AWP (nolock)
INNER JOIN AllDocs AD (nolock) ON AWP.tp_SiteId = AD.SiteId AND AWP.tp_PageUrlID = AD.Id
LEFT JOIN @WPTypes WPT ON AWP.tp_WebPartTypeId = WPT.Id
LEFT JOIN Lists L (nolock) ON AWP.tp_SiteId = L.tp_SiteId AND AWP.tp_ListId = L.tp_ID
LEFT JOIN Webs W (nolock) ON AWP.tp_SiteId = W.SiteId AND L.tp_WebId = W.Id
WHERE ISNULL(AWP.tp_Class, WPT.ClassName) LIKE ‘%ListViewWebPart’

By including the list title or the web URL in the WHERE clause (or the ID of the list or the web if you wish) you can further limit the items returned by the query.

If there are records returned with NULL in the ListTitle and WebUrl columns it means typically that the list was deleted, but yet available in the Recycle Bin. See my comments regarding the tp_Deleted field above. Note, that despite the name of the FullUrl column in the Web view, it is actually a server relative URL.

I hope this overview has helped you to better understand what and how is stored in these tables of the SharePoint content database, as well, how the “magical” IDs of the well-known web part types do fit into the whole picture.

Advertisements

July 16, 2017

Find Your Scripts in SharePoint within Seconds – the Effective, but Unsupported Way

Filed under: JavaScript, SP 2013, SQL, Tips & Tricks — Tags: , , , — Peter Holpar @ 18:41

The SharePoint environment I’m working on contains hundreds of webs. I create test sites for various tasks (like prototyping JSLink-based solutions) including the necessary lists, and store the .js and .css files typically in the Site Assets library of that web site to keep the solution artifacts (lists / files) together. It is rather common, that after I’ve finished the proof of concept, I don’t need it for months, then suddenly I should return to it, but I don’t find it anymore, as I don’t remember, which site I used for that solution.

For that kind of search I’ve created a simply SQL query to find the script directly in the content database of the site collection. Yes, I know it is officially unsupported to access the SharePoint databases directly, but I’m OK with that in my test system. Use it on your own risk.

SELECT [Id]
      ,[SiteId]
      ,[DirName]
      ,[LeafName]
      ,[TimeLastModified]
      ,[DeleteTransactionId]
  FROM [dbo].[AllDocs]
  WHERE LeafName LIKE ‘%.js’
  AND DirName LIKE ‘%SiteAssets%’
  ORDER BY TimeLastWritten DESC

This script lists the file name (LeadName) and path (DirName) of the scripts stored in various sites in their Site Assets library. The name of script and the date of last modification (TimeLastModified) is usually enough to identify the script I need. Note, that the records, that have a value other that 0x in the DeleteTransactionId column are recycled and located in the Recycle Bin. Of course, this method works only in the case of on-premise installations, and only as long as you have access to the SharePoint databases.

Create a free website or blog at WordPress.com.