Decoding the content of the BinarySerializedWebPart – The theory

When you save your SharePoint site as a template, the files within the generated WSP package contain the customized web parts as BinarySerializedWebPart, a format not easy to work with (see an example in my former post: Publishing files stored in the file system through external list).

Up to know I found no easy way to look inside the real content of such a web part. You can import the solution into Visual Studio 2010 (see example here), but I feel it a bit uncomfortable. If you can attach to the server using SharePoint Designer 2010, you can check the content there, but sometimes you might be offline, for example, the WSP was created in a restricted network and was sent via e-mail.

The following snippet illustrates a BinarySerializedWebPart:

1. <?xml version="1.0" encoding="utf-8"?>
2. <Elements xmlns="http://schemas.microsoft.com/sharepoint/">
3.   <Module Name="FileSystemForms" Url="Lists/Files" RootWebOnly="FALSE" SetupPath="pages">
4.     <File Url="DispForm.aspx" Type="Ghostable" Path="form.aspx">
5.       <BinarySerializedWebPart>
6.         <GUIDMap>
7.           <GUID Id="33ff2881_489d_4ce2_ac94_e81d64689d2a" ListUrl="Lists/Files" />
8.         </GUIDMap>
9.         <WebPart ID="{035cec7d-5f69-4dbf-a551-0b8203467c41}" WebPartIdProperty="" List="{$ListId:Lists/Files;}" Type="4"
10.           Flags="0" DisplayName="" Version="4" Url="Lists/Files/DispForm.aspx" WebPartOrder="1" WebPartZoneID="Main"
11.           IsIncluded="True" FrameState="0" WPTypeId="{feaafd58-2dc9-e199-be37-d6cdd7f84690}"
12.           SolutionId="{00000000-0000-0000-0000-000000000000}" Assembly="" Class="" Src=""
13.           AllUsers="B6Dt/kMAAAABAAAAAAAAAAIAAAAvX2xheW91dHMvaW1hZ2VzL2l0ZWJsLnBuZwAvRjFTaXRlL0xpc3RzL0ZpbGVzAP8BFCsAJQICAgMCAwEEAAICAhICFAEBAAIEBQtDb250cm9sTW9kZQspiAFNaWNyb3NvZnQuU2hhcmVQb2ludC5XZWJDb250cm9scy5TUENvbnRyb2xNb2RlLCBNaWNyb3NvZnQuU2hhcmVQb2ludCwgVmVyc2lvbj0xNC4wLjAuMCwgQ3VsdHVyZT1uZXV0cmFsLCBQdWJsaWNLZXlUb2tlbj03MWU5YmNlMTExZTk0MjljAQUIRm9ybVR5cGUCBAEAAAIWAoYBCyo0U3lzdGVtLldlYi5VSS5XZWJDb250cm9scy5XZWJQYXJ0cy5XZWJQYXJ0RXhwb3J0TW9kZQICggEFGi9fbGF5b3V0cy9pbWFnZXMvaXRlYmwucG5nAn0FEy9GMVNpdGUvTGlzdHMvRmlsZXMFCFBhZ2VUeXBlCyl3TWljcm9zb2Z0LlNoYXJlUG9pbnQuUEFHRVRZUEUsIE1pY3Jvc29mdC5TaGFyZVBvaW50LCBWZXJzaW9uPTE0LjAuMC4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2VuPTcxZTliY2UxMTFlOTQyOWMEBQdMaXN0VXJsZQUGTGlzdElkKClYU3lzdGVtLkd1aWQsIG1zY29ybGliLCBWZXJzaW9uPTIuMC4wLjAsIEN1bHR1cmU9bmV1dHJhbCwgUHVibGljS2V5VG9rZW49Yjc3YTVjNTYxOTM0ZTA4OSQzM2ZmMjg4MS00ODlkLTRjZTItYWM5NC1lODFkNjQ2ODlkMmEFD0xpc3REaXNwbGF5TmFtZWUClQEFJnszM0ZGMjg4MS00ODlELTRDRTItQUM5NC1FODFENjQ2ODlEMkF9BQ1YbWxEZWZpbml0aW9uBcUPDQo8VXNlckNvbnRyb2wgeDpDbGFzcz0iRm9ybVhtbFRvWGFtbC5Vc2VyQ29udHJvbDIiIHhtbG5zOng9Imh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vd2luZngvMjAwNi94YW1sIiB4bWxuczpTaGFyZVBvaW50PSJNaWNyb3NvZnQuU2hhcmVQb2ludC5XZWJDb250cm9scyIgeG1sbnM6c3lzdGVtPSJjbHItbmFtZXNwYWNlOlN5c3RlbTthc3NlbWJseT1tc2NvcmxpYiI+PFN0YWNrUGFuZWwgeDpOYW1lPSJGb3JtIj4NCjxTdGFja1BhbmVsLlJlc291cmNlcz4NCjxzeXN0ZW06U3RyaW5nIHg6S2V5PSJGb3JtTW9kZSI+RGlzcGxheTwvc3lzdGVtOlN0cmluZz4NCjxzeXN0ZW06U3RyaW5nIHg6S2V5PSJGb3JtVHlwZSI+TGlzdEZvcm08L3N5c3RlbTpTdHJpbmc+DQo8L1N0YWNrUGFuZWwuUmVzb3VyY2VzPg0KPFN0YWNrUGFuZWwgeDpOYW1lPSJNYWluU2VjdGlvbnMiPjxHcmlkPjxHcmlkLkNvbHVtbkRlZmluaXRpb25zPg0KPENvbHVtbkRlZmluaXRpb24gU3R5bGU9IntTdGF0aWNSZXNvdXJjZSBtcy1mb3JtbGFiZWx9Ii8+DQo8Q29sdW1uRGVmaW5pdGlvbiBTdHlsZT0ie1N0YXRpY1Jlc291cmNlIG1zLWZvcm1ib2R5fSIvPg0KPC9HcmlkLkNvbHVtbkRlZmluaXRpb25zPjxHcmlkLlJvd0RlZmluaXRpb25zPg0KPFJvd0RlZmluaXRpb24gLz4NCjxSb3dEZWZpbml0aW9uIC8+DQo8Um93RGVmaW5pdGlvbiAvPg0KPFJvd0RlZmluaXRpb24gLz4NCjwvR3JpZC5Sb3dEZWZpbml0aW9ucz4NCjxTaGFyZVBvaW50OkZpZWxkTGFiZWwgR3JpZC5Db2x1bW49IjAiIEdyaWQuUm93PSIwIiBDb250cm9sTW9kZT0iRGlzcGxheSIgRmllbGROYW1lPSJOYW1lIiAvPg0KPENvbW1lbnQgRmllbGROYW1lPSJOYW1lIiBGaWVsZEludGVybmFsTmFtZT0iTmFtZSIgRmllbGRUeXBlPSJUZXh0IiAvPg0KPFNoYXJlUG9pbnQ6Rm9ybUZpZWxkIEdyaWQuQ29sdW1uPSIxIiBHcmlkLlJvdz0iMCIgQ29udHJvbE1vZGU9IkRpc3BsYXkiIEZpZWxkTmFtZT0iTmFtZSIgSW5jbHVkZURlc2NyaXB0aW9uPSJUcnVlIi8+DQo8U2hhcmVQb2ludDpGaWVsZExhYmVsIEdyaWQuQ29sdW1uPSIwIiBHcmlkLlJvdz0iMSIgQ29udHJvbE1vZGU9IkRpc3BsYXkiIEZpZWxkTmFtZT0iU2l6ZSIgLz4NCjxDb21tZW50IEZpZWxkTmFtZT0iU2l6ZSIgRmllbGRJbnRlcm5hbE5hbWU9IlNpemUiIEZpZWxkVHlwZT0iSW50ZWdlciIgLz4NCjxTaGFyZVBvaW50OkZvcm1GaWVsZCBHcmlkLkNvbHVtbj0iMSIgR3JpZC5Sb3c9IjEiIENvbnRyb2xNb2RlPSJEaXNwbGF5IiBGaWVsZE5hbWU9IlNpemUiIEluY2x1ZGVEZXNjcmlwdGlvbj0iVHJ1ZSIvPg0KPFNoYXJlUG9pbnQ6RmllbGRMYWJlbCBHcmlkLkNvbHVtbj0iMCIgR3JpZC5Sb3c9IjIiIENvbnRyb2xNb2RlPSJEaXNwbGF5IiBGaWVsZE5hbWU9IkNyZWF0ZWQiIC8+DQo8Q29tbWVudCBGaWVsZE5hbWU9IkNyZWF0ZWQiIEZpZWxkSW50ZXJuYWxOYW1lPSJDcmVhdGVkIiBGaWVsZFR5cGU9IkRhdGVUaW1lIiAvPg0KPFNoYXJlUG9pbnQ6Rm9ybUZpZWxkIEdyaWQuQ29sdW1uPSIxIiBHcmlkLlJvdz0iMiIgQ29udHJvbE1vZGU9IkRpc3BsYXkiIEZpZWxkTmFtZT0iQ3JlYXRlZCIgSW5jbHVkZURlc2NyaXB0aW9uPSJUcnVlIi8+DQo8U2hhcmVQb2ludDpGaWVsZExhYmVsIEdyaWQuQ29sdW1uPSIwIiBHcmlkLlJvdz0iMyIgQ29udHJvbE1vZGU9IkRpc3BsYXkiIEZpZWxkTmFtZT0iTGFzdE1vZGlmaWVkIiAvPg0KPENvbW1lbnQgRmllbGROYW1lPSJMYXN0IG1vZGlmaWVkIiBGaWVsZEludGVybmFsTmFtZT0iTGFzdE1vZGlmaWVkIiBGaWVsZFR5cGU9IkRhdGVUaW1lIiAvPg0KPFNoYXJlUG9pbnQ6Rm9ybUZpZWxkIEdyaWQuQ29sdW1uPSIxIiBHcmlkLlJvdz0iMyIgQ29udHJvbE1vZGU9IkRpc3BsYXkiIEZpZWxkTmFtZT0iTGFzdE1vZGlmaWVkIiBJbmNsdWRlRGVzY3JpcHRpb249IlRydWUiLz4NCjwvR3JpZD4NCjwvU3RhY2tQYW5lbD4NCjwvU3RhY2tQYW5lbD4NCjwvVXNlckNvbnRyb2w+AktkBRFQYXJhbWV0ZXJCaW5kaW5ncwXoAg0KPFBhcmFtZXRlckJpbmRpbmcgTmFtZT0iZHZ0X2Fwb3MiIExvY2F0aW9uPSJQb3N0YmFjaztDb25uZWN0aW9uIi8+DQogICAgICAgIDxQYXJhbWV0ZXJCaW5kaW5nIE5hbWU9IlVzZXJJRCIgTG9jYXRpb249IkNBTUxWYXJpYWJsZSIgRGVmYXVsdFZhbHVlPSJDdXJyZW50VXNlck5hbWUiLz4NCiAgICAgICAgPFBhcmFtZXRlckJpbmRpbmcgTmFtZT0iVG9kYXkiIExvY2F0aW9uPSJDQU1MVmFyaWFibGUiIERlZmF1bHRWYWx1ZT0iQ3VycmVudERhdGUiLz4NCiAgICAgICAgPFBhcmFtZXRlckJpbmRpbmcgTmFtZT0iTGlzdEl0ZW1JZCIgTG9jYXRpb249IlF1ZXJ5U3RyaW5nKElEKSIgRGVmYXVsdFZhbHVlPSIwIi8+DQogICAgICAgIA==" />
14.       </BinarySerializedWebPart>
15.       </File>
16.     </Module>
17. </Elements>

You can see at the first sight that the most critical part of the XML is the AllUsers attribute of the WebPart node.

Other BinarySerializedWebPart nodes may contain View or PerUser attributes with similar format, and another attribute, WPTypeId that is a GUID.

In this post I try to help you to understand the decoding (or we can call it deserialization but it is not really deserialization in its traditional meaning) process SharePoint applies when reading web parts from content database and on the other side the encoding (or serialization) of the web part when creating the WSP package on site template export.

If you don’t like to track call chains you might want to skip the following section although I think it may help to understand the whole story and gives you a picture about what happens under the cover of the export process and some other areas related to the solution of the problem, like how web parts are reconstructed from content database by a web part manager. In this case you can check my next post that contains the actual code.

First, the public static ExportWeb method of SPSolutionExporter class (Microsoft.SharePoint namespace in Microsoft.SharePoint assembly) is called. The sole task of this method is to instantiate a SPSolutionExporter object to call its private ExportWebAsSolution method. This method calls forward to another private method GenerateSolutionFiles and this later one calls the private ExportWebWebPart method calls the proc_EnumerateWebPartsForWeb stored procedure and reads the web parts from the database including the properties we need.

If you are a regular visitor of the SharePoint content database you may be familiar with the AllWebParts table and its fields. The four fields we focus on are neighbors in this table:  tp_View, tp_WebPartTypeId, tp_AllUsersProperties, tp_PerUserProperties. These fields are bound to the above-mentioned attributes.

The ExportWebWebPart method aggregates the web parts in the WebWebParts property (that is of type SortedList<string, List<WebWebPart>>) of the SPSolutionExporter class.

The information about the web part is stored in the members of the private WebWebPart class, the relevant ones are:

1. public byte[] AllUsersProperties;
2. public byte[] PerUserProperties;
3. public byte[] View;
4. public Guid WebpartTypeId;

Later the GenerateSolutionFiles method calls ExportModules and ExportLists methods, then both of these methods call the WriteModuleInnerElementsOfElements method. This method calls the WriteModuleElementIntoFeatureManifest method: once for the customized and once for the uncustomized entries. In this method the EmitAllUsersWebpart method is called, that calls forward into the WriteBlobWebpart method.

In WriteBlobWebpart method you can see that most of the attributes are written as simple strings, however byte arrays are written as Base64 encoded values. It means that the attributes with more complex values contain the same binary context as their corresponding database fields in the AllWebParts table.

OK, we already know that the values of these properties must be Base64 decoded to get a byte array. That is not a surprise for most of us, it could have been said at first sight having a bit of experience in Base64. But the question still remains: how could we reconstruct a web part using these binary values?

To answer the question, let’s see what happens internally in the SPWebPartManager (Microsoft.SharePoint.WebPartPages namespace, Microsoft.SharePoint assembly) when it reads the web parts from the content database.

For the sake of simplicity I list only the methods involved in the process as their follow each other, and limit myself only to the methods that lead us to the “decoding algorithm”:

OnInit -> OnPageInitComplete -> LoadWebParts -> CreateWebPartsFromRowSetData

The CreateWebPartsFromRowSetData method calls the internal static  CreateBinaryDeserializer method of the internal  BinaryWebPartDeserializer class (Microsoft.SharePoint.WebPartPages namespace in Microsoft.SharePoint assembly) that returns an instance of BinaryWebPartDeserializer or its derived internal class SPUserCodeWebPartDeserializer. Then its internal Deserialize method is called that finally returns a WebPart instance.

When you follow the call chain in Reflector, at this point you might be a bit misled, as when you click on the Deserialize method in the CreateWebPartsFromRowSetData method, the virtual Deserialize method of the BinaryWebPartDeserializer class is displayed, however, since the SPUserCodeWebPartDeserializer class might be instantiated within the CreateBinaryDeserializer method, you may have to follow the override Deserialize method of the SPUserCodeWebPartDeserializer class. The first step in this method is to call the Deserialize method of the base class.

The Deserialize method of the BinaryWebPartDeserializer class calls the LoadInitialWebPart method that decodes the byte array properties by calling the ByteArrayToObjectArray method of the BinaryWebPartSerialization class (Microsoft.SharePoint.WebPartPages namespace, Microsoft.SharePoint assembly). This method then calls the static DeserializeByteArrayToObject method (same namespace and assembly as former one). Finally the Deserialize and DeserializeValue methods of the ObjectStateFormatter class (System.Web.UI namespace, System.Web assembly) do the actual work of reading object from stream.

Important thing to note. The Deserialize method contains a check that the first byte of the stream must be 255, and the second one must be 1, otherwise an exception of “The serialized data is invalid” will be thrown. The binary data stored in the fields of AllWebParts table do not always start with these bytes, similar to the Base64 decoded XML attributes of the BinarySerializedWebPart node.

In my tests I simply ignored the bytes in the byte array before this specific byte pattern, and using the code showed that splitting the leading bytes produces a useable result. How the actual code does look like will be shown in the next part of the post.