Second Life of a Hungarian SharePoint Geek

June 29, 2011

How to limit the auto-generated code to SharePoint lists really used when working with ADO.NET Data Services?

Filed under: ADO.NET Data Services, SP 2010, Visual Studio — Tags: , , — Peter Holpar @ 01:08

Assume you have a rather complex SharePoint 2010 site with several lists and document libraries and would like to create a client application that access a single list using REST / ADO.NET Data Services.

What do you do in this case? If you use the Visual Studio built-in tools, like Data / Add New Data Source… menu, then the auto-generated code includes all of the entities of your site, most of them unused, and some might be even irrelevant to your actual solution.

For example, we need only the title of a single Tasks list item:

  1. HomeDataContext dc = new HomeDataContext(new Uri("http://sp2010/_vti_bin/listdata.svc"));
  2. dc.Credentials = CredentialCache.DefaultCredentials;
  4. var tasks = from task in dc.Tasks
  5.             select new
  6.             {
  7.                 task.Title
  8.             };
  9. var firstTask = tasks.FirstOrDefault();
  11. if (firstTask != null)
  12. {
  13.     Console.WriteLine(firstTask.Title);
  14. }

…and the generated file contains more than 30,000 lines of code, most of them are for classes you will never use, and maybe would not even like to be visible for users of your assembly (for example, through Reflector).

I have not found built-in tools to help developers to limit the code generation only to lists they really need. But you can do that with some fairly simple tricks.

In the following example I added the data source as SharePointData to my project:


If you switch Solution Explorer to Show All Files, then you will find two files (Reference.cs and service.edmx) that are automatically generated when adding the data source and each time you click on Update Service Reference menu in Visual Studio (it will get importance a bit later).


Reference.cs contains the classes you use when connecting to the data source and query data in your code.

These classes are generated from the list service metadata, that is returned by (replace sp2010 with the address of your site):


The request above returns an XML data document that serves as the service.edmx for the project. The Reference.cs is generated from the file. If we could (and we can!) remove the unused data from the XML document, we might be able (and we can do that either!) to generate code only for the remaining entities.

Let’s see some code how it works in practice!

The first code block is to retrieve the metadata from the list data service and store it in an XmlDocument instance for further processing:

  1. WebClient webClient = new WebClient();
  2. webClient.UseDefaultCredentials = true;
  3. // or you can set another credential
  4. //webClient.Credentials = new NetworkCredential("user", "password", "domain");
  6. byte[] metaData = webClient.DownloadData("http://sp2010/_vti_bin/listdata.svc/$metadata");
  7. MemoryStream metaDataStream = new MemoryStream(metaData);
  9. XmlDocument xmlMeta = new XmlDocument();
  10. xmlMeta.Load(metaDataStream);
  12. // this could be used as well instead of the stream-based approach illustrated above,
  13. // but seems to have issues with non-ASCII characters in entity names
  14. //String metaData = webClient.DownloadString("http://sp2010/_vti_bin/listdata.svc/$metadata");
  15. //xmlMeta.LoadXml(metaData);

The goal of the next main block is to process the XML document and remove the entities that is not needed:

  1. XmlNamespaceManager nsmgr = new XmlNamespaceManager(xmlMeta.NameTable);
  2. nsmgr.AddNamespace("edmx", "");
  3. nsmgr.AddNamespace("edm", "");
  5. // include your target lists and related lists and values here
  6. List<String> lists = new List<String> { "Tasks", "Attachments", "UserInformationList", "TasksPriority", "TasksStatus" };
  8. XmlNode schemaNode = xmlMeta.SelectSingleNode("edmx:Edmx/edmx:DataServices/edm:Schema", nsmgr);
  9. XmlNode entityContainerNode = schemaNode.SelectSingleNode("edm:EntityContainer", nsmgr);
  11. XmlNodeList entitySetNodes = entityContainerNode.SelectNodes("edm:EntitySet", nsmgr);
  13. foreach (XmlNode entitySetNode in entitySetNodes)
  14. {
  15.     XmlAttribute entityTypeAttr = entitySetNode.Attributes["EntityType"];
  16.     XmlAttribute nameAttr = entitySetNode.Attributes["Name"];
  17.     if ((entityTypeAttr != null) && (nameAttr != null))
  18.     {
  19.         String name = nameAttr.Value;
  21.         String entityType = entityTypeAttr.Value;
  22.         int lastDotPos = entityType.LastIndexOf(".");
  23.         if (lastDotPos > -1)
  24.         {
  25.             entityType = entityType.Substring(lastDotPos + 1);
  26.         }
  28.         if (!lists.Contains(name))
  29.         {
  30.             XmlNodeList entityTypeNodes = schemaNode.SelectNodes(
  31.                 String.Format("edm:EntityType[@Name='{0}']", entityType), nsmgr);
  32.             foreach (XmlNode entityTypeNode in entityTypeNodes)
  33.             {
  34.                 schemaNode.RemoveChild(entityTypeNode);
  35.             }
  37.             XmlNodeList associationSetNodes = entityContainerNode.SelectNodes(
  38.             String.Format("edm:AssociationSet[edm:End/@EntitySet='{0}']", name), nsmgr);
  39.             foreach (XmlNode associationSetNode in associationSetNodes)
  40.             {
  41.                 entityContainerNode.RemoveChild(associationSetNode);
  42.             }
  44.             XmlNodeList associationNodes = schemaNode.SelectNodes(
  45.                  String.Format("edm:Association[edm:End/@Role='{0}']", entityType), nsmgr);
  46.             foreach (XmlNode associationNode in associationNodes)
  47.             {
  48.                 schemaNode.RemoveChild(associationNode);
  49.             }
  51.             entityContainerNode.RemoveChild(entitySetNode);
  52.         }
  53.     }
  54. }

You should include the lists you want to work with in the lists list, including any related lists, for example UserInformationList that is bound through the Person or Group field types, like Created By and Modified By, or any other lists you have reference to through Lookup fields, and any other lists that those lists refer to, etc.

Note, that although the Tasks list contains a Lookup field called Predecessors we don’t have to include another list as this lookup refers to the same list, Tasks.

If your lists (or the related lists) contain Choice field types, you have to include those entities as well. See TasksPriority and TasksStatus in the case of Tasks list, corresponding to the Priority and Status choice fields.

If attachments to list items are enabled, you have to include the Attachments entity too.

The final block of code is about the code generation, but it stores the shortened edmx file as well. For code generation we create a EntityClassGenerator instance (System.Data.Services.Design namespace in System.Data.Services.Design.dll assembly in GAC) and use its GenerateCode method to generate the required classes. You have to add System.Data.Entity assembly as well to build the code. Try to find it at location like C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0\Profile\Client. BTW, my project was built targeting the .NET Framework 4 Client Profile.

  1. // replace with your intended output folder
  2. String outputPath = @"C:\YourOutputFolder\";
  3. File.WriteAllText(outputPath + "service.edmx", xmlMeta.OuterXml);
  4. // use LanguageOption.GenerateVBCode for VB code
  5. EntityClassGenerator cg = new EntityClassGenerator(LanguageOption.GenerateCSharpCode);
  6. StringReader sr = new StringReader(xmlMeta.OuterXml);
  7. XmlReader reader = XmlReader.Create(sr);
  8. try
  9. {
  10.     // replace with your namespace used in the data source reference
  11.     String nameSpace = "TestConsole.SharePointData";
  12.     using (FileStream outputStream = new FileStream(outputPath + "Reference.cs", FileMode.Create))
  13.     {
  14.         StreamWriter writer = new StreamWriter(outputStream, Encoding.Unicode);
  16.         // results an IList<System.Data.Metadata.Edm.EdmSchemaError> on warnings,
  17.         // but throws MetadataException on schema errors
  18.         var warnings = cg.GenerateCode(reader, writer, nameSpace);
  19.         foreach (var warning in warnings)
  20.         {
  21.             Console.WriteLine("Error severity '{3}' code: {1} ({2}) at line {0}",
  22.                 warning.Line, warning.ErrorCode, warning.Message, warning.Severity);
  23.         }
  24.         writer.Flush();
  25.     }
  26. }
  27. catch (MetadataException ex)
  28. {
  29.     Console.WriteLine(ex.Message);
  30. }

Although the GenerateCode method returns an IList<EdmSchemaError> you have to be prepared to handle exceptions. If you omitted any related entity from the lists list you will probably get a MetadataException (“Schema specified is not valid”) with the following stack trace:

at System.Data.Metadata.Edm.EdmItemCollection.LoadItems(IEnumerable`1 xmlReaders, IEnumerable`1 sourceFilePaths, SchemaDataModelOption dataModelOption, DbProviderManifest providerManifest, ItemCollection itemCollection, Boolean throwOnError)
at System.Data.Metadata.Edm.EdmItemCollection.Init(IEnumerable`1 xmlReaders, IEnumerable`1 filePaths, Boolean throwOnError)
at System.Data.Metadata.Edm.EdmItemCollection..ctor(IEnumerable`1 xmlReaders)
at System.Data.Services.Design.EntityClassGenerator.GenerateCode(XmlReader sourceReader, LazyTextWriterCreator target, String namespacePrefix)
at System.Data.Services.Design.EntityClassGenerator.GenerateCode(XmlReader sourceReader, String targetFilePath)

That behavior is due to the fact that the EdmItemCollection class is initialized (see its constructor calling the Init method) with throwOnError = true. So only warnings are returned, errors will be thrown as MetadataException. The Message property of the exception contains the missing entities.

Having the limited service.edmx and Reference.cs files generated we should overwrite the original versions in our project.

The Data Source view shows the new schema that contains only the Tasks and strongly related lists.


Our generated Reference.cs file contains “only” 1,600 lines that is a significant step from the original (30,000 lines in my case) version. Our original code that displays a single task entity (see the beginning of the post) should work with this limited version as well.

Don’t forget that you mustn’t use the Update Service Reference menu in Visual Studio if you want to refresh your data source. In this case the auto-generated files will be overwritten by the original version. Instead, you should refresh the code using the tool.

The tool introduced in this post can be used to generate a shorter code for your data sources. Of course, it would be more comfortable to use a tool integrated into Visual Studio. Based on the method shown here one can create such tool, but probably it requires a bit more time.

Alternatively, one could probably create a server side solution, like an ashx handler, that pre-process the result of metadata request on the server and returns only the necessary entities. In this case the integrated, out-of-the-box tools of Visual Studio could be used. My original goal was to create a tool that does not require installation on the server, but assuming a developer environment, where the SharePoint server is installed locally, it could not be a problem as well.

If it is, one can create a kind of proxy as well that is installed on the developer environment and intercepts requests / responses sent / received to / from the server. As you can see, there are a lot of options, the question is how many time you have to create the solution.

I feel a significant improvement would be to add a functionality to the current tool that requires only the entities you really need and explore and include its dependencies automatically.

I should note that I have a serious dislike regarding the auto generated code. The entity names in the code seems to be generated from the display name of the fields, so if a user changes a field name on the UI your code will fail. But assume users do not change field names. There are special, non-ASCII characters in the Hungarian language, so we use them on the SharePoint sites as well as field display names. The auto-generated C# code contains these non-ASCII characters either that I don’t like at all. Just to mention a few examples: “Responsible” will be “Felelős”, “Supplier” will be “Szállító”. How might it look like in case of sites created for Far Eastern languages like Japanese, Chinese or Korean? (I may check how it works for sites having field names in multiple languages.) Wouldn’t it be better to enable some kind of mapping of field names to names contain only ASCII characters?


Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at

%d bloggers like this: