Second Life of a Hungarian SharePoint Geek

February 17, 2016

Visual Studio Build Error: Could not copy the file because it was not found

Filed under: Bugs, Visual Studio — Tags: , — Peter Holpar @ 22:03

Recently we got a bunch of errors when building a project in Visual Studio 2013, complaining about missing content files:

Could not copy the file "C:\projects\OurProject\images\image1.png" because it was not found.

We checked the files in the file system, but they have really disappeared. We had luck, as we had the original files under source control, so we could restore them from the repository. But it did not provide a long-term solution, as we got the very same error on the next build, so we had to restore them again. Restarting Visual Studio did not help as well.

To prohibit further deletion I set the files as read-only. It led me to the solution, as we got this time another build error message:

Unable to copy file "C:\projects\OurProject\images\image1.png" to ".\image1.png". Access to the path ‘.\image1.png’ is denied.

The problem was that the Output path (under project properties Build / Output) was accidentally deleted, so Visual Studio was to create the build at the source file location. Resetting the original value, the default bin/debug path resolved our issue.

One can simple reproduce the issue, simply create a new project (let it be a Console Application, for example), add an image as content file (Build Action = Content) to it and set its Copy to Output Directory property to Copy always. Then delete the content of the Output path property, save the project and try to build.

I know that it was our mistake, but to tell the truth, I would expect Visual Studio not to delete my source files, but rather validate my input, and not allow to leave the Output path empty.

March 5, 2013

Deleting files from the IE cache as part of the Visual Studio deployment process

Filed under: CKS.Dev, Internet Explorer, SP 2010, Visual Studio, VSX — Tags: , , , , — Peter Holpar @ 21:44

In the past months I had again a lot to do with client side SharePoint development, that means in practice mainly projects including ribbon extensions and tons of JavaScript files. One of the issues with that type of development is the bad habit of Internet Explorer called caching. For example, when you make modifications to the .js files, and re-deploy your project, it is rather common, that IE executes the former versions of the scripts, that I find pretty annoying.

Some of the workarounds I found for that issue in the past years:

Option Nr.1: Disable caching in IE (Internet options / General / Browsing history / Settings). It is OK if you use your environment only for development (like a Dev-VM), but not optimal if you should use the same browser to fulfill your other daily tasks.


Option Nr.2: Type the full URL of the script in the address text box of IE, download a local copy of the script to a temporary folder, and refresh the page with Ctrl+F5. Rather cumbersome method, especially if you have to refresh several scripts .

Option Nr.3: Open the location of the cache (Internet options / General / Browsing history / Settings / View files, marked with blue on the screenshot above), and delete the file(s) manually. The location in the file system is typically C:\Users\UserName\AppData\Local\Microsoft\Windows\Temporary Internet Files. Works quite good, but it takes some time and is still something you can forget.

Wouldn’t it be more comfortable to automate the process, for example, as a deployment step in Visual Studio? Definitely, but how to achieve that? Well, deleting files from the IE cache programmatically is far from being straightforward, but fortunately I found a nice example on MSDN for a sample wrapper class in C#, including a lot of PInvoke calls. Having this solution, the Visual Studio Extensibility part of the exercise was a routine task.

My first idea was to contribute this VS extension to the CKS.Dev project on CodePlex (see some posts on my other contributions to CKS.Dev here), but the project is just being upgraded to SharePoint 2013 / Visual Studio 2012, so I extended my own extension (documented here) instead.

Expected functionality:

I would like to specify through Visual Studio project properties, which files should be deleted from the cache (e.g. from the Temporary Internet Files folder).

First, I introduce a new property called IE cache delete rule that determines the scope of the action and has the following possible values:

Only from current site – only matching files cached from the active SharePoint site (the one you specified as the target of the deployment in VS) are deleted.

All from current sites – all files cached from the active SharePoint site are deleted, other filters are ignored.

No site specific – all cached files that fulfill the filters are deleted, independently from the origin URL.

Filters (there is a logical OR between the filters that means a cached file should fulfill at least one of the conditions to be deleted):

IE cache file list – It is a list of the names of the files to be deleted from the cache.

IE cache file pattern – It is a regular expression pattern. All files matching the pattern will be deleted from the cache, as long as it also matches the scope of the action (see IE cache delete rule property above)

Let’s have a look at the implementation!

My original SPVSProjectProps class was extended with the new properties:

Code Snippet
  1. // IECacheFilePattern property related members
  2. private const string IECacheFilePatternPropertyId = "IECacheFilePattern";
  3. private const string IECacheFilePatternPropertyDefaultValue = "";
  5. [DisplayName("IE cache file pattern")]
  6. [DescriptionAttribute("This property specifies a regular expression pattern to determine which files should be deleted from the IE cache when the Clear IE cache deployment step is activated")]
  7. [DefaultValue(IECacheFilePatternPropertyDefaultValue)]
  8. // we want our custom property to show up in the SharePoint section in the project properties property grid
  9. [Category("SharePoint")]
  10. public string IECacheFilePattern
  11. {
  12.     get
  13.     {
  14.         string propertyValue;
  15.         int hr = projectStorage.GetPropertyValue(IECacheFilePatternPropertyId, string.Empty,
  16.             (uint)_PersistStorageType.PST_PROJECT_FILE, out propertyValue);
  18.         // Try to get the current value from the project file; if it does not yet exist, return a default value.
  19.         if (!ErrorHandler.Succeeded(hr) || String.IsNullOrEmpty(propertyValue))
  20.         {
  21.             propertyValue = IECacheFilePatternPropertyDefaultValue;
  22.         }
  24.         return propertyValue;
  25.     }
  27.     set
  28.     {
  29.         projectStorage.SetPropertyValue(IECacheFilePatternPropertyId, string.Empty,
  30.             (uint)_PersistStorageType.PST_PROJECT_FILE, value);
  31.     }
  32. }
  34. // IECacheFileList property related members
  35. private const string IECacheFileListPropertyId = "IECacheFileList";
  36. private const string IECacheFileListPropertyDefaultValue = "";
  38. [DisplayName("IE cache file list")]
  39. [Description("This property specifies which files should be deleted from the IE cache when the Clear IE cache deployment step is activated")]
  40. [DefaultValue(RelatedTimerJobsPropertyDefaultValue)]
  41. // we want our custom property to show up in the SharePoint section in the project properties property grid
  42. [Category("SharePoint")]
  43. // use custom property editor to avoid "Constructor on type 'System.String' not found." error in design mode
  44. [Editor("System.Windows.Forms.Design.StringArrayEditor, System.Design", typeof(UITypeEditor))]
  45. [TypeConverter(typeof(CsvArrayConverter))]
  46. public String[] IECacheFileList
  47. {
  48.     get
  49.     {
  50.         String propertyValue;
  51.         int hr = projectStorage.GetPropertyValue(IECacheFileListPropertyId, string.Empty,
  52.             (uint)_PersistStorageType.PST_PROJECT_FILE, out propertyValue);
  54.         // Try to get the current value from the project file; if it does not yet exist, return a default value.
  55.         if (!ErrorHandler.Succeeded(hr) || String.IsNullOrEmpty(propertyValue))
  56.         {
  57.             propertyValue = IECacheFileListPropertyDefaultValue;
  58.         }
  60.         // remove accidental whitespaces
  61.         String[] fileNames = propertyValue.Split(new char[] { '|' }, StringSplitOptions.RemoveEmptyEntries);
  62.         fileNames = Array.ConvertAll<String, String>(fileNames, fileName => fileName.Trim());
  63.         return fileNames;
  64.     }
  66.     set
  67.     {
  68.         String propertyValue =
  69.             (value == null) ?
  70.             String.Empty :
  71.             // remove accidental whitespaces
  72.             String.Join("|", Array.ConvertAll<String, String>(value, fileName => fileName.Trim()));
  73.         projectStorage.SetPropertyValue(IECacheFileListPropertyId, string.Empty,
  74.             (uint)_PersistStorageType.PST_PROJECT_FILE, propertyValue);
  75.     }
  76. }
  78. // IECacheDeleteRule property related members
  79. private const string IECacheDeleteRulePropertyId = "IECacheDeleteRule";
  80. private const IECacheDeleteRules IECacheDeleteRulePropertyDefaultValue = IECacheDeleteRules.OnlyCurrentSite;
  82. [DisplayName("IE cache delete rule")]
  83. [Description("This property specifies the relation between the current SharePoint site and files to be deleted from IE cache")]
  84. [DefaultValue(IECacheDeleteRulePropertyDefaultValue)]
  85. // we want our custom property to show up in the SharePoint section in the project properties property grid
  86. [Category("SharePoint")]
  87. [TypeConverter(typeof(IECacheDeleteRuleConverter))]
  88. public IECacheDeleteRules IECacheDeleteRule
  89. {
  90.     get
  91.     {
  92.         // set default value
  93.         IECacheDeleteRules propertyValue = IECacheDeleteRulePropertyDefaultValue;
  94.         string propertyValueString;
  95.         int hr = projectStorage.GetPropertyValue(IECacheDeleteRulePropertyId, string.Empty,
  96.             (uint)_PersistStorageType.PST_PROJECT_FILE, out propertyValueString);
  98.         // Try to get the current value from the project file; if it does not yet exist, return a default value.
  99.         if (ErrorHandler.Succeeded(hr) && !String.IsNullOrEmpty(propertyValueString))
  100.         {
  101.             Enum.TryParse<IECacheDeleteRules>(propertyValueString, out propertyValue);
  102.         }
  104.         return propertyValue;
  105.     }
  107.     set
  108.     {
  109.         projectStorage.SetPropertyValue(IECacheDeleteRulePropertyId, string.Empty,
  110.             (uint)_PersistStorageType.PST_PROJECT_FILE, value.ToString());
  111.     }
  112. }
  114. public enum IECacheDeleteRules
  115. {
  116.     [Description("Only from current site")]
  117.     OnlyCurrentSite,
  118.     [Description("All from current site")]
  119.     AllFromCurrentSite,
  120.     [Description("No site specific")]
  121.     NoSiteSpecific
  122. }
  124. // based on EnumConverter example from
  125. //
  126. class IECacheDeleteRuleConverter : EnumConverter
  127. {
  128.     private Type enumType;
  130.     public IECacheDeleteRuleConverter(Type type) : base(type)
  131.     {
  132.         enumType = type;
  133.     }
  135.     public override bool CanConvertTo(ITypeDescriptorContext context, Type destType)
  136.     {
  137.         return destType == typeof(string);
  138.     }
  140.     public override object ConvertTo(ITypeDescriptorContext context, CultureInfo culture, object value, Type destType)
  141.     {
  142.         FieldInfo fi = enumType.GetField(Enum.GetName(enumType, value));
  143.         DescriptionAttribute dna = (DescriptionAttribute)Attribute.GetCustomAttribute(fi, typeof(DescriptionAttribute));
  144.         if (dna != null)
  145.             return dna.Description;
  146.         else
  147.             return value.ToString();
  148.     }
  150.     public override bool CanConvertFrom(ITypeDescriptorContext context, Type srcType)
  151.     {
  152.         return srcType == typeof(string);
  153.     }
  155.     public override object ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, object value)
  156.     {
  157.         foreach (FieldInfo fi in enumType.GetFields())
  158.         {
  159.             DescriptionAttribute dna =
  160.             (DescriptionAttribute)Attribute.GetCustomAttribute(fi, typeof(DescriptionAttribute));
  161.             if ((dna != null) && ((string)value == dna.Description))
  162.                 return Enum.Parse(enumType, fi.Name);
  163.         }
  164.         return Enum.Parse(enumType, (string)value);
  165.     }
  166. }

We have two new methods to the static ExtensionHelper class. The ShouldDeleteIECacheFile method encapsulates the logic of cache file deletion as function of our new project properties. The ClearIECacheFile method calls the core cache file deletion (see ClearIEFiles method of the DeleteIECache class below) with this logic injected as a parameter. We do some logging in both of these methods to inform users in the Output window of VS about the progress and result of the deployment. We should keep the list of deleted files in the local filesDeleted variable, otherwise each files would be listed twice due to the logic implemented in the ClearIEFiles method.

Code Snippet
  1. internal static void ClearIECacheFile(ISharePointProject project)
  2. {
  3.     try
  4.     {
  5.         LogToOutputWindows(project, "Clearing files from IE cache");
  6.         SPVSProjectProps propertiesObject;
  7.         if (project.Annotations.TryGetValue<SPVSProjectProps>(out propertiesObject))
  8.         {
  9.             String ieCacheFilePattern = propertiesObject.IECacheFilePattern;
  10.             IEnumerable<String> ieCacheFileList = propertiesObject.IECacheFileList;
  11.             SPVSProjectProps.IECacheDeleteRules ieCacheDeleteRule = propertiesObject.IECacheDeleteRule;
  12.             List<String> filesDeleted = new List<String>();
  14.             Func<Uri, bool> shouldDelete = new Func<Uri, bool>(u => ShouldDeleteIECacheFile(u,
  15.                                                                                         ieCacheFilePattern,
  16.                                                                                         ieCacheFileList,
  17.                                                                                         ieCacheDeleteRule,
  18.                                                                                         filesDeleted,
  19.                                                                                         project));
  20.             DeleteIECache.ClearIEFiles(shouldDelete);
  22.             LogToOutputWindows(project, String.Format("Number of files deleted from IE cache: {0}", filesDeleted.Count));
  23.         }
  24.     }
  25.     catch (Exception ex)
  26.     {
  27.         LogToOutputWindows(project, String.Format("Clearing files from IE cache failed. Exception: {0}", ex.Message));
  28.     }
  29. }
  31. internal static bool ShouldDeleteIECacheFile(Uri uri, String filePattern, IEnumerable<String> fileList,
  32.     SPVSProjectProps.IECacheDeleteRules ieCacheDeleteRule, List<String> filesDeleted, ISharePointProject project)
  33. {
  34.     bool result = false;
  36.     Uri siteUrl = project.SiteUrl;
  37.     Regex filePatternRegex = string.IsNullOrEmpty(filePattern) ? null : new Regex(filePattern);
  38.     List<string> fileListEx = (fileList == null) ? new List<string>() : fileList.ToList();
  39.     string fileName = uri.Segments[uri.Segments.Length – 1];
  41.     bool isFromCurrentSite = uri.AbsoluteUri.IndexOf(siteUrl.AbsoluteUri, StringComparison.InvariantCultureIgnoreCase) == 0;
  42.     if (ieCacheDeleteRule == SPVSProjectProps.IECacheDeleteRules.AllFromCurrentSite)
  43.     {
  44.         result = isFromCurrentSite;
  45.     }
  46.     else
  47.     {
  48.         result = ((fileListEx.Any(f => (f.ToUpper() == fileName.ToUpper()))) ||
  49.                 (filePatternRegex != null) && (filePatternRegex.IsMatch(fileName)));
  50.         if (ieCacheDeleteRule == SPVSProjectProps.IECacheDeleteRules.OnlyCurrentSite)
  51.         {
  52.             result = result && isFromCurrentSite;
  53.         }
  54.     }
  56.     if ((result) && (!filesDeleted.Contains(uri.AbsoluteUri)))
  57.     {
  58.         filesDeleted.Add(uri.AbsoluteUri);
  59.         LogToOutputWindows(project, String.Format("Deleting file from IE cache: '{0}'", uri));
  60.     }
  62.     return result;
  63. }

I kept the DeleteIECache class and its ClearIEFiles method as it was published on MSDN, except this method has a Func<Uri, bool> shouldDelete parameter that we use to decide if a specific file should be deleted from the IE cache:

returnValue = shouldDelete(uri) ? DeleteUrlCacheEntry(internetCacheEntry.lpszSourceUrlName) : false;

The code for the deployment step is pretty straightforward, it simply calls the ClearIECacheFile method of our ExtensionHelper class. Since we don’t have to call any SharePoint-specific code on the server side (that means no x64 process), there is no need for SharePoint commands in this case. That makes our life easier.

Code Snippet
  1. [DeploymentStep("PHolpar.ClearIECache")]
  2. [Export(typeof(IDeploymentStep))]
  3. internal class ClearIECacheDeployStep : IDeploymentStep
  4. {
  6.     public bool CanExecute(IDeploymentContext context)
  7.     {
  8.         return true;
  9.     }
  11.     public void Execute(IDeploymentContext context)
  12.     {
  13.         ISharePointProject project = context.Project;
  14.         ExtensionHelper.ClearIECacheFile(project);
  15.     }
  17.     public void Initialize(IDeploymentStepInfo stepInfo)
  18.     {
  19.         stepInfo.Name = "Clear IE cache";
  20.         stepInfo.Description = "This step deletes the specified files from the Internet Explorer cache of the current user.";
  21.         stepInfo.StatusBarMessage = "IE cache is being cleared…";
  22.     }
  23. }

To test the new extension, I’ve created a deployment configuration called Clear IE cache.


This DC has only a single deployment step, our new Clear IE cache DS.


Using the following project properties (reg. exp. used: ^.*\.(js|JS)$) we can delete all .js files that were cached from the active SharePoint site:


We can specify the files to be deleted explicitly in the IE cache file list property:


In this case these files would be deleted from the cache independently from the URL they were downloaded from:


The next screenshot displays a sample deployment for the previous configuration:


Using the All from current sites value of IE cache delete rule (not illustrated here) the deployment process clears all files cached from the active SharePoint site, filtering properties (IE cache file list and IE cache file pattern) are ignored.

You can download the updated version (v 1.1) of my VS extension from the original location.

July 15, 2012

Automatically setting the next statement to a specific position when debugging in Visual Studio

Filed under: Debugging, Macros, Tips & Tricks, Visual Studio — Tags: , , , — Peter Holpar @ 21:58

Have you ever faced to the problem during debugging in Visual Studio, that you find a problem in a code block, and would like to avoid this code without altering the code and rebuilding / restarting  the application? In this case you need to set the next statement to a specific line of code using the context menu (see below).


It is trivial when you need to do it just  a few time, but it can be rather tedious, if not impossible, if you are – for example – in a loop processing several hundreds of items, and should do it for each items.

Fortunately, it is easy to achieve this goal using Visual Studio macros. The following method – defined in a module called in this example DebugHelpers – jumps to the next bookmark, and sets the next statement to that position.

Sub GoToNextBookmark()
End Sub

Having this method, all you need to do is to set a new breakpoint with When Hit… option at the line of code you want to jump from,


and set the GoToNextBookmark method in the Run a macro list:


Then mark the target of the jump with a bookmark, and you are ready.

October 6, 2011

Determining the path of the virtual directory for a SharePoint web application from Visual Studio

Filed under: CKS.Dev, SP 2010, Visual Studio — Tags: , , — Peter Holpar @ 11:41

The other day I was to add a minor extension to my customized version of CKS.Dev. If I select a deployable item (or a folder containing such items) in Solution Explorer, the extension should display a custom menu item that opens the deployment location using Windows Explorer.

It is similar to the standard Visual Studio 2010 menu item Open Folder in Windows Explorer, but instead of opening the original location, this menu should open the corresponding folder in the 14 hive (for items deployed to {SharePointRoot}) or in the InetPub the (for items deployed to {WebApplicationRoot}).

// BTW, I already added my own version of Open Folder in Windows Explorer to Visual Studio 2008 (and another one that opens the folder in the command shell) a few years ago and so was happy to welcome it built into VS 2010. Seems that other developers missed that feature as well.

It was easy to determine the root of the SharePoint installation (that is the 14 hive). Assuming you have a reference to an ISharePointProject (let’s call it project)  this path is available as project.ProjectService.SharePointInstallPath (the property is populated from the registry hive HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\14.0 with the Location value ). Actually, the current version of CKS.Dev uses exactly this way to get the path in the QuickDeploymentUtilities.SubstituteRootTokens static method. However, there is nothing that handles the {WebApplicationRoot} token.

So my first idea was to create a custom SharePoint command to retrieve the path of the virtual directory for the SharePoint application set for the current project in Visual Studio. But before doing so, I should have known how to get the value on the server side.

The path we need is available as a DirectoryInfo value in the Path property of the SPIisSettings class (Microsoft.SharePoint.Administration namespace, Microsoft.SharePoint assembly). To get the SPIisSettings instance corresponding to our web application we could use either the IisSettings property or the GetIisSettingsWithFallback method of the SPWebApplication class. In both cases we should specify the zone we need info about.

But after analyzing the Path property using Reflector it turned out that Microsoft has already created the command we need: the GetWebApplicationLocalPath method can be found in the SharePointCommands class (Microsoft.VisualStudio.SharePoint.Commands namespace, Microsoft.VisualStudio.SharePoint.Commands.Implementation.v4 assembly in GAC).

Note: If you check the SharePointCommands class you can see there are a lot of useful commands implemented in this class, and there are many more in other (usually not public) classes provided by Visual Studio 2010. I spent a few days last year with discovering the possibilities of these commands and I can assure you there are a few real gems among them. I’ve been planning a series of blog posts about this topic since then, but unfortunately I missed the time to publish them at the level I would like to do. So might be I will publish them in a kind of raw format to help your work. I think if you are at least aware of the existence of these commands it is a large step into the right direction.

Having the GetWebApplicationLocalPath method found (that is the Microsoft.VisualStudio.SharePoint.Commands.GetWebApplicationLocalPath command) it is quite trivial to get the path of the web application root:

String webApplicationRoot = project.SharePointConnection.ExecuteCommand<String>("Microsoft.VisualStudio.SharePoint.Commands.GetWebApplicationLocalPath");

September 21, 2011

Strong name verification issues when creating a customized build of the CKS.Dev solution

Filed under: CKS.Dev, SP 2010, Visual Studio — Tags: , , — Peter Holpar @ 22:44

Recently I made a minor modification to my local CKS.Dev version to enable accessing the Quick Deploy menu functionality through the keyboard shortcuts even when working on a code file. The standard behavior of version 2.1 is that you should select a SharePoint project in the Solution Explorer prior to using any of the key combination, otherwise you receive a message like this one:

The key combination (Alt + C, G) is bound to command (Copy to GAC/BIN) which is not currently available.

I found it to be a bit uncomfortable, so I loaded the solution into Visual Studio and made the required changes in the code. It turned out that altering the code was the easier part of the task.

When I tried to build the project, the following build errors were generated for the file CreatePkgDef:

Error    211    Could not load file or assembly ‘CKS.Dev, Version=, Culture=neutral, PublicKeyToken=487fd6341a5c701f’ or one of its dependencies. Strong name validation failed. (Exception from HRESULT: 0x8013141A)
Error    212    Strong name validation failed. (Exception from HRESULT: 0x8013141A)

Fortunately, I found a good description of the problem and the solution in a blog post from Charlie Holland.

Based on the post (and the Readme.txt in the CKS Developer Edition\Signing folder) I’ve started an instance of Visual Studio Command Prompt (2010) and run the DisableStrongNameVerification.bat command. It runs the sn command that adds a verification entry for assembly ‘*,487fd6341a5c701f’ to disable strong name verification for the CKS.Dev assemblies.

I’ve uninstalled the original version of CKS.Dev and started debugging the modified one using the experimental instance of VS 2010.

I pressed Alt + C, G, the command was fired through the shortcut as expected, deployed my assembly to the GAC, but finally generated the following alert:

Command ID ‘Deployment.GetApplicationPoolName’ is invalid.

And the following trace message was displayed in the Output window:

Could not load file or assembly ‘CKS.Dev.Commands.Implementation.v4, Version=, Culture=neutral, PublicKeyToken=487fd6341a5c701f’ or one of its dependencies. Strong name validation failed. (Exception from HRESULT: 0x8013141A)

So we have turned verification off, or might be not? What is the reason for that?

As you may know (if you don’t, you should read this post first), to bridge the gap between Visual Studio 2010 (x86 process) and SharePoint 2010 (x64 process), VS runs SharePoint commands through the x64 VSSPHOST4 process.

In the former steps we disabled strong name verification only for the x86 process (Visual Studio), but to disable it for x64 (VSSPHOST4) as well, we should start Visual Studio x64 Win64 Command Prompt (2010) and run the DisableStrongNameVerification.bat command again.

Note: You can check the independency of the different versions by starting a command prompt for both x86 and x64, and adding / removing / listing entries using the sn command and the –Vr / -Vx / -Vl switches.

Then kill the VSSPHOST4 process, or simply restart VS, and things work as we planned.

June 29, 2011

How to limit the auto-generated code to SharePoint lists really used when working with ADO.NET Data Services?

Filed under: ADO.NET Data Services, SP 2010, Visual Studio — Tags: , , — Peter Holpar @ 01:08

Assume you have a rather complex SharePoint 2010 site with several lists and document libraries and would like to create a client application that access a single list using REST / ADO.NET Data Services.

What do you do in this case? If you use the Visual Studio built-in tools, like Data / Add New Data Source… menu, then the auto-generated code includes all of the entities of your site, most of them unused, and some might be even irrelevant to your actual solution.

For example, we need only the title of a single Tasks list item:

  1. HomeDataContext dc = new HomeDataContext(new Uri("http://sp2010/_vti_bin/listdata.svc&quot;));
  2. dc.Credentials = CredentialCache.DefaultCredentials;
  4. var tasks = from task in dc.Tasks
  5.             select new
  6.             {
  7.                 task.Title
  8.             };
  9. var firstTask = tasks.FirstOrDefault();
  11. if (firstTask != null)
  12. {
  13.     Console.WriteLine(firstTask.Title);
  14. }

…and the generated file contains more than 30,000 lines of code, most of them are for classes you will never use, and maybe would not even like to be visible for users of your assembly (for example, through Reflector).

I have not found built-in tools to help developers to limit the code generation only to lists they really need. But you can do that with some fairly simple tricks.

In the following example I added the data source as SharePointData to my project:


If you switch Solution Explorer to Show All Files, then you will find two files (Reference.cs and service.edmx) that are automatically generated when adding the data source and each time you click on Update Service Reference menu in Visual Studio (it will get importance a bit later).


Reference.cs contains the classes you use when connecting to the data source and query data in your code.

These classes are generated from the list service metadata, that is returned by (replace sp2010 with the address of your site):


The request above returns an XML data document that serves as the service.edmx for the project. The Reference.cs is generated from the file. If we could (and we can!) remove the unused data from the XML document, we might be able (and we can do that either!) to generate code only for the remaining entities.

Let’s see some code how it works in practice!

The first code block is to retrieve the metadata from the list data service and store it in an XmlDocument instance for further processing:

  1. WebClient webClient = new WebClient();
  2. webClient.UseDefaultCredentials = true;
  3. // or you can set another credential
  4. //webClient.Credentials = new NetworkCredential("user", "password", "domain");
  6. byte[] metaData = webClient.DownloadData("http://sp2010/_vti_bin/listdata.svc/$metadata&quot;);
  7. MemoryStream metaDataStream = new MemoryStream(metaData);
  9. XmlDocument xmlMeta = new XmlDocument();
  10. xmlMeta.Load(metaDataStream);
  12. // this could be used as well instead of the stream-based approach illustrated above,
  13. // but seems to have issues with non-ASCII characters in entity names
  14. //String metaData = webClient.DownloadString("http://sp2010/_vti_bin/listdata.svc/$metadata&quot;);
  15. //xmlMeta.LoadXml(metaData);

The goal of the next main block is to process the XML document and remove the entities that is not needed:

  1. XmlNamespaceManager nsmgr = new XmlNamespaceManager(xmlMeta.NameTable);
  2. nsmgr.AddNamespace("edmx", ";);
  3. nsmgr.AddNamespace("edm", ";);
  5. // include your target lists and related lists and values here
  6. List<String> lists = new List<String> { "Tasks", "Attachments", "UserInformationList", "TasksPriority", "TasksStatus" };
  8. XmlNode schemaNode = xmlMeta.SelectSingleNode("edmx:Edmx/edmx:DataServices/edm:Schema", nsmgr);
  9. XmlNode entityContainerNode = schemaNode.SelectSingleNode("edm:EntityContainer", nsmgr);
  11. XmlNodeList entitySetNodes = entityContainerNode.SelectNodes("edm:EntitySet", nsmgr);
  13. foreach (XmlNode entitySetNode in entitySetNodes)
  14. {
  15.     XmlAttribute entityTypeAttr = entitySetNode.Attributes["EntityType"];
  16.     XmlAttribute nameAttr = entitySetNode.Attributes["Name"];
  17.     if ((entityTypeAttr != null) && (nameAttr != null))
  18.     {
  19.         String name = nameAttr.Value;
  21.         String entityType = entityTypeAttr.Value;
  22.         int lastDotPos = entityType.LastIndexOf(".");
  23.         if (lastDotPos > -1)
  24.         {
  25.             entityType = entityType.Substring(lastDotPos + 1);
  26.         }
  28.         if (!lists.Contains(name))
  29.         {
  30.             XmlNodeList entityTypeNodes = schemaNode.SelectNodes(
  31.                 String.Format("edm:EntityType[@Name='{0}']", entityType), nsmgr);
  32.             foreach (XmlNode entityTypeNode in entityTypeNodes)
  33.             {
  34.                 schemaNode.RemoveChild(entityTypeNode);
  35.             }
  37.             XmlNodeList associationSetNodes = entityContainerNode.SelectNodes(
  38.             String.Format("edm:AssociationSet[edm:End/@EntitySet='{0}']", name), nsmgr);
  39.             foreach (XmlNode associationSetNode in associationSetNodes)
  40.             {
  41.                 entityContainerNode.RemoveChild(associationSetNode);
  42.             }
  44.             XmlNodeList associationNodes = schemaNode.SelectNodes(
  45.                  String.Format("edm:Association[edm:End/@Role='{0}']", entityType), nsmgr);
  46.             foreach (XmlNode associationNode in associationNodes)
  47.             {
  48.                 schemaNode.RemoveChild(associationNode);
  49.             }
  51.             entityContainerNode.RemoveChild(entitySetNode);
  52.         }
  53.     }
  54. }

You should include the lists you want to work with in the lists list, including any related lists, for example UserInformationList that is bound through the Person or Group field types, like Created By and Modified By, or any other lists you have reference to through Lookup fields, and any other lists that those lists refer to, etc.

Note, that although the Tasks list contains a Lookup field called Predecessors we don’t have to include another list as this lookup refers to the same list, Tasks.

If your lists (or the related lists) contain Choice field types, you have to include those entities as well. See TasksPriority and TasksStatus in the case of Tasks list, corresponding to the Priority and Status choice fields.

If attachments to list items are enabled, you have to include the Attachments entity too.

The final block of code is about the code generation, but it stores the shortened edmx file as well. For code generation we create a EntityClassGenerator instance (System.Data.Services.Design namespace in System.Data.Services.Design.dll assembly in GAC) and use its GenerateCode method to generate the required classes. You have to add System.Data.Entity assembly as well to build the code. Try to find it at location like C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0\Profile\Client. BTW, my project was built targeting the .NET Framework 4 Client Profile.

  1. // replace with your intended output folder
  2. String outputPath = @"C:\YourOutputFolder\";
  3. File.WriteAllText(outputPath + "service.edmx", xmlMeta.OuterXml);
  4. // use LanguageOption.GenerateVBCode for VB code
  5. EntityClassGenerator cg = new EntityClassGenerator(LanguageOption.GenerateCSharpCode);
  6. StringReader sr = new StringReader(xmlMeta.OuterXml);
  7. XmlReader reader = XmlReader.Create(sr);
  8. try
  9. {
  10.     // replace with your namespace used in the data source reference
  11.     String nameSpace = "TestConsole.SharePointData";
  12.     using (FileStream outputStream = new FileStream(outputPath + "Reference.cs", FileMode.Create))
  13.     {
  14.         StreamWriter writer = new StreamWriter(outputStream, Encoding.Unicode);
  16.         // results an IList<System.Data.Metadata.Edm.EdmSchemaError> on warnings,
  17.         // but throws MetadataException on schema errors
  18.         var warnings = cg.GenerateCode(reader, writer, nameSpace);
  19.         foreach (var warning in warnings)
  20.         {
  21.             Console.WriteLine("Error severity '{3}' code: {1} ({2}) at line {0}",
  22.                 warning.Line, warning.ErrorCode, warning.Message, warning.Severity);
  23.         }
  24.         writer.Flush();
  25.     }
  26. }
  27. catch (MetadataException ex)
  28. {
  29.     Console.WriteLine(ex.Message);
  30. }

Although the GenerateCode method returns an IList<EdmSchemaError> you have to be prepared to handle exceptions. If you omitted any related entity from the lists list you will probably get a MetadataException (“Schema specified is not valid”) with the following stack trace:

at System.Data.Metadata.Edm.EdmItemCollection.LoadItems(IEnumerable`1 xmlReaders, IEnumerable`1 sourceFilePaths, SchemaDataModelOption dataModelOption, DbProviderManifest providerManifest, ItemCollection itemCollection, Boolean throwOnError)
at System.Data.Metadata.Edm.EdmItemCollection.Init(IEnumerable`1 xmlReaders, IEnumerable`1 filePaths, Boolean throwOnError)
at System.Data.Metadata.Edm.EdmItemCollection..ctor(IEnumerable`1 xmlReaders)
at System.Data.Services.Design.EntityClassGenerator.GenerateCode(XmlReader sourceReader, LazyTextWriterCreator target, String namespacePrefix)
at System.Data.Services.Design.EntityClassGenerator.GenerateCode(XmlReader sourceReader, String targetFilePath)

That behavior is due to the fact that the EdmItemCollection class is initialized (see its constructor calling the Init method) with throwOnError = true. So only warnings are returned, errors will be thrown as MetadataException. The Message property of the exception contains the missing entities.

Having the limited service.edmx and Reference.cs files generated we should overwrite the original versions in our project.

The Data Source view shows the new schema that contains only the Tasks and strongly related lists.


Our generated Reference.cs file contains “only” 1,600 lines that is a significant step from the original (30,000 lines in my case) version. Our original code that displays a single task entity (see the beginning of the post) should work with this limited version as well.

Don’t forget that you mustn’t use the Update Service Reference menu in Visual Studio if you want to refresh your data source. In this case the auto-generated files will be overwritten by the original version. Instead, you should refresh the code using the tool.

The tool introduced in this post can be used to generate a shorter code for your data sources. Of course, it would be more comfortable to use a tool integrated into Visual Studio. Based on the method shown here one can create such tool, but probably it requires a bit more time.

Alternatively, one could probably create a server side solution, like an ashx handler, that pre-process the result of metadata request on the server and returns only the necessary entities. In this case the integrated, out-of-the-box tools of Visual Studio could be used. My original goal was to create a tool that does not require installation on the server, but assuming a developer environment, where the SharePoint server is installed locally, it could not be a problem as well.

If it is, one can create a kind of proxy as well that is installed on the developer environment and intercepts requests / responses sent / received to / from the server. As you can see, there are a lot of options, the question is how many time you have to create the solution.

I feel a significant improvement would be to add a functionality to the current tool that requires only the entities you really need and explore and include its dependencies automatically.

I should note that I have a serious dislike regarding the auto generated code. The entity names in the code seems to be generated from the display name of the fields, so if a user changes a field name on the UI your code will fail. But assume users do not change field names. There are special, non-ASCII characters in the Hungarian language, so we use them on the SharePoint sites as well as field display names. The auto-generated C# code contains these non-ASCII characters either that I don’t like at all. Just to mention a few examples: “Responsible” will be “Felelős”, “Supplier” will be “Szállító”. How might it look like in case of sites created for Far Eastern languages like Japanese, Chinese or Korean? (I may check how it works for sites having field names in multiple languages.) Wouldn’t it be better to enable some kind of mapping of field names to names contain only ASCII characters?

June 28, 2011

Creating WSP packages for all of your SharePoint projects

Filed under: Deployment, SP 2010, Visual Studio — Tags: , , — Peter Holpar @ 09:19

Assume you have a solution in Visual Studio 2010 with several SharePoint projects. You have to create the WSP files for the SharePoint artifacts, but no deployment.

In this case you can use the built-in Package menu on the project Build menu as described here. Of course, you have to package the projects one by one.

If you have to do it several times and you have many SharePoint projects in your solution it might be quicker to create your own deployment configuration as illustrated below:


and set it as the default deployment step:


After you repeat these steps for each of the SharePoint projects, you can select the Deploy Solution in the solution menu. Your project will be packaged at a single action, but without deployment.

BTW deployment configurations: Interesting behavior (bug?) in Visual Studio 2010 that you can delete the active deployment configuration (see image below), that results the following error on deployment:

The active deployment configuration ‘Package Only – No Deployment’ cannot be located. Specify a valid deployment configuration in the Project Properties.


June 25, 2011

An error occurred while signing: Key not valid for use in specified state

The other day I had to create an Excel 2010 add-in. I wouldn’t have liked to start from scratch, so I took one of my similar projects from the beginning of last year, and started to alter it.

However, when I was to build the project in Visual Studio 2010, I got this error:

An error occurred while signing: Key not valid for use in specified state

I’ve checked the original version of the project, but I found the same error there.

First I thought the issue is with the signing of the assembly, but it turned out to be false. The real reason was related to the pfx certificate  used to sign the manifest for ClickOnce.

The original version of the project was created using a CTP version of Visual Studio 2010 on another machine, so even the certificate was expired.

The following  section contains the related settings in (csproj) file:

<ManifestKeyFile>Former Add-in_TemporaryKey.pfx</ManifestKeyFile>

You can find the same settings on the VS 2010 UI as well, see the Signing /Sign the ClickOnce manifests setting at project properties.


First I tried to simply remove the section from the csproj (after Unload Project, then Edit MyAddInProject.csproj, and Reload Project after editing the file). This helped to remove the build error, but caused error on deployment:

Reference in the deployment does not match the identity defined in the application manifest

So I created a new pfx, and used that certificate for ClickOnce signing.

Later I tried to reproduce the issue that turned out to be not easy, but I think it would have been enough to simply uncheck the Sign the ClickOnce checkbox. At least, VS seems to check it back on project build for me and it solved the issue as well. I’m still not sure what the real reason for this error was, but probably something related to VS was not able to import the certificate to my certificate store, so it could not use it to sign the manifests.

June 20, 2011

An error occured while attempting to find services

Filed under: ADO.NET Data Services, SP 2010, Visual Studio — Tags: , , — Peter Holpar @ 22:04

Today I was unable to add a new, SharePoint 2010 based data source to my Silverlight 4.0 project in Visual Studio 2010.

An error (Details) occured while attempting to find services at

The details of the error suggested there were issues with the authentication:

The document at the url http://yourserver/_vti_bin/listdata.svc was not recognized as a known document type.
The error message from each known type may help you fix the problem:
– Report from ‘XML Schema’ is ‘Root element is missing.’.
– Report from ‘DISCO Document’ is ‘Root element is missing.’.
– Report from ‘WSDL Document’ is ‘There is an error in XML document (0, 0).’.
  – Root element is missing.
Metadata contains a reference that cannot be resolved: ‘http://yourserver/_vti_bin/listdata.svc&#8217;.
The HTTP request is unauthorized with client authentication scheme ‘Anonymous’. The authentication header received from the server was ‘NTLM’.
The remote server returned an error: (401) Unauthorized.
If the service is defined in the current solution, try building the solution and adding the service reference again.

Even the Fiddler trace of the request contained 401 HTTP status results, but there was an internal server error (500) as well.


When I tried to connect to a subsite in the same site collection it caused no error.

As it turned out after some investigation the real reason was not security related. The site contained “orphaned” lists. These lists were created based on a list template that was activated by a feature that was deactivated in the meantime.

When navigating to the list in the browser the following exception was displayed:

Server Error in ‘/’ Application.

<nativehr>0x8107058a</nativehr><nativestack></nativestack>Feature ‘524f4fa9-7e07-4ba3-ab31-73d5b42aa849’ for list template ‘10000’ is not installed in this farm.  The operation could not be completed.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Runtime.InteropServices.COMException: <nativehr>0x8107058a</nativehr><nativestack></nativestack>Feature ‘524f4fa9-7e07-4ba3-ab31-73d5b42aa849’ for list template ‘10000’ is not installed in this farm.  The operation could not be completed.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. 

Stack Trace:

[COMException (0x8107058a): <nativehr>0x8107058a</nativehr><nativestack></nativestack>Feature ‘524f4fa9-7e07-4ba3-ab31-73d5b42aa849’ for list template ‘10000’ is not installed in this farm.  The operation could not be completed.]
   Microsoft.SharePoint.Library.SPRequestInternalClass.GetViewsSchemaXml(String bstrUrl, String bstrListName, Boolean bFullBlown, Boolean bNeedInitAllViews, ISP2DSafeArrayWriter p2DWriter, Int32& plDefaultViewIndex, Int32& plMobileDefaultViewIndex) +0
   Microsoft.SharePoint.Library.SPRequest.GetViewsSchemaXml(String bstrUrl, String bstrListName, Boolean bFullBlown, Boolean bNeedInitAllViews, ISP2DSafeArrayWriter p2DWriter, Int32& plDefaultViewIndex, Int32& plMobileDefaultViewIndex) +187

[SPException: Feature ‘524f4fa9-7e07-4ba3-ab31-73d5b42aa849’ for list template ‘10000’ is not installed in this farm.  The operation could not be completed.]
   Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx) +27445106
   Microsoft.SharePoint.Library.SPRequest.GetViewsSchemaXml(String bstrUrl, String bstrListName, Boolean bFullBlown, Boolean bNeedInitAllViews, ISP2DSafeArrayWriter p2DWriter, Int32& plDefaultViewIndex, Int32& plMobileDefaultViewIndex) +27822691
   Microsoft.SharePoint.SPViewCollection.EnsureViewSchema(Boolean fullBlownSchema, Boolean bNeedInitallViews) +283
   Microsoft.SharePoint.SPList.get_Views() +58
   Microsoft.SharePoint.SPList.get_DefaultViewUrl() +136
   Microsoft.SharePoint.Publishing.CachedList..ctor(SPList list, CachedObjectFactory factory) +420
   Microsoft.SharePoint.Publishing.CachedArea.GetChildListByGuid(Guid listIdGuid) +498
   Microsoft.Office.DocumentManagement.MetadataNavigation.MetadataNavigationSettings.GetCachedList(SPWeb web, Guid listId) +160
   Microsoft.Office.DocumentManagement.MetadataNavigation.MetadataNavigationContext.GetCurrentListRootFolderUrl() +374
   Microsoft.Office.DocumentManagement.MetadataNavigation.MetadataNavigationContext.CompareFolderUrlToListUrl() +97
   Microsoft.Office.DocumentManagement.MetadataNavigation.MetadataNavigationContext.get_FolderPath() +28
   Microsoft.Office.DocumentManagement.MetadataNavigation.MetadataNavigationContext.InitializeSelectedPathAndTreeValues(NameValueCollection queryString) +488
   Microsoft.Office.DocumentManagement.MetadataNavigation.MetadataNavigationContext.Page_InitComplete(Object sender, EventArgs e) +1400
   System.EventHandler.Invoke(Object sender, EventArgs e) +0
   System.Web.UI.Page.OnInitComplete(EventArgs e) +11041550
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +1674

After deleting the lists, I was able to add the new data source to the project.

May 25, 2011

Using Visual Studio Watch window to discover managed client object model internals

Filed under: Managed Client OM, SP 2010, Visual Studio — Tags: , , — Peter Holpar @ 18:01

In my former post I wrote about using Fiddler to analyze the network traffic generated by the client object model. Although personally I think it is a great way to discover the object model internal working, there are tools that many of .NET developers are more familiar with, for example, Visual Studio itself.

The Watch window in Visual Studio a useful tool to look into the details of objects, especially as it allows you to check the values of non-public class members as well as public ones.

In this post I show you some of the objects / properties I suggest you to watch when you would like to have a deeper knowledge about the client OM.

For the example I use one of the simple codes used in the previous post (of course, you should replace the URL before running the code):

  1. ClientContext clientContext =
  2.   new ClientContext("http://pholparlaptop&quot;);
  3. Web web = clientContext.Web;
  4. ListCollection lists = web.Lists;
  5. List list = lists.GetByTitle("Links");
  6. clientContext.ExecuteQuery();

Put a breakpoint to the first line, then start debugging the application.

When the execution is stopped at the breakpoint, add the following items to the Watch window:

  • clientContext
  • clientContext.ObjectPaths
  • clientContext.PendingRequest.Actions
  • list.Path

Then step through the code line by line and see how the values are changing.

In the clientContext you can see (among others) the version on the OM, the application name, the timeout value (in milliseconds), and a single static object, that is the RequestContext (Microsoft.SharePoint.SPContext). As we have learnt that is sent as a standard part of each request.


As you step through the code you can see how the number of items in the clientContext.ObjectPaths generic Dictionary and clientContext.PendingRequest.Actions generic List changes. Usually as you refer new objects in the code, a new item is created for both. This way the context aggregates what it should do and with which objects on execution. The figures below show the state just before calling the ExecuteQuery method.



After ExecuteQuery has finished, the clientContext.PendingRequest.Actions list is reset, as there are no more pending actions for the current context.

There is an other change either after query execution (see below), that the last item of the ObjectPaths is changed from ObjectPathMethod to ObjectPathIdentity. Remember, that in Fiddler it was an action of ObjectIdentityQuery (it is Microsoft.SharePoint.Client.ObjectIdentityQuery as shown in the Watch window above for the last item of clientContext.PendingRequest.Actions) that referred to a Method subnode of ObjectPaths (related .NET class is Microsoft.SharePoint.Client.ObjectPathMethod as displayed in the Watch window above for the last item of clientContext.ObjectPaths).


The next screenshot illustrates that the relation between the actions and object paths we observed in Fiddler is reflected in the object hierarchy as well. The key for the relation is the Path property of the action that refers to the object path.


The image bellow shows some useful information observable in the Watch window. This time the static root object (m_parentId = –1, Parent = null) is displayed. Note the non-public m_propertyName and m_typeId properties having the same values we saw in the former post.


And the next one is for a standard ObjectPathProperty. See the non-public m_PropertyName with value “Web”, and the m_ParentId = 1. It means the object path refers to the Web property of the root context.


The next two figures show the response, the “standalone” ObjectPathIdentity object (m_parentId = –1, Parent = null)created on successful query execution. Note the Identity property that refers the list using the canonical ID.

The first screenshot has been taken from clientContext.ObjectPaths dictionary (note the item has an index of 3), while the second one is from a direct “addressing” the same item from the Dictionary by adding clientContext.ObjectPaths[3] to the watched names.

In fact, if you happen to use an index that refers to an ID that does not exist in the object paths (like clientContext.ObjectPaths[2]), an exception of type ‘System.Collections.Generic.KeyNotFoundException’   is thrown, although there is an item having index 2 in the clientContext.ObjectPaths watch. It is due to a general Dictionary behavior, not a client OM specific one, but you should be aware of this. The reason for it is that the Key of the dictionary is the Id property of the Value.



Now compare this values to the ones you can see for list.Path. The Path property is the same as above. It means the result is loaded to the list object.


In the next post I plan to dig even deeper describing the internal working of the client OM (both client and server side). This knowledge will help us to do some really fancy things using the framework provided by the managed client object model infrastructure.

Older Posts »

Create a free website or blog at