Second Life of a Hungarian SharePoint Geek

November 29, 2017

How to detect if your DbConext is already disposed

Filed under: Entity Framework, Tips & Tricks — Tags: , — Peter Holpar @ 23:34

A little background (or let’s say context) to the story. Recently I had to create a tool (of course it was SharePoint-related) that needs to import a quite large number of entries into a SQL database for further analysis. I chose Entity Framework (DB-first) as a wrapper to our database entities. Having the knowledge that I already have now, I should admit, it was not the perfect choice, but rather a great opportunity to learn from our failures. In the development environment, where we have a smaller amount of data, the tool was quite quick, so I anticipated, it would be quick enough in the test and productive environment either. Just to have an idea about the order of magnitude, in these environments we have around 0,5 Mio. entries to export. It is a large number of entries, but I don’t think it is extreme large. After the first run in the test environment I knew, I was completely wrong with my assumption.

After letting the tool run on the weekend, and even a few day after, we stopped it, because it simply have not finished, and we haven’t seen the end of the tunnel. I analyzed the log data and found, that at the beginning the tool saved a batch of 100 entries in approximately 50 milliseconds. But later it was slower and slower, and after having saved already about 15.000 entries, the saving of the same batch of 100 entries took more than 7 seconds. What was the problem, and how have we solved it? You can read more about the issue here and here, and about the solution we applied here.

As part of the solution, we disposed and re-created our data context after saving every batch of items. However, this solution has introduced a new problem.

In the original version we have called the SaveChanges method at the end of our processing in an outer code block to ensure saving any remaining items that do not completed an entire batch. But in the new version we got this exception at this point:

InvalidOperationException: The operation cannot be completed because the DbContext has been disposed.
at System.Data.Entity.Internal.LazyInternalContext.InitializeContext()
at System.Data.Entity.Internal.LazyInternalContext.get_ObjectContext()
at System.Data.Entity.Internal.InternalContext.DetectChanges(Boolean force)
at System.Data.Entity.Internal.InternalContext.GetStateEntries(Func`2 predicate)
at System.Data.Entity.Infrastructure.DbChangeTracker.Entries()
at System.Data.Entity.DbContext.GetValidationErrors()
at System.Data.Entity.Internal.InternalContext.SaveChanges()

Note: you have this error on a disposed DbContext object only if there were something to save. If no object has been changed, no exception is thrown on calling SaveChanges on a disposed DbContext object.

This exception was due to disposing of the data context we created in a using block. We passed the data context to a method, that disposed (and re-created it) as part of the pattern suggested for performance tuning for importing of large number of items via the Entity Framework. See the sample code with Exporter class and it static DoSomething method further down. The data context was disposed in the DoSomething method. In the original version we tried to invoke dbContext.SaveChanges after returning from the DoSomething method, although the data context was already disposed in the DoSomething method, and it resulted the exception above.

Although some disposable object types might provide a kind of check if the object has been already disposed (see the IsDisposed property of the System.Windows.Forms.Control class, for example), it is definitely not part of the IDisposable interface. If the object belongs to your solution, you can implement a similar property yourself. If you can’t change the source code of the class (it is a class in the .NET libraries or any 3rd party class), you can derive a subclass from it, and extend it by the functionality as shown in this answer. This method works of course only as long as the base class is not marked as sealed.

In the case of DbContext I found a private field of type System.Data.Entity.Internal.InternalContext called _internalContext. InternalContext is an abstract internal class, you can find references for it and for its derived class LazyInternalContext in the error stack trace above as well. The InternalContext class has a public property called IsDisposed. The LazyInternalContext class invokes in its InitializeContext method the CheckContextNotDisposed method of the base class, that throws the InvalidOperationException above if the context has been already disposed.

I created the extension method below using Reflection to read the value of the IsDisposed property of the _internalContext field.

  1. public static bool IsDisposed(this DbContext context)
  2. {
  3.     var result = true;
  4.  
  5.     var typeDbContext = typeof(DbContext);
  6.     var typeInternalContext = typeDbContext.Assembly.GetType("System.Data.Entity.Internal.InternalContext");
  7.  
  8.     var fi_InternalContext = typeDbContext.GetField("_internalContext", BindingFlags.NonPublic | BindingFlags.Instance);
  9.     var pi_IsDisposed = typeInternalContext.GetProperty("IsDisposed");
  10.  
  11.     var ic = fi_InternalContext.GetValue(context);
  12.  
  13.     if (ic != null)
  14.     {
  15.         result = (bool)pi_IsDisposed.GetValue(ic);
  16.     }
  17.  
  18.     return result;
  19. }

You can test the functionality in a code block like this one:

  1. using (dbContext = new YourDbContext())
  2. {
  3.     Console.WriteLine("Disposed: {0}", dbContext.IsDisposed());
  4.     Exporter.DoSomething(dbContext);
  5.     Console.WriteLine("Disposed: {0}", dbContext.IsDisposed());
  6. }
  7. Console.WriteLine("Disposed: {0}", dbContext.IsDisposed());

The Exporter class and its static DoSomething method in the code above are just some arbitrary functionality you would like to perform.

Having the IsDisposed method defined, you can check if the context is disposed before calling SaveChanges (or any other method that are dangerous to be invoked on a disposed context):

  1. if ((dbContext != null) && (!dbContext.IsDisposed()))
  2. {
  3.     dbContext.SaveChanges();
  4. }

Note: The code in this post was tested with Entity Framework 6.1.3. Other versions might behave in another way, so there is no guarantee that the same code works with those versions too.

Advertisements

November 23, 2017

Displaying Filtered or Sorted SharePoint Web Properties from PowerShell

Filed under: PowerShell, SP 2013, Tips & Tricks — Tags: , , — Peter Holpar @ 22:47

Specific SharePoint applications and services as well as tools, like SharePoint Designer tend to store their own settings in the property bag of web sites. Especially root webs of site collection have a lot of properties.

You can display the web properties from PowerShell like:

$web = Get-SPWeb http://YourSharePointSite
$web.AllProperties

The problem, that the above script displays really all of the properties, and the list is not alphabetically sorted, so it might be challenging  to find a specific property.

The properties and their values are stored as key-value pairs in a Hashtable object, you can filter these entries as described in this thread. For example, the script below displays the search related entries, the ones whose names begin with SRCH.

$web.AllProperties.GetEnumerator() | ? Key -like ‘SRCH*’

image

Although it is not so common, you can filter the entries based on its values as well. For example, this script display all properties having a value of True:

$web.AllProperties.GetEnumerator() | ? Value -eq ‘True’

If all you want is to display the properties sorted alphabetically by their names, it is easy to achieve either:

$web.AllProperties.GetEnumerator() | Sort-Object -Property Key

image

Of course, all of these possibilities apply not only to the property bags of web objects (SPWeb.AllProperties), but to other property bags as well, like the properties of the folders (SPFolder.Properties) and files (SPFile.Properties) or lists (SPLists.Properties) and list items (SPListItem.Properties).

October 16, 2017

Editing the PWS Site Address does not work if the URL is very long

Filed under: Bugs, PowerShell, Project Server — Tags: , , — Peter Holpar @ 19:52

As part of our daily jobs, we should rename projects on our Project Server occasionally. For this kind of change, we have already a “human workflow” or a check list: tasks, we should perform one after another.

The Standard Process

These steps include:

1. Changing the project name on the Project Details page:

image

2. Setting the project web site (PWS) title and URL on the Title, Description, and Logo page of Site Settings:

image

(Note, there is a bug already on this page. If the URL is long enough, it is displayed duplicated in the path under URL name, once in the fix part, and once in the text box. It is already part of the example URL bottom on the left as well. See the screenshot above.)

3. Re-binding the project to the relocated PWS via the PWA Settings / Connected SharePoint Sites / Edit Address.
(For more information, see The Edit Site Address settings on this page)

image

image

image

4. Beyond the steps described above we have some extra steps, like renaming project groups, setting further PWS properties, and so on, but these steps are all custom to our current solution.

The Problem

A few month ago one of the Project Server administrators complained that he is not able to change the site address of  PWS. He was to change the URL from (let’s say) VeryVeryLongProjectSiteUrl to VeryVeryLongProjectSiteUrlNew. Although he has not got any error message, and I’ve not found any related entry in the ULS logs either, the original URL of the PWS remained unchanged.

Changing the Site Address via PSI and PowerShell

First, I wrote a PowerShell script that uses the PSI to change the PWS binding via the UpdateProjectWorkspaceAddress method.

  1. $pwsCurrentUrl = "http://YourProjectServer/PWA/VeryVeryLongProjectSiteUrl"
  2. $projUrl = "PWA/VeryVeryLongProjectSiteUrlNew" # that is the destination URL of the PWS, it should be the server relative URL, including PWA in the path!
  3.  
  4. $web = Get-SPWeb $pwsCurrentUrl
  5.  
  6. # if you already know the IDs (project ID and site ID of the PWA site)
  7. # $projId = [Guid]"99894c16-7a03-e411-83c6-005056b45654"
  8. # $siteId = [Guid]"e1b9fba5-09ad-441a-8679-6286dde059ab"
  9.  
  10. # or get the IDs from the PWS properties
  11. $projId = $web.AllProperties["MSPWAPROJUID"]
  12. $siteId = $web.Site.Id
  13.  
  14. # figure out the PWA url dinamically
  15. # $pwaUrl = $web.AllProperties["PWAURL"] # or
  16. $pwaUrl = $web.Site.Url
  17.  
  18. # we are using the Project PSI service
  19. $svcPath = "/_vti_bin/psi/Project.asmx?wsdl"
  20.  
  21. # https://social.technet.microsoft.com/Forums/scriptcenter/en-US/9d0d73bb-b2bf-4528-beea-321cf82a9b89/problem-executing-a-script-what-uses-namespace-parameter-in-the-newwebserviceproxy-cmdlet
  22. If ($global:svcPSProxy -eq $null)
  23. {
  24.   Write-Host "Connecting PSI proxy at $pwaUrl …"
  25.   $global:svcPSProxy = New-WebServiceProxy -Namespace PSIProxy -Uri ($pwaUrl + $svcPath) -UseDefaultCredential
  26. }
  27. Else
  28. {
  29.   Write-Host "Reusing existing PSI proxy"
  30. }
  31.  
  32. # change the project – PWS binding, or create a new PWS if there is no PWS at the destination
  33. $svcPSProxy.UpdateProjectWorkspaceAddress($projId, $projUrl, $siteId)

Note, that based on my tests, the script not only maps an existing PWS to the project, but it creates a new PWS if there is no PWS at the destination URL specified.

Finding the Bug on the Web Page

After completing the task via the script above, I decided to find out the reason, the UI does not work in this case.

As far as I see, it is a simple silly error in the JavaScript on the Edit Site Address page (\TEMPLATE\LAYOUTS\PWA\ADMIN\EditSiteAddressDlg.aspx).

It is the Init function on that page:

  1. function Init()
  2. {ULSH9J:;
  3.    oArgs = window.frameElement.dialogArgs;
  4.    sProjName = oArgs.sProjName;
  5.    sServerAddr = ((oArgs.sServerAddr != null) ? oArgs.sServerAddr : "");
  6.    sSubwebName = oArgs.sSubwebName;
  7.  
  8.    idProjectNameTD.title = sProjName;
  9.    if(sProjName.length > 40)
  10.    {
  11.       sProjName = sProjName.slice(0,40) + "…";
  12.    }
  13.    XUI.Html.SetText(idProjectNameTD, sProjName);
  14.  
  15.    if((sServerAddr != "") && (sSubwebName != ""))
  16.    {
  17.       var sUrl = sServerAddr + "/" + sSubwebName;
  18.       idServerAddressTD.title = sUrl;
  19.       if(sUrl.length > 40)
  20.       {
  21.          sUrl = sUrl.slice(0,40) + "…";
  22.       }
  23.       XUI.Html.SetText(idServerAddressTD, sUrl);
  24.    }
  25.  
  26.    idSubwebName.value = sSubwebName;
  27.  
  28.    RecalculateTargetURL();
  29.  
  30.    origTargetUrl = XUI.Html.GetText(idTargetURL);
  31. }

This function invokes the RecalculateTargetURL function (see below) to trim the end of the URL of the PWS if it is longer then 50 characters, and to append … to it. This value is displayed then on the page as Destination URL. In the Init function we store the original value in the origTargetUrl variable.

  1. function RecalculateTargetURL()
  2. {ULSH9J:;
  3.    var sURL = idVirtualServerDropdown[idVirtualServerDropdown.selectedIndex].text;
  4.    sURL += "/" + TrimSpaces(idSubwebName.value);
  5.  
  6.    idTargetURL.title = sURL;
  7.  
  8.    if(sURL.length > 50)
  9.    {
  10.       sURL = sURL.slice(0,50) + "…";
  11.    }
  12.  
  13.    XUI.Html.SetText(idTargetURL, sURL);
  14. }

image

The very same RecalculateTargetURL function is invoked on each key press or on changes in the Site URL text box to keep the value of the Destination URL on the page current:

<input DIR="ltr" type="text" id="idSubwebName" name="idSubwebName" style="width: 160px" onchange="RecalculateTargetURL()" onkeyup="RecalculateTargetURL()" …

Note, that the RecalculateTargetURL function is registered for the onchange event of Web Application dropdown either.

The problem is, that the script uses the trimmed values for comparison in the OkBtn_OnClick function (see the method below, including some server side code) to decide, if there is any change in the URL (see the condition with the comment “If nothing changed then we don’t have to do anything” below). Of course, if you have long site (and project) names, and you change something only at the end of the name, this comparison won’t detect the change.

  1. function OkBtn_OnClick()
  2. {ULSH9J:;
  3.    if(idSiteEnabled.checked && (TrimSpaces(idSubwebName.value) == ""))
  4.    {
  5.       XUI.Html.SetText(idAlertBox, PJUnescape("<%=PJEscape(PJUtility.GetLocalizedString(IDS.ADMIN_EDITSITEADDRESSDLG_WEB_NAME_BLANK_ALERT))%>"));
  6.       XUI.Html.SetText(idRequiredFieldIndicator, "*");
  7.       idSubwebName.focus();
  8.       return;
  9.    }
  10.    
  11.    // If nothing changed then we don't have to do anything.
  12.    if((origTargetUrl == XUI.Html.GetText(idTargetURL)) && !idSiteNotEnabled.checked)
  13.    {
  14.       window.frameElement.commonModalDialogClose(0, null);
  15.       return;
  16.    }
  17.    //if we remove the site
  18.    else if(idSiteNotEnabled.checked)
  19.    {
  20.       idSubwebName.value = "";
  21.       oArgs.sNewSubwebName = "";
  22.       oArgs.sNewServerUID = "<%=Guid.Empty%>";
  23.    }
  24.    //we change the site
  25.    else
  26.    {
  27.       oArgs.sNewServerUID = idVirtualServerDropdown[idVirtualServerDropdown.selectedIndex].value;
  28.       var sTemp = TrimSpaces(idSubwebName.value);
  29.  
  30.       // Remove the trailing slash.
  31.       if(sTemp.charAt(sTemp.length – 1) == '/')
  32.       {
  33.          sTemp = sTemp.substr(0, sTemp.length – 1);
  34.       }
  35.       oArgs.sNewSubwebName  = sTemp;
  36.       window.returnValue    = true;
  37.    }
  38.  
  39.    window.frameElement.commonModalDialogClose(1, oArgs);
  40. }

Note however, that if you click on the Test URL button, a new browser tab would be opened with the right destination URL (and not the trimmed one). The right new URL is displayed as a tooltip as well, when you move the mouse pointer over the URL right to the Destination URL title.

function TestUrl_OnClick(event)
{ULSH9J:;
   window.open(idTargetURL.title);
}

As you can see, the TestUrl_OnClick function uses the tooltip of the Destination URL (idTargetURL.title) to open the site. It is important to point out, that the value of  idTargetURL.title is set to the full URL, and not to the trimmed one in the RecalculateTargetURL function (see above).

image

A Quick Workaround via the Web Page

If you don’t want (or not allowed) to use the PowerShell script above to relocate your PWS, there is a simple workaround that uses the standard web admin UI. Start the F12 Developer Tools in Internet Explorer, and set a breakpoint on the line

if((origTargetUrl == XUI.Html.GetText(idTargetURL)) && !idSiteNotEnabled.checked)

on the Edit Site Address page. If the breakpoint get hit, jump over the condition by setting the next statement of execution direct onto the line:

oArgs.sNewServerUID = idVirtualServerDropdown[idVirtualServerDropdown.selectedIndex].value;

The Long-Term (but Dirty) Solution

Although it is not supported, you can change the code in the EditSiteAddressDlg.aspx page as well. I strongly suggest you to take a backup of this file first.

There are two options to fix the error, the first one is to modify the Init function to save the original full (!) URL instead of the trimmed one:

//origTargetUrl = XUI.Html.GetText(idTargetURL);
origTargetUrl = idTargetURL.title;

Then use this value to compare with the current untrimmed URL value in the OkBtn_OnClick function:

// If nothing changed then we don’t have to do anything.
//if((origTargetUrl == XUI.Html.GetText(idTargetURL)) && !idSiteNotEnabled.checked)
if((origTargetUrl == idTargetURL.title) && !idSiteNotEnabled.checked)

The other option is to forget origTargetUrl, and take the original full URL from the tooltip of the Current site address. As you can see on the screenshot after the RecalculateTargetURL function code snippet above, this tooltip contains the untrimmed URL version.

In this case, the new comparison in the OkBtn_OnClick function:

if((idServerAddressTD.title == idTargetURL.title) && !idSiteNotEnabled.checked)

How to query your working hours from Windows Event Log via PowerShell

Filed under: PowerShell, Tips & Tricks — Tags: , — Peter Holpar @ 19:49

At my company we have a kind of time reporting application. If I book the activities the same day, there is no problem. But after a week, it is not always straightforward to remember what I exactly did on a given day. To have a rough estimate, how many hours I overall and separated for projects worked, I usually make use of data sources like Event Viewer (first and last entries daily in the Windows Logs / System), Exchange (mails sent and receive), Internet Explore (sites visited in Browser History) and TFS (check-ins and task history).

To be able to query the Event Viewer Logs without starting the application and browsing through the entries, I wrote a PowerShell script that perform these tasks automatically for me. It’s nothing extra, but I thought it might be useful for others as well:

$startDay = Get-Date -Date ‘2017/09/01’
$endDay = Get-Date -Date ‘2017/09/11’

$days = New-Object System.Collections.Generic.List“1[System.DateTime]
For ($today = $startDay; $today -le $endDay; $today = $today.AddDays(1)) {
  $days.Add($today)
}

$days | % {
  $day = $_
  $events = Get-EventLog -Log System | ? { $_.TimeGenerated.Date -eq $day.Date }
  $maxDate = ($events | Measure-Object -Property TimeGenerated -Maximum).Maximum
  $minDate = ($events | Measure-Object -Property TimeGenerated -Minimum).Minimum
  select -Input $_ -Prop `
    @{ Name=’Day’; Expression={$day.ToShortDateString()} },
    @{ Name=’From’; Expression={ $minDate.ToLongTimeString() } },
    @{ Name=’To’; Expression={ $maxDate.ToLongTimeString() } },
    @{ Name=’Working Hours’; Expression={ $maxDate – $minDate } }
} | Export-Csv -Path C:\Temp\TimeReport.csv -Delimiter ";" -Encoding UTF8 -NoTypeInformation

The script writes the results in a .csv file, but without the last part (Export-Csv) you can direct the output to the screen as well.

September 9, 2017

Approving all pending documents (and folders) of a specified library using PowerShell on the Client Side

Filed under: Managed Client OM, PowerShell, SP 2013 — Tags: , , — Peter Holpar @ 07:00

A few years ago I already wrote about how to approve all pending document in a document library via PowerShell. That time I achieved that using the server side object model of SharePoint. Recently we had a situation, where we were not allowed to log on the server, so we had to do the approval from the client side. To achieve that, I’ve adapted the script to the requirements of the client object model.

Here is the result:

  1. $url = "http://YourSharePointServer/Web/SubWeb&quot;
  2.  
  3. # set path according to your current configuration
  4. Add-Type -Path "c:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
  5. Add-Type -Path "c:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.dll"
  6.  
  7.  
  8. # set credentials, if the current credentials would not be appropriate
  9. #$domain = "YourDomain"
  10. #$userName = "YourUserName"
  11. #$pwd = Read-Host -Prompt ("Enter password for $domain\$userName") -AsSecureString
  12. #$credentials = New-Object System.Net.NetworkCredential($userName, $pwd, $domain);
  13.  
  14. $ctx = New-Object Microsoft.SharePoint.Client.ClientContext($url)
  15. #$ctx.Credentials  = $credentials
  16.  
  17. $web = $ctx.Web
  18.  
  19.  
  20. function approveItems($listTitle)  
  21. {
  22.   Write-Host Processing $listTitle
  23.   $list = $web.Lists.GetByTitle($listTitle)
  24.   $query = New-Object Microsoft.SharePoint.Client.CamlQuery
  25.   $query.ViewXml = "<View Scope = 'RecursiveAll'><ViewFields><FieldRef Name=\'Name\'/><FieldRef Name=\'_ModerationStatus\'/></ViewFields><Query><Where><Eq><FieldRef Name='_ModerationStatus' /><Value Type='ModStat'>2</Value></Eq></Where></Query></View>"
  26.   $items = $list.GetItems($query)
  27.   $ctx.Load($items)
  28.   $ctx.ExecuteQuery()
  29.  
  30.   $items | % {
  31.       Write-Host Approving:$_["FileLeafRef"]
  32.     $_["_ModerationStatus"] = 0
  33.     $_.Update()
  34.     # if you have an error "The request uses too many resources", call ExecuteQuery here
  35.     # $ctx.ExecuteQuery()
  36.   }
  37.  
  38.   $ctx.ExecuteQuery()
  39.   Write-Host —————————
  40. }
  41.  
  42. approveItems "TitleOfYourList"

The script assumes, that your current credentials allow you to perform the approval. If it would be not the case, you can comment out the section with credentials in the script, and read for the password of the user having permission to the task. I don’t suggest storing the password in the script.

If the library contains a lot of items waiting for approval, you may get an error message “The request uses too many resources” (see details here). In this case you should call the ExecuteQuery method in the loop for each item, instead of sending the request in a single batch.

July 20, 2017

How to Export a SharePoint List View to Excel Automatically Using PowerShell

Filed under: PowerShell, SP 2013, Tips & Tricks — Tags: , , — Peter Holpar @ 21:17

Note: This post is actually only a minor modification of the post I wrote recently about the URL of the Edit View page.

We can easily export the content of a SharePoint List View via the UI, simply by clicking the Export to Excel button on the ribbon:

image

You can achieve that automatically as well, for example from PowerShell.

Note: In the text below I describe the solution for a situation if you work locally on the server, but it is possible to apply the same technique to a remote solution as well, one should only transfer the code to the Managed Client Object Model.

Assume you have a list called YourList in a SharePoint site with URL http://YourSharePoint/Web/SubWeb.

It is easy to find out (for example, by monitoring the network traffic by Fiddler) that the URL generated when you click the Export to Excel button is like this:

http://YourSharePoint/Web/SubWeb/_vti_bin/owssvr.dll?CS=65001&Using=_layouts/15/query.iqy&List=%7B5315A0C9%2DAA6A%2D4598%2DA1D4%2D99B1BBCBF8C7%7D&View=%7B8E449E17%2D593C%2D4218%2DA4A4%2D43A8B47382BC%7D&RootFolder=%2FWeb%2FSubWeb%2FLists%2FYourList&CacheControl=1

The values of the List and View query string parameters are the encoded IDs of your list and list view respectively.

The following code generates the same URL from PowerShell:

$web = Get-SPWeb http://YourSharePoint/Web/SubWeb
$list = $web.Lists[‘YourList’]

# get the default view of the list
$view = $list.DefaultView
# or get an arbitrary view by its name
# $view = $list.Views[‘All Items’]
$viewId = $view.ID

function EscapeGuid($guid)
{
  return "{$guid}".ToUpper().Replace(‘-‘, ‘%2D’).Replace(‘{‘, ‘%7B’).Replace(‘}’, ‘%7D’)
}

$escapedListId = EscapeGuid $list.ID
$escapedViewId = EscapeGuid $view.ID
$escapedRootFolder = $list.RootFolder.ServerRelativeUrl.Replace(‘/’, ‘%2F’)

$url = $web.Url + "/_vti_bin/owssvr.dll?CS=65001&Using=_layouts/15/query.iqy&List=$escapedListId&View=$escapedViewId&RootFolder=$escapedRootFolder&CacheControl=1"

The URL above is actually no URL for the data or its schema, it’s a URL for a descriptor file (with the extension .iqy, see more about that here), that contains the URL for that list data and its schema.

The content of an .iqy file looks like this (you can capture it by Fiddler as well, or have a look at the content of the file we saved in our script further below) :

WEB
1
http://YourSharePoint/Web/SubWeb/_vti_bin/_vti_bin/owssvr.dll?XMLDATA=1&List={5315A0C9-AA6A-4598-A1D4-99B1BBCBF8C7}&View={8E449E17-593C-4218-A4A4-43A8B47382BC}&RowLimit=0&RootFolder=%2fWeb%2fSubWeb%2fLists%2fYourList

Selection={5315A0C9-AA6A-4598-A1D4-99B1BBCBF8C7}-{8E449E17-593C-4218-A4A4-43A8B47382BC}
EditWebPage=
Formatting=None
PreFormattedTextToColumns=True
ConsecutiveDelimitersAsOne=True
SingleBlockTextImport=False
DisableDateRecognition=False
DisableRedirections=False
SharePointApplication=http://YourSharePoint/Web/SubWeb/_vti_bin
SharePointListView={8E449E17-593C-4218-A4A4-43A8B47382BC}
SharePointListName={5315A0C9-AA6A-4598-A1D4-99B1BBCBF8C7}
RootFolder=/Web/SubWeb/Lists/YourList

The URL we have in line 3 refers to the endpoint that returns the data schema and the data itself.. Based on this information, Excel can import and display the data of the list view.

Let’s save the file from the URL of the .icq file we have already from the first script, and start Excel to open the list view data. The script below assumes the extension .iqy is associated with Excel in your system:

$path = "C:\temp\owssvr.iqy"

$request = [System.Net.WebRequest]::Create($url)
$request.UseDefaultCredentials = $true
$request.Accept = "text/html, application/xhtml+xml, */*"

$response = $request.GetResponse()
$reader = New-Object System.IO.StreamReader $response.GetResponseStream()
$data = $reader.ReadToEnd()

$writer = [System.IO.StreamWriter] $path
$writer.WriteLine($data)
$writer.Close()

# the .iqy file will be opened by Excel
Invoke-Expression $path
# optionally delete the file
# Remove-Item $path

Bulk Deletion of Events from a SharePoint Calendar

Filed under: Calendar, SP 2013 — Tags: , — Peter Holpar @ 21:15

Assume you have a SharePoint Calendar with thousands of events, including recurring events, and recurring event exceptions. These ones are the result of aggregating events from several years, most of them are no more relevant.

Problems

If the number of events is over the List View Threshold limit, your users will not be able to access the All Events view (see sample screenshot below taken from another calendar including a standard and a recurring event, a deleted recurring event and a recurrence exception),

image

they receive instead that an error message:

The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.

They won’t be able even to delete the calendar, either from the Site Content page (they get a warning like “We’re sorry, we had some trouble removing this. You can try again from the settings page.”):

image

image

or from the Settings page of the list (same error as above):

image

Of course, you as an administrator, can increase the List View Threshold limit, delete the list via PowerShell (see below), or even from the web UI if you log in using the farm account, but it takes more time for you and for your users.

$web = Get-SPWeb http://YourSharePoint/Web/SubWeb
$calendar= $web.lists["YourCalendar"]
$calendar.Delete()

Even if the number of events stays below the List View Threshold limit, the users might have experience performance problems.

Solution

Note, that we did not want to delete the list, the limitation with the deletion was only one of the examples. Instead of deletion, we wanted to drastically reduce the number of events by deleting all events that are no more relevant in the current year. That means, deleting all single events that were finished earlier than this year, and repeating events whose repetitions is finished earlier than the current year.

Based on my experience the deletion of such high number of items is not very performant, so I decided to delete the items in batches. There are several examples for that on the web, including solutions for C# (see here or here) or for PowerShell (see here) or for both of them (see this one). I planned to use PowerShell, but the examples I found were typically simple translations of the C# version, without using the structures and features available in PowerShell. Even worse, both the C# and PowerShell implementations out there have a serious limitation: they either output all of the results returned by the ProcessBatchData method, or they display no information at all. It might be OK as long as you successfully delete all of the items, but if you have any problem there, I wish you a good luck to find any usable information about it, if there are really of thousands of items in your list. So I created my own PowerShell implementation using the samples available. Note, that because I don’t want to wait for a feedback about the success of the deletion until all items are processed (we had over some 10k items there), I’m deleting the items in smaller batches, in the code below it is 1000 items / batch.

To get the items I should delete, I used the following CAML query:

<Where>
    <Lt>
    <FieldRef Name=’EndDate’ />
    <Value Type=’DateTime’>2017-01-01 00:00:00</Value>
    </Lt>
</Where>

Here is the full code of the first version of my script, see how I process the results by splitting the response XML to successfully deleted items an failures, and displaying only the latter ones:

  1. # modify the $url and $listTitle values to match your configuration
  2. $url = "http://YourSharePoint/Web/SubWeb&quot;
  3. $listTitle = "YourCalendar"
  4.  
  5. $web = Get-SPWeb $url
  6. $list = $web.Lists[$listTitle]
  7.  
  8. $query = New-Object Microsoft.SharePoint.SPQuery
  9.   $query.Query =
  10.           "<Where>
  11.              <Lt>
  12.                <FieldRef Name='EndDate' />
  13.                <Value Type='DateTime'>2017-01-01 00:00:00</Value>
  14.              </Lt>
  15.            </Where>"
  16. $query.ViewFields = "<FieldRef Name='ID' />"
  17. $query.ViewFieldsOnly = $true
  18. $query.RowLimit = 1000;
  19.  
  20. $itemCount = 0
  21. $listId = $list.ID
  22.  
  23. do
  24. {
  25.     $listItems = $list.GetItems($query)
  26.     $itemIds = $listItems | % { [String]$_.ID }
  27.     [System.Text.StringBuilder]$batchXml = New-Object "System.Text.StringBuilder"
  28.     [Void]$batchXml.Append("<?xml version=`"1.0`" encoding=`"UTF-8`"?><Batch>")
  29.     $itemIds | % {
  30.       $itemId = $_
  31.       [Void]$batchXml.Append("<Method ID=`"$itemId`"><SetList>$listId</SetList><SetVar Name=`"ID`">$itemId</SetVar><SetVar Name=`"Cmd`">Delete</SetVar></Method>")
  32.     }
  33.     [Void]$batchXml.Append("</Batch>")
  34.     Write-Host Deleting next $listItems.Count entries…
  35.  
  36.     $result = [Xml]$web.ProcessBatchData($batchXml.ToString())
  37.     $success = @(Select-Xml -Xml $result -XPath '//Results/Result[@Code="0"]')
  38.     $failure = @(Select-Xml -Xml $result -XPath '//Results/Result[@Code!="0"]')
  39.     $itemCount += $success.Count
  40.     # list errors
  41.     $failure | % {
  42.         $errorNode = $_.Node
  43.         Write-Host Error deleting entry with ID $errorNode.ID error code: $errorNode.Code error text: $errorNode.ErrorText
  44.     }
  45. }
  46. while ($listItems.ListItemCollectionPosition -ne $null)
  47.  
  48. Write-Host Summary: $itemCount entries deleted

Although a few events have been really deleted, I started to get the following errors very quickly, and no deletion was performed after that:

image

The error messages above correspond to the following XML response:

<Result ID="" Code="-2147023673">
<ErrorText>The operation failed because an unexpected error occurred. (Result Code: 0x800704c7)</ErrorText></Result>

In the ULS logs I found a lot of such entries:

Batchmgr Method error. Errorcode: 0x1c32cbb0. Error message: The operation failed because an unexpected error occurred. (Result Code: 0x800704c7)

and at the top of the a single entry like this:

Batchmgr Method error. Errorcode: 0x1c32cbb0. Error message: Item does not exist.  The page you selected contains an item that does not exist.  It may have been deleted by another user.

After that, I’ve tried to delete the items using the same CAML query, hoping that I get more information about the failure. I used this script:

  1. # modify the $url and $listTitle values to match your configuration
  2. $url = "http://YourSharePoint/Web/SubWeb&quot;
  3. $listTitle = "YourCalendar"
  4.  
  5. $web = Get-SPWeb $url
  6. $list = $web.Lists[$listTitle]
  7.  
  8. $query = New-Object Microsoft.SharePoint.SPQuery
  9.   $query.Query =
  10.           "<Where>
  11.              <Lt>
  12.                <FieldRef Name='EndDate' />
  13.                <Value Type='DateTime'>2017-01-01 00:00:00</Value>
  14.              </Lt>
  15.            </Where>"
  16. $query.ViewFields = "<FieldRef Name='ID' />"
  17. $query.ViewFieldsOnly = $true
  18. $listItems = $list.GetItems($query)
  19. $itemIDsToDelete = $listItems | % { $_["ID"] }
  20. $totalCount = $itemIDsToDelete.Count   
  21. Write-Host $totalCount item will be deleted
  22. $counter = 1
  23. $itemIDsToDelete | % {
  24.   $itemID = $_
  25.   Write-Host Deleting item with ID $itemID `($counter / $totalCount`)
  26.   $item = $list.GetItemById($itemID)
  27.   $item.Delete()
  28.   $counter++
  29. }

I’ve got similar error messages as earlier for specific items, but the deletion went further than:

Exception calling "Delete" with "0" argument(s): "Item does not exist.
The page you selected contains an item that does not exist.  It may have been
deleted by another user."
At line:19 char:3
+   $item.Delete()
+   ~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException 
    + FullyQualifiedErrorId : SPException

image

I stopped the script after a while, and checked the ULS logs:

Item does not exist.  The page you selected contains an item that does not exist.  It may have been deleted by another user.<nativehr>0x81020016</nativehr><nativestack></nativestack>
SPRequest.GetListItemDataWithCallback2: UserPrincipalName=i:0).w|s-1-5-21-3634847118-1559816030-2180994487-3194, AppPrincipalName= ,pSqlClient=<null> ,bstrUrl=http://YourSharePoint/Web/SubWeb ,bstrListName={2A67D5C3-7AC1-4F3E-AB47-2051CDB94237} ,bstrViewName=<null> ,bstrViewXml=<View Scope="RecursiveAll" ModerationType="Moderator"><Query><Where><Eq><FieldRef Name="ID"></FieldRef><Value Type="Integer">2328</Value></Eq></Where></Query><RowLimit Paged="TRUE">1</RowLimit></View> ,fSafeArrayFlags=SAFEARRAYFLAG_NONE
System.Runtime.InteropServices.COMException: Item does not exist.  The page you selected contains an item that does not exist.  It may have been deleted by another user.<nativehr>0x81020016</nativehr><nativestack></nativestack>, StackTrace:    at Microsoft.SharePoint.SPListItemCollection.EnsureListItemsData()     at Microsoft.SharePoint.SPListItemCollection.get_Count()     at Microsoft.SharePoint.SPList.GetItemById(String strId, Int32 id, String strRootFolder, Boolean cacheRowsetAndId, String strViewFields, Boolean bDatesInUtc, Boolean bExpandQuery)     at Microsoft.SharePoint.SPList.GetItemById(String strId, Int32 id, String strRootFolder, Boolean cacheRowsetAndId, String strViewFields, Boolean bDatesInUtc)     at Microsoft.SharePoint.SPList.GetItemById(String strId, Int32 id, String strR…
…ment.Automation.Runspaces.RunspaceBase.RunActionIfNoRunningPipelinesWithThreadCheck(Action action)     at System.Management.Automation.ScriptBlock.InvokeWithPipe(Boolean useLocalScope, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo invocationInfo, Object[] args)     at System.Management.Automation.ScriptBlock.InvokeUsingCmdlet(Cmdlet contextCmdlet, Boolean useLocalScope, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Object[] args)     at Microsoft.PowerShell.Commands.ForEachObjectCommand.ProcessRecord()     at System.Management.Automation.CommandProcessor.ProcessRecord()     at System.Management.Automation.CommandProcessorBase.DoExecute()     at System.Mana…
Item does not exist.  The page you selected contains an item that does not exist.  It may have been deleted by another user.<nativehr>0x81020016</nativehr><nativestack></nativestack>
SPRequest.DeleteItem: UserPrincipalName=i:0).w|s-1-5-21-3634847118-1559816030-2180994487-3194, AppPrincipalName= ,bstrUrl=http://YourSharePoint/Web/SubWeb ,bstrListName={2A67D5C3-7AC1-4F3E-AB47-2051CDB94237} ,lID=2327 ,dwDeleteOp=3 ,bUnRestrictedUpdateInProgress=False
System.Runtime.InteropServices.COMException: Item does not exist.  The page you selected contains an item that does not exist.  It may have been deleted by another user.<nativehr>0x81020016</nativehr><nativestack></nativestack>, StackTrace:    at Microsoft.SharePoint.SPListItem.DeleteCore(DeleteOp deleteOp)     at Microsoft.SharePoint.SPListItem.Delete()     at CallSite.Target(Closure , CallSite , Object )     at <ScriptBlock>(Closure , FunctionContext )     at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)     at System.Management.Automation.ScriptBlock.InvokeWithPipeImpl(Boolean createLocalScope, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInfo invocationInfo, Object[] args)     a…

What should it mean, that specific items were deleted? I was sure, I work alone on the calendar. Fortunately, I already worked a lot previously with calendar entries, so it took not very long time to find a reason for the problem.

You should know, that if you delete the “master” item of the recurring events in SharePoint (and we had a lot of them in this case), all of the series exception are deleted the same time. As our original query returned this exception items as well, we had errors when we wanted to delete the already deleted items. The second script (without batch deletion) was able to survive it, but the batch deletion script got mad because of that.

What’s the solution for this issue? Let’s fix our CAML query and select only the items that are not recurring event exceptions:

<Where>
    <And>
    <Lt>
        <FieldRef Name=’EndDate’ />
        <Value Type=’DateTime’>2017-01-01 00:00:00</Value>
    </Lt>
    <Lt>
        <FieldRef Name=’EventType’/>
        <Value Type=’Integer’>2</Value>
    </Lt>
    </And>
</Where>

The query for EventType less than 2 originates from my practice, an unofficial documentation can be found here. The event types are defined (at least more or less) in the internal static class Microsoft.SharePoint.ApplicationPages.Calendar.EventType, for example Deleted has a value 3 and Updated has a value of 4.

The updated script:

  1. # modify the $url and $listTitle values to match your configuration
  2. $url = "http://YourSharePoint/Web/SubWeb&quot;
  3. $listTitle = "YourCalendar"
  4.  
  5. $web = Get-SPWeb $url
  6. $list = $web.Lists[$listTitle]
  7.  
  8. $query = New-Object Microsoft.SharePoint.SPQuery
  9.   $query.Query =
  10.           "<Where>
  11.              <And>
  12.                <Lt>
  13.                  <FieldRef Name='EndDate' />
  14.                  <Value Type='DateTime'>2017-01-01 00:00:00</Value>
  15.                </Lt>
  16.                <Lt>
  17.                  <FieldRef Name='EventType'/>
  18.                  <Value Type='Integer'>2</Value>
  19.                </Lt>
  20.              </And>
  21.            </Where>"
  22. $query.ViewFields = "<FieldRef Name='ID' />"
  23. $query.ViewFieldsOnly = $true
  24. $query.RowLimit = 1000;
  25.  
  26. $itemCount = 0
  27. $listId = $list.ID
  28.  
  29. do
  30. {
  31.     $listItems = $list.GetItems($query)
  32.     $itemIds = $listItems | % { [String]$_.ID }
  33.     [System.Text.StringBuilder]$batchXml = New-Object "System.Text.StringBuilder"
  34.     [Void]$batchXml.Append("<?xml version=`"1.0`" encoding=`"UTF-8`"?><Batch>")
  35.     $itemIds | % {
  36.       $itemId = $_
  37.       [Void]$batchXml.Append("<Method ID=`"$itemId`"><SetList>$listId</SetList><SetVar Name=`"ID`">$itemId</SetVar><SetVar Name=`"Cmd`">Delete</SetVar></Method>")
  38.     }
  39.     [Void]$batchXml.Append("</Batch>")
  40.     Write-Host Deleting next $listItems.Count entries…
  41.  
  42.     $result = [Xml]$web.ProcessBatchData($batchXml.ToString())
  43.     $success = @(Select-Xml -Xml $result -XPath '//Results/Result[@Code="0"]')
  44.     $failure = @(Select-Xml -Xml $result -XPath '//Results/Result[@Code!="0"]')
  45.     $itemCount += $success.Count
  46.     # list errors
  47.     $failure | % {
  48.         $errorNode = $_.Node
  49.         Write-Host Error deleting entry with ID $errorNode.ID error code: $errorNode.Code error text: $errorNode.ErrorText
  50.     }
  51. }
  52. while ($listItems.ListItemCollectionPosition -ne $null)
  53.  
  54. Write-Host Summary: $itemCount entries deleted

Using the modified CAML query I had no more errors, BUT according to the comment of Andrey Markeev in this thread, batch deleting only moves the items to the Recycle Bin, instead of really deleting them. That is not optimal for me.

To delete the items from the Recycle Bin as well, there are two main alternative solutions. Either you extend the batch deletion script with removing recycled items from the Recycle Bin, or you perform a cleanup in a second step.

The code samples in the forum thread referred to earlier as well are deleting either all items from the Recycle Bin, or trying to delete the items using the original list item ID, although the ID in the Recycle Bin differs from the original one. That is either not ideal or does not function. If you invoke the Recycle method of a SPListItem instance, the new ID, the transaction ID is returned to you (as a Guid), but this information is unfortunately not available by batch deletion. We had no folder structure in our calendar, so the LeafName property of the SPRecycleBinItem correspond to the server relative URL of the root folder of the source list (where the item was deleted) without the leading slash, and the DirName property corresponds to the list item ID (ending with ‘_.000’), so we can simply make a query for the IDs in the Recycle Bin, and remove them permanently:

$recBin = $web.Site.RecycleBin
$recBinIds = @($recBin | ? { $itemIds -contains $_.LeafName.Trim(‘_.000’) -and $_.DirName -eq $list.RootFolder.ServerRelativeUrl.Trim(‘/’) } | % { $_.ID })
$recBin.Delete($recBinIds)

Note, that this script is not universal. It might not function, if you have deleted an item from a list including folder structure or a document from a library. For example, in the case of a document library the LeafName property correspond to the file name without the extension and not to the ID.

The updated script:

  1. # modify the $url and $listTitle values to match your configuration
  2. $url = "http://YourSharePoint/Web/SubWeb&quot;
  3. $listTitle = "YourCalendar"
  4.  
  5. $web = Get-SPWeb $url
  6. $list = $web.Lists[$listTitle]
  7.  
  8. $query = New-Object Microsoft.SharePoint.SPQuery
  9.   $query.Query =
  10.           "<Where>
  11.              <And>
  12.                <Lt>
  13.                  <FieldRef Name='EndDate' />
  14.                  <Value Type='DateTime'>2017-01-01 00:00:00</Value>
  15.                </Lt>
  16.                <Lt>
  17.                  <FieldRef Name='EventType'/>
  18.                  <Value Type='Integer'>2</Value>
  19.                </Lt>
  20.              </And>
  21.            </Where>"
  22. $query.ViewFields = "<FieldRef Name='ID' />"
  23. $query.ViewFieldsOnly = $true
  24. $query.RowLimit = 1000;
  25.  
  26. $itemCount = 0
  27. $listId = $list.ID
  28.  
  29. do
  30. {
  31.     $listItems = $list.GetItems($query)
  32.     $itemIds = $listItems | % { [String]$_.ID }
  33.     [System.Text.StringBuilder]$batchXml = New-Object "System.Text.StringBuilder"
  34.     [Void]$batchXml.Append("<?xml version=`"1.0`" encoding=`"UTF-8`"?><Batch>")
  35.     $itemIds | % {
  36.       $itemId = $_
  37.       [Void]$batchXml.Append("<Method ID=`"$itemId`"><SetList>$listId</SetList><SetVar Name=`"ID`">$itemId</SetVar><SetVar Name=`"Cmd`">Delete</SetVar></Method>")
  38.     }
  39.     [Void]$batchXml.Append("</Batch>")
  40.     Write-Host Deleting next $listItems.Count entries…
  41.  
  42.     $result = [Xml]$web.ProcessBatchData($batchXml.ToString())
  43.     $success = @(Select-Xml -Xml $result -XPath '//Results/Result[@Code="0"]')
  44.     $failure = @(Select-Xml -Xml $result -XPath '//Results/Result[@Code!="0"]')
  45.     $itemCount += $success.Count
  46.     # list errors
  47.     $failure | % {
  48.         $errorNode = $_.Node
  49.         Write-Host Error deleting entry with ID $errorNode.ID error code: $errorNode.Code error text: $errorNode.ErrorText
  50.     }
  51.  
  52.     # delete items from Recycle Bin as well
  53.     $recBin = $web.Site.RecycleBin
  54.     $recBinIds = @($recBin | ? { $itemIds -contains $_.LeafName.Trim('_.000') -and $_.DirName -eq $list.RootFolder.ServerRelativeUrl.Trim('/') } | % { $_.ID })
  55.     Write-Host Deleting $recBinIds.Count entries from recycle bin…
  56.     $recBin.Delete($recBinIds)
  57. }
  58. while ($listItems.ListItemCollectionPosition -ne $null)
  59.  
  60. Write-Host Summary: $itemCount entries deleted

As the second option, you can delete all items you recycled from the calendar after you have finished the batch deletion. The script below removes items from the Recycle Bin that were deleted from the calendar in the past hour by the current user:

$recBin = $web.Site.RecycleBin
$ids = $recBin | ? { $_.DeletedDate -gt (Get-Date).ToUniversalTime().AddHours(-1) -and $_.DirName -eq ‘Web/SubWeb/Lists/YourCalendar’ -and $_.DeletedById -eq $web.CurrentUser.ID } | % { $_.ID }
Write-Host Deleting $ids.Count item from recycle bin…
$recBin.Delete($ids)

If the deletion was quicker than 60 minutes, you can reduce the time span used in the script, to reduce the possibility you delete an item inadvertently from the Recycle Bin.

July 16, 2017

Find Your Scripts in SharePoint within Seconds – the Effective, but Unsupported Way

Filed under: JavaScript, SP 2013, SQL, Tips & Tricks — Tags: , , , — Peter Holpar @ 18:41

The SharePoint environment I’m working on contains hundreds of webs. I create test sites for various tasks (like prototyping JSLink-based solutions) including the necessary lists, and store the .js and .css files typically in the Site Assets library of that web site to keep the solution artifacts (lists / files) together. It is rather common, that after I’ve finished the proof of concept, I don’t need it for months, then suddenly I should return to it, but I don’t find it anymore, as I don’t remember, which site I used for that solution.

For that kind of search I’ve created a simply SQL query to find the script directly in the content database of the site collection. Yes, I know it is officially unsupported to access the SharePoint databases directly, but I’m OK with that in my test system. Use it on your own risk.

SELECT [Id]
      ,[SiteId]
      ,[DirName]
      ,[LeafName]
      ,[TimeLastModified]
      ,[DeleteTransactionId]
  FROM [dbo].[AllDocs]
  WHERE LeafName LIKE ‘%.js’
  AND DirName LIKE ‘%SiteAssets%’
  ORDER BY TimeLastWritten DESC

This script lists the file name (LeadName) and path (DirName) of the scripts stored in various sites in their Site Assets library. The name of script and the date of last modification (TimeLastModified) is usually enough to identify the script I need. Note, that the records, that have a value other that 0x in the DeleteTransactionId column are recycled and located in the Recycle Bin. Of course, this method works only in the case of on-premise installations, and only as long as you have access to the SharePoint databases.

July 9, 2017

Import-SPWeb: Failed to read package file

Filed under: PowerShell, SP 2013 — Tags: , — Peter Holpar @ 21:41

A few days ago I wanted to restore a SharePoint list (including data) on a development server via the Import-SPWeb cmdlet based on a backup created by the Export-SPWeb cmdlet in the productive farm:

$url = "http://YourSharePoint/Web/SubWeb&quot;
$filePath = "E:\Data\ListExport.cmp"

Import-SPWeb $url -Path $filePath

Surprisingly, I’ve received a “Failed to read package file” exception, although the same .cmp file could have been imported in the test farm, having the same SharePoint version as the development and productive systems using the same user and same permissions.

PS C:\Users\pholpar> Import-SPWeb $url -Path $filePath

Log file generated:
        E:\Data\ListExport.cmp.import.log

Import-SPWeb : Failed to read package file.
At line:1 char:1
+ Import-SPWeb $url -Path $filePath
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (Microsoft.Share…CmdletImportWeb:
   SPCmdletImportWeb) [Import-SPWeb], SPException
    + FullyQualifiedErrorId : Microsoft.SharePoint.PowerShell.SPCmdletImportWeb

image

Having a look into the generated log file, I found the next surprise in the inner exception: “Failure writing to target file”. Actually, I wanted to read the .cmp file, not to write, what’s the problem then with writing?

[03.07.2017 21:03:28] Start Time: 03.07.2017 21:03:28.
[03.07.2017 21:03:28] Progress: Initializing Import.
[03.07.2017 21:03:28] Error: Failure writing to target file
[03.07.2017 21:03:28] Debug:    at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
   at Microsoft.SharePoint.Library.SPRequest.ExtractFilesFromCabinet(String bstrTempDirectory, String bstrCabFileLocation)
   at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3()
   at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)
   at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)
   at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)
   at Microsoft.SharePoint.Deployment.ImportDataFileManager.Uncompress(SPRequest request)
[03.07.2017 21:03:28] FatalError: Failed to read package file.
*** Inner exception:
Failure writing to target file
[03.07.2017 21:03:28] Debug:    at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)
   at Microsoft.SharePoint.Library.SPRequest.ExtractFilesFromCabinet(String bstrTempDirectory, String bstrCabFileLocation)
   at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3()
   at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)
   at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)
   at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)
   at Microsoft.SharePoint.Deployment.ImportDataFileManager.Uncompress(SPRequest request)
[03.07.2017 21:03:28] Progress: Import did not complete.
[03.07.2017 21:03:28] Finish Time: 03.07.2017 21:03:28.
[03.07.2017 21:03:28] Duration: 00:00:00
[03.07.2017 21:03:28] Finished with 0 warnings.
[03.07.2017 21:03:28] Finished with 2 errors.

Checking the ULS logs provided me further details:

Entering BeginProcessing Method of Import-SPWeb.
Leaving BeginProcessing Method of Import-SPWeb.
Entering ProcessRecord Method of Import-SPWeb.
SecurityTokenServiceSendRequest: RemoteAddress: ‘
http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc’ Channel: ‘Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustChannelContract’ Action: ‘http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue’ MessageId: ‘urn:uuid:41e822d8-9b0f-4a48-93a2-003eaf92c8dc’
Leaving Monitored Scope (Build the X509Chain.). Execution Time=168,6921
Leaving Monitored Scope (SPCertificateValidator.Validate). Execution Time=168,7904
SecurityTokenServiceSendRequest: RemoteAddress: ‘
http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc’ Channel: ‘Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustChannelContract’ Action: ‘http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue’ MessageId: ‘urn:uuid:e3155df3-9409-4d5a-9391-bf938f9b7007’
Initializing Import.
SecurityTokenServiceSendRequest: RemoteAddress: ‘
http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc’ Channel: ‘Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustChannelContract’ Action: ‘http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue’ MessageId: ‘urn:uuid:953853ec-af7a-4100-91f8-b49533a7a986’
<nativehr>0x81070266</nativehr><nativestack></nativestack>Failure writing to target file
SPRequest.ExtractFilesFromCabinet: UserPrincipalName=i:0).w|s-1-5-21-3634847118-1559816030-2180994487-3194, AppPrincipalName= ,bstrTempDirectory=C:\Users\pholpar\AppData\Local\Temp\2\48463715-4bf7-4ca6-8aa1-3b92128d6789 ,bstrCabFileLocation=E:\Data\ListExport.cmp
System.Runtime.InteropServices.COMException: <nativehr>0x81070266</nativehr><nativestack></nativestack>Failure writing to target file, StackTrace:    at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3()     at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)     at Microsoft.SharePoint.Deployment.ImportDataFileManager.Uncompress(SPRequest request)     at Microsoft.SharePoint.Deployment.SPImport.Run()     at Microsoft.SharePoint.PowerShell.SPCmdletImportWeb.InternalProcessRecord()     at Microsoft.SharePoint.PowerShell.S…
…PCmdlet.ProcessRecord()     at System.Management.Automation.CommandProcessor.ProcessRecord()     at System.Management.Automation.CommandProcessorBase.DoExecute()     at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input, Hashtable errorResults, Boolean enumerate)     at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)     at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)     at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)     at System.Management.Automatio…
…n.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)     at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame)     at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)     at System.Management.Automation.DlrScriptCommandProcessor.RunClause(Action`1 clause, Object dollarUnderbar, Object inputToProcess)     at System.Management.Automation.CommandProcessorBase.DoComplete()     at System.Management.Automation.Internal.PipelineProcessor.DoCompleteCore(CommandProcessorBase commandRequestingUpstreamCommandsToStop)     at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input, Hashtable errorResults, Boolean enumerate)     at System.Management.Automation.Runspaces.LocalPipeline.Inv…
…okeHelper()     at System.Management.Automation.Runspaces.LocalPipeline.InvokeThreadProc()     at System.Management.Automation.Runspaces.PipelineThread.WorkerProc()     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)     at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)     at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)     at System.Threading.ThreadHelper.ThreadStart() 
Failure writing to target file
Failed to read package file.  *** Inner exception:  Failure writing to target file
Import did not complete.
Microsoft.SharePoint.SPException: Failed to read package file. —> Microsoft.SharePoint.SPException: Failure writing to target file —> System.Runtime.InteropServices.COMException: <nativehr>0x81070266</nativehr><nativestack></nativestack>Failure writing to target file     at Microsoft.SharePoint.Library.SPRequestInternalClass.ExtractFilesFromCabinet(String bstrTempDirectory, String bstrCabFileLocation)     at Microsoft.SharePoint.Library.SPRequest.ExtractFilesFromCabinet(String bstrTempDirectory, String bstrCabFileLocation)     — End of inner exception stack trace —     at Microsoft.SharePoint.SPGlobal.HandleComException(COMException comEx)     at Microsoft.SharePoint.Library.SPRequest.ExtractFilesFromCabinet(String bstrTempDirectory, String bstrCabFileLocation)     at Microsoft.Sha…
…rePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3()     at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)     at Microsoft.SharePoint.Deployment.ImportDataFileManager.Uncompress(SPRequest request)     — End of inner exception stack trace —     at Microsoft.SharePoint.Deployment.ImportDataFileManager.Uncompress(SPRequest request)     at Microsoft.SharePoint.Deployment.SPImport.Run()     at Microsoft.SharePoint.PowerShell.SPCmdletImportWeb.InternalProcessRecord()     at Microsoft.SharePoint.PowerShell.SPCmdlet.ProcessRecord()
Error Category: InvalidData    Target Object  Microsoft.SharePoint.PowerShell.SPCmdletImportWeb  Details  NULL  RecommendedAction NULL
Leaving ProcessRecord Method of Import-SPWeb.
Entering EndProcessing Method of Import-SPWeb.
Leaving EndProcessing Method of Import-SPWeb.

Searching for the error code and text on the web, I found this (not SharePoint-related) forum entry, with a bit more descriptive error text “Failure writing to the target file. Please check that you have enough disk space.”, that led me to the rather trivial solution: the C: drive was full, so the process was not able to create a temporary folder by extracting the .cmp file (that is actually a .cab file).

Note 1: As you see at the top of the post, the Import-SPWeb cmdlet was executed from the path C:\Users\pholpar. It does not help however,if you simply change the path to an other drive that has more place, as a temporary folder in the user profile (located in our case on the C: drive) is used to extract the files, see the value of the bstrTempDirectory parameter in the ULS logs above.

Note 2: It does help however, if you create and restore the backup file using the NoFileCompression switch of the Import-SPWeb / Export-SPWeb cmdlets, as in this case there is no need for the extraction process and the temporary folder mentioned above.

The ideal solution is of course, to keep your servers clean and healthy, providing always enough resources (including but not limited to disk space and memory) to fulfill their tasks.

June 25, 2017

How to get the Url of the “Edit View” Page of a Specific SharePoint List View from PowerShell

Filed under: PowerShell, SP 2013, Tips & Tricks — Tags: , , — Peter Holpar @ 07:33

There might be cases when you can’t access the Edit View page of a specific list view from the SharePoint UI. For example, there is no such direct link in the case of Survey lists. There is no ribbon including the Manage Views group, and the Views area is missing from the List settings page as well.

You can, however access the Edit View page from your browser if you know its URL. The standard URL of this page has this pattern:

http://YourSharePoint/Web/SubWeb/_layouts/15/ViewEdit.aspx?List=%7BDC913804%2DB28E%2D4F52%2DAF53%2DDEC490A1C83D%7D&View=%7B2E7DF707%2D42BA%2D44EE%2D87C6%2D0919CA38BDF1%7D

As you see, the ViewEdit.aspx page is responsible for this functionality. The encoded Ids (Guid) of the List and the View are passed as query string parameters (List and View respectively).

You can get the URL of the page using this PowerShell script easily:

$web = Get-SPWeb ‘http://YourSharePoint/Web/SubWeb&#8217;
$list = $web.Lists[‘YourList’]
# get the default view of the list
$view = $list.DefaultView
# or get an arbitrary view by its name
# $view = $list.Views[‘All Items’]
$viewId = $view.ID

function EscapeGuid($guid)
{
  return "{$guid}".ToUpper().Replace(‘-‘, ‘%2D’).Replace(‘{‘, ‘%7B’).Replace(‘}’, ‘%7D’)
}

$url = $web.Url + ‘/_layouts/15/ViewEdit.aspx?List=’ + (EscapeGuid $list.ID) + ‘&View=’ + (EscapeGuid $view.ID)

You can even start the page in Internet Explorer from PowerShell if you wish:

$ie = New-Object -ComObject InternetExplorer.Application
$ie.Navigate2($url)
$ie.Visible = $true

Older Posts »

Create a free website or blog at WordPress.com.