Second Life of a Hungarian SharePoint Geek

March 29, 2017

Working with the REST / OData Interface from PowerShell

Filed under: OData, PowerShell, REST, SP 2013 — Tags: , , , — Peter Holpar @ 20:56

If you follow my blog you might already know that I am not a big fan of the REST / OData interface. I prefer using the client object model. However there are cases, when REST provides a simple (or even the only available) solution.

For example, we are working a lot with PowerShell. If you are working with SharePoint on the client side at a customer, and you are not allowed to install / download / copy the assemblies for the managed client object model (CSOM), you have a problem.

Some possible reasons (you should know, that the SharePoint Server 2013 Client Components SDK is available to download as an .msi, or you can get the assemblies directly from an on-premise SharePoint installation):

  • You might have no internet access, so you cannot download anything from the web.
  • If you happen to have internet access, you are typically not allowed to install such things without administrator permissions on the PC. It’s quite rare case, if you or the business user you are working with has this permission.
  • You have no direct access on the SharePoint server, so you cannot copy the assemblies from it.
  • You are not allowed to use your own memory stick (or other storage device) to copy the assemblies from it.
  • Even if there is no technical barrier, company policies might still prohibit you using external software components like the CSOM assemblies.

In this case, using the REST interface is a reasonable choice. You can have a quick overview of the REST-based list operations here.

The main questions I try to answer in this post:

  • Which object should I use to send the request?
  • How to authenticate my request?
  • How to build up the payload for the request?

First of all, I suggest you to read this post to learn some possible pitfalls when working with REST URLs from PowerShell and how to avoid them with escaping.

Reading data with the SharePoint REST interface

Reading data with a GET request

Sending a GET request for a REST-based service in PowerShell is not really a challenge, might you think, and you are right, it is really straightforward most of the cases. But take the following example, listing the Id and Title fields of items in a list:

$listTitle = "YourList"
$url = "http://YourSharePoint/_api/Web/Lists/GetByTitle(‘$listTitle‘)/Items?`$select=Id,Title"

$request = [System.Net.WebRequest]::Create($url)
$request.UseDefaultCredentials = $true
$request.Accept = ‘application/json;odata=verbose’

$response = $request.GetResponse()
$reader = New-Object System.IO.StreamReader $response.GetResponseStream()
# ConvertFrom-Json : Cannot convert the Json string because a dictionary converted from it contains duplicated keys ‘Id’ and ‘ID’.
#$response = $reader.ReadToEnd()
$response = $reader.ReadToEnd() -creplace ‘"ID":’, ‘"DummyId":’

$result = ConvertFrom-Json -InputObject $response
$result.d.results | select Id, Title

If you would use

$response = $reader.ReadToEnd()

instead of

$response = $reader.ReadToEnd() -creplace ‘"ID":’, ‘"DummyId":’

then you became this exception, when trying to convert the JSON response:

ConvertFrom-Json : Cannot convert the Json string because a dictionary converted from it contains duplicated keys ‘Id’ and ‘ID’.

The reason, that the JSON response of the server contains the fields Id and ID. JSON is case-sensitive, but PowerShell is not, so it is an issue if you want to convert the JSON response to a PowerShell object. You can read more about it in this post, although I don’t like the solution proposed there. Although it really helps to avoid the error, but it uses the case insensitive replace operator instead of the case sensitive creplace, so it converts both fields into a dummy field. PowerShell seems to have no problem with the duplicated properties.

Instead of using a System.Net.WebRequest object, we can achieve a shorter version using the Invoke-RestMethod cmdlet. Note, that we don’t select and display the Id property in this case to avoid complications. See my comments about that in the next section discussing the POST request.

$listTitle = "YourList"
$url = "http://YourSharePoint/_api/Web/Lists/GetByTitle(‘$listTitle‘)/Items?`$select=Title"
$headers = @{ ‘Accept’ = ‘application/json; odata=verbose’}
$result = Invoke-RestMethod -Uri $url -Method Get -Headers $headers -UseDefaultCredentials
$result.d.results | select Title

Reading data with a POST request

There are cases when you have to use the POST method instead of GET to read some data from SharePoint. For example, if you need to filter the items via a CAML query. In the following example I show you how to query the file names all documents in a library recursively that are older than a threshold value:

$listTitle = "YourDocuments"
$offsetDays = -30

$urlBase = "http://YourSharePointSite/"
$urlAuth = $urlBase +"_api/ContextInfo"
$url = $urlBase + "_api/Web/Lists/GetByTitle(‘$listTitle’)/GetItems?`$select=FileLeafRef"

$viewXml = "<View Scope=’Recursive’><ViewFields><FieldRef Name=’Created’/><FieldRef Name=’FileLeafRef’/></ViewFields><Query><Where><Lt><FieldRef Name=’Created’ /><Value Type=’DateTime’><Today OffsetDays=’$offsetDays’ /></Value></Lt></Where></Query></View>"

$queryPayload = @{ 
                   ‘query’ = @{
                          ‘__metadata’ = @{ ‘type’ = ‘SP.CamlQuery’ };                      
                          ‘ViewXml’ = $viewXml
                   }
                 } | ConvertTo-Json

# authentication
$auth = Invoke-RestMethod -Uri $urlAuth -Method Post -UseDefaultCredentials
$digestValue = $auth.GetContextWebInformation.FormDigestValue

# the actual request
$headers = @{ ‘X-RequestDigest’ = $digestValue; ‘Accept’ = ‘application/json; odata=verbose’ }
$result = Invoke-RestMethod -Uri $url -Method Post -Body $queryPayload -ContentType ‘application/json; odata=verbose’ -Headers $headers –UseDefaultCredentials

# displaying results
$result.d.results | select FileLeafRef

Just for the case of comparison I include the same payload in JavaScript format:

var queryPayload = {
                     ‘query’ : {
                        
‘__metadata’ : { ‘type’ : ‘SP.CamlQuery’ },
                         ‘ViewXml’ : viewXml
                    
}
                   };

As you can see, these are the most relevant differences in the format we need in PowerShell:

  • We use an equal sign ( = ) instead of  ( : ) to separate the name and its value.
  • We use a semicolon ( ; ) instead of the comma ( , ) to separate object fields.
  • We need a leading at sign ( @ ) before the curly braces ( { ).

The Invoke-RestMethod tries to automatically convert the response to the corresponding object based on the content type of the response. If it is an XML response (see the authentication part above) then the result will be a XmlDocument. If it is a JSON response then the result will be a PSCustomObject representing the structure of the response. However, if the response can not be converted, it remains a single String.

For example, if we don’t limit the fields we need in response via the $select query option:

$url = $urlBase + "_api/Web/Lists/GetByTitle(‘$listTitle’)/GetItems"

then the response includes the fields Id and ID again. In this case we should remove one of these fields using the technique illustrated above with the simple GET request, before we try to convert the response via the ConvertFrom-Json cmdlet.

Note: If you still use PowerShell v3.0 you get this error message when you invoke Invoke-RestMethod setting the Accept header:

Invoke-RestMethod : The ‘Accept’ header must be modified using the appropriate property or method.
Parameter name: name

So if it is possible, you should consider upgrading to PowerShell v4.0. Otherwise, you can use the workaround suggested in this forum thread, where you can read more about the issue as well.

If you are not sure, which version you have, you can use $PSVersionTable.PSVersion to query the version number, or another option as suggested here.

Creating objects

In this case we send a request with the POST method to the server. The following code snippet shows, how you can create a new custom list:

$listTitle = "YourList"

$urlBase = "http://YourSharePoint/&quot;
$urlAuth = $urlBase +"_api/ContextInfo"
$url = $urlBase + "_api/Web/Lists"

$queryPayload = @{ 
                    ‘__metadata’ = @{ ‘type’ = ‘SP.List’ }; ‘AllowContentTypes’ = $true; ‘BaseTemplate’ = 100;
                    ‘ContentTypesEnabled’ = $true; ‘Description’ = ‘Your list description’; ‘Title’ = $listTitle                      
    } | ConvertTo-Json

$auth = Invoke-RestMethod -Uri $urlAuth -Method Post -UseDefaultCredentials
$digestValue = $auth.GetContextWebInformation.FormDigestValue

$headers = @{ ‘X-RequestDigest’ = $digestValue; ‘Accept’ = ‘application/json; odata=verbose’ }

$result = Invoke-RestMethod -Uri $url -Method Post -Body $queryPayload -ContentType ‘application/json; odata=verbose’ -Headers $headers –UseDefaultCredentials

The response we receive in the $result variable contains the properties of the list we just created. For example, the Id (GUID) of the list is available as $result.d.Id.

Updating objects

In this case we send a request with the POST method to the server and set the X-HTTP-Method header to MERGE. The following code snippet shows, how to change the title of the list we created in the previous step:

$listTitle = "YourList"

$urlBase = "http://YourSharePoint/&quot;
$urlAuth = $urlBase +"_api/ContextInfo"
$url = $urlBase + "_api/Web/Lists/GetByTitle(‘$listTitle’)"

$queryPayload = @{ 
                    ‘__metadata’ = @{ ‘type’ = ‘SP.List’ }; ‘Title’ = ‘YourListNewTitle’                      
    } | ConvertTo-Json

$auth = Invoke-RestMethod -Uri $urlAuth -Method Post -UseDefaultCredentials
$digestValue = $auth.GetContextWebInformation.FormDigestValue

$headers = @{ ‘X-RequestDigest’ = $digestValue; ‘Accept’ = ‘application/json; odata=verbose’; ‘IF-MATCH’ = ‘*‘; ‘X-HTTP-Method’ = ‘MERGE’ }

$result = Invoke-RestMethod -Uri $url -Method Post -Body $queryPayload -ContentType ‘application/json; odata=verbose’ -Headers $headers –UseDefaultCredentials

Deleting objects

In this case we send a request with the POST method to the server and set the X-HTTP-Method header to DELETE. The following code snippet shows, how you can delete a list item:

$listTitle = "YourList"

$urlBase = "http://YourSharePoint/&quot;
$urlAuth = $urlBase +"_api/ContextInfo"
$url = $urlBase + "_api/Web/Lists/GetByTitle(‘$listTitle’)/Items(1)"

# authentication
$auth = Invoke-RestMethod -Uri $urlAuth -Method Post -UseDefaultCredentials
$digestValue = $auth.GetContextWebInformation.FormDigestValue

# the actual request
$headers = @{ ‘X-RequestDigest’ = $digestValue; ‘IF-MATCH’ = ‘*’; ‘X-HTTP-Method’ = ‘DELETE’ }
$result = Invoke-RestMethod -Uri $url -Method Post -Headers $headers -UseDefaultCredentials

Note: Although the documentation states, that “in the case of recyclable objects, such as lists, files, and list items, this results in a Recycle operation”, based on my tests it is false, as the objects got really deleted.

Final Note: This one applies to all of the operations discussed in the post. If the SharePoint site you are working with available via HTTPS and there is an issue with the certificate, you can turn off the certificate validation, although it is not recommended in a production environment. You should include this line in your code before making any web requests:

[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }

March 26, 2017

Generating Pseudo GUIDs for Your Project Server Entities

Filed under: PowerShell, Project Server, Tips & Tricks — Tags: , , — Peter Holpar @ 06:24

As you might have known, since the version 2013, Project Server utilizes pseudo-GUIDs to improve Project Server performance. These ones has the format of a “classical” GUID, but actually generated sequentially. As Microsoft states in this TechNet article:

"We handle GUIDs a little better in Project Server 2013 – and in many places they are sequential GUIDs which cause less index fragmentation"

This topic is quite good described in the Project Conference 2014 presentation Project Worst Practice – Learning from other peoples mistakes by Brian Smith. See the video recording between 6:08-13:54, or the slides 10-14.

One of the main components of the pseudo-GUID generation is the NewSequentialUid method of the Microsoft.Office.Project.Server.Library.PSUtility class:

public static Guid NewSequentialUid() 

  Guid guid; 
  if (NativeMethods.UuidCreateSequential(out guid) != 0) 
    return Guid.NewGuid(); 
  byte[] b = guid.ToByteArray(); 
  Array.Reverse((Array) b, 0, 4); 
  Array.Reverse((Array) b, 4, 2); 
  Array.Reverse((Array) b, 6, 2); 
  return new Guid(b); 
}

So if you want to use the same kind of pseudo-GUIDs for your own custom entities you create from code, you can get the IDs by invoking the method (for example, via PowerShell). The code sample below illustrates, how to get a single ID, or a batch of  IDs (in this case, 5 of them):

# load the necessary assembly
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Project.Shared")
# generate a single sequential ID
[Microsoft.Office.Project.Server.Library.PSUtility]::NewSequentialUid()
# or generate a range of sequential IDs, in this case, five of them
(1..5) | % { [Microsoft.Office.Project.Server.Library.PSUtility]::NewSequentialUid().Guid }

May 29, 2016

Project Publishing Failed due to Deleted SharePoint User

Filed under: Bugs, Event receivers, PowerShell, PS 2013 — Tags: , , , — Peter Holpar @ 05:53

In my recent post I wrote about a project publishing issue that was a result of a scheduling conflict.

The other day we had a similar problem with project publishing, but in this special case failed an other sub-process of the publishing process, the task synchronization. Another important difference from the former one is that at the scheduling conflict it was an end-user issue (a business user caused the conflict in the project plan scheduling), and in the case I’m writing about now, it was a mistake of an administrator plus a suboptimal code block in Project Server, that we can consider as a bug as well. But more on that a bit later…

First the symptoms we experienced. On the Manage Queue Jobs page in our PWA (http://YourProjectServer/PWA/_layouts/15/pwa/Admin/queue.aspx) we saw an entry of Job TypeSharePoint Task List Project” and Job State Failed And Blocking Correlation”.

Clicking on the entry displayed this information:

Queue: GeneralQueueJobFailed (26000) – ManagedModeTaskSynchronization.SynchronizeTaskListInManagedModeMessage. Details: id=’26000′ name=’GeneralQueueJobFailed’ uid=’46918ff3-3719-e611-80f4-005056b44e32′ JobUID=’adcad466-44bd-444b-a803-073fd12a2426′ ComputerName=’4fc61930-ef50-461b-b9ef-084a666c61ca’ GroupType=’ManagedModeTaskSynchronization’ MessageType=’SynchronizeTaskListInManagedModeMessage’ MessageId=’1′ Stage=” CorrelationUID=’cd56b408-a303-0002-d428-98cd03a3d101′.

The corresponding entries in the ULS logs:

PWA:http://YourProjectServer/PWA, ServiceApp:ProjectServerApplication, User:i:0#.w|YourDomain\FarmAccount, PSI: [QUEUE] SynchronizeTaskListInManagedModeMessage failed on project 5c21bf1b-c910-e511-80e5-005056b44e34. Exception: System.NullReferenceException: Object reference not set to an instance of an object.     at Microsoft.Office.Project.Server.BusinessLayer.ProjectModeManaged.UpdateAssignedToField(SPWeb workspaceWeb, DataSet taskDS, Guid taskUID, SPListItem listItem)     at Microsoft.Office.Project.Server.BusinessLayer.ProjectModeManaged.SynchronizeTask(SPList list, DataSet taskDS, Dictionary`2 taskMapping, DataRow row, DataView secondaryView, Dictionary`2 redoEntries)     at Microsoft.Office.Project.Server.BusinessLayer.ProjectModeManaged.<>c__DisplayClass1.<SynchronizeTaskListI…
…nManagedMode>b__0(SPWeb workspaceWeb)     at Microsoft.Office.Project.Server.BusinessLayer.Project.<>c__DisplayClass3d.<TryRunActionWithProjectWorkspaceWebInternal>b__3c()     at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3()     at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)     at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)     at Microsoft.Office.Project.Server.BusinessLayer.Project.TryRunActionWithProjectWorkspaceWebInternal(IPlatformContext context, Guid projectUid, Action`1 method, Boolean noThrow, DataRow row)     at Microsoft.Office.Project.Server.Busine…
…ssLayer.ProjectModeManaged.SynchronizeTaskListInManagedMode(Guid projectUid)     at Microsoft.Office.Project.Server.BusinessLayer.Queue.ProcessPublishMessage.ProcessSynchronizeTaskListInManagedModeMessage(Message msg, Group messageGroup, JobTicket jobTicket, MessageContext mContext), LogLevelManager Warning-ulsID:0x000CE687 has no entities explicitly specified.

So we have a NullReferenceException in the UpdateAssignedToField method of the Microsoft.Office.Project.Server.BusinessLayer.ProjectModeManaged class (Microsoft.Office.Project.Server assembly).

From the job message type “ManagedModeTaskSynchronization.SynchronizeTaskListInManagedModeMessage” it was obvious, that we have an issue with the synchronization between the project tasks and the Tasks list of the Project Web Site (PWS) of the project having the ID 5c21bf1b-c910-e511-80e5-005056b44e34”,  and from the method name “UpdateAssignedToField” we could assume, that the problem is caused either by an existing value of the “Assigned To” field, or by constructing a new value we want to update the field with.

We can use the following script to find out, which PWS belongs to the project ID above:

$pwa = Get-SPWeb http://YourProjectServer/PWA
$pwa.Webs | ? { $_.AllProperties[‘MSPWAPROJUID’] -eq ‘5c21bf1b-c910-e511-80e5-005056b44e34’ }

If we have a look at the code of the UpdateAssignedToField method, we see it begins with these lines. These lines are responsible for removing users from the “Assigned To” field (of type SPFieldUserValueCollection) that are no longer responsible for the task. The second part of method (not included below) is responsible for inserting new user entries. I highlighted the line that may cause (and in our case in fact has caused) an error if the value of the assignedTo[i].User expression is null.

bool isModified = false;
SPFieldUserValueCollection assignedTo = listItem["AssignedTo"] as SPFieldUserValueCollection;
DataRowView[] source = taskDS.Tables[1].DefaultView.FindRows(taskUID);
if (assignedTo != null)
{
    for (int i = assignedTo.Count – 1; i >= 0; i–)
    {
        string userName = ClaimsHelper.ConvertAccountFormat(assignedTo[i].User.LoginName);
        if (!source.Any<DataRowView>(resourceRow => (string.Compare(userName, resourceRow.Row.Field<string>("WRES_CLAIMS_ACCOUNT"), StringComparison.OrdinalIgnoreCase) == 0)))
        {
            assignedTo.RemoveAt(i);
            isModified = true;
        }
    }
}

The expression may be null if the user it refers to was deleted from the site. Note, that the expression assignedTo[i].LookupId even in this case returns the ID of the deleted user, and the expression assignedTo[i].LookupValue return its name.

How to detect which projects and which users are affected by the issue? I wrote the script below to display the possible errors:

  1. $rootWeb = Get-SPWeb http://YourProjectServer/PWA
  2.  
  3. $rootWeb.Webs | % {
  4.  
  5.     $web = $_
  6.  
  7.  
  8.     Write-Host ——————————-
  9.     Write-Host $web.Title
  10.  
  11.  
  12.     $foundMissingUsers = New-Object 'Collections.Generic.Dictionary[int,string]'
  13.  
  14.     $list = $web.Lists["Tasks"]
  15.  
  16.     if ($list -ne $null)
  17.     {
  18.         $list.Items | % {
  19.             $_["AssignedTo"] | ? {
  20.                  ($_.User -eq $null) -and (-not $foundMissingUsers.ContainsKey($_.LookupId)) } | % {
  21.                      if ($_ -ne $null ) { $foundMissingUsers.Add($_.LookupId, $_.LookupValue) }
  22.                  }
  23.         }
  24.  
  25.         $foundMissingUsers | % { $_ }
  26.     }
  27. }

Assuming

$allUserIds = $rootWeb.SiteUsers | % { $_.ID }

we could use

$allUserIds -NotContains $_.LookupId

instead of the condition

$_.User -eq $null

in the script above.

Indeed, we could identify two users on two separate projects, that were deleted by mistake, although they have assignments in the project Tasks lists.

We have recreated the users (and assigned the new users to the corresponding enterprise resources), but they have now another IDs. What can we do to fix the problem? The synchronization does not work anymore on these projects (making the project publishing impossible as well) so it does not provide a solution. We could replace the users in the “Assigned To” field, or simply remove the wrong one (it would be re-inserted by the second part of the UpdateAssignedToField method during the next synchronization), but there is an event receiver (Microsoft.Office.Project.PWA.ManagedModeListItemEventHandler) registered on this list, that cancels any changes in the list items when you want to persist the changes via the Update method. To avoid that, we could temporary disable the event firing, as described here.

We used the following script to fix the errors.

  1. $rootWeb = Get-SPWeb http://YourProjectServer/PWA
  2. $siteUsers = $rootWeb.SiteUsers
  3.  
  4.  
  5. # disable event firing to prevent cancelling updates by PreventEdits method (Microsoft.Office.Project.PWA.ManagedModeListItemEventHandler)
  6. # http://sharepoint.stackexchange.com/questions/37614/disableeventfiring-using-powershell
  7. $receiver = New-Object "Microsoft.SharePoint.SPEventReceiverBase"
  8. $type = $receiver.GetType()
  9. [System.Reflection.BindingFlags]$flags = [System.Reflection.BindingFlags]::Instance -bor [System.Reflection.BindingFlags]::NonPublic
  10. $method = $type.GetMethod("DisableEventFiring", $flags)
  11. $method.Invoke($receiver, $null)
  12.  
  13.  
  14. $rootWeb.Webs | ? { $_.Title -eq 'YourProjectName' } | % {
  15.  
  16. $web = $_
  17.  
  18. Write-Host ——————————-
  19. Write-Host $web.Title
  20.  
  21. $userPairs = ((122, 3421), (145, 2701))
  22.  
  23. $userPairsResolved = $userPairs | Select-Object -Property `
  24.   @{ Name="OldUserId"; Expression = { $_[0] }},
  25.   @{ Name="NewUser"; Expression = { $up = $_; $siteUsers | ? { $_.ID -eq $up[1] } }}
  26.  
  27. $list = $web.Lists["Tasks"]
  28.  
  29. if ($list -ne $null)
  30. {
  31.     $list.Items | % { $list.Items | % {
  32.         $item = $_
  33.         [Microsoft.SharePoint.SPFieldUserValueCollection]$assignedTo = $item["AssignedTo"]
  34.         if ($assignedTo -ne $null)
  35.         {
  36.             $isModified = $false
  37.  
  38.             # iterate through the assignments
  39.             for($i = 0; $i -lt $assignedTo.Count; $i++)
  40.             {
  41.                 if ($assignedTo[$i].User -eq $null)
  42.                 {
  43.                     $userName = $assignedTo[$i].LookupValue
  44.                     $userid = $assignedTo[$i].LookupId
  45.                     $taskTitle = $item.Title.Trim()
  46.                     Write-Host Task """$taskTitle""" assigned user """$userName""" "($userId)" missing
  47.                     $newUser = $userPairsResolved | ? { $_.OldUserId -eq $userid } | % { $_.NewUser }
  48.                     if ($newUser -ne $null)
  49.                     {
  50.                         $newUserId = $newUser.Id
  51.                         $newUserName = $newUser.Name
  52.                         do { $replaceAssignedTo = Read-Host Would you like to replace the assignment of the missing user with """$newUserName""" "($newUserId)"? "(y/n)" }
  53.                         until ("y","n" -contains $replaceAssignedTo )
  54.  
  55.                         if ($replaceAssignedTo -eq "y")
  56.                         {
  57.                             # step 1: removing the orphaned entry
  58.                             $assignedTo.RemoveAt($i)
  59.  
  60.                             # step 2: create the replacement
  61.                             [Microsoft.SharePoint.SPFieldUserValue]$newUserFieldValue = New-Object Microsoft.SharePoint.SPFieldUserValue($web, $newUser.Id, $newUser.Name)     
  62.                             $assignedTo.Add($newUserFieldValue)
  63.  
  64.                             # set the 'modified' flag
  65.                             $isModified = $true
  66.                         }
  67.                     }
  68.                     else
  69.                     {
  70.                         Write-Host WARNING No user found to replace the missing user with -ForegroundColor Yellow
  71.                     }
  72.                       }
  73.             }
  74.  
  75.             # update only if it has been changed
  76.             if ($isModified)
  77.             {
  78.             $item["AssignedTo"] = $assignedTo
  79.             $item.Update()
  80.             Write-Host Task updated
  81.             }
  82.         }
  83.     }}
  84. }
  85.  
  86. }
  87.  
  88. # re-enabling event fireing
  89. $method = $type.GetMethod("EnableEventFiring", $flags)
  90. $method.Invoke($receiver, $null)

The variable $userPairs contains the array of old user IDnew user ID mappings. In step 1 we remove the orphaned user entry (the one referring the deleted user), in step 2 we add the entry for the recreated user. If you plan to run the synchronization (for example, by publishing the project) after the script, step 2 is not necessary, as the synchronization process inserts the references for the users missing from the value collection.

Note 1: The script runs only on the selected project (in this case “YourProjectName”), to minimize the chance to change another project unintentionally.

Note 2: The script informs a user about the changes it would perform, like to replace a reference to a missing user to another one, and waits a confirmation (pressing the ‘y’ key) for the action on behalf on the user executes the script. If you have a lot of entries to change, and you are sure to replace the right entries, you can remove this confirmation and make the script to finish faster.

April 20, 2016

Reusing PSI Proxy Objects from PowerShell

Filed under: PowerShell, PS 2013, PSI — Tags: , , — Peter Holpar @ 22:06

Assume you create a PowerShell script that invokes PSI to perform some actions on Project Server. For example, creating a custom field as described in my former post. You save the script as a .ps1 file and invoke it from the PowerShell shell. Assume it has some parameters and your goal is to invoke it multiple times with various parameter sets. On of the first step in the script is of course the creation in the PSI proxy object, as shown in the original version:

$pwaUrl = "http://YourProjectServer/pwa&quot;
$svcPath = "/_vti_bin/psi/CustomFields.asmx?wsdl"

$svcPSProxy = New-WebServiceProxy -Namespace PSIProxy -Uri ($pwaUrl + $svcPath) -UseDefaultCredential

later in your code you invoke a method on the proxy object:

$svcPSProxy.CreateCustomFields($customFieldDataSet, $false, $true)

On the first run of the script it performs the actions without error, however on the next (and on each later) run it gives you an exception like this:

Cannot convert argument "cfds", with value: "PSIProxy.CustomFieldDataSet", for
"CreateCustomFields" to type "PSIProxy.CustomFieldDataSet": "Cannot convert
the "PSIProxy.CustomFieldDataSet" value of type "PSIProxy.CustomFieldDataSet"
to type "PSIProxy.CustomFieldDataSet"."
At line:1 char:1
+ $svcPSProxy.CreateCustomFields($customFieldDataSet, $false, $true)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodException
    + FullyQualifiedErrorId : MethodArgumentConversionInvalidCastArgument

Do you see the strange error message?

Cannot convert the "PSIProxy.CustomFieldDataSet" value of type "PSIProxy.CustomFieldDataSet" to type "PSIProxy.CustomFieldDataSet".

An object having a specific type cannot be converted to a type having the same type name. Very weird. PowerShell seems to cache the object types created dynamically by the New-WebServiceProxy cmdlet on the first run, and these types seem to be not compatible (at least, in the .NET-sense) with the ones created on the next runs. The single (or at least the most simple) solution seems to be to restart the shell after each run, but it is not very nice, to say the least.

Fortunately, I’ve found a better way in this thread for the “recycling” of the proxy object created on the first execution. Note, that the solution I find there is not the accepted answer as I wrote this post. See the answer from existenz7 on February 07, 2013 1:08 PM.

So I’ve changed the proxy creation part in my script to the form:

If ($global:svcPSProxy -eq $null)
{
  Write-Host "Connecting PSI proxy at $pwaUrl …"
  $global:svcPSProxy = New-WebServiceProxy -Namespace PSIProxy -Uri ($pwaUrl + $svcPath) -UseDefaultCredential
}
Else
{
  Write-Host "Reusing existing PSI proxy"
}

You can invoke the proxy method just like earlier:

$svcPSProxy.CreateCustomFields($customFieldDataSet, $false, $true)

Note: I typically omit this kind of proxy creation from my code posted here on the blog just not to disturb you with details that are not relevant to the problem discussed actually in the post. However, I suggest you to apply the same technique to avoid the type incompatibility issue mentioned above.

Handling PSI Errors in PowerShell Scripts

Filed under: PowerShell, PS 2013, PSI — Tags: , , — Peter Holpar @ 22:03

Recently I work pretty much with PSI calls from PowerShell, to automate such administrative tasks, that are not available in the Project Server Client Object Model, like setting the rules of graphical indicators for the custom fields (see full code here). When using such code, sooner or later you receive an PSI exception, either due to lack of data, an invalid data, or simply because the entity you are working with are not checked out to you.

For example, if you call the CreateCustomFields as shown in the former post:

$svcPSProxy.CreateCustomFields($customFieldDataSet, $false, $true)

you may receive a PSI error that is displayed by default so in PowerShell:

Exception calling "CreateCustomFields" with "3" argument(s):
"ProjectServerError(s) LastError=CustomFieldLowerOrderBitsOutOfRange
Instructions: Pass this into PSClientError constructor to access all error
information"
At line:1 char:1
+ $svcPSProxy.CreateCustomFields($customFieldDataSet, $false, $true)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : SoapException

To display the details of the error, I’ve implemented the error handling similar to the one you find in several PSI-related C# article on MSDN (see this one, for example):

Try {
    $svcPSProxy.CreateCustomFields($customFieldDataSet, $false, $true)
}
Catch [System.Web.Services.Protocols.SoapException] {
    $ex = $_.Exception
    $clientError = New-Object Microsoft.Office.Project.Server.Library.PSClientError($ex)   
    $psiErrors = $clientError.GetAllErrors()
    $psiErrors | % {
        $err = $_
        Write-Host ([int]$err.ErrId) $err.ErrName
        For($i = 0; $i -lt $err.ErrorAttributes.Length; $i++) {
            Write-Host $err.ErrorAttributeNames()[$i] $err.ErrorAttributes[$i];
        }
    }
}

In my case, the specific error information is displayed as error code – error name pairs, like:

11521 CustomFieldMaskDoesNotMatchEntityType
11522 CustomFieldLowerOrderBitsOutOfRange

Note 1: I typically omit this kind of error handling from my code posted here on the blog just not to disturb you with details that are not relevant to the problem discussed actually in the post. It may be OK for you to run the code as it is posted as long as there is no error, however if an exception is thrown, it is good to know how to access the details of the problem. Alternatively, you can use Fiddler to capture the server response, and check the raw XML for the error information.

Note 2: A comprehensive list of PSI error codes is available here, including a short description for each error type. See the Microsoft.Office.Project.Server.Library.PSErrorID enumeration as well.

April 18, 2016

“Cannot find an SPSite object” Error From PowerShell

Filed under: PowerShell, SP 2013 — Tags: , — Peter Holpar @ 22:41

Last week I learned something new about a well known error message. We’ve tried to access a SharePoint site collection / site via the Get-SPSite / Get-SPWeb cmdlets in a test environment, but got the errors:

Get-SPWeb : Cannot find an SPSite object that contains the following Id or Url: http://YourSharePoint

and

Get-SPSite : Cannot find an SPSite object that contains the following Id or Url: http://YourSharePoint

In my practice this means either you mistyped the URL (what the message really suggests), using the wrong protocol (HTTP instead of  HTTPS or the opposite one), or your user has not the required permission (like membership in the the SharePoint_Shell_Access role, you can add via the Add-SPShellAdmin cmdlet) in the content database. In this case the URL was right, and I have db_owner permissions on the DBs, so I had to look for another reason.

Calling the Get-SPWebApplication cmdlet returned the URL of  every web applications in the farm, however when I tried to list the site collections in any (!) of these web applications, like this:

$wa = Get-SPWebApplication http://YourSharePoint
$wa.Sites

it gave me this error on the screen:

An error occurred while enumerating through a collection: The HTTP service
located at
http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc is
unavailable.  This could be because the service is too busy or because no
endpoint was found listening at the specified address. Please ensure that the
address is correct and try accessing the service again later..

Similarly, by checking the stack trace (via the value of  $StackTrace) after invoking the Get-SPSite / Get-SPWeb cmdlets, I received an other evidence of the problem with the communication with token service:

   at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.Http
ChannelRequest.WaitForReply(TimeSpan timeout)

The corresponding details from ULS logs:

Entering BeginProcessing Method of Get-SPSite.
Leaving BeginProcessing Method of Get-SPSite.
Entering ProcessRecord Method of Get-SPSite.
SecurityTokenServiceSendRequest: RemoteAddress: ‘http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc&#8217; Channel: ‘Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustChannelContract’ Action: ‘http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue&#8217; MessageId: ‘urn:uuid:b7dfe754-fdd4-4177-bec6-f8992a097a6b’
SPSecurityContext: Request for security token failed with exception: System.ServiceModel.ServerTooBusyException: The HTTP service located at http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc is unavailable.  This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. —> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.     at System.Net.HttpWebRequest.GetResponse()     at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)     — End of inner exception stack trace —    Server stack trace:      at System.ServiceModel.Chann…
…els.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)     at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)     at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]:      at System.Runtime.Remoting….
…Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)     at Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustContract.Issue(Message message)     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr)     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst)     at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo, SPRequestSecurityTokenProperties properties)
An exception occurred when trying to issue security token: The HTTP service located at http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc is unavailable.  This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later..
Microsoft.SharePoint.PowerShell.SPCmdletPipeBindException: Cannot find an SPSite object with Id or Url: http://YourSharePoint. —> System.ServiceModel.ServerTooBusyException: The HTTP service located at http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc is unavailable.  This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. —> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.     at System.Net.HttpWebRequest.GetResponse()     at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)     — End of inner exception stack…
… trace —    Server stack trace:      at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)     at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)     at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage mess…
…age)    Exception rethrown at [0]:      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)     at Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustContract.Issue(Message message)     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr)     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst)     at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo, SPRequestSecurityTokenProperties properties)     at Microsoft.SharePoi…
…nt.SPSecurityContext.SecurityTokenForLegacyLoginContext(Uri context)     at Microsoft.SharePoint.SPSite.InitUserToken(SPRequest request)     at Microsoft.SharePoint.SPSite.SPSiteConstructor(SPFarm farm, Guid applicationId, Guid contentDatabaseId, Guid siteId, Guid siteSubscriptionId, SPUrlZone zone, Uri requestUri, String serverRelativeUrl, Boolean hostHeaderIsSiteName, SPUserToken userToken, Boolean appWebRequest, String appHostHeaderRedirectDomain, String appSiteDomainPrefix, String subscriptionName, String appSiteDomainId, Uri primaryUri)     at Microsoft.SharePoint.SPSite..ctor(SPFarm farm, Uri requestUri, Boolean contextSite, Boolean swapSchemeForPathBasedSites, SPUserToken userToken)     at Microsoft.SharePoint.SPSite..ctor(SPFarm farm, Uri requestUri, Boolean contextSite, SPUserToke…
…n userToken)     at Microsoft.SharePoint.SPSite..ctor(String requestUrl)     at Microsoft.SharePoint.PowerShell.SPSitePipeBind.Read(Boolean exactUrl)     — End of inner exception stack trace —     at Microsoft.SharePoint.PowerShell.SPSitePipeBind.Read(Boolean exactUrl)     at Microsoft.SharePoint.PowerShell.SPCmdletGetSite.InternalValidate()     at Microsoft.SharePoint.PowerShell.SPCmdlet.ProcessRecord()
Error Category: InvalidData    Target Object  Microsoft.SharePoint.PowerShell.SPCmdletGetSite  Details  NULL  RecommendedAction NULL
Leaving ProcessRecord Method of Get-SPSite.
Entering EndProcessing Method of Get-SPSite.
Leaving EndProcessing Method of Get-SPSite.

Checking the web server I found that the application pool “SecurityTokenServiceApplicationPool” was accidentally stopped, causing issues with the authentication. Restarting it solved my problem.

After solving the issue I played further with the token service, and stopped the web site for "SharePoint Web Services" in IIS, just to learn how the error messages differ in this case. Here is what I found.

The message for the failure of the the Get-SPSite / Get-SPWeb cmdlets is the same as the original case, however the stack trace ($StackTrace) is an other one:

   at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecut
eEnumerate(Object input, Hashtable errorResults, Boolean enumerate)
   at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boo
lean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] p
ipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext fun
cContext)
   at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(Inte
rpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.
Run(InterpretedFrame frame)

Trying to enumerate the site collection in the web application (see above) results in this error:

An error occurred while enumerating through a collection: There was no
endpoint listening at
http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc that
could accept the message. This is often caused by an incorrect address or SOAP
action. See InnerException, if present, for more details..
At line:1 char:1
+ $wa.Sites
+ ~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Microsoft.Share…rePoint.SPS
   ite]:SPEnumerator`1) [], RuntimeException
    + FullyQualifiedErrorId : BadEnumeration

And finally the ULS trace for this scenario (as you can see this time for Get-SPWeb instead of Get-SPSite) :

Entering BeginProcessing Method of Get-SPWeb.
Leaving BeginProcessing Method of Get-SPWeb.
Entering ProcessRecord Method of Get-SPWeb.
SecurityTokenServiceSendRequest: RemoteAddress: ‘http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc&#8217; Channel: ‘Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustChannelContract’ Action: ‘http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue&#8217; MessageId: ‘urn:uuid:bc713eb8-c1f1-453c-b333-86ef8e4411a5’
SPSecurityContext: Request for security token failed with exception: System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:32843     at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)     at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address,…
… ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)     — End of inner exception stack trace —     at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)     at System.Net.HttpWebRequest.GetRequestStream()     at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()     — End of inner exception stack trace —    Server stack trace:      at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()     at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)     at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) …
…    at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]:      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)     at Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustContract.Issue(Message message)     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, Re…
…questSecurityTokenResponse& rstr)     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst)     at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo, SPRequestSecurityTokenProperties properties)
An exception occurred when trying to issue security token: There was no endpoint listening at http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details..
Microsoft.SharePoint.PowerShell.SPCmdletPipeBindException: Cannot find an SPSite object that contains the following Id or Url: http://YourSharePoint. —> System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. —> System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:32843     at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)     at System.Net.ServicePoint.Conne…
…ctSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)     — End of inner exception stack trace —     at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)     at System.Net.HttpWebRequest.GetRequestStream()     at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()     — End of inner exception stack trace —    Server stack trace:      at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream()     at System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)     at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeo…
…ut)     at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]:      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)     at Microsoft.IdentityModel.Protocols.WSTrust.IWSTrustContract.Issue(Message message) …
…    at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst, RequestSecurityTokenResponse& rstr)     at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustChannel.Issue(RequestSecurityToken rst)     at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForContext(Uri context, Boolean bearerToken, SecurityToken onBehalfOf, SecurityToken actAs, SecurityToken delegateTo, SPRequestSecurityTokenProperties properties)     at Microsoft.SharePoint.SPSecurityContext.SecurityTokenForLegacyLoginContext(Uri context)     at Microsoft.SharePoint.SPSite.InitUserToken(SPRequest request)     at Microsoft.SharePoint.SPSite.SPSiteConstructor(SPFarm farm, Guid applicationId, Guid contentDatabaseId, Guid siteId, Guid siteSubscriptionId, SPUrlZone zone, Uri requestUri, String s…
…erverRelativeUrl, Boolean hostHeaderIsSiteName, SPUserToken userToken, Boolean appWebRequest, String appHostHeaderRedirectDomain, String appSiteDomainPrefix, String subscriptionName, String appSiteDomainId, Uri primaryUri)     at Microsoft.SharePoint.SPSite..ctor(SPFarm farm, Uri requestUri, Boolean contextSite, Boolean swapSchemeForPathBasedSites, SPUserToken userToken)     at Microsoft.SharePoint.SPSite..ctor(SPFarm farm, Uri requestUri, Boolean contextSite, SPUserToken userToken)     at Microsoft.SharePoint.SPSite..ctor(String requestUrl)     at Microsoft.SharePoint.PowerShell.SPSitePipeBind.Read(Boolean exactUrl)     — End of inner exception stack trace —     at Microsoft.SharePoint.PowerShell.SPSitePipeBind.Read(Boolean exactUrl)     at Microsoft.SharePoint.PowerShell.SPCmdletGetW…
…eb.InternalValidate()     at Microsoft.SharePoint.PowerShell.SPCmdlet.ProcessRecord()
Error Category: InvalidData    Target Object  Microsoft.SharePoint.PowerShell.SPCmdletGetWeb  Details  NULL  RecommendedAction NULL
Leaving ProcessRecord Method of Get-SPWeb.
Entering EndProcessing Method of Get-SPWeb.
Leaving EndProcessing Method of Get-SPWeb.

Hopefully these details help someone to solve the issue easier and faster when it comes.

Re-creating the Missing Out-of-the-Box Search Locations using PowerShell

Filed under: PowerShell, Reflection, Search, SP 2013 — Tags: , , , — Peter Holpar @ 22:40

Recently we migrated a SharePoint 2010 web application into an existing SharePoint 2013 farm to free up the old SharePoint 2010 farm for “recycling” (it was the last application running in the old farm). The web application has a single site collection, and contains a single business application with views and list forms having a lot of customizations via “jQuery magic”. Since a replacement of the business application is planed for the near future, we decided not to upgrade the site collection to the SharePoint 2013 user interface (version 15). Leaving it in the SharePoint 2010 mode (version 14) ensures the views and forms are working further without any modifications in the JavaScript codes. After a few days a user complained, that when searching the local web site an error is displayed instead of the search results on the _layouts/OSSSearchResults.aspx page:

Unable to display this Web Part. To troubleshoot the problem, open this Web page in a Microsoft SharePoint Foundation-compatible HTML editor such as Microsoft SharePoint Designer. If the problem persists, contact your Web server administrator

In the ULS logs we found these entries:

CoreResultsWebpart: Couldnt find location with internal name LocalSearchIndex
CoreResultsDatasourceView: Couldnt find location with internal name LocalSearchIndex

On the web we found a post from Sushant Dukhande with the description of the issue, and a suggestion for the solution.

Using the script on that site it turned out, that the LocationConfigurations property of the search proxy is really empty. In an other environment, where we tested the migration we had no such issue.

Sushant Dukhande suggests to re-provision the search application. It might really solve the problem, however in our case I felt it to be an intense change, and searched for an alternative solution. Having a look into what happens under the cover of a provisioning process, I found the method responsible for provisioning the missing search locations. It is the internal static CreateOOBLocations method of the Microsoft.Office.Server.Search.Administration.LocationFactory class (in the Microsoft.Office.Server.Search assembly).

First, we need a reference to the search service application. You can get it like this (assuming it is named "Search Service Application"):

$ssa = Get-SPEnterpriseSearchServiceApplication "Search Service Application"

or via this script (as long as you are sure, you have a single instance of this service application type in your farm):

[Microsoft.Office.Server.Search.Administration.SearchServiceApplication]$ssa = Get-SPServiceApplication | ? { $_.TypeName -eq "Search Service Application" }

To display the names of the existing search locations:

$locConfigs = $ssa.LocationConfigurations
$locConfigs | % { $_.InternalName }

The following PowerShell script shows how to invoke the CreateOOBLocations method passing the search service application as parameter using PowerShell and Reflection:

$searchAssembly = [Microsoft.Office.Server.Search.Administration.SearchServiceApplication].Assembly
$locationFactory_Type = $searchAssembly.GetType("Microsoft.Office.Server.Search.Administration.LocationFactory")

$bindingFlags = [Reflection.BindingFlags]::NonPublic -bor [Reflection.BindingFlags]::Static
$mi_CreateOOBLocations = $locationFactory_Type.GetMethod("CreateOOBLocations", $bindingFlags)
$mi_CreateOOBLocations.Invoke($null, @([Microsoft.Office.Server.Search.Administration.SearchServiceApplication]$ssa))

Invoking the CreateOOBLocations method might be not always the solution for you. The same is true for the re-provisioning process suggested by Sushant Dukhande, since it invokes the same method as well. The problem, that this method has a condition, before provisioning all of the default search locations:

if (searchApp.LocationConfigurations.Count < 1)
{
    LocationConfigurationCollection locationConfigurationsInternal = searchApp.GetLocationConfigurationsInternal(true);
    CreateLiveLocation(locationConfigurationsInternal);
    CreateLiveSuggestionsLocation(locationConfigurationsInternal);
    CreateLocalSharepointLocation(locationConfigurationsInternal);
    CreateLocalPeopleLocation(locationConfigurationsInternal);
    CreateLocalFSSharePointLocation(locationConfigurationsInternal);
}

I don’t see, how our farm “lost” its search locations, but if it is possible to “lose” only a subset of the search locations (for example, only the one called LocalSearchIndex), it won’t be re-created by the CreateOOBLocations method, as the count of search location is still not zero.

In this case, the solution may be to re-create only the missing search location via the corresponding method. In the case of the LocalSearchIndex search location it is the CreateLocalSharepointLocation method of the LocationFactory class:

$locConfigs = $ssa.LocationConfigurations
$mi_CreateLocalSharepointLocation = $locationFactory_Type.GetMethod("CreateLocalSharepointLocation", $bindingFlags)
$mi_CreateLocalSharepointLocation.Invoke($null, @([Microsoft.Office.Server.Search.Administration.LocationConfigurationCollection]$locConfigs))

After fixing the issue in the farm, I’ve tested our other farms as well to find out, whether they are affected by the same problem or not. In one of the farm, the script provided in the post I mentioned earlier detected the issue, although I was sure, there is no problem with the search. It turned out to be a false positive test. This farm has its search service as a shared service from another farm, and the user account the script was run with had no permission on the search service in that remote farm. The script simply hid away the access denied error.

However, if we create a LocationConfigurationCollection instance via its internal constructor (either with a parameter of type SearchServiceApplication or of type SearchServiceApplicationProxy), the access denied error is displayed in the case the user has no permissions, and the items of the collection can be accessed if there is no problem with the permissions.

Let’s see first the script using the SearchServiceApplication:

$bindingFlags = [Reflection.BindingFlags]::NonPublic -bor [Reflection.BindingFlags]::Instance
$ci_LocationConfigurationCollection = [Microsoft.Office.Server.Search.Administration.LocationConfigurationCollection].GetConstructor($bindingFlags, $null, @([Microsoft.Office.Server.Search.Administration.SearchServiceApplication]), $null)
$locConfigs = $ci_LocationConfigurationCollection.Invoke(@([Microsoft.Office.Server.Search.Administration.SearchServiceApplication]$ssa))
$locConfigs | % { $_.InternalName }

As I wrote, you can achieve the same via a service proxy. It is useful for example, if the application is connected to a shared search service of another farm. First, we get the proxy as:

$url = "http://YourSharePointApp/&quot;
$site = Get-SPSite $url
$serviceContext = [Microsoft.SharePoint.SPServiceContext]::GetContext($site)
$ssaAppProxy = $serviceContext.GetDefaultProxy([Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy])

Next, we can use the same script as earlier, but in this case we invoke the internal constructor having the SearchServiceApplicationProxy parameter type:

$bindingFlags = [Reflection.BindingFlags]::NonPublic -bor [Reflection.BindingFlags]::Instance
$ci2_LocationConfigurationCollection = [Microsoft.Office.Server.Search.Administration.LocationConfigurationCollection].GetConstructor($bindingFlags, $null, @([Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy]), $null)
$locConfigs = $ci2_LocationConfigurationCollection.Invoke(@([Microsoft.Office.Server.Search.Administration.SearchServiceApplicationProxy]$ssaAppProxy))
$locConfigs | % { $_.InternalName }

February 27, 2016

Test On Demand PDF Conversion without Document Libraries via PowerShell

Filed under: PowerShell, SP 2013, Word Automation Services — Tags: , , — Peter Holpar @ 15:52

If you had to convert Word documents to PDFs using the Word Automation Services in SharePoint 2010, you had to store the files at least temporary in document libraries. That was true, even if you have created the documents dynamically, for example based on SharePoint list item data using the Open XML SDK and sent as response to a web request (as illustrated in this example). Typically you could solve this requirement using streams or byte arrays otherwise, but the ConversionJob class supported in SharePoint 2010 only working with files stored in document libraries via the AddFile method. A further limitation of that only asynchronous conversation was supported, if you needed a synchronous operation, you should have to find workaround, like this one.

In SharePoint 2013 we have the SyncConverter class, that – as its name already suggests – provides the synchronous conversion as well stream and byte array support via its Convert method.

Recently we are working on a document conversation solution again (similar to the SharePoint 2010 example I referred to above), but we did not want to deploy the SharePoint solution each time we change the logic in the Word document creation, but still wanted to have the ability to test:

  • If the WAS is able to convert the word document to PDF.
  • Does the output of the conversation fulfill our expectations.
  • How much time the conversion takes.

So we called the appropriate methods of our Word document creation assembly from PoweShell, and stored the result in a byte array. This step is specific to the current customer project and as such out of the scope of this post. Instead of this we simulate the step simply by reading the static content of a .docx file in the script below. Next we got a reference to the Word Automation Services Proxy, and converted the Word document from the byte array into another byte array, holding the content of the PDF conversion. In the original SharePoint solution this byte array would be sent back in the web server response to the client. In the script we simply save it into a file into the file system. If you run this script, don’t forget, that if the target file already exists, it is overwritten without any confirmation request!

  1. $url = "http://YourSharePointSite&quot;
  2. $inputFilePath = "C:\Data\input.docx"
  3. $outputFilePath = "C:\Data\output.pdf"
  4.  
  5. [byte[]]$inputContentBytes = Get-Content $inputFilePath -Encoding Byte
  6. [byte[]]$outputContentBytes = $null
  7.  
  8. # get the WAS proxy assigned to the site
  9. $site = Get-SPSite $url
  10. $serviceContext = [Microsoft.SharePoint.SPServiceContext]::GetContext($site)
  11. $wasAppProxy = $serviceContext.GetDefaultProxy([Microsoft.Office.Word.Server.Service.WordServiceApplicationProxy])
  12. # or if you have a single WAS proxy in your farm you can use this as well:
  13. #$wasAppProxy = Get-SPServiceApplicationProxy | ? { $_.TypeName -eq "Word Automation Services Proxy" }
  14.  
  15. [void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Word.Server")
  16. $job = New-Object Microsoft.Office.Word.Server.Conversions.SyncConverter($wasAppProxy)
  17. $job.Settings.OutputFormat = "PDF"
  18. $job.Settings.UpdateFields = $true
  19. $job.UserToken = ($site.RootWeb).CurrentUser.UserToken
  20.  
  21. $jobInfo = $job.Convert($inputContentBytes, [ref] $outputContentBytes)
  22. [System.IO.File]::WriteAllBytes($outputFilePath, $outputContentBytes)
  23.  
  24. If ($jobInfo.Succeeded)
  25. {
  26.   $duration = ($jobInfo.CompleteTime $jobInfo.StartTime).TotalSeconds
  27.   Write-Host Job completed in $duration seconds
  28. }
  29. Else
  30. {
  31.   Write-Host Job completed with error $jobInfo.ErrorCode
  32. }

You should have the Word Automation Services infrastructure (I mean the service application and the related services) up and running. I was sure, that it is OK in our case, but wanted to keep the script simple, so I did not check these things in the script above. If you are not sure that your farm was set up correctly, you should have a look at this script and extend my version with the test steps if you wish.

February 15, 2016

How to Set a SharePoint Document Library to Read-Only Mode

Filed under: PowerShell, Security, SP 2013 — Tags: , , — Peter Holpar @ 23:49

Recently we had to archive a few document libraries on our SharePoint farm. The document libraries may have they own permission setting on the library level, and / or at the folder, and / or the document levels either. We do not have any custom permission level defined in our sites.

We defined “archive” as this: all users that have read / write permission to an item, should keep the access on the item, but it has to be restricted to read permission in the future. The users having permission to the documents should be able to download them, and work on them locally (if they wish) but are not allowed to save them back to the library.

I wrote a PowerShell script that processes the permissions, and replaces write permissions with read permissions on demand.

Note: the solution implemented in this post is a one-way street. It’s not a read-only switch you can turn on or off as you can do for example in the case of a site collection (like Set-SPSite http://YourSharePointSite -LockState ReadOnly). You won’t be able to reproduce the original permissions once you run the script.

  1. $url = "http://YourSharePointSite/SubSite&quot;
  2. $docLibName = "Documents"
  3.  
  4. $web = Get-SPWeb $url
  5. $docLib = $web.Lists[$docLibName]
  6.  
  7. $limitedAccess = $web.RoleDefinitions.GetByType([Microsoft.SharePoint.SPRoleType]::Guest)
  8. $readAccess = $web.RoleDefinitions.GetByType([Microsoft.SharePoint.SPRoleType]::Reader)
  9.  
  10. $allowedAccess = $limitedAccess.Name, $readAccess.Name
  11.  
  12. function Replace-Permissions($securable)  
  13. {
  14.   $securable.RoleAssignments | ? { $_.RoleDefinitionBindings | ? { $allowedAccess -notcontains $_.Name }} | % {
  15.     $_.RoleDefinitionBindings.RemoveAll()
  16.     $_.RoleDefinitionBindings.Add($readAccess)
  17.     $_.Update()
  18.   }
  19. }
  20.  
  21. # if the doc. lib. inherits the permissions and if there is any role assignments that contains an access level beyond read only
  22. # we should break the inheritance first, to be able to change the permissions on the library level
  23. If (($doclib.HasUniqueRoleAssignments) -and (($docLib.RoleAssignments | ? { $_.RoleDefinitionBindings | ? { $allowedAccess -notcontains $_.Name }}).Count -gt 0))
  24. {
  25.   $doclib.BreakRoleInheritance($true);
  26. }
  27.  
  28. # set permissions on the doc. lib level
  29. Replace-Permissions($docLib)
  30. # set permissions on all folders having its own role assignment
  31. $doclib.Folders | ? { $_.HasUniqueRoleAssignments } | % { Replace-Permissions $_ }
  32. # set permissions on all documents having its own role assignment
  33. $doclib.Items | ? { $_.HasUniqueRoleAssignments } | % { Replace-Permissions $_ }

In the Replace-Permissions function I replace any permissions  on a securable object other than Read or Limited Access (Guest) permissions with Read permissions.

Note: If you remove a Limited Access permission using the web UI, or the corresponding role assignment from code, you will loose the permissions set explicitly for that user anywhere in the hierarchy below that level. As described here, you can call the RemoveAll method on the RoleDefinitionBindings without such side effects. Then we can add the read permissions in place of the removed permissions.

I invoke the Replace-Permissions function once for the document library, then once for each folder and document having its own unique role assignments.

December 1, 2015

Delete a SharePoint List Template Programmatically

Filed under: PowerShell, SP 2013 — Tags: , — Peter Holpar @ 01:08

Last week I had to “reproduce” a custom list having a rather complex structure from a root web site into a sub web site. I mean, I had to create another list having the same fields and views like the original one, but located in another web site.

I fulfilled the requirements via a PowerShell script by creating a temporary list template (without including content), creating a list based on the template, and finally deleting the list template (as the title of this post promises), since we didn’t want to allow others to create the same list in the webs they have write access to.

Note: There are several alternative ways to copy a list from one site to another, for example via the SharePoint Content Deployment and the Migration API (see samples here, here and here), or via the Export-SPWeb and Import-SPWeb PowerShell Cmdlets. as illustrated in this example. Of course, each of these solutions has its own pros and cons, and it really depends on the exact requirements (do you need to copy the content, or just the structure; do you need to copy security or not; do you need to copy the versions or not, etc.) which way we may or should choose. The comparison of these methods is beyond the scope of the current post, but it is useful to know these alternatives.

But back to the deletion of the list template. The solutions I found on the web after a quick search were IMHO a bit complicated as it should be (see this and this one for example), as they iterate through the list items in the List Template Gallery list to find the file / list item that corresponds to our template.

Based on my experience, we can find the template directly via the indexer of the SPListTemplateCollection by name. It results in solution that is so simple:

$listTemplates = $web.Site.GetCustomListTemplates($web)
$listTemplate = $listTemplates[$listName]
$listTemplateFile = $web.GetFile("_catalogs/lt/" + $listTemplate.InternalName)
$listTemplateFile.Delete()

If you need the full code we used to copy the list, it is here as well:

$url = "http://YourSharePointSite&quot;

$targetWebUrl = "/SubSite"

$listName = "CustomListName"

$site = Get-SPSite $url
$sourceWeb = $site.RootWeb
$targetWeb = Get-SPWeb ($url + $targetWebUrl)

$list = $sourceWeb.Lists[$listName]
# create a temporary list template
# we copy the list without data, so the last param is 0
$list.SaveAsTemplate($listName, $listName, $listName, 0)

$listTemplates = $site.GetCustomListTemplates($sourceWeb)
$listTemplate = $listTemplates[$listName]
# create the list in the target web
# get the list description from the source list
$targetWeb.Lists.Add($listName, $list.Description, $listTemplate)

# delete the temporary list template, we don’t need it anymore
$listTemplateFile = $sourceWeb.GetFile("_catalogs/lt/" + $listTemplate.InternalName)
$listTemplateFile.Delete()

Older Posts »

Create a free website or blog at WordPress.com.