Second Life of a Hungarian SharePoint Geek

March 25, 2017

Microsoft.Workflow.Client.InvalidRequestException: Failed to query the OAuth S2S metadata endpoint – The remote server returned an error: (400) Bad Request

Filed under: Certificates, SP 2013, Workflow — Tags: , , — Peter Holpar @ 21:11

Recently we installed a new Workflow Manager farm (a single-server one) on the front-end server of one of our SharePoint farms.

I wanted to register the Workflow Manager for a web application in the SharePoint farm via the PowerShell cmdlet:

Register-SPWorkflowService -SPSite https://YourSharePointSite -WorkflowHostUri https://YourWorkflowManagerServer:12290 -ScopeName YourScope –Force

But I received an error like this one:

Register-SPWorkflowService : Failed to query the OAuth S2S metadata endpoint
at URI ‘https://YourSharePointSite/_layouts/15/metadata/json/1’.
Error details: ‘An error occurred while sending the request’. HTTP headers received from the server – ActivityId:
d10c4cbb-bde4-4040-b09f-1ace1491dc87. NodeId: YourWFNode. Scope: /YourScope.
Client ActivityId : b89c2ff9-8560-458e-9ea2-31ec6c8fde36.
At line:1 char:1
+ Register-SPWorkflowService -SPSite https://YourSharePointSite/  -W …

In the Event Viewer (Application and Services Logs / Microsoft-Workflow / Operational) we had this error:

image

Failed to query the remote endpoint for the S2S metadata document. Details: System.Net.Http.HttpRequestException: An error occurred while sending the request. —> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. —> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
   at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
   — End of inner exception stack trace —
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   — End of inner exception stack trace —

In the ULS logs we had this error message:

Microsoft.Workflow.Client.InvalidRequestException: Failed to query the OAuth S2S metadata endpoint at URI ‘https://YourSharePointSite/_layouts/15/metadata/json/1’. Error details: ‘An error occurred while sending the request.’. HTTP headers received from the server – ActivityId: d10c4cbb-bde4-4040-b09f-1ace1491dc87. NodeId: YourWFNode. Scope: /YourScope. Client ActivityId : b89c2ff9-8560-458e-9ea2-31ec6c8fde36. —> System.Net.WebException: The remote server returned an error: (400) Bad Request.     at Microsoft.Workflow.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)     at Microsoft.Workflow.Client.HttpGetResponseAsyncResult`1.End(IAsyncResult result)     at Microsoft.Workflow.Client.ClientHelpers.SendRequest[T](HttpWebRequest request, T content)     — End of inner exceptio…

The SharePoint site https://YourSharePointSite and the Workflow Manager endpoint URL https://YourWorkflowManagerServer:12290 were both available without any issue (e.g. no problem with the certificate too), on both nodes (front-end and application servers) of the SharePoint farm, as well as from client computers.

The articles I found about the issue (like this one or this one) explained the problem with the reason, that the SharePoint endpoint URL (in our case ‘https://YourSharePointSite/_layouts/15/metadata/json/1‘) is not accessible, probably because of a name resolution issue. In our case that was definitely not the issue, because if I switched the SharePoint URL from HTTPS to HTTP (via changing the Alternate Access Settings for the site + bindings in IIS manager), I was able to run the registration script successfully:

Register-SPWorkflowService -SPSite http://YourSharePointSite -WorkflowHostUri https://YourWorkflowManagerServer:12290 -ScopeName YourScope –Force -AllowOAuthHttp

After switching back the URL to HTTPS we had the problem again.

My next assumption was, that the service account for the Workflow Manager does not have the root certificate of the SSL certificate under the Trusted Root Certification Authorities.

So I’ve started the Microsoft Management Console (mmc.exe) and added the Certificates snap-in for the service account of the Workflow Manager Backend service:

image

image

image

I found that the list of Trusted Root Certification Authorities contains the root certificate of the SSL, so it could not be a problem either.

As next step, I’ve logged in on the Workflow Manager server (that is the front-end server of the SharePoint farm) the using the Workflow Manager service account to test the connection to the SharePoint site interactively via Internet Explorer. In this case I was faced with the problem, that the SharePoint site https://YourSharePointSite has a certificate warning. As I opened the certificate for the site in Internet Explorer, I saw only the very last entry in the certificate chain (for example, the entry for YourSharePointSite), but none of the certificates above. I’ve found it either, that the account has configured not to use a proxy server. I enabled the proxy connection, then restarted Internet Explorer, and voila no more issues with the certificate. I was able to register the Workflow Manager as well. I don’t exactly know, what was the problem, but I assume, the certificate revocation list was not available without the proxy, and that prohibited the certificate validation necessary for the registration of the Workflow Manager.

Advertisements

March 4, 2017

How to Change the Service Account for the Workflow Manager

Filed under: SP 2013, Workflow — Tags: , — Peter Holpar @ 21:49

A few weeks ago we made a mistake when installing Workflow Manager in a new environment, as we have chosen a wrong account name as the service account for Workflow Manager.

As a first try, we simply changed the identity of the application pool assigned to the Workflow Manager (called WorkflowMgmtPool) in IIS and restarted the pool, but after the change we had an error when accessing the workflow related pages in SharePoint:

Application error when access /_layouts/15/Workflow.aspx, Error=The remote server returned an error: (500) Internal Server Error.   at Microsoft.Workflow.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)     at Microsoft.Workflow.Client.HttpGetResponseAsyncResult`1.End(IAsyncResult result)     at Microsoft.Workflow.Client.ClientHelpers.SendRequest[T](HttpWebRequest request, T content)    9d19d89d-48f7-c052-732f-a59123539aa3
System.Net.WebException: The remote server returned an error: (500) Internal Server Error.    at Microsoft.Workflow.Common.AsyncResult.End[TAsyncResult](IAsyncResult result)     at Microsoft.Workflow.Client.HttpGetResponseAsyncResult`1.End(IAsyncResult result)     at Microsoft.Workflow.Client.ClientHelpers.SendRequest[T](HttpWebRequest request, T content)    9d19d89d-48f7-c052-732f-a59123539aa3

In the Workflow Manager event logs (Event Viewer/Applications and Services Logs/Microsoft-Workflow/Operational) we found this error message:

Error processing management request. Method: GET, RequestUri: https://YourSharePoint:12290/YourScope, Error: System.Security.Cryptography.CryptographicException: Keyset does not exist

   at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
   at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
   at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
   at Microsoft.Workflow.Common.EncryptionHelper.DecryptStringWithCertificate(X509Certificate2 encryptionCertificate, String encryptedText)
   at Microsoft.Workflow.Management.WorkflowEncryptionSettings.InitializeInternal()
   at Microsoft.Workflow.Management.WorkflowServiceConfiguration.get_EncryptionSettings()
   at Microsoft.Workflow.Management.WorkflowServiceConfiguration.GetResourceManagementConnectionStringFromConfig()
   at Microsoft.Workflow.Management.WorkflowServiceConfiguration.get_ConfigProvider()
   at Microsoft.Workflow.Management.WorkflowServiceConfiguration.GetWorkflowServiceConfiguration()
   at Microsoft.Workflow.Gateway.HttpConfigurationInitializer.CreateServiceContext(String nodeId, NamespaceSender namespaceSender)
   at Microsoft.Workflow.Gateway.HttpConfigurationInitializer.EnsureInitialized(String nodeId, NamespaceSender namespaceSender)
   at Microsoft.Workflow.Gateway.HttpConfigurationInitializer.Initialize(HttpConfiguration config, String nodeId, NamespaceSender namespaceSender)
   at Microsoft.Workflow.Gateway.Global.EnsureConfigInitialized(String nodeId)
   at Microsoft.Workflow.Gateway.Global.Application_BeginRequest(Object sender, EventArgs e)

image

It seems the account had no permission to access a certificate or something like this, so we changed back the application pool identity an searched for a better solution.

We found a few useful resources on the web, discussing how the account change should be performed (see here, here and here).

So we run this script from Workflow Manager PowerShell console on our single-node workflow farm:

Stop-SBFarm
Set-SBFarm –RunAsAccount <YourDomain\UserName>
$RunAsPassword = ConvertTo-SecureString -AsPlainText -Force ‘<Password>’
Update-SBHost -RunAsPassword $RunAsPassword
Start-SBFarm

As the result of the script above, the identity of the following Windows services has been changed to the account specified in the script:

  • Service Bus Gateway
  • Service Bus Message Broker
  • Service Bus Resource Provider
  • Service Bus VSS
  • Windows Fabric Host Service

The identity of the Workflow Manager Backend service was not changed, nor the application pool identity of the Workflow Manager in IIS

The script grant the following database roles in the Service Bus databases:

  • Workflow_SB_Container (role granted: ServiceBus.Operators)
  • Workflow_SB_Gateway (roles granted: SBProjectStore.Operators, ServiceBus.Operators)
  • Workflow_SB_Management (role granted: Strore.Operators)

There was however no permission granted on the following workflow-related databases:

  • Workflow_Farm
  • Workflow_Instance
  • Workflow_Resource

As a next step of the identity change (following the suggestion from one of the above referenced forum threads), we changed manually the account of the Workflow Manager Backend service, and restarted it. It caused however further problems, granting permissions for the account on the before mentioned three WF databases (WFServiceOerators role, or db_owner) did not helped either.

The symptoms we faced to were:

  • We were able to start workflow (at least, no error message at this place) from the SharePoint UI, but happened  nothing, we can not stop the workflows from the UI.
  • At the web-endpoint of the Workflow Manager (https://YourSharePoint:12290/YourScope) we had this error message:

<Error xmlns:i="http://www.w3.org/2001/XMLSchema-instance"&gt;
  <Code>UnexpectedError</Code>
  <Message>The data or messaging layer is unavailable. Please retry after 300 seconds.</Message> 
</Error>

In the Event Viewer we had a lot of errors like:

The Workflow Manager cannot contact Service Bus service after retrying for ’28’ minutes. Please verify if the Service Bus service is up and running. The Workflow Manager failed at location ‘ServiceBusNamespaceListener.GetSessionAndStateWithRetryAsyncResult.HandleException’ due to exception: System.UnauthorizedAccessException: 40100: Unauthorized.TrackingId:b006a351-d6bc-4b4e-a178-a4a1d689fee9_GYourSharePoint_GYourSharePoint,TimeStamp:27.02.2017 11:04:31 —> System.ServiceModel.FaultException: 40100: Unauthorized.TrackingId:b006a351-d6bc-4b4e-a178-a4a1d689fee9_GYourSharePoint_GYourSharePoint,TimeStamp:27.02.2017 11:04:31

image

and warnings like:

Service Bus exception swallowed at location ServiceBusNamespaceListener.GetSessionAndStateWithRetryAsyncResult.HandleException. System.UnauthorizedAccessException: 40100: Unauthorized.TrackingId:c0f820e5-bc7f-4186-8d8f-41899f014c84_GYourSharePoint_GYourSharePoint,TimeStamp:27.02.2017 11:05:19 —> System.ServiceModel.FaultException: 40100: Unauthorized.TrackingId:c0f820e5-bc7f-4186-8d8f-41899f014c84_GYourSharePoint_GYourSharePoint,TimeStamp:27.02.2017 11:05:19

image

The few discussions related to similar problems we found on the web (like this one or this one) did not help to much, so we decided to set back the original  account of the Workflow Manager Backend service, and restarted it again. Our workflows are functioning now, but I am really keen to know, how we could change the identity of the Workflow Manager Backend service as well.

March 3, 2017

SharePoint Designer Workflow Gets Suspended after Task Completion – How to Get Field Value from a Workflow Task via Lookup

Filed under: SP 2013, SPD, Workflow — Tags: , , — Peter Holpar @ 06:21

Nowadays we are working quite a lot with SharePoint Designer 2013 based workflows. On workflows I mean the “new”, Workflow Manager based ones.

Recently we wanted to access a workflow task field beyond the standard outcome to use its value in another part of the workflow. For example, we need the value of the Description field, as the explanation of the decision made on the form (rejection vs. approval).

image

To achieve that, we stored the workflow task Id in a variable called TaskID (see above), and planned to use it as a lookup value from the task list (see below). Note, that we used the ID field in the lookup list, Data Source is Assocciation: Task List, that is the standard Worklow Tasks list in our case.

image

The value of the TaskID variable is returned as integer:

image

After publishing the workflow and creating an item to test it, the workflow task was created. We entered some text in the Description field, and approved the task. We found, that the workflow gets stuck in the Suspended status. Resuming it has not helped either.

image

The error description we had:

RequestorId: 3c361109-ce76-de39-0000-000000000000. Details: An unhandled exception occurred during the execution of the workflow instance. Exception details: System.FormatException: Input string was not in a correct format. at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal) at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info) at Microsoft.Activities.Expressions.ParseNumber`1.Execute(CodeActivityContext context) at System.Activities.CodeActivity`1.InternalExecute(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager) at System.Activities.Runtime.ActivityExecutor.ExecuteActivityWorkItem.ExecuteBody(ActivityExecutor executor, BookmarkManager bookmarkManager, Location resultLocation)

The resources we found on the web here, here and there did not help to much, but the error message itself did.

The reason of the error was, that the TaskID (a variable of type String) we have from the Assign a task action is actually the Guid of the task item, but we wanted to use it to look up the task based on its ID field (an Integer). Of curse, the workflow engine was not able to convert the Guid to an integer value.

The correct lookup is illustrated below. We use the GUID field for as the lookup field, and TaskID is returned as a string:

image

image

With this “minor” modification the workflow runs as expected.

After we solved the problem I found that the the original requirement (getting field value from a specific workflow task as data source via lookup) was already discussed and solved earlier, see this thread and this one.

June 2, 2013

"There is already a workflow association with this name" on WorkflowAssociations.Update

Filed under: SP 2010, Workflow — Tags: , — Peter Holpar @ 21:02

Recently I had to enhance an existing workflow solution. When I tried to (re)activate the feature that contains the workflow I received an error that states, that “There is already a workflow association with this name”.

I found that the workflow is associated to a specific content type through code in a feature receiver, using a code similar to this sample code on MSDN.

The code can be outlined with these main steps:

SPWorkflowAssociation workflowAssociation = SPWorkflowAssociation.CreateSiteContentTypeAssociation(workflowTemplate,
                                                                       workflowName,
                                                                       taskListTitle,
                                                                       historyListTitle);
// several workflow association properties are set here
if (siteContentType.WorkflowAssociations.GetAssociationByName(workflowAssociation.Name, site.Locale) == null)
{
    siteContentType.WorkflowAssociations.Add(workflowAssociation);
}
else
{
    siteContentType.WorkflowAssociations.Update(workflowAssociation);
}

First, a new instance of SPWorkflowAssociation is created. Next, if a workflow association with the same name not found, we add the new one, otherwise, we try to update it (the new one?!). I found this logic erroneous. It seems that others had problems with this approach as well.

If we check the getter method of the WorkflowAssociations property in the SPContentType class, we found, that although the return type is SPWorkflowAssociationCollection, its runtime type is in fact a subclass, namely the internal SPContentTypeWorkflowAssociationCollection type.

this.m_wac = new SPContentTypeWorkflowAssociationCollection(this);

Having a look at the Update method of the SPContentTypeWorkflowAssociationCollection type, it turns out that the read-only ParentAssociationId property of the SPWorkflowAssociation in the parameter is compared with the same property of the existing SPWorkflowAssociation instances in the collection to find out which instance should be updated. If no matching instance is found, then instead an update, the Add method is called and the new instance is – at least theoretically – inserted into the collection.

SPWorkflowAssociation waOld = null;
foreach (SPWorkflowAssociation association2 in this)
{
    if (association2.ParentAssociationId == workflowAssociation.ParentAssociationId)
    {
        waOld = association2;
        break;
    }
}
workflowAssociation.ParentCollection = this;
if (waOld == null)
{
    SPWorkflowTemplate templateByBaseID = base.ParentWeb.WorkflowTemplates.GetTemplateByBaseID(workflowAssociation.BaseTemplate.BaseId);
    if (workflowAssociation.ContentTypePushDown && ((!workflowAssociation.IsDeclarative || (templateByBaseID != null)) || workflowAssociation.BaseTemplate.IsRootPublic))
    {
        this.Add(workflowAssociation);
    }
}

However, as there is already another workflow association having the very same name, a conflict occurs and an exception is thrown. See the Add method of the SPContentTypeWorkflowAssociationCollection class:

if (base.GetAssociationByName(workflowAssociation.Name, base.ParentWeb.Locale) != null)
{
       throw new ArgumentException(SPResource.GetString("DuplicateWorkflowNameDetected", new object[] { workflowAssociation.Name }));
}

The correct order of execution would be:

SPWorkflowAssociation workflowAssociation = siteContentType.WorkflowAssociations.GetAssociationByName(workflowAssociation.Name, site.Locale);
if (workflowAssociation  == null)
{
    workflowAssociation = SPWorkflowAssociation.CreateSiteContentTypeAssociation(workflowTemplate,
                                                                       workflowName,
                                                                       taskListTitle,
                                                                       historyListTitle);
    // set workflow association properties as you wish here
    siteContentType.WorkflowAssociations.Add(workflowAssociation);
}
else
{
    // set workflow association properties as you wish here
    siteContentType.WorkflowAssociations.Update(workflowAssociation);
}

A nice sample implementation can be found here. After altering the logic in the feature receiver according to the new logic, the error was eliminated and one was able to reactivate the feature successfully.

Note: In the example above I used the CreateSiteContentTypeAssociation method, however, the same should apply all content type related methods of the SPWorkflowAssociation type, that means CreateWebContentTypeAssociation and CreateListContentTypeAssociation methods as well.

Blog at WordPress.com.