Second Life of a Hungarian SharePoint Geek

February 17, 2016

List View Threshold Causing Project Server Synchronization Failures

Filed under: Project Server — Tags: — Peter Holpar @ 22:01

Yesterday we found a lot of errors in the Project Server queue job history (PWA Settings / Manage Queue Jobs). The Job State was Failed But Not Blocking Correlation, the Job Type was of type Reporting (Project Sync).

image

To understand the reason, you should know that Project Server synchronizes changes in several lists from the project web sites (PWS) to its reporting database. This kind of synchronization is performed by the Project Server Queue Service (Microsoft.Office.Project.Server.Queuing.exe).

By clicking on the Click to view the error details link we became more information about the background of the problem.

Queue Job Error Details
General
Reporting Wss list sync failed
ReportingWssSyncListFailed (24018) – 101. Details: id=’24018′ name=’ReportingWssSyncListFailed’ uid=’5b39b9cd-bad4-e511-8c5c-005056b45654′ SPListType=’0e2d89a0-e57a-e411-9568-005056b45654′ Error=’101′.
Reporting message processor failed
ReportingWSSSyncMessageFailed (24016) – RDS failed while trying to sync one or more SP lists. The RDS queue message will be retried.. Details: id=’24016′ name=’ReportingWSSSyncMessageFailed’ uid=’6239b9cd-bad4-e511-8c5c-005056b45654′ QueueMessageBody=’ProjectUID=’0e2d89a0-e57a-e411-9568-005056b45654′. ForceFullSync=’False’. SynchronizationType=’Documents” Error=’RDS failed while trying to sync one or more SP lists. The RDS queue message will be retried.’.
Queue
GeneralQueueJobFailed (26000) – ReportingWSSSync.WSSSyncMessageEx. Details: id=’26000′ name=’GeneralQueueJobFailed’ uid=’ade7bdd3-bad4-e511-8c5c-005056b45654′ JobUID=’485a0869-b8d4-e511-8c5c-005056b45654′ ComputerName=’d30c825e-996d-4a92-a60d-1d66f5340b1b’ GroupType=’ReportingWSSSync’ MessageType=’WSSSyncMessageEx’ MessageId=’1′ Stage=” CorrelationUID=’0fca5f9d-75e8-507f-40b1-bf88f739c79d’. For more details, check the ULS logs on machine d30c825e-996d-4a92-a60d-1d66f5340b1b for entries with JobUID 485a0869-b8d4-e511-8c5c-005056b45654.

The value 101 in the Error property and the value Documents in the SynchronizationType suggest that it is something wrong with the synchronization of the document library list (101 is the ID of the document library list template).

The relevant part from the ULS logs:

02/16/2016 15:37:35.19     Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Server                 General                           a2tj    Medium      RowLimit unset…using default RowLimit for query: <Where><Geq><FieldRef Name=’Modified’/><Value Type=’DateTime’ IncludeTimeValue=’TRUE’ StorageTZ=’TRUE’>2016-02-16T14:29:19</Value></Geq></Where>    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19     Microsoft.Office.Project.Server (0x136D8)    0x125EC    Project Server                    Project Server Database           ah91z    Medium      Successfully got the connection string (database name=[YourProjectServerDB], id=ae38d4f0-3d16-430b-8cec-c76290a640df, type=Consolidated). Requested access level=ReadWrite: Data Source=YourProjectServerServiceApp;Initial Catalog=YourProjectServerDB;Integrated Security=True;Enlist=False;Pooling=True;Min Pool Size=0;Max Pool Size=100;Connect Timeout=15    
02/16/2016 15:37:35.19     Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             Health                            46ri    High        Throttled:Big list slow query. List item query elapsed time: 0 milliseconds, Additional data (if available): Query HRESULT: 80070024 List internal name, flags, and URL: {7903E344-47BC-42B5-A1BF-14E3E58D817F}, flags=0x0080000000001008, URL="
http://YourProjectServer/PWA/Proj1&quot; Current User: 1073741823 Query XML: "<Query><Where><Geq><FieldRef Name="Modified"/><Value Type="DateTime" IncludeTimeValue="TRUE" StorageTZ="TRUE">2016-02-16T14:29:19</Value></Geq></Where></Query>" SQL Query: "N/A"     0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19     Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             General                           xxpm    High        Unable to execute query: Error 0x80070024    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19     Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             General                           8kh7    High        The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.<nativehr>0x80070024</nativehr><nativestack></nativestack>    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19     Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             General                           aix9j    High        SPRequest.GetListItemDataWithCallback2: UserPrincipalName=i:0).w|s-1-5-21-3634847118-1559816030-2180994487-3302, AppPrincipalName= ,pSqlClient=<null> ,bstrUrl=
http://YourProjectServer/PWA/Proj1 ,bstrListName={7903E344-47BC-42B5-A1BF-14E3E58D817F} ,bstrViewName=<null> ,bstrViewXml=<View Scope=’RecursiveAll’><Query><Where><Geq><FieldRef Name=’Modified’/><Value Type=’DateTime’ IncludeTimeValue=’TRUE’ StorageTZ=’TRUE’>2016-02-16T14:29:19</Value></Geq></Where></Query><ViewFields><FieldRef Name=’ID’/><FieldRef Name=’UniqueId’/><FieldRef Name=’FileRef’/><FieldRef Name=’FSObjType’/> ,fSafeArrayFlags=SAFEARRAYFLAG_NONE    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19     Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             General                           ai1wu    Medium      System.Runtime.InteropServices.COMException: The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.<nativehr>0x80070024</nativehr><nativestack></nativestack>, StackTrace:    at Microsoft.SharePoint.SPListItemCollection.EnsureListItemsData()     at Microsoft.SharePoint.SPListItemCollection.get_Count()     at Microsoft.Office.Server.Utilities.ContentIterator.ProcessListItems(SPList list, SPQuery query, ItemsProcessor itemsProcessor, ItemsProcessorErrorCallout errorCallout)     at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.WSSDataTransfer.BuildIncrementalTransfer(Guid projectUid, SPList splist, SPChangeToken previousChangeToken, WSSListMetadata wssListMetadata, Int32& syncItemsCount)     at Microsoft.Office.Project.Ser…    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19*    Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             General                           ai1wu    Medium      …ver.BusinessLayer.ReportingLayer.WSSDataTransfer.BuildTransferData(Guid projectUid, SPList splist, WSSListMetadata wssListMetadata, Boolean forceFullSync, Boolean& syncWasIncremental, Int32& syncItemsCount)     at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RdsWssSyncProcessor.TransferWSSListData(Guid projectUid, SPList splist, WSSListMetadata wssListMetadata, DateTime dtStartSyncTime, Boolean forceFullSync, SPChangeToken currentChangeToken)     at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RdsWssSyncProcessor.SyncWSSList(Guid projectUid, SPWeb spweb, WSSListMetadata wssListMetadata, Boolean forceFullSync)     at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RdsWssSyncProcessor.SyncWSSData(Guid projectUid, SPWeb spweb, WSSTransferMetadata w…    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19*    Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             General                           ai1wu    Medium      …ssTransferMetadata, Boolean forceFullSync)     at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RdsWssSyncProcessor.runRDSTransformation(WSSSyncMessageEx wssSyncMessage)     at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.RdsWssSyncProcessor.HandleMessage(Message msg, Group messageGroup, JobTicket jobTicket, MessageContext mContext)     at Microsoft.Office.Project.Server.BusinessLayer.Queue.ProcessMessageThread.ThreadEntry()     at Microsoft.Office.Project.Server.BusinessLayer.Queue.MessageProcessor.ProcessMessage(Message msg, MessageContext mContext, ProcessMessageThread processMessageThread)     at Microsoft.Office.Project.Server.BusinessLayer.Queue.MessageProcessor.HandleMessages()     at Microsoft.Office.Project.Server.BusinessLayer.Queue.MessageProcessor…    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19*    Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             General                           ai1wu    Medium      ….ThreadEntry()     at Microsoft.Office.Project.Server.BusinessLayer.Queue.MessageProcessor.<>c__DisplayClassb.<Execute>b__a()     at Microsoft.Office.Project.Server.Library.SafeCallbackManager.RunCallbackAction(Action action, Func`1 safePredicate)     at Microsoft.Office.Project.Server.BusinessLayer.Queue.MessageProcessor.Execute(Group& nextGroupInCorrelation, JobTicket& nextGroupJobTicket)     at Microsoft.Office.Project.Server.BusinessLayer.Queue.Receiver.ProcessMessageGroup(CancellationToken schedulerCancellationToken, Group messageGroup, JobTicket jobTicket, Group& nextGroupInCorrelation, JobTicket& nextGroupJobTicket)     at Microsoft.Office.Project.Server.BusinessLayer.Queue.Receiver.ExecuteJobInternal(CancellationToken schedulerCancellationToken, Group& messageGroup)     at Microsof…    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19*    Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             General                           ai1wu    Medium      …t.Office.Project.Server.BusinessLayer.Queue.Receiver.<>c__DisplayClass1d.<>c__DisplayClass21.<ExecuteJob>b__1b()     at Microsoft.Office.Project.Server.BusinessLayer.Queue.QueueUtils.RunActionWithIgnoreExceptionPredicate(Action action, Predicate`1 ignoreExceptionPredicate, Boolean& isIgnorableException)     at Microsoft.Office.Project.Server.BusinessLayer.Queue.Receiver.<>c__DisplayClass1d.<ExecuteJob>b__1a()     at Microsoft.Office.Project.Server.Library.SafeCallbackManager.RunCallbackAction(Action action, Func`1 safePredicate)     at Microsoft.Office.Project.Server.BusinessLayer.Queue.Receiver.ExecuteJob(CancellationToken schedulerCancellationToken, Int32 siteJobCount)     at Microsoft.Office.Project.Server.Services.SiteQueue.<>c__DisplayClass2.<ExecuteJob>b__1()     at Microsoft.Office….    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19*    Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             General                           ai1wu    Medium      …Project.Server.Library.SafeCallbackManager.RunCallbackAction(Action action, Func`1 safePredicate)     at Microsoft.Office.Project.Server.Services.SiteQueue.ExecuteJob(CancellationToken ct, Int32 siteJobCount)     at Microsoft.Office.Project.Server.Services.QueueJobRunner.ExecuteSiteJob(Object data)     at System.Threading.Tasks.Task.Execute()     at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)     at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)     at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)     at System.Threading.Tasks.Task.ExecuteEntry(Boolean bPreventDoubleExecut…    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19*    Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             General                           ai1wu    Medium      …ion)     at System.Threading.ThreadPoolWorkQueue.Dispatch()      0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19     Microsoft.Office.Project.Server (0x136D8)    0x16F88    SharePoint Foundation             Health                            46ri    High        big list slow query, list:{7903E344-47BC-42B5-A1BF-14E3E58D817F}  viewname: viewxml: <View Scope=’RecursiveAll’><Query><Where><Geq><FieldRef Name=’Modified’/><Value Type=’DateTime’ IncludeTimeValue=’TRUE’ StorageTZ=’TRUE’>2016-02-16T14:29:19</Value></Geq></Where></Query><ViewFields><FieldRef Name=’ID’/><FieldRef Name=’UniqueId’/><FieldRef Name=’FileRef’/><FieldRef Name=’FSObjType’/><FieldRef Name=’Author’/><FieldRef Name=’Created’/><FieldRef Name=’Editor’/><FieldRef Name=’Modified’/><FieldRef Name=’Title’/></ViewFields><RowLimit Paged="TRUE">2000</RowLimit></View>    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19     Microsoft.Office.Project.Server (0x136D8)    0x16F88    Project Server                    Reporting                         auos    Warning     Standard Information:PSI Entry Point:   Project User: PROJECTSERVER\system  Correlation Id: 495a0869-b8d4-e511-8c5c-005056b45654  PWA Site URL:
http://YourProjectServer/PWA/Proj1  SA Name: ProjectServerApplication  PSError: NoError (0) RDS: Problem transferring SharePoint data to the Project Server. Unable to prepare the incremental transfer data for the SharePoint list  ‘101’ associated with the project ‘0e2d89a0-e57a-e411-9568-005056b45654’. Error: The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator..  Full transfer will be used.    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19     Microsoft.Office.Project.Server (0x136D8)    0x16F88    Project Server                    Reporting                         auos    High        PWA:
http://YourProjectServer/PWA/Proj1, ServiceApp:ProjectServerApplication, User:PROJECTSERVER\system, PSI: ULS Event: WSSIncrementalTransferFailed, tag 1635086195 was associated with exception: Microsoft.SharePoint.SPQueryThrottledException: The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator. —> System.Runtime.InteropServices.COMException: The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.<nativehr>0x80070024</nativehr><nativestack></nativestack>     at Microsoft.SharePoint.Library.SPRequestInternalClass.GetListItemDataWithCallback2(IListItemSqlClient pSqlClient, String bstrUrl, String bstrListName, String bstrViewName, String bstrViewXml, SAFEARRAYFLAGS fSafeAr…    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19*    Microsoft.Office.Project.Server (0x136D8)    0x16F88    Project Server                    Reporting                         auos    High        …rayFlags, ISP2DSafeArrayWriter pSACallback, ISPDataCallback pPagingCallback, ISPDataCallback pPagingPrevCallback, ISPDataCallback pFilterLinkCallback, ISPDataCallback pSchemaCallback, ISPDataCallback pRowCountCallback, Boolean& pbMaximalView)     at Microsoft.SharePoint.Library.SPRequest.GetListItemDataWithCallback2(IListItemSqlClient pSqlClient, String bstrUrl, String bstrListName, String bstrViewName, String bstrViewXml, SAFEARRAYFLAGS fSafeArrayFlags, ISP2DSafeArrayWriter pSACallback, ISPDataCallback pPagingCallback, ISPDataCallback pPagingPrevCallback, ISPDataCallback pFilterLinkCallback, ISPDataCallback pSchemaCallback, ISPDataCallback pRowCountCallback, Boolean& pbMaximalView)     — End of inner exception stack trace —     at Microsoft.SharePoint.SPGlobal.HandleThrottleException(…    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19*    Microsoft.Office.Project.Server (0x136D8)    0x16F88    Project Server                    Reporting                         auos    High        …COMException comEx)     at Microsoft.SharePoint.Library.SPRequest.GetListItemDataWithCallback2(IListItemSqlClient pSqlClient, String bstrUrl, String bstrListName, String bstrViewName, String bstrViewXml, SAFEARRAYFLAGS fSafeArrayFlags, ISP2DSafeArrayWriter pSACallback, ISPDataCallback pPagingCallback, ISPDataCallback pPagingPrevCallback, ISPDataCallback pFilterLinkCallback, ISPDataCallback pSchemaCallback, ISPDataCallback pRowCountCallback, Boolean& pbMaximalView)     at Microsoft.SharePoint.SPListItemCollection.EnsureListItemsData()     at Microsoft.SharePoint.SPListItemCollection.get_Count()     at Microsoft.Office.Server.Utilities.ContentIterator.ProcessListItems(SPList list, SPQuery query, ItemsProcessor itemsProcessor, ItemsProcessorErrorCallout errorCallout)     at Microsoft.Office.P…    0fca5f9d-75e8-507f-40b1-bf88f739c79d
02/16/2016 15:37:35.19*    Microsoft.Office.Project.Server (0x136D8)    0x16F88    Project Server                    Reporting                         auos    High        …roject.Server.BusinessLayer.ReportingLayer.WSSDataTransfer.BuildIncrementalTransfer(Guid projectUid, SPList splist, SPChangeToken previousChangeToken, WSSListMetadata wssListMetadata, Int32& syncItemsCount)     at Microsoft.Office.Project.Server.BusinessLayer.ReportingLayer.WSSDataTransfer.BuildTransferData(Guid projectUid, SPList splist, WSSListMetadata wssListMetadata, Boolean forceFullSync, Boolean& syncWasIncremental, Int32& syncItemsCount), LogLevelManager Warning-ulsID:0x61756F73 has no entities explicitly specified.    0fca5f9d-75e8-507f-40b1-bf88f739c79d

The most important message from the logs:

The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator

That means, the value set for the List View Threshold (configurable in Resource Throttling in Web Application Management site of the Central Administration, defaults to 5000) is less than the document count in the library. You should increase the value to re-enable synchronization, using PowerShell, for example:

$webApp = Get-SPWebApplication http://YourProjectServer
$webApp.MaxItemsPerThrottledOperation = 10000
$webApp.Update()

Or alternatively convince your users not to store so many files in the document library.

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: