Analyzing XML File Content written by Lync Server Storage Service (LYSS)

This post has been republished via RSS; it originally appeared at: Skype for Business Blog articles.

First published on TECHNET on Jan 17, 2018
In Skype for Business Server 2015, we added functionality, where contents flushed by Lyss (EVENT ID 32089) is automatically imported every 30 minutes. This configuration can be enabled or disabled by using Set-CsStorageServiceConfiguration and when the contents are successfully imported, we would see EVENT ID 32097 and if the import fails, EVENT ID 32099 would be written to disk
 

        Log Name:      Lync Server

        Source:        LS Storage Service

        Event ID:      32097

        Computer:      lyncstd01.contoso.com

        Description:

        One or more files were automatically imported back into Storage Service successfully.

 

        The following automatic flushed file import success events occurred.

        #CTX#{ctx:{traceId:10001, activityId:"a35a9699-53fb-440d-bbd4-058729dd284f"}}#CTX# File:
        \\contoso.com\LyncRootDFS\RTCShare\1-WebServices-1\StorageService\DataExport\ 2014

        0625\SKYPEFE01.contoso.com\1c5de83aaf4e5d41a767576ed27f333f__0.xml, items imported: 249

        :

        Cause: A background task will look for eligible flushed files from the web service file store or from       the local front end which were created at least a certain time ago and import back into Storage             Service. These items could have been flushed out from the DB due to high DB size or from
      manually invoking pool failover or flushing cmdlets.

        Resolution:

        No action needed.

        Log Name: Lync Server
        Source: LS Storage Service  
        Event ID: 32099
        Computer: SKYPEFE01.contoso.com
        Description:
        Attempt to automatically import a flushed file back into Storage Service encountered error.
        The following automatic flushed file import error events occurred.
        #CTX#{ctx:{traceId:10001, activityId:"8454c5d4-57f6-437a-9cf7-46fc15960492"}}#CTX# File:
       
\\contoso.com\LyncRootDFS\RTCShare\1-WebServices-1\StorageService\DataExport\ 2014

      0625\SKYPEFE01.contoso.com\0640daf8d97b5199b82663737356b525__14.xml , items deserialized 3, items       failing re-import: 3

      #CTX#{ctx:{traceId:10001, activityId:"8454c5d4-57f6-437a-9cf7-46fc15960492"}}#CTX# File:
         
\\contoso.com\LyncRootDFS\RTCShare\1-WebServices-1\StorageService\DataExport\ 2014
       
0625\SKYPEFE03.contoso.com\b20f02d8943b53dc89ddcb2ff106f912__29.xml
, items deserialized 1, items
        failing re-import: 1


      #CTX#{ctx:{traceId:10001, activityId:"8454c5d4-57f6-437a-9cf7-46fc15960492"}}#CTX# File: 
       
\\contoso.com\LyncRootDFS\RTCShare\1-WebServices-1\StorageService\DataExport\ 2014
       
0625\SKYPEFE03.contoso.com\b20f02d8943b53dc89ddcb2ff106f912__6.xml
, items deserialized 2, items       failing re-import: 2
        :
        Cause: Bad input data, or error calling Storage Service, or other errors.
        Resolution:
        Please look at event details and use the correlation ID to view corresponding traces to resolve the error.


To investigate, I would get started with the XML files. First, I would simply view them in a Browser or another application to view the contents. A quick visual spot check could provide information, about the failures

<?xml version="1.0"?>
-<LyssQueueItem Version="1" xmlns="http://schemas.microsoft.com/RtcServer/2012/11/lyssimpexp">
-<QueueItems>
+<ItemQueue ItemQueueID="0282d738-9468-e711-8108-0050569e79b5" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="e081e54a-9468-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="82c21bcb-9768-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="3cd27fce-9a68-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="48b5359a-a368-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="44bb9767-a768-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="8b35a1e5-be68-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="4ff14355-c068-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="b6e98870-c168-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="c0204b47-c568-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="6c59d707-c668-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="90482b96-c768-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
+<ItemQueue ItemQueueID="6c76a50e-ca68-e711-8108-0050569e79b6" GroupID="38e63176-1723-5d4d-8a40-ff2e2c436899">
</QueueItems>
</LyssQueueItem>


Here we can count the number of items as 13 items, and may be able to investigate individually. Let's say, if there were many items, we would want to not manually count the errors and then manually check what’s wrong.

So we can run use PowerShell to help us out, this can be accomplished by running

[XML] $a=Get-Content '.\LYSS_Sample.XML'
$a.LyssQueueItem.QueueItems.ChildNodes.Count


1289


Disclaimer: Importing a XML file in PowerShell can be very resource extensive, and it highly recommended to not be performed on a LYNC or Skype for Business Server

Next, we can look into the characteristics about the issue, simply by running the below script. We can see that the content in this XML file is all tagged as Item Status 3 with a particular AdapterID.

$a.LyssQueueItem.QueueItems.ItemQueue | ft ItemStatus,AdapterID

ItemStatus AdapterID
---------- ---------
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f
5          cde2bace-f515-444d-a3f1-858a7fc8728f


If all the items have ItemStatus 5 and the AdapterID is cde2bace-f515-444d-a3f1-858a7fc8728f, and you have users that are enabled for Skype for Business, but have mailboxes hosted in an email system that’s either non-Microsoft Solution ( EWS doesn’t exist) or doesn’t allow for OAuth ( Exchange Server 2010 and earlier) then these messages mostly would be Server-Side Conversation History for the account, but conversations cannot be saved. If the goal is not to move the mailbox to Exchange Server 2013 or higher, then these files can be ignored. (Updated 08-March 2018) In-order to prevent more data to be written to the file-share, it is recommended that you use New-csClientPolicy to create a new policy for these users and  then disable Server-Side Conversation History for the users.

$a.LyssQueueItem.QueueItems.ItemQueue | Group-Object AdapterID | FT Count, Name
Count       Name
-----       ----
60         cde2bace-f515-444d-a3f1-858a7fc8728f
12         36AA818F-00BB-43BC-88E7-6840ECA732C6
5          0947BCF3-7D50-40A7-9E3A-F07B9DC4CEF1

If the name matches "CDE2BACE-F515-444D-A3F1-858A7FC8728F" you might want to follow instructions The LCSLog SQL Database is not logging any archiving content .
If the name matches "36AA818F-00BB-43BC-88E7-6840ECA732C6" it could be possible that you also have issue with EVENT ID 56208 – Resolving Issues with CDR Throttling
If the name matches “0947BCF3-7D50-40A7-9E3A-F07B9DC4CEF1”, if could be possible that you have issues with EVENT ID 56416 – Failed to post QoE report to External Consumer

If there are more than 2,000 items in any XML file ( shouldn’t typically happen), then it could happen that the Auto-Import functionality may fail parsing the file. In such a case, we would recommend to use ImportStorageServiceData.exe or reach out to Microsoft Premier Support

If the contents of the folder have data older than your retention period ( CDR, QoE, IM&WebConf retention) say for example the XML files are generated 12 months ago, but the retention periods are 90 days, then it would be safe to delete the XML files from  >90 days ago.

As always, when in doubt, we request you to open a Service Request with Microsoft Premier Support

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.