Exception in alarm clock event subscriber

We are on Sitecore 9.2 with SXA 1.9 hosted on Azure PaSS using XP0 topology. We are noticing numerous exceptions in the logs files about Alarm Clock Event Subscriber. Please have a look at the stack trace below:

ManagedPoolThread #9 2022-07-07 13:11:55 ERROR Exception in alarm clock event subscriber.
Exception: System.Runtime.Serialization.SerializationException
Message: Attempting to deserialize an empty stream.
Source: mscorlib
   at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
   at Sitecore.Analytics.Data.DataAccess.SubmitQueue.FileSubmitQueue.ReadHeader(FileStream fileStream, Int64& entryCount, Int64& currentReadPosition)
   at Sitecore.Analytics.Data.DataAccess.SubmitQueue.FileSubmitQueue.GetDequeueingStream()
   at Sitecore.Data.DataProviders.Retryer.Execute[T](Func`1 action, Action recover)
   at Sitecore.Analytics.Data.DataAccess.SubmitQueue.FileSubmitQueue.Dequeue()
   at Sitecore.Analytics.SubmitQueueService.WakeUp()
   at Sitecore.Services.AlarmClock.<>c__DisplayClass11_0.<Heartbeat_Beat>b__0(Object state)
 

First lets see what is Submit Queue?

When the tracker submits a session to xConnect, but xConnect is temporarily unavailable, the tracker classifies this as a transient error (meaning it assumes xConnect becomes available later). The tracker adds the session to a submit queue that, by default, serializes the session to disk. The submit queue service later re-submits the session to xConnect at configurable intervals. If you open the ShowConfig.aspx, you will notice the setting below:

<setting name="Analytics.UseSubmitQueue" value="true" patch:source="Sitecore.Analytics.Tracking.config"/>

Also in the Sitecore.Analytics.Tracking.config you will see the further configuration:

<submitQueue patch:source="Sitecore.Analytics.Tracking.config">
    <queue type="Sitecore.Analytics.Data.DataAccess.SubmitQueue.FileSubmitQueue, Sitecore.Analytics" singleInstance="true"/>

Submit Queue is stored on the physical location under app_data (unless it is customized). If you notice in the stack trace above the error is related to Alarm Clock triggering the SubmitQueue service. It is to process the submit queue files in your submit queue folder, but currently it appears that it is throwing an exception because it is trying to serialize an empty file.

Usually if you are seeing the exceptions like above please check the items in the /App_Data/Submit Queue if the content of the files are empty or not. Additionally, please consider creating a backup and remove the files in the Submit Queue folder.

Hope this helps.