What is the Y2K22 Microsoft Exchange bug?
While I initially assumed that Microsoft Exchange set a New Year's resolution to be the first problematic tech of 2022, it turns out there's a much more logical reason for the breakage.
Similar to the Y2K dilemma faced 22 years ago, the new Exchange bug is due to the year changing from 2021 to 2022. As security researcher and Exchange admin Joseph Roosen pointed out on Twitter, the issue is happening because Microsoft is using a signed Int32 variable to store a date value. Signed Int32 variables have a maximum value of 2,147,483,647. However, the new 2022 date value added to the Int32 is 2,201,010,001, putting it over the max value an Int32 can contain.
As soon as the clock struck midnight on January 1st, 5300 and 1106 error messages began rolling in from the FIP-FS engine, causing email messages to be stuck in the transport queues.
Log Name: Application
Source: FIPFS
Logged: 1/1/2022 1:03:42 AM
Event ID: 5300
Level: Error
Computer: server1.contoso.com
Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.Log Name: Application
Source: FIPFS
Logged: 1/1/2022 11:47:16 AM
Event ID: 1106
Level: Error
Computer: server1.contoso.com
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.Who's affected?
According to Microsoft, this issue only impacts on-premises versions of Microsoft Exchange 2016 and 2019 configured with the mailbox role using the FIP-FS malware engine. If you've disabled the FIP-FS engine and utilize a different email hygiene solution, this issue shouldn't impact you. Also, Edge Transport Servers are not impacted.
Microsoft's official solution
Microsoft engineers putting in extra time during the holiday weekend wasted no time providing an official solution to resolve the Y2K22 Exchange bug. In fact, they provided both a manual solution and an automated solution for those of us who shun the word "manual".
From Microsoft:
We have addressed the issue causing messages to be stuck in transport queues of on-premises Exchange Server 2016 and Exchange Server 2019. The problem relates to a date check failure with the change of the new year and it not a failure of the AV engine itself. This is not an issue with malware scanning or the malware engine, and it is not a security-related issue. The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues.
We have now created a solution to address the problem of messages stuck in transport queues on Exchange Server 2016 and Exchange Server 2019 because of a latent date issue in a signature file used by the malware scanning engine within Exchange Server. Customer action is required to implement this solution.
The automated solution
The automated solution comes in the form of a script that needs to run on Exchange mailbox servers using an elevated Exchange Management Shell. Please note that prior to running the script, you will need to change the execution policy for PowerShell scripts on the server to RemoteSigned by running Set-ExecutionPolicy -ExecutionPolicy RemoteSigned.
The manual solution
If you're the type of sysadmins that prefers to take matters into your own hands, then this manual solution should be right up your alley.
Remove existing engine and metadata
Stop the Microsoft Filtering Management service. When prompted to also stop the Microsoft Exchange Transport service, click Yes.
Use Task Manager to ensure that updateservice.exe is not running.
Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.
Update to the latest engine
Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.
Verify engine update info
In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001.
After updating the engine, it's recommended that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.
These solutions and several FAQ's can be found on Microsoft's tech community site here: https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-exchange-on-premises-transport-queues/ba-p/3049447
Wrapping up
Resolving the Microsoft Exchange issues is certainly not the most exciting way to ring in 2022, and hopefully this bug isn't a sign of things to come for 2022. Thankfully, Microsoft provided solutions are easy to implement, allowing you plenty of time to continue recovering from your questionable decisions made on New Year's Eve.
