Microsoft Exchange Fixes Disruptive 'Y2K22' BugIssue Affects Email Delivery Because of Date Rendering Format
Remember Y2K? Widespread disruption was feared since systems that rendered dates as two digits needed to be updated to work with four digits. Well, Microsoft Exchange has just issued a workaround to fix a fatal error that disrupted email delivery due to a date check failure with the change of the New Year.
"We have addressed the issue causing messages to be stuck in transport queues of on-premises Exchange Server 2016 and Exchange Server 2019. The problem relates to a date check failure with the change of the new year and it is not a failure of the AV engine itself," according to the Microsoft Exchange team.
Microsoft in a post on its Tech Community forum assures users that the issue has nothing to do with malware scanning or the malware engine, and it is not a security-related issue.
What is The Issue?
"The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues," Microsoft says.
Microsoft states that it created a solution to address the problem of messages stuck in transport queues on Exchange Server 2016 and Exchange Server 2019, because of a latent date issue in a signature file used by the malware scanning engine within Exchange Server. However, it says, “Customer action is required to implement this solution.”
"E-mail is a critical communication tool for every organization, and we expect it to work like we expect the lights to turn on when we flip the switch," Says John Bambenek, principal threat hunter at digital IT and security operations company Netenrich.
"This bug highlights the difficulty of robust coding where straight-forward functions, like date-checks, may look good to code scanners or in code review, but still cause critical failures," Bambenek tells Information Security Media Group.
The issue began as the year 2022 kicked in, which caused servers to not deliver email messages. An error message was displayed, which states, "The FIP-FS 'Microsoft' Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert '2201010001' to long."
A Reddit user first reported this issue and dubbed it the “Y2K22” bug said, "Just wondering if any other Exchange admins had their new year’s celebration interrupted due to the 'Microsoft Filtering Management Service' being stopped and reports of issues with mail flow?"
"The “long” type allows for values up to 2,147,483,647. It appears that Microsoft uses the first two numbers of the update version to denote the year of the update. So when the year was 2021, the first two numbers was “21”, and everything was fine. Now that it’s 2022 (GMT), the update version, converted to a “long” would be 2,201,01,001 - - which is above the maximum value of the “long” data type. @Microsoft: If you change it to an ‘unsigned long’, then the max value is 4,294,967,295 and we’ll be able to sleep easy until the year 2043," the user states.
Microsoft recommends an automated solution to fix the issue. Users are advised to download a script from here and before running it users should change the execution policy for PowerShell scripts by running 'Set-ExecutionPolicy -ExecutionPolicy RemoteSigned'.
"Run the script on each Exchange mailbox server that downloads antimalware updates in your organization (use elevated Exchange Management Shell). Edge Transport servers are unaffected by this issue. You can run this script on multiple servers in parallel," Microsoft says.
However, the firm also released a manual solution to resolve the issue and restore service. Microsoft says that to manually solve the issue, a user must first verify which impacted version it has installed. It adds that to check the current version, a user has to run Get-EngineUpdateInformation and check the UpdateVersion information.
If it starts with "22..." then proceed, but if the installed version starts with "21...", a user need not take any action.
The next step involves removing the existing engine and metadata. "Stop the Microsoft Filtering Management service. When prompted to, also stop the Microsoft Exchange Transport service, click Yes. Use the task manager to ensure that updateservice.exe is not running," Microsoft recommends.
The company also advises deleting the folder: Exchange ServerV15FIP-FSDataEnginesamd64Microsoft and removing all the files from the following folder: FIP-FSDataEnginesmetadata.
Lastly, the company says users should update to the latest engine. It says, "Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service, then open the Exchange Management Shell, navigate to the Scripts folder (Exchange ServerV15Scripts), and run Update-MalwareFilteringServer.ps1
In the Exchange Management Shell, Microsoft recommends running Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell and run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001 (or higher).