Backward compatibility of SMAT files

Unicode stores raw inbound data in SMAT files, but stores UTF-8 Unicode in outbound SMAT files. This causes some backward compatibility issues with existing SMAT files because they contain raw binary data for both inbound and outbound. A single SMAT file could contain raw and UTF-8 data if a site is upgraded with outbound SMAT files in place.

When the Unicode engine is started, it continues writing to the existing SMAT files using UTF-8. This results in mixed raw/UTF-8 files. To handle this properly, the Unicode engine inserts an additional setting into the outbound .idx file. This setting is {ENCODING UTF-8}.

These settings are stored on a per-message basis, so there is no problem even with a mixed file. When a SMAT file is viewed or resent, the ENCODING setting must be checked. If it does not exist, then the message data is assumed to be raw. An automatic binary to UTF-8 conversion is then required when the message is resent to any point other than Inbound pre-TPS.

An automatic UTF-8 to binary conversion is required for any UTF-8 message resent to Inbound pre-TPS. The only valid value for ENCODING is UTF-8 and it is only saved for an outbound SMAT file. This feature ensures that both old and new SMAT files can be used without any requirement for explicitly converting the files.