Digital Signatures in Windows Installer
Digitally signing files helps protect against changes to a file (or any data, really) by validating that a hash of the current file matches the hash stored in the digital signature. Digital signatures also help verify that a package came from a particular publisher by encrypting the hash with the publisher's private key. Verifying the signature using the publisher's public key or a trusted certificate authority that signed their public key validates the publisher.
You can sign Windows Installer packages, such as .msi and .msp files, to help guarantee that users know if your files have been modified and that they came from you, the publisher. Windows Installer validates that a package hasn't been changed if it contains a digital signature when attempting to install it. It does this by calling the WinVerifyTrust function both directly and indirectly. Signing a patch can also enable privileged installs for non-privileged users. This was called LUA patching and is now known as UAC patching.
But how does Windows Installer know if a package is signed? Many different file types can be signed, including .exe and .dll, .cab, .msi and .msp, and more. The basic file types all store digital signatures differently. PE/COFF executables store signatures in the virtual certificates directory in the optional headers. Signatures fill reserved space within a cabinet file. With a simple modification to my patch files extractor, you can see both a _SummaryInformation stream and - in signed files - a _DigitalSignature stream. (The tool actually changes the character \005 to an underscore for readability, so the digital signature stream is actually named \005DigitalSignature.)
So how does WinVerifyTrust determine how to find the signature? Download and run regmon.exe from Sysinternals and run the following command from within a Platform SDK (now Windows SDK) command shell:
signtool verify /pa <signed.msp>
You'll noticed a lot of requests to registry keys under HKEY_LOCAL_MACHINE\Software\Microsoft\Cryptography\OID\EncodingType 0, including the sub-key CryptSIPDllIsMyFileType2 and more. SIPs, or Subject Interface Packages, implement exported callback functions to help determine if a file type is supported, to get the signature from a supported file type, and more. You can see such a file, listed as msisip.dll, and various entry points, each under a different sub-key. Looking in the Windows SDK in mssip.h, you can see signatures and comments for these functions. Using SIPs, WinVerifyTrust can get and verify the signature for any given file type that has an associated SIP provider. You can even add your own providers using the CryptSIPAddProvider function.