Apps/WebApplicationReceipt/GenerationService

Receipt Generation Service

Project Plan

The Receipt Generation Service is a high-availability service that generates digitally-signed receipts for objects vended by the Marketplace. The receipts are intended to be a complete proof of purchase that can be independently verified by third parties.

System Overview

Receipts are essentially an RSA 2048-bit signed triple of (user ID, store ID, product URL/ID). They contain other useful metadata, but from a trust standpoint those are the important pieces.

The receipt trust system is rooted in a small number (1, or 2-4; see appendix A) of asymmetric keypairs. The private keys are held in HSMs, and require k-of-n procedures to change. The root receipt trust nodes are responsible for generating daily receipt signing keys, and certifying them. Within a given site, only one of these nodes is active; the other is on hot standby. Each site is responsible for generating its daily keys, since a receipt generation cluster could become active on short notice.

The public keys are carefully exported from the HSMs and conveyed to a distribution point that provides 3rd parties with confidence (see Appendix B).

A horizontally-scalable cluster of receipt signing nodes receives one of these keys from the root nodes every day. These signing nodes are responsible for receiving unsigned receipts from the Marketplace, signing them, logging the fact of signing, and returning the signed receipt as the response.

The certification of a signing key will cover the key itself, its identifier, and the date range for which it is valid (as a nbf/exp pair, as defined in JSON Web Token).

When an authenticated user session makes a receipt request, the Marketplace issues a request-to-sign to the receipt signing cluster, logs the request, and returns the signed receipt to the user.

The user agent makes requests for receipts when Marketplace content uses the Application DOM API to request an install. The receipts are stored, along with application metadata, in the user agent, where they are made available to the developer's domain through the Application DOM API. The developer is then expected to upload the receipt to their own server, where they can perform validation.

Validation of the receipt can be done by the developer's server, by verifying the signature, and then verifying the chain of certificates. The root of trust of the chain will be one of the public certificates previously advertised by the Marketplace. The date range on each key must be checked against the issue date on the receipt. If the application's server decides that the receipt is cryptographically invalid, it can take appropriate action. (XXX Can the server prompt the user agent to refresh the certificate? See appendix D).

Alternatively, the developer can POST the receipt to a URL specified in the receipt (the verify_url). This is a service, hosted by Marketplace, which will:

1. Verify the receipt and certificate chain
2. If verification succeeds, retrieve the business state of the receipt (one of okay, refunded, rejected, invalid) (XX language check here)
3. Report back the verification of the receipt and the business state of the receipt.

(NB to Marketplace team: If there is going to be a developer-facing API to query the state of individual receipts by user ID, it will need to have a developer-facing authentication piece. This system does not require authentication; possession of the receipt is considered sufficient to find out the user's status).

The user agent should endeavor to never present an expired receipt. This has obvious implications for the UA's behavior, and is developed in depth in Appendix D.

Software Components

The root signing nodes are responsible for generating a daily list of keys, certifying them with a hardware-protected private key. They push these certified keys to the signing nodes through a secure, intranet-only link. There is no internet access to the root signing nodes. The public half of these keys is exported and delivered very carefully to the advertising point. The receipt signing nodes are responsible for signing receipt-requests that comes form Marketplace nodes. They receive new private keys once a day. XXX Best case would be to hold the key in memory only - but how do we handle unexpected process termination and restarts? If there's no way to request a new key from the root nodes we're stuck.

The Marketplace Verification service needs to receive the public keys, and to perform verification of the receipt and the certificate chain.

Threat Model

1. Threat: An attacker could capture a signing key and export it from the signing node. They could make as many receipts as they like.
   1. Countermeasures:
      1. The attacker needs to attach the certification for the key for it to be accepted, and the certification contains a not-after date. The attacker would not be able to produce certificates that would be acceptable past that date. That said, we might want the not-after date to be substantially in the future, so we take additional steps if the theft is detected.
      2. If we detect the theft of the signing key, we revoke it internally and begin a root key reset and reissuance as described in Appendix C.
      3. If we do not know that a signing key has escaped into the wild, the only clue we would receive is when the verify_url is invoked with a receipt that passes cryptographic verification but does not match the transaction log. This condition would be indistinguishable from database corruption but should trigger immediate warnings.
2. Threat: An attacker could compromise a signing node and create as many receipts as they like.
   • Countermeasure: Infrasec will correlate the signing activity log with actual requests from the Marketplace.
3. Threat:An attacker could create their own receipt, sign it with their own key, certify that key with their own private key, and attach that.
   • Countermeasure: The developer must verify that the chain is based on a public key that comes from the store the receipt claims to be from.
4. Threat:An attacker could compromise the public key distribution point of Marketplace, place their own key there, and produce their own receipts rooted in that key. Or: Attacker could compromise the SSL chain leading to the public key distribution point and MITM their own key.
   • Countermeasure: Changes to the public key should be accompanied by out-of-band signaling from Marketplace; any other change is an attack. Geographic tripwiring would be needed to catch the MITM.
5. Threat:An attacker could compromise k-of-n Mozilla employees, and export the private key.
   • Countermeasure:
6. Threat:An attacker could compromise the channel through which the public keys are relayed from the root trust nodes to the distribution point, and place their own public key in the list.
   • Countermeasure:

Operational Procedures

• The public keys need to be advertised very securely; any tampering with this trust root enables receipt-impersonation attacks.
• Revoked keys need to be advertised just as securely; the ability to disrupt revocation distribution could allow an attacker to use a stolen key.
• A receipt-generation site would need to detect if a root node has stopped generating private keys, and failover to the backup node.
• Marketplace would need to detect if a receipt signing site has stopped responding to requests from Marketplace, and fail over to a backup site.
• Marketplace would need to keep a log of receipt-generation requests; this should be compared to the receipt-generation jobs executed by the signing nodes to detect unexpected requests. (OR: all Marketplace nodes would need to authenticate to the signing cluster)

Appendices

Appendix A: One key or 2-4?

The HSMs have mechanisms to sync a private key pair across all nodes; if we do this we have only one private key. If not, we have four,

Appendix B: How to distribute the public keys?

The straightforward approach is to place them on an SSL-protected website. We would then be vulnerable to any attack which can compromise SSL.

Rather than invent a new scheme, mhanson proposes to use an RFC5785 / RFC6415 compliant ".well-known/host-meta" file, which is an XRD containing a LINK with a REL of "receipt-verification-keys", which points to a file containing an array of JSON Web Keys.

Appendix C: Key Compromise and Receipt Reissuance

If a signing key is compromised, and the theft is detected, the Marketplace will revoke the key. This will require a reissuance of all receipts that included that key.

While we could advertise the need for this reissuance by hosting a list of revoked certificates, we believe that the evidence has shown that it is hard to get certificate revocation right. So we adopt a scheme that makes us do more work, but is simpler for the rest of the web to interoperate with, based on resetting the root key.

Specifically, the actions to be taken following a key compromise are:

• Infrasec alerts Marketplace and client teams
• A new root key is generated.
• The new root key is advertised from .well-known (see Appendix B)
• [optional] A hotfix is issued to user agents indicating the need to refresh receipts "soon"
• As receipts expire, or more immediately, user agents request new receipts from the Marketplace
• New receipts are generated with the new root key

If we don't do the hotfix step, there will be a window of time where user agents are presenting receipts that look good, but contain the revoked key. During this window, applications could be fooled by an attacker using the stolen key. Since the key is certified with a date range, the attacker's ability to produce new receipts would eventually expire - that is, the attacker could still produce valid-looking receipts, but only for date ranges whose not-after date was in the past.

This interaction will require new logic in the user agent and should be communicated to the webapps and B2G teams for review.

Appendix D: User Agent Receipt Refresh

When a user agent notices that a receipt is approaching its expiry date, or when a getSelf() call is made by application code to retrieve a receipt that has expired, the user agent must contact the issuing authority from which it received the receipt and request a fresh one.

XXX note that this suggests that UAs need to parse the receipt and check the date inside getSelf(). This is a change from current behavior.

XXX also note that this could describe a race condition, where a periodic expiry-date checker and one or more getSelf() calls could collide and all attempt to refresh a certificate.

In the happy path, this could be as simple as making a GET request to a URL that was specified in the original receipt, and could be handled automatically and silently. In the less happy path, this could require authentication of the user agent to the issuing authority (e.g. Marketplace); if the user fails to authenticate, the receipt cannot be refreshed. This could also fail because the user does not actually have a valid purchase record; the common case is probably user account confusion (signed in as wrong user) but this is how the stolen-key case would manifest to the user as well.

XXX Note that if the user agent checks dates periodically, it could decide that receipts are invalid at a time when the user is not authenticated to the Marketplace. Spontaneously requesting user authentication is bad UX and bad security, so we shouldn't do that.

Comments / Concerns / Clarification Needed

• [mcoates - 2012-03-28] We refer to sites e.g. "Within a given site". Do we mean data centers?
  • [mhanson 2012-03-29] yes. or whatever unit of operational-stability-and-granularity makes sense.

• [mcoates - 2012-03-28] Need to specify that the receipt verfication service hosted by mozilla is only accessible via HTTPS

• [mcoates - 2012-03-28] review Bug needed - review implementation to ensure daily private keys are correctly destroyed

• [mcoates - 2012-03-28] Threat #2 - More discussion is needed here to understand this requested control "Infrasec will correlate the signing activity log with actual requests from the Marketplace. "

• [clouserw - 2012-03-29] Am I correct in my understanding that the HSM is used to generate a key once a day which lives on a .well-known distribution point. From there the receipt signing nodes pick up on the new key and sign receipts with it? So the HSMs aren't actually being used to sign the receipts, just to hold the private key securely?
  • [mhanson - 2012-03-29] That is correct; that is the most-secure pattern, according to badida and bwarner.

• [clouserw - 2012-03-29] In order to use the /.well-known/ prefix we'd technically have to register it with the standards folks (after debate). Obviously we can do what we want, but we'd be out of spec at that point.
  • [mhanson 2012-03-29] added clarification about using RFC 6415 too. If we can find an existing REL to use the XRD, great.

• [clouserw - 2012-03-29] The receipt reissuance moves the authority of a valid receipt from the original certificates to the AMO database. I don't know a way around that, but it's worth noting.
  • [mhanson 2012-03-29] I think the AMO database has always been the ground truth since it's the place that knows about refunds and chargebacks. Don't think it changes the picture in the big scheme.

• [clouserw - 2012-03-29] The receipt validation and reissuance checks are complicated. We should definitely provide server side code examples for developers trying to implement that.
  • [mhanson 2012-03-29] Most recent draft attempts to simplify by eliminating cert revocation from validation.

• [clouserw - 2012-03-29] I haven't seen plans for a way to query for receipts by User ID. I'm happy to skip that until it's shown we need it.
  • [mhanson 2012-03-29] Good, a service to do that would have weird/bad privacy characteristics.

• [clouserw - 2012-03-29] In Appendix B you ask if the public keys and the revoked keys should be in the same file, but in "Software Components" you say that the public keys are on an intranet-only URL. In "System Overview" you mention that the developer's servers can retrieve the list of revoked keys but they won't have access to an intranet-only URL.
  • [mhanson 2012-03-29] the private keys are intranet only - the public keys are "delivered carefully to the advertising point" - e.g. the public website
• [joes/kang - 2012-04-06] is the python glue on the HSM verifying that the keys it's signing are expiring after a day maximum?
  • [mhanson 2012-04-06] No, that logic hasn't been written yet - all the HSM python glue knows how to do today is sign some next. I think we should use JWK as the serialization format - pinging benadida for expertise on that.

• [andym 2012-04-08] "When a user agent notices that a receipt is approaching its expiry date" I haven't seen anything to suggest that receipts themselves have expiry dates.

• [kang 2012-05-11] In Appendix B, the public key which is the root of the trust is stored on a HTTPS webserver. This means if the webserver is compromised, the key can be replaced by any key and the applications will trust it (which may be what you meant by "compromise SSL"). A possible recommendation (note: this is *only* a possible recommendation) would be to generate 2 keys in the HSM, lets call them master key and sub key. The master key sign the sub pub key, and we place it on the HTTPS webserver. All apps verify up to the master key, enforcing that, any key must be signed by the HSM master key in order to be trusted (so if someone replace the key on the HTTPS webserver it won't work. he would have to also compromise the HSM). It also means the trust is established at the first fetch of the master pub key (and that the same threat still exists at this level, but once trusted you would keep it, while the sub key can be replaced. This also means devices such as B2G devices could ship with it, and that also means the master key can be on a different server (so you would have to compromise 2, to compromise the first fetch). Yet another possibility is to simply have a warning on the application side prompting to accept the key when the master/root key changes (which should only occur when something really wrong happened, aka server or key has been compromised)

Security Review Notes

Review Notes:

Items to be reviewed

• bug 734445
• https://wiki.mozilla.org/Apps/WebApplicationReceipt

• How we are going to authenticate the signers.
• authenticate server (marketplace) requesting signing.
• Operations Security Tasks

1. write ocs/acs (hsm smartcards) operational procedures and policy
2. program ocs/asc (hsm smartcards)
3. publish system and network security requirements for these servers and hosts
4. verify that CEF logging in place for each receipt signing operation. <joes><ray>

risk

• server ip compromise could allow push of signing cert from root cert to malicious server.
• multiple refunds against non-valid transactions or just too many refunds.

Action Items

• Who :: What :: By when
• Bill to verify with Justin about plan for receipt revocation
• Need to design and implement a receipt reissue system
• Review to verify daily keys are correctly destroyed each day
• Need to alter receipt verification to cope with the proposed signing chain
• need to define process for recovation/re-issue of root key