One of the main benefits of consolidating on a single crypto library is doing so create allows consolidation of management of certificates. In order to reap this benefit, however, we not only need to use a common crypto library, but also agree on a common way to use the crypto resources (keys/certs/tokens). This document attempts to explain how applications using NSS as a system library should initialize NSS for maximum effect.

Application Programmer's Summary

The short summary: Except as noted below, all applications should initialize NSS with the following call:

rv = NSS_InitReadWrite(“sql:/etc/pki/nssdb”);

[Can we have a macro for this, e.g. NSS_DEFAULT_DB or somesuch?]

There are 3 classes of Exceptions.

1. The first exception is applications which must set special flags. The two most common flags are NSS_INIT_READONLY == 1 and NSS_INIT_OPTIMIZESPACE == 0. NOTE: applications that used to initialize READONLY because of the old dbm semantics no longer need to do so.

These applications should initialize with:

rv = NSS_Initialize(“sql:/etc/pki/nssdb”, “”, “”, “”, flags);

where flags are 0, NSS_INIT_READONLY, or NSS_INIT_READONLY|NSS_INIT_OPTIMIZESPACE. (Note: flags = NSS_INIT_OPTIMIZESPACE is the same as NSS_InitReadWrite() above. Also NSS_INIT_READONLY|NSS_INIT_OPTIMIZESPACE is the same as NSS_Init().).

2. The second exception is applications which want to perform automatic migration of an old NSS database from an old location to the new system location. These applications should initialize NSS with:

rv = NSS_InitWithMerge("sql:/etc/pki/nssdb”, “”, “”, “”, oldNSSdirectory, uniqueIDForOldDB, nameOfOldDBForPrompt, flags);

The application should then follow the automatic merge instructions in the shared database documentation.

3. The third exception is applications that must have a separately managed environment. Typically these applications would have a mode (probably default) which would open NSS normally, but could be configured to use their own NSS environment. The options to use the system or private environment should not be hard coded in the application, but should be under user/administrator control. Applications of this class would include Server products or test modes of application products (like Firefox profiles). In this mode the application should initialize NSS with it's traditional Init call, passing in the appropriate private database.

On Linux, the effect of first initialization as well as exceptions 1 or 2 will be to:

1. Open a database in the home directory of the user running. With the flags specified by the application (R/W, Optimize space, etc).
2. Load any User specified PKCS #11 modules (chosed by the OS).
3. Open the system database in /etc/pki/nssdb readonly.
4. Load any System specified PKCS #11 modules (specified in /etc/pki/nssdb/pkcs11.txt).
5. Load the NSS builtins module.

How this happens will be explained later, but first we discuss where the requirements for the above come from.

[#1 is alarming, and looks like black magic. Why, when I tell the API to look in /etc/pki, should I expect it to look anywhere else? This always happens, /etc/pki/nssdb has a file called pkcs11.txt that loads all the databases/modules related to opening that directory. The change here is softoken can now automatically open more then one itself.]

Requirements by Package Type

There are 3 typical package types that may use NSS and it's databases:

Type 1: User Application. These are applications like browsers, mail clients, etc. that individual users run. The user expects to be able to access all of his/her certificates from any application. The user also expects that system wide configuration is automatically available, though overridable by the user.

Type 2: System/Service Applications. These include things like web servers, mail servers, etc. These applications may want to share keys and certs with other servers, or they may want to have their own private copies of certs and keys.

Type 3: Libraries. Libraries are a special case. The library may run in any kind of application or services.

Type 1 packages: User applications

User applications should open NSS using a shared database stored in ~/.pki/nssdb in the user's home directory. If the application needs to store new certificates (like a web browser), then it should open this database read/write. User's local preferences would be stored in this database. Changes the application wants to make will occur in this database. Any user specified tokens would also be stored in this database.

In addition, the application should open the system database /etc/pki/nssdb. This database should be opened read only. The user will typically not have permission to modify this database. This database will provide system level defaults for tokens to load and root certs to trust. This gives us hooks form things like IPA to manage and distribute trusted root certs system wide.

[I'm almost certain there's a typo there - how can every application *own* the system database? Don't you mean *open*? - bob: you are correct, changed.]

[How does an application open first the user database at ~/.pki/nssdb, with NSS_InitReadWrite I presume, and then opens the system database in /etc/pki/nssdb? Since the NSS intialization calls are idempotent, does it need to use NSS_InitContext to accomplish this?]

Type 2 packages: Services applications

Option 1: Services applications, again, should open both sets of databases. The difference is service applications may actually have read permission on the key database in /etc/pki/nssdb. This means it can use /etc/pki/nssdb to share key material with other system services. Each service could start as it's own user or several services could run as a single user and share keys that way.

Option 2: instead of sharing service level keys in /etc/pki/nssdb, we would specify a different database that services would open for shared key support (call it /etc/pki/services for now). In this case all three would be opened, but there would never be any keys in /etc/pki/nssdb, and User applications would not have access to /etc/pki/services.

Option 3: Either option 1 or option 2 except the 'user' database for the server isn't necessarily stored in ~/.pki/{server_user}, but is configurable for each server.

[Option 1 certainly makes the least sense. The point behind PK crypto is that public keys can be public and private keys must be private. Storing every services' private keys in a shared DB that everyone has equal access to seems counterproductive. - bob: agree, it's probably a bad idea to put key material in /etc/pki/nssdb for more than one reason, though it may make sense for servers to have a shared key store (between other servers) along with a private one.]

Type 3 packages: Libraries

Type 3 packages should open ~/.pki/nssdb and /etc/pki/nssdb read only. If the packages are stand alone, this should be sufficient. If the packages are part of a bigger application, then it should follow the needs of that bigger application.

[Is the order of initialization important? order only has a tangential effect (what slot the DB's wind up in. What happens when the library opens things read-only, but it's linked into an application that wants read-write access? What happens when the lib is init'd first? What happens when the app inits NSS first? Most of these issues are handled in the multi init call, but the short answer is if a database is opened ro and it needs to be opened r/w, we will open the database again, r/w]

Summary of Requirements

In short, all applications will need to open ~/.pki/nssdb and /etc/pki/nssdb. Now, how can this most easily happen?

How NSS decides what to load

NSS delegates the responsibility for managing the list of PKCS#11 modules in the running process to a shared library, sometimes called the "Module DB library". This shared library has a small API (NOT the PKCS#11 API) with functions to output a list of PKCS#11 modules, add a new PKCS#11 module to the list, and delete a PKCS#11 module from the list. The details of how the Module DB library stores its information about the PKCS#11 modules is entirely up to the Module DB library.

In NSS 3.11 and earlier releases, and in NSS 3.12 (when in dbm compatibility mode), NSS's Module DB library uses a Berkeley database file (commonly named secmod.db) to keep the information about the configured PKCS#11 modules.

In NSS 3.12 (when in sql mode), NSS Module DB library uses a plain text file named pkcs11.txt to store the information about configured PKCS#11 modules. All the information in that file is stored as a plain text string.

In NSS 3.x releases to date, the shared library for the Module DB library is also the shared library for NSS's softoken PKCS#11 module, but that is merely a convention, and is not architecturally required. [Bob: In general this is correct. Module DB's are loaded through the same mechanism as PKCS #11 modules. There are flags to say if a module is a Module DB, PKCS #11 module or both. Softoken is, in fact, loaded as both a Module DB and a PKCS #11 module (rather than a single module that can do both)]

During NSS initialization, NSS passes a string of application configuration information to the Module DB library, indicating whether the list is to be managed read-only or read-write, and certain other application preferences, such as whether the PKCS#11 modules should optimize their behavior for maximum speed or minimum memory footprint.

This document proposes to modify the pk11wrap code which loads and invokes the Module DB library. The proposed modification will make it possible for NSS to use a substitute module DB library, different than the one NSS normally uses in the softoken shared library. Such a substitute module DB library will receive the same string that the module DB library in softoken receives, the string which contains the application's NSS preferences. The substitute library can act on this string.

In the simplest case, a Module DB library simply retrieves and outputs a list of PKCS #11 modules and information about them. In the future, a substitute library could utilize some sort of managed database, select PKCS#11 modules according to the appropriate policy, and make system wide decisions on what softoken databases to load. Such decisions could be based on the application's request, the application itself, and the system administrator's policy.

Note: I gather that Bob is about to propose a new substitute Module DB library implementation that will cause softoken to be initialized with multiple slots, one containing the user's own cert and key DBs (e.g. ~/.pki/nssdb) and one containing some "system" DBs, likely containing trusted root CAs (e.g. /etc/pki/nssdb).

[short answer from Bob: yes -- long answer deleted because it was a non sequitor, look in the history if you care about mechanics.]

Certificate/Key Migration

[[ Most secure applications had an existence before the crypto consolidation effort. If those applications already had ways to store keys and certs, then there needs to be a way to automatically or semi-automatically migrate the keys and certs from those applications to the central NSS database. To assist in this NSS provides some built in primitives. --- work in progress ]]

Issues to be resolved

1. 32-bit vs 64-bit packages: Do 32-bit and 64-bit packages both want to use the same directories, ~/.pki/nssdb and /etc/pki/nssdb, or do we want to have 64-bit versions of those directories, such as ~/.pki64/nssdb and /etc/pki64/nssdb or ~/.pki/nssdb64 and /etc/pki/nssdb64, or perhaps yet some other means?
   • [We are already dealing with this by path. The databases and config files stored in /etc/pki/nssdb are platform independent (data sizes and endianness prespecified). The only issue is the load modules. Currently we are specifying loadmodules with implicit paths and depending on the correct libpath to load them. We probably need something better like defining a arch specific path component (like $ARCH) which points to 32bit or 64bit directories as appropriate. Linux supposedly parses these in dlopen, but I haven't had any success. Perhaps including such code in pk11wrap before we load the module would be appropriate)]
2. what about the case of an NFS-mounted home directory that is used from different machines on different OS platforms. Perhaps the path should also attempt to differentiate by architecture more broadly than just 32/64 bits, e.g. linux-x86-32, linux-x86-64, linux-S360, Linux-sparc-v8plus (32), Linux-sparc-v9 (64), Solaris-sparc-v9, etc., etc.
   • [ We are currently dealing with this by the fact that the wrong binary won't load on a foreign OS -- our solution we've been living with for over a decade. This proposal mitigates this quite a bit. It moves configuration of PKCS #11 modules (particularly hardware modules) to a machine dependent directory, not a user dependent directory. Implementing $ARCH above would also provide a strong possible solution this issue as well. ]
3. More work is still needed on the problems faced by libraries.

• How does a library call NSS initialization functions in a way that works well, whether it is the first caller of NSS_Initialize in the process, or is not the first caller.
  • [ Correct, this does not address these issues, though it mitigates some of them. Primarily the issue of how do you deal with a library opening NSS with one set of directories while the app opens them with another. Under this scheme this will only happen with a few apps. We still have the problem that a library opens NSS, uses it, then unilaterally closes it. We could say 'don't close it if NSS isn't open but then you trip over the cause where the library opens NSS first, then the app opens NSS, then the library closes NSS while the app is still using it ]
• How does a library know if it is the first caller of NSS? Does it need to know?
  • [ There is already a call to tell if NSS is already initialized. I don't think knowing that info, however, solves all the problems. The primary problem it doesn't solve is how to deal with shutdown. It probably solves the 97% case, which is bad because we may not trip over the 3% case until someone makes it a firedrill. ]
    • This is the approach I've taken with OpenLDAP libldap - if NSS was already initialized when we started, we will not call NSS_Shutdown when we're torn down. If whoever initialized NSS doesn't clean up properly, it's not our fault and there's nothing we can do to help the situation. But obviously that means setup and teardown have to occur in the correct order in the app.

Other comments

• I don't see anything about this proposal that uniquely ties it to Linux. Seems to me that an improved Module DB Library would be useful on all platforms.
  • [The primarily tie is the suggestion that apps open /etc/pki/nssdb, and the expectation that the OS will supply appropriate code so that a user specific database will be opened when one opened /etc/pki/nssdb. It's true that any OS can implement this proposal. Some OS's may chose a different path for /etc/pki/nssdb [Windows will certainly have to, Mac probably would as well, though they wouldn't need to]. The purpose was to nail down the exact specifics for Linux, and thus the title I initially gave it.]
• in the old dbm days, secmod.db was combined in the softoken PKCS #11 module primarily because that's where the dbm code was. With the sql database, it's possible to separate the moduleDB load from softoken. That might be a good plan for 3.13, removing yet more code from the cryptographic boundary of NSS.
• IMO, hardcoding magic behavior like this (open two locations when only one path was passed into the API) is a mistake. It's obvious that /etc/pki/nssdb is "special" and the API ought to highlight this fact. There should be a distinct API call for initializing the system default location, separate from the call for accessing a user-specified configuration, and every app/caller that needs instance-specific configuration should explicitly call both APIs.
  • [bob:It's only special because the 'system' configured it to be special. I can set *any* directory up to do exactly this without change to NSS. What's important is it's a single roundevous point. If your app has a config file that selects the NSS database, you can force that app into the new scheme by adding sql:/etc/pki/nssdb in that config file. NOTE: you will not find the string /etc/pki/nssdb anywhere inside NSS itself]

• The overall feel I get from reading this page is still that NSS was designed for single-user client usage, and server/library usage are a hasty afterthought. The only way to get away from the NSS_Shutdown problem is by introducing new APIs that are specifically designed for these new use cases, and punting backward compatibility. (I.e., stability of apps that only use the existing APIs cannot be guaranteed in this new environment.)

[bob: This is not supposed to address *that* issue. The idea here is that control of what gets opened or not now rests with the OS (under administrator control). Many of the issues you were looking for are in the multinit proposal]

E.g.: use a distinct API call to initialize the default /etc/pki/nssdb database. Refcount all DB opens. Use a distinct API call to close each database, instead of a one-call-destroys-everything Shutdown. Use an atexit() or similar destructor to make sure that Shutdown on hardware tokens occurs if an app exits without cleaning up nicely. (You're still SOL if you get SIGKILL but you always were...)

[bob: several of these are in the multi init proposal] here