At 4:44 AM +1300 2/26/08, Peter Gutmann wrote:
... Ten years ago the rule-of-thumb estimate for a FIPS 140 level 1 eval was $100K and 6-12 months no matter who you went to. In 2006 (last time I checked) the rule-of-thumb estimate for a FIPS 140 level 1 eval was $100K and 6-12 months no matter who you went to. So it looks like we already have a well- established oligopoly with FIPS 140 evals, the only difference is the nomenclature.
BBN developed the first crypto module evaluated at level 3 under FIPS 140-1. The fee we paid to the eval lab was about $40K, and the elapsed time was about 6 months. That was more than ten years ago, but we're also talking about a much more thorough eval. The cost to a developer is, of course, higher, when you add in the time required to prepare the necessary documentation. Also, if the developer is not experienced in secure development practices, the likelihood increases that the eval will not go smoothly, and thus the costs will be higher. Maybe what you're reporting is the result of more developers, with less experience secure crypto module development experience submitting products that are "not quite ready."
...
>If you think that FIPS-140-* is a target-rich environment, then please try toseriously propose something better. I understand NIST and its partners are looking to evolve into FIPS 140-3 from FIPS 140-2.It depends on what you want from FIPS 140 (and I'm talking specifically FIPS 140 for software modules here, which is the form that 99.9% of users encounter it in ). Some people have said you get "assurance", but the assurance you're getting here is a high level of assurance that the vendors were desperate enough for government sales that they shelled out a large amount of money in order to get a ticket to ride the gravy train.
Sotware modules are easier for a developer to create and submit, and this too may cause more submissions that are less well thought out. If you're building hardware there may be more of an effort to "get it right" in the planning stages, because the costs of redesign are potentially higher.
The problem is that like "religion" or "morals", "assurance" is a loaded term
and can be interpreted to mean almost anything. Without some basic definition
of what's meant by "assurance", it's not really possible to reason about this.
So here's an attempt to nail it down: For me (and I would guess most end
users) assurance is being able to sleep at night knowing there's little chance
of anyone compromising my system ("things that should happen do happen, things
that shouldn't happen don't happen").
Assurance (in the security context) is not as squishy a term as your suggest. Security assurance is generally characterized as confidence that an implementation meet certain objective security-relevant criteria. These are not interoperability criteria. It is distinguished from security functionality, which refers to the set of functions (features) ascribed to a product (without confidence that these features operate as promised).
Let's see what FIPS 140 gives you to help you sleep at night. It tests some crypto mechanisms, but ignores others. If you're buying an IPsec product it doesn't test all the mechanisms needed for IPsec. If you're buying a TLS product it doesn't test the mechanisms needed for TLS. If you're buying... well, you get the idea.
For a protocol such as TLS or IPsec, crypto module security assurance does not encompass most aspects of what a users sees as "correct" operation. It makes sense to look for both functionality testing and security assurance evaluation, if you care about both functionality and security.
Compare this to the example I gave earlier of performing a TLS exchange with Amazon. This performs an in-depth test of all the crypto algorithms (corresponding to the FIPS algorithm tests, including ones that FIPS ignores), and the crypto mechanisms (many/most of which FIPS again ignores). In other words simply by connecting to Amazon using TLS and ordering a "Scrubs" DVD for $19.95 I'm getting more comprehensive algorithm testing than I can for a five- figure sum with the FIPS algorithm tests.
nonsense! what you get is confidence that the SSL/TLS implementation at Amazon and in your browser work compatibly, even if both happen to be "wrong." In fact, the test you propose checks only a small subset of the SSL/TLS spec, as it verifies only that the RSA key transport method of key management works, none of the other key management methods defined by the RFCs. So, maybe what you meant to say was that the proposed test regimen verifies that a protocol and associated crypto work compatibility with an implementation of the most common subset of the crypto modes and protocol features deployed. That's a useful finding, but let's not confuse it with protocol standard conformance or security assurance.
I won't bother to respond to the rest of your observations, since they repeat many of the themes I touched upon above.
It is silly to ask FIPS 140 to address protocol interop issues; that is simply not in scope for a generic crypto security eval criteria. So let's forgo all the arguments along those lines.
Vendor claims about security sometimes (often?) prove to be wrong. Thus is makes sense to seek external, independent assurances about security, if you care. The fact that FIPS 140 addresses only a subset of those claims is also not a basis for criticizing a generic crypto module evaluation criteria like FIPS 140. Common Criteria eval of products is more encompassing, but also MUCH more expensive.
So, IF one cares about crypto security, THEN FIPS 140 eval is the best option for now. That does not make it ideal, but it also does not make it as bad as some comments have suggested, and it is unfair to criticize it for not doing something that it has never promised to do.
Steve
Note Well: Messages sent to this mailing list are the opinions of the senders and do not imply endorsement by the IETF.