https://blog.startcom.org/?p=145

--
Regards

Signer: Eddy Nigg, StartCom Ltd.
Jabber: start...@startcom.org
Blog:   https://blog.startcom.org

Eddy,
        That reseller's ability to sell Comodo certificates has been
suspended while we investigate why they are apparently not fulfilling their
contractual obligations to us.
        We revoked your certificate for mozilla.com.

Regards
Robin Alden
Comodo

On 12/23/2008 03:15 AM, Robin Alden:

How can you outsource such a critical part as domain control validation
to a reseller is a complete mystery to me! Your controls (if any) have
completly failed.

And apparently if the fish stinks at the tail, chances that the head is
rotten too are pretty high.

I suggest to revoke ALL certificates from this reseller and perform an
urgent review of your policies and implementations in relation to
resellers at all. Other steps might be needed as well, you know better
than me.

--
Regards

Signer: Eddy Nigg, StartCom Ltd.
Jabber: start...@startcom.org
Blog:   https://blog.startcom.org

Eddy,

As I noted in my prior correspondence, Comodo has undertaken an internal
review of the Certstar reseller account.  We have informed CertStar that
their email violates their contractual obligation to refrain from sending
unsolicited emails and that their email could be interpreted as misleading
and confusing to the customer.  During our review, we discovered that
Certstar had apparently issued a certificate to mozilla.com without
validating control of the domain.  We immediately revoked the certificate
(prior to your posting) and have suspended Certstar's reseller activities
until our investigation has been completed.  Please let me know if you have
any further problems.  

Regards
Robin Alden
Comodo

On 12/23/2008 03:38 AM, Robin Alden:

A.) The certificate was revoked after posting to this list.
B.) Your CA also issued a certificate to startcom.org without validating
anything.
C.) Yes, I think we have a problem  of a wider scale here. This is not
about me personally.

--
Regards

Signer: Eddy Nigg, StartCom Ltd.
Jabber: start...@startcom.org
Blog:   https://blog.startcom.org

I believe that this is a symptom of faulty internal control, and that
this is direct evidence of not living up to the Mozilla CA policy.

I advocate at least temporarily removing the trust bits from Comodo
until a new external audit can be completed, with an eye toward
ensuring that Comodo, not the reseller, perform the domain
validations.

-Kyle H

There are two general reasons for pulling a root, to address a clear and
present danger to Mozilla users, and to punish a CA and deter others. My
concern right now is with the former. I see at least three issues in
relation to that:

1. Issuance of further non-validated certs by this reseller. Comodo
seems to have addressed this by suspending the reseller's ability to get
certs issued. (I can testify that this is the case, as I tried to
duplicate Eddy's feat earlier today and got my uploaded CSR rejected.)

2. Potential problems with certs already sold through this reseller.
Comodo should investigate this and take action if needed. (This need not
necessarily require revoking all certificates associated with the
reseller; for example, the existing certs and their associated domains
could be re-validated, the registered domain owners could be notified of
the potential for bogus certs floating around, etc.)

3. Potential problems with other Comodo resellers. I'm not going to tell
Comodo how to operate its reseller network, but they certainly should
take a look at whether and where this might be a problem with other
resellers, and how they could revamp their systems to reduce potential
problems with resellers.

Pulling a Comodo root will knock out Firefox, etc., access to thousands
of SSL sites, maybe tens of thousands. Given the disruption that would
cause, the final decision on this IMO should be made in conjunction with
the Firefox security folks. From my point of view I'd wait on more
information regarding items 2 and 3 above before making a recommendation.

Frank

--
Frank Hecker
hec...@mozillafoundation.org

I agree with all three of these issues -- it's not *just* this
reseller, it's potentially the entire reseller network, and it's
entirely possible that the problem exists across multiples.
(Especially if Comodo delegates full Registration Authority capability
without verification, which seems to be the case -- though they could
have simply issued a sub-CA certificate.)

It occurs to me that there is no facility in Firefox or other Mozilla
products to provide an explanatory dialog that there's an issue, and
such a facility would be extremely useful at this point.  Being able
to print a message to the user like "The Mozilla Foundation has
identified issues with the trusted root that issued this certificate
which prevent Firefox from being able to guarantee that this is truly
the site to which you intended to go.  While it is unlikely that this
is a widespread problem, and an attack would rely on more technical
intrusions into the network, the nature of these issues requires that
you be warned of this circumstance so that you can exercise
appropriate levels of caution.  The Mozilla Foundation is working with
the trusted root to resolve these issues." would help a lot.

(I word it like that because in order for an attacker to succeed he
would need to also hijack DNS, or place a entry in the user's hosts
file.)

Of course, this would be an NSS change (the addition of a 'trust
suspended' bit, in addition to the trust flags), a Firefox/Thunderbird
code change (to look for that bit), and a chrome change (to explain
what's going on).  Or even a check for the name in the hosts file to
see if it's overridden from DNS.  Placing an entry in a user's hosts
file is easier than hijacking DNS, or at least it's supposed to be.
I'm pretty sure we're all aware of the fairly recent cache poisoning
attack, but the people who write BIND pushed out a change to protect
against it very rapidly.

The addition of a 'trust suspended' bit is primarily unlikely to
happen, though, because there's too much inertia in the way things are
to do what needs to be done to fix the authentication system to allow
a root to stay in while the user is potentially at risk.

-Kyle H

On Mon, Dec 22, 2008 at 9:09 PM, Frank Hecker

Hi all,

A glitch in our validation system has today caused a certificate to be
issued to a person who successfully abused our system.

We have now strengthened our domain validation system so that such
abuse cannot happen again. Comodo has handled this issue in a
professional way by invoking the certificate immediately after issuing
and contacting Certstar.

Again, I cannot stress enough how seriously we take this issue and I
would like to apologize to the Mozilla organization for the mis-issue.

--
kind regards,
Patricia, Certstar ApS

Hi Patricia,

patri...@certstar.com schrieb:

just curious: How do you normally validate domain ownership?

TIA,

Thorsten

On 12/23/2008 10:48 AM, patri...@certstar.com:

It's not me who abused your system, it's your company which sent out
illegal, misleading emails to our customers. Since I couldn't find out
who stands behind your site I had to get a certificate. There was no
abuse, there was only use-and-get.

- No affiliation published, the fact that you are a reseller for Comodo
is nowhere noted.
- No subscriber agreement or notice presented to subscribers.
- No control validation of the domain name. Just pay-and-go.

I didn't had to do anything to overcome any validation whatsoever. There
was no abuse!

Yes, after I posted my article here at the mailing list.

--
Regards

Signer: Eddy Nigg, StartCom Ltd.
Jabber: start...@startcom.org
Blog:   https://blog.startcom.org

For those interested, Frank opened a bug to investigate this incident:

https://bugzilla.mozilla.org/show_bug.cgi?id=470897

--
Regards

Signer: Eddy Nigg, StartCom Ltd.
Jabber: start...@startcom.org
Blog:   https://blog.startcom.org

On 12/23/2008 07:09 AM, Frank Hecker:

As long as this site keeps operating, our customers are still being let
to believe that they have to renew their certificate with them. This is
only a reminder about how it started at all. CAs and their customers are
still taking damage from the previously sent messages.

You shouldn't notify the subscribers or domain name owners, but the
relying parties. How to do that is up to you and Comodo I guess.

Comodo not only shouldn't just investigate and take action, Comodo needs
to report publicly about their findings and full report about the
actions taken. This isn't a suggestion of resolution about this
incident, it's the transparency I expect from them at this hour.

I'm not advocating removing their root, however we must assess the risk
which is potentially caused to the relying parties. There may be
thousands of sites which received certificates without validating them.

Disabling the trust bits of "AddTrust External CA Root" could be a
temporary measure to prevent damage to relying parties until Mozilla
receives full report and disclosure from Comodo about its resellers and
conclusion of their investigation.

Additionally instead of just yanking a root as a deterrent and
punishment, as you mentioned above, I'd prefer to receive a commitment
from Comodo to address other issues noted during the last discussion -
mainly those listed in the "Problematic Practices" document. Considering
that OCSP in Firefox is set to soft-fail, an issue with their OCSP
server could still circumvent bogus sites for potentially the next five
years. A full review of their current status in NSS and their practices
might be advocated too.

--
Regards

Signer: Eddy Nigg, StartCom Ltd.
Jabber: start...@startcom.org
Blog:   https://blog.startcom.org

On 12/23/2008 09:09 AM, Kyle Hamilton:

Or be a WiFi operator. This was the attack vector of
https://bugzilla.mozilla.org/show_bug.cgi?id=460374

I think this to be an interesting idea and should be considered.

--
Regards

Signer: Eddy Nigg, StartCom Ltd.
Jabber: start...@startcom.org
Blog:   https://blog.startcom.org

Actually Nelson opened this bug.

Frank

--
Frank Hecker
hec...@mozillafoundation.org

Patricia, I believe it's important to realize a couple of things:

1) An unsolicited commercial email (UCE) message was sent from your
company to the party in question suggesting that there already existed
a relationship between your company and the party in question.  This
is obvious from the verb 'renew' in the original message -- a
non-party to the original certificate can't renew "on behalf of" the
original certificate issuer.  If they can, there's a major problem.

1a) The message from your company, and in fact the entire process up
to and including paying for the certificate which your company issued,
did not expressly claim an affiliation, and the party in question went
through the process of "renewal" with the intent of figuring out which
CA's services were being advertised via UCE.

2)  The party in question (Eddy Nigg) is the founder of another
Certifying Authority, with a root which participates in the Mozilla
root program and which is also included in the root store.

3) It falls within the concept of "due diligence" for a
security-conscious warden of a public trust to recognize the lack of
actual domain control verification in a trusted certificate issuance
process, and *for the purposes of verification and public
notification* obtain a certificate which could be shown absolutely to
have been issued in violation of the standards of at least one of the
root programs that the issuer (or issuer's trust-delegator)
participates in.

3a) If this had not been identified, someone else would have found it
eventually -- and the reaction to a gross violation of security
standards is to immediately believe that the worst-case scenario has
occurred: that it had already been found and exploited by at least one
other person.

4) The end-users (referred to as "relying parties") are well-served by
this identification of completely bogus credentialing.

5) It's also important to recognize the following (the first comment
on https://bugzilla.mozilla.org/show_bug.cgi?id=470897):

  -------  Comment #1 From  Reed Loden [:reed]   2008-12-23 01:35:37
PST  -------

The same company that Eddy was able to get the mozilla.com cert from (Certstar)
has been endlessly spamming webmas...@mozilla.org since the beginning of
December complaining that one of our SSL certs had "expired" and needed to be
"renewed" (both of which were false). They have continued to spam us almost
daily. :(

  ------ end comment ------

Yes, the bare facts are that a user exploited the lack of verification
in your certificate issuance process.  However, the lack of
verification in your process violates at least one contractual
obligation that your company is required to uphold, and reduces the
value of both Comodo's root AND the commercial Certifying Authority
structure in general.  Because the user founded another commercial CA
which is trusted by Mozilla NSS by-default, this user was operating in
a manner to verify that the commercial CA structure was in fact
secure; the trust afforded this user's CA is demonstrably harmed by
your lack of verification.

I do not take this situation as lightly as you appear to wish that
everyone would.  The initial fault is YOUR COMPANY'S (and that fault
reverbrates up to Comodo, since it delegated trust to your company),
and the fact that you're attempting to shift the blame onto the user
shows that you are absolutely untrustworthy to run or be the public
face of any commercial certificate-issuance service.

Because of this, my recommendation that Comodo's trust bits be removed
until a full audit of their practices (and a full audit of all issued
certificates) stands, and I am that much more resolute in my belief.
It may be that the integrity of Comodo's root has been irreparably
damaged by your company's malfeasance; if this is the case, I
certainly hope they rake you over the coals.

-Kyle H

Do you mean the UTN-UserFirst-Hardware root? According to the screenshot
on your blog post, that's the root the bogus cert chains up to. Also, if
we were to take action of this general sort (as a hypothetical), what
about adding the PositiveSSL CA cert to NSS with the SSL trust bit
disabled; wouldn't that accomplish the same purpose, without interfering
with other parts of the hierarchy under the UTN-UserFirst-Hardware root?
(I seem to recall we've discussed this sort of thing in the past.)

Also note that any "suspension" of a root would last at last 1-3 months,
since that the typical interval between security updates for Firefox and
other Mozilla-based products.

Frank

--
Frank Hecker
hec...@mozillafoundation.org

On 12/23/2008 03:05 PM, Frank Hecker:

Thanks for that. More into this story...

...all our employees coming the our network are able to retrieve the
certificates and all the rest from the "control panel" of this site.
Again, no questions asked. Apparently the only things one needs is to
fake our gateway IP address to get there.

It's getting weirder with the second. If they don't shut that site, we
can perhaps just publish the private key for the mozilla.com certificate
as well so everybody can enjoy it.

--
Regards

Signer: Eddy Nigg, StartCom Ltd.
Jabber: start...@startcom.org
Blog:   https://blog.startcom.org

Either way, it's enough of a problem that a bug needs to be opened,
and it needs to be followed upon sooner than later.

(My opinion, of course.  Comodo's gotta be breathing a sigh of relief
that I don't work for the Mozilla Foundation, else I would have yanked
the trust bits first and figured it out later.)

This incident shows anew the lack of policy/procedure of what to do in
the case of a gross breach of the root agreement, though.  The entire
point of certificates in the first place is "user security", not "user
convenience".  (If this weren't the case, none of us would have any
desire to get certificates in the first place.)  Your reaction seems
to be based on placing "user convenience" over "user security", and
(again, my opinion) I don't believe that this is appropriate at all.

-Kyle H

On Tue, Dec 23, 2008 at 5:05 AM, Frank Hecker

In the past, lots of good stuff has been done that handles the ascension
to the root list of Mozilla.  c.f. the policy.  But not so much is
written about *what happens afterwards*.  This recent thread has been
such a case, and has afforded an opportunity to make some notes on what
might be suitable as a practices page on the wiki.

The reason for doing this, and adding yet more boring verbage to the
mass of words, is that it can prevent uneccessary legal costs by
providing clear guidepaths, and clear signals of seriousness.

Here's what I see:

1.  How to file a dispute against Mozilla and/or a CA.  This seems
fairly easy;  file a bug in bugzilla and mark it as already described
here: https://wiki.mozilla.org/CA:How_to_apply .  (With mods:  Severity:
  as appropriate.)

2.  What is the practice on revocation or un-trusting on roots?  This is
is perhaps a headline case of a dispute, so possible merits its own
notes.  Frank has suggested the test:

    a.  there is a clear and present danger to Mozilla users, or
    b.  to punish a CA, and/or to deter others.

I would suggest that (b) be modifed slightly to give it a basis, which
in this case might be "for breaches of policy or practices".  The
Policy, pt 4, gives the authority, as well as some examples, and adds
another test case:

    c.  where certificates cause technical problems with mozo software.

3.  How to resolve a dispute.  This is a Mozilla action &
responsibility.  Reverse-engineering and referring, I would suggest this
as a teaser:

   a.  The CA certificate "module owner" at Mozilla foundation is
responsible.  Ref, the policy, pt 15.
   b.  The dispute is investigated and ruled on by module owner.
   c.  The ruling is listed in the bug report above.
   d.  Many disputes will be dealt with by communication, and no ruling
will be required.  This will create a default "closed, no action" ruling.

4.  Finality.  What happens if we disagree with the decision of the
module owner?  In the policy, it says "CAs or others objecting to a
particular decision may appeal to mozilla.org staff, who will make a
final decision."  Ref, policy, pt 15.

I would wonder about this;  google suggests that "staff" is as listed here:
http://www.mozilla.org/about/staff
but that seems out of date.  Also, due to the absence of this forum in
the public eye, I doubt it musters the credibility we need in dispute
review where the legal and contractual significance is high.  E.g., is
there any way we can review the decisions they made in the past?

There are several possibilities:

   (i)   Ruling is final.
   (ii)  Mozilla.org staff, policy, pt 15.
   (iii) Review by board of Mozilla Foundation.
   (iv)  Review by some independent party.
   (v)   Review by forum at law: courts, or Arbitrator.

Personally, I would plumb for (iii) and suggest the Mozo Foundation
board as the next step.  It is expensive, but available.  The directors
already have fiduciary responsibility, and can thus deal with the
significance.  It is also aligned with the review of the manager
concerned, the policy and the general contractual issues.

End!  I'd encourage others to comment!

If nobody has any objections, I can add that as a page into the wiki as
an informal document of practices, including or without those questions.

iang

On 23/12/08 06:09, Frank Hecker wrote:

My intent is to balance the disruption that would be caused by pulling a
root vs. the actual security threat to users. Right now we have no real
idea as to the extent of the problem (e.g., how many certs might have
been issued without proper validation, how many of those were issued to
malicious actors, etc.).

Frank

--
Frank Hecker
hec...@mozillafoundation.org

I don't have time right now to address your whole post, but I did want
to comment on one point.

There is no longer a "mozilla.org staff" as traditionally constituted.
 From a Mozilla community point of view the closest present-day
equivalent is the system described at

https://wiki.mozilla.org/Module_Owners_Activities_Modules

Basically there is a group of people that oversee the Mozilla module
ownership system and the module owners within it. That group in turn
operates under general policy structures set by Mitchell Baker and
Brendan Eich, and as you note the ultimate legal authority resides in
the board of the Mozilla Foundation (of which Mitchell and Brendan are
members).

For more background on this see

http://www.mozilla.org/hacking/module-ownership
http://blog.lizardwrangler.com/2008/05/30/governance-and-module-owner...

Frank

--
Frank Hecker
hec...@mozillafoundation.org

This was the first thing that occurred to me as well. Can anyone
(Nelson?) tell us definitively if this is possible? I would assume it
is, but it would be good to get confirmation.

Gerv

Are we going to receive information from Comodo regarding how many  
other Comodo resellers may be in a similar position to Certstar?

Are we going to receive information from Certstar as to how many other  
certs may have been issued in error?

How do we verify the claims from Comodo or Certstar?

Gen

These are, of course, the reason why I want the trust bits removed.
It's going to take more than a month to get this all straightened out,
and that's "over a month" that we have no idea if Comodo certificates
can be trusted.

-Kyle H