Poll
On Windows Vista I use
Firewall Challenge
- Introduction
- Results and comments
- Testing levels
- List of products
- My leaks
- Frequently asked questions
Results and comments
Contents:
Products' ratings
The table below sorts the tested products by their total score, which is displayed in the Product score column.
The second number in this column, separated by a slash, represents a number of tests that were in the system when the given
product was tested, i.e. the number of tests with a valid test result value (other value than N/A).
This table also shows the exact version of every tested product. The Level reached column presents the highest
level that the product reached in Firewall Challenge. If it passed all levels, this number is suffixed with a plus sign.
For products that score at least 80% in Firewall Challenge, the Recommendation column contains links to the online stores
or products' webpages of the vendors that we have affiliate agreements with. If you click on any of these links and then buy
the target product or other product offered on the target webpage, we will profit from it. This is one of the ways how you can
support this project. The PDF document icon allows you to download the testing report in PDF format for the tested product.
| Product | Product score | Level reached | Protection level | Recommendation | Report | |
|---|---|---|---|---|---|---|
| Online Armor Personal Firewall 3.0.0.190 | 93% / 84 | 10+ | Excellent | GET IT NOW! |
||
| Comodo Internet Security 3.5.55810.432FREE | 90% / 84 | 10+ | Very good | GET IT NOW! | ||
| Online Armor Personal Firewall 2.1.0.131 FreeFREE | 89% / 73 | 10 | Very good | GET IT NOW! |
||
| Outpost Firewall Pro 2009 6.5.2358.316.0607 | 89% / 84 | 9 | Very good | GET IT NOW! | ||
| Kaspersky Internet Security 2009 8.0.0.454 | 87% / 73 | 10+ | Very good | GET IT NOW! |
||
| Netchina S3 2008 3.5.5.1FREE | 86% / 73 | 9 | Very good | N/A | ||
| ZoneAlarm Pro 2009 8.0.020.000 | 86% / 73 | 10+ | Very good | GET IT NOW! | ||
| PC Tools Firewall Plus 4.0.0.45FREE | 85% / 73 | 10+ | Very good | GET IT NOW! |
||
| Privatefirewall 6.0.20.9 | 82% / 84 | 10+ | Very good | GET IT NOW! | ||
| System Safety Monitor 2.3.0.612 | 77% / 62 | 7 | Good | Not recommended | ||
| Jetico Personal Firewall 2.0.2.7.2311 | 72% / 84 | 10+ | Good | Not recommended | ||
| Norton Internet Security 2009 16.0.0.125 | 71% / 73 | 7 | Good | Not recommended | ||
| Lavasoft Personal Firewall 3.0.2293.8822 | 70% / 73 | 7 | Good | Not recommended | ||
| Dynamic Security Agent 2.0.11.22FREE | 62% / 71 | 7 | Poor | Not recommended | ||
| Comodo Firewall Pro 2.4.18.184FREE | 55% / 73 | 7 | Poor | Not recommended | ||
| Webroot Desktop Firewall 5.8.0.25FREE | 54% / 84 | 7 | Poor | Not recommended | ||
| Trend Micro Internet Security Pro 17.0.1224 | 27% / 73 | 4 | None | Not recommended | ||
| G DATA InternetSecurity 2008 | 19% / 73 | 3 | None | Not recommended | ||
| FortKnox Personal Firewall 2008 3.0.195.0 | 16% / 62 | 2 | None | Not recommended | ||
| Look 'n' Stop 2.06 | 15% / 62 | 2 | None | Not recommended | ||
| McAfee Internet Security 2009 10.0.209 | 12% / 73 | 2 | None | Not recommended | ||
| F-Secure Internet Security 2008 8.00.101 | 12% / 73 | 2 | None | Not recommended | ||
| Panda Internet Security 2008 12.01.00 | 12% / 73 | 2 | None | Not recommended | ||
| Avira Premium Security Suite 8.1.00.206 | 11% / 70 | 2 | None | Not recommended | ||
| Rising Personal Firewall 2008 20.59.10 | 11% / 73 | 2 | None | Not recommended | ||
| BitDefender Internet Security 2009 12.0.10.2 | 7% / 73 | 1 | None | Not recommended | ||
| AVG Internet Security 8.0.93 | 6% / 62 | 1 | None | Not recommended | ||
| Ashampoo FireWall FREE 1.20FREE | 5% / 73 | 1 | None | Not recommended | ||
| Windows Live OneCare 2.0.2500.22 | 5% / 62 | 1 | None | Not recommended | ||
| Sunbelt Personal Firewall 4.6.1861.0 | 5% / 84 | 1 | None | Not recommended | ||
| BullGuard Internet Security 8.0.0.13 | 4% / 70 | 1 | None | Not recommended | ||
| ESET Smart Security 3.0.672.0 | 4% / 73 | 1 | None | Not recommended | ||
| iolo Personal Firewall 1.5.2.7 | 3% / 62 | 1 | None | Not recommended | ||
| Filseclab Personal Firewall 3.0.3.8982FREE | 3% / 73 | 1 | None | Not recommended | ||
| Steganos Internet Security 2008 7.5.509 | 3% / 70 | 1 | None | Not recommended | ||
| Mamutu 1.7.0.23 | 2% / 84 | 1 | None | Not recommended | ||
Detailed results
The following links take you to pages with detailed products' results on each level. The level pages also contain important information about the given level and short information about its tests.
- Level 1 – Breakout2, Coat, ECHOtest, Kill1, Kill2, Leaktest, PerfTCP, PerfUDP, Tooleaky, Wallbreaker1, Yalta
- Level 2 – AWFT1, DNStest, Ghost, Jumper, Kill3, Kill3b, Kill6, Wallbreaker3, Wallbreaker4
- Level 3 – AWFT3, AWFT4, DNStester, Kernel1, Kill3f, Kill4, Kill7, SSS2, Suspend1, Thermite
- Level 4 – CopyCat, CPIL, CPILSuite1, Kernel1b, Keylog1, Kill3e, Kill8, Kill9, SSS, Suspend2
- Level 5 – Breakout1, CPILSuite2, Crash1, Crash2, Crash3, Crash4, Kernel2, Kernel3, Keylog2, Kill3c, Kill3d, VBStest
- Level 6 – CPILSuite3, Crash5, Crash6, DDEtest, ECHOtest2, FireHole, Flank, Kernel4, Keylog3, Keylog4, Kill10, Kill11, Runner
- Level 7 – BITStest, FireHole2, Keylog5, Keylog6, Kill12, OSfwbypass, Runner2, Schedtest, SSS3
- Level 8 – Kernel4b, Kernel5, Keylog7, Kill5, NewClass, Schedtest2, SockSnif, SSS4
- Level 9 – Crash7, Driver Verifier
- Level 10 – BSODhook, ShadowHook
Interpretation of results
The paid version of Online Armor Personal Firewall 3.0.0.190 leads the challenge with 93% and it is the only product that reached the Excellent level of protection. Online Armor 3 is followed by Comodo Internet Security 3.5.55810.432 with 90%, and Online Armor Personal Firewall 2.1.0.131 Free and Outpost Firewall Pro 2009 6.5.2358.316.0607 with 89%. However, Online Armor Personal Firewall 2.1.0.131 Free was not tested against the latest version of the tests, hence it is likely that its comparable result would be worse.
It seems that Firewall Challenge tests make a big difference between really good products and the rest of the world. Most of the products are filtered in very low levels which means that they probably miss some critical features.
However, it is crucial to know what does it mean if a product succeeds in our tests and what does it mean if it fails. Before you start interpreting the results, you should be familiar with the information on the index page, especially with the methodology and rules. You should also know which kind of products do we test before you start to interpret the results. We have received a lot of reactions from people who are not familiar with that information and simply do not understand the results and misinterpret them. All the tested products have one common feature – the application based security model. In combination with their packet filtering capabilities, the tested products attempt to block attacks from other machines on the network as well as attacks performed by malicious codes that might run inside the protected machine. This is definitely not an unusual situation. People who use email clients, instant messengers, or web browsers face attacks that exploit the vulnerabilities in this kind of software very often. It happens that a malicious code gets inside the machine. And then it may try to install itself silently to the system, to steal users' data or sniff their passwords, or to join the target machine to the botnet. This is what the products we test, called personal firewalls, want to prevent. This is why they are used. The problem is that although the goal is common, not all the products implement a sufficient protection.
We require the personal firewalls to prevent data and identity theft. They should implement a packet filter functionality to prevent direct online attacks – i.e. not to let the malware get in. Personal firewalls should control the software installed on the computer to prevent the malware to integrate itself into the operating system. Then the malware should not be able to get the user's private data, thus anti-sniffing, anti-keylogging and personal data protection features should be implemented too. And even if the malware succeeded to collect the information it should not be allowed to send it outside the protected system and this means the implementation of the outbound network traffic control. To achieve all these is much harder task than it seems. The protection system also has to prevent attacking trusted processes and other components in the system. Otherwise, the malware would be able to use trusted parts of the system to integrate into the operating system, to collect or steal sensitive data and/or to send the data outside the system without being noticed. So the next feature that is required here is a control of untrusted processes' activities and that is the hardest task for personal firewalls. It also includes the implementation of self-protection mechanisms because the malware should not be able to terminate the protection, which implies some other features to be implemented and so on.
So, what does it mean if the product fails even the most basic tests of our challenge? It means that it is unable to do what its vendor claims it can. Such a product can hardly protect you against the mentioned threats. On the other hand, if the product succeeds in all our tests, it does not mean that it is perfect. Our tests are focused on the security and performance, but there are many other aspects important for the users. It should be also noted that although our testing suite is quite large, it is not complete and there are many other ways to bypass personal firewalls. We are working constantly on extending the suite to be able to provide more accurate information about the security of the tested products. If the tested product fails only a few tests in our challenge, it still might be a great product. This is why we can recommend, from the security point of view, the products that reached at least 80% score in the challenge. You should try them yourself and choose the one best for you, the one that you would be happy with, the one you would be able to configure and use everyday.
Vendors' responses
We have received the following responses to Firewall Challenge:
Agnitum Ltd. – the vendor of Outpost Firewall Pro
2008-07-04 (Outpost Firewall Pro scored 99% and took 1st place): As always, we are grateful to the Matousec team for their job! The latest tests show there is still a gap for improvements in Outpost's proactive protection, but it's just a 1% gap, and we feel fully capable to accomplish the 100% result shortly.
Pavel Goryakin,
PR Manager, Agnitum Ltd.
2008-03-18 (Outpost Firewall Pro scored 91% and took 4th place): We'd like to thank Matousec – Transparent Security Lab for doing research on Agnitum's product – Outpost Firewall Pro 2008. Thanks to the results revealed in the latest tests (based on the lab's new methodology), we'll make related corrections and updates in the next version of our software. Those should help us resist new threat types discovered in Matousec's report – to the benefit of security products users.
Alexey Belkin,
Chief Software Architect,
Agnitum
AVG Technologies – the vendor of AVG Internet Security
2008-04-24 (AVG Internet Security scored 6%): Thank you for including our product in your test. Our firewall was never designed to be used in a stand-alone environment, but as a part of an integrated endpoint security solution that delivers our recommended layered approach to security. We will, however, review these results and implement those changes we believe to be necessary in future upgrades.
Larry Bridwell
Global Security Strategist
AVG Technologies
BitDefender – the vendor of BitDefender Internet Security
2008-06-07 (BitDefender Internet Security scored 4%): Thank you for testing BitDefender Total Security 2008. Bitdefender Firewall doesn't exist as a stand-alone product because it has been designed to work and be sold with the Antivirus as a package. These tools you have been using in your test are automatically caught by our Antivirus module. We will analyze the results in this test and plan to cover this type of test in the future.
Rudi-Gabriel Bedy
BitDefender
Our response: According to VirusTotal, not even the latest version of BitDefender detects our tests. But it is a good thing that it does not detect them because they are not malware and you would scare your users with false positive alerts if they were marked as infected.
Comodo Security Solutions, Inc. – the vendor of Comodo Internet Security
2008-12-01 (Comodo Internet Security scored 90% and took 2nd place): Thank you for testing our product. We are happy to report that the bug identified during your tests is fixed.
Egemen TAS,
Sr. Research Scientist,
COMODO
2008-05-21 (Comodo Firewall Pro scored 95% and took 1st place): Thank you very much for pointing out the "windows shutdown race condition" bug introduced with the tested version, which is the main reason for failing in SSS tests. We will be addressing this with the planned release on 05/20/2008. Keep up the good work.
Egemen TAS,
Sr. Research Scientist,
COMODO CP Inc
2008-03-18 (Comodo Firewall Pro scored 98% and took 1st place): We do appreciate the good work. We believe security researchers such as you guys, who try to poke holes in our products, are going to make them stronger and make us give our users even better products. Keep up the good work!
Egemen TAS,
Sr. Research Scientist,
COMODO Research Labs.
Emsi Software GmbH – the vendor of Mamutu
2008-11-29 (Mamutu scored 2%): In our opinion Mamutu is completely misplaced in this test. Mamutu is not a firewall, but a behavior blocker, designed to detect and block real malware samples only, not to pass a firewall leaktest. For us, the test results are useless because the product was tested for features that Behavior Blockers are not intended to provide in general. In oposite to Matousec, we think that Firewalls, Behavior Blockers and HIPS are not the same type of software and therefore they can not be tested and compared as they were the same.
Christian Mairoll
Managing Director
Emsi Software GmbH
Our response: We are testing a specific kind of security software for which we defined the term "personal firewall". A product must meet some fixed criteria in order to be included to our project. The main criterion is to implement a process-based security. Firewall Challenge is designed to test personal firewalls, HIPS products, behavior blockers and other behavior based systems. Mamutu met all the required criteria and hence there was no why not to include Mamutu to our project after we received several requests from our visitors. All the products included to our project implement similar features. These security features are tested in our project. We believe that using a set of open tests is the only objective way to compare all the products that implement the very same features. There are various tests used in Firewall Challenge, only a part of the used testing suite is based on leak-tests.
Filseclab Corporation – the vendor of Filseclab Personal Firewall
2008-06-18 (Filseclab Personal Firewall 3.0.0.8686 scored 1%): Thank you for testing our firewall. Filseclab Personal Firewall 3.0.0.8686 is designed for inbound protection, it is worse for outbound protection, we recommend our users to use the Anti-virus or Anti-spyware software to protect the outbound. However, we will make a new version to improve those features in the end of this year.
Bright Chu
Filseclab
Jetico, Inc. – the vendor of Jetico Personal Firewall
2008-07-07 (Jetico Personal Firewall 2.0.2.4.2264 scored 78%): We do appreciate your testing efforts. Your test suite reveals weaknesses in security software and helps vendors to make more reliable products. As soon as you added performance tests, please publish full technical specifications of your test system along with test results.
Nail Kaipov
Jetico, Inc.
Our response: Currently, we have only two performance tests in FWC, which are designed to compare the network performance of the products on 100 Mbit full duplex Ethernet. These tests are performed on the testing system without the tested product installed and also with the tested product installed. The measured results are compared then.
2008-03-27 (Jetico Personal Firewall 2.0.1.5.2216 scored 29%): Thank you for testing our software. Results can be explained easily: JPF lacks self-protection. We shall add it in upcoming versions. We greatly appreciate your work on new test suite creation.
Nail Kaipov
Jetico, Inc.
Lavasoft – the vendor of Lavasoft Personal Firewall
2008-05-20 (Lavasoft Personal Firewall 3.0.2293.8822 scored 70%): Thank you very much for your inclusion of the Lavasoft Personal Firewall 3.0 in the Matousec research. Upon review of the results, we were surprised to find that the Lavasoft Personal Firewall program received a 'good' rating and could not be a recommended firewall, whereas our firewall technology partner, Agnitum, received an 'excellent' score with recommendations for the same firewall technology. We hope that you will take this under consideration with your next round of research, and continue your good work.
Michael Helander
Vice President
Lavasoft
Our response: Outpost Firewall Pro was tested when there were 62 tests in the system, Lavasoft Personal Firewall was tested when there were 73 tests in the system. This is the first major difference. The second one is that Outpost Firewall Pro usually contains a newer version of their engine and so Outpost firewall may pass a few more tests than Lavasoft firewall. To mitigate the differences, we intend to retest Outpost Firewall Pro in the near future.
NETGATE Technologies s.r.o. – the vendor of FortKnox Personal Firewall
2008-04-08 (FortKnox Personal Firewall 2008 3.0.195.0 scored 16%): We really appreciate testing of our product and we will try to incorporate protection for new tests in the next build releases. Thank you.
Martin Pekar
NETGATE Technologies s.r.o.
PC Tools Pty Ltd – the vendor of PC Tools Firewall Plus
2008-09-05 (PC Tools Firewall Plus 4.0.0.45 scored 85%): We would like to thank Matousec for conducting the Firewall Challenge tests. PC Tools has invested substantial resources to improve its support in this latest version, and will continue to do so in the future. We are pleased to hear that in this test PC Tools Firewall Plus came out as the fastest firewall. We recognize the importance of minimizing the firewalls influence on the user experience which is a testament to this latest test result. With regard to Keyloggers, PC Tools Firewall Plus does not focus on the detection of keyloggers as these types of threats fall into the Anti-Spyware category of protection. However, there are circumstances in which PC Tools Firewall Plus will detect a Keylogger, this being the point at which it attempts to access the Internet.
Hanoch Ben-David
PC Tools Firewall Plus Team Leader
Soft4Ever – the vendor of Look 'n' Stop
2008-04-08 (Look 'n' Stop 2.06 scored 15%): Look 'n' Stop Firewall is a pure firewall, which filters at TDI & NDIS levels. Look 'n' Stop is not an HIPS application and therefore is not designed to block most of the leaktests considered by the "Firewall Challenge" project from Matousec.
The Look 'n' Stop Team.
Sunbelt Software – the vendor of Sunbelt Personal Firewall
2008-03-24 (Sunbelt Personal Firewall scored 18%): Thanks for including us in your testing. Sunbelt Software is currently testing a new version of Sunbelt Personal Firewall (SPF), planned for release in Q2 2008, that addresses the majority of the leak tests that SPF 4.5.916 fails.
Phil Owens
Product Manager
Sunbelt Personal Firewall
Sunbelt Software
Tall Emu Pty Ltd – the vendor of Online Armor Personal Firewall
2008-11-29 (Online Armor Personal Firewall scored 93% and took 1st place): We're pleased that our effort to produce a powerful, yet easy to use firewall are proved to be paying dividends. Our consisently good result in the firewall challenge is very pleasing.
Mike Nash,
CEO Tall Emu Pty Ltd
2008-05-17 (Online Armor Personal Firewall Free scored 89% and took 4th place): Online Armor Free edition does not provide protection against keyloggers. We would recommend the paid version of our product for keylogger protection. Our current Beta version of Online Armor (both free and paid) addresses the BSOD issues noted, and the fixes will be included in our next release.
Mike Nash,
CEO Tall Emu Pty Ltd
2008-03-18 (Online Armor Personal Firewall Free scored 94% and took 2nd place): We're very pleased to receive this position in the first round of Firewall Challenge tests. Matousec's work on security is a service to both users and vendors of personal firewall products and we are pleased to participate.
Mike Nash,
CEO Tall Emu Pty Ltd