Test methodology
Testing was carried out on an experimental PC running licensed Windows XP with SP1 installed (testing was carried out under idealized conditions - "operating system + Firewall" to exclude the influence of other programs on the purity of the experiment). The APS utility was used as an indicator of successful access to services. The following were used as means of external influence:- scanner XSpider 6.5 and 7.0
- Retina Network Security Scanner 4.9
- several scanners of my design.
On the test PC, IE 6 was used as a means of accessing the network and the Internet, Outlook Express 6, TheBat 1.60, MSN Messenger 6.1. In addition to them, simulators of Trojans and real Trojans / Backdoors from my collection participated in the test (in particular, Backdoor.Antilam, Backdoor.AutoSpy, Backdoor.Death, Backdoor.SubSeven, Backdoor.Netbus, Backdoor.BO2K), network / mail viruses ( I-Worm.Badtrans, I-Worm.NetSky, I-Worm.Sircam, I-Worm.Mydoom, I-Worm.MSBlast), TrojanDownloader downloaders (in particular TrojanDownloader.IstBar) and SpyWare spy components. The main task of the tests was an attempt to look at the Firewall through the eyes of the user, to note its strengths and weaknesses from my point of view.
Kerio Technologies WinRoute Pro v4.2.5
Installation and uninstallation:Passes without problems.
Installation with "default" settings, no rules - only NAT works. Networking - no problems, scan results - APS does not show an alarm status, the scanner thinks that all ports are closed. Winroute itself does not issue alarms and does not visually identify the fact of scanning.
Outpost Firewall Pro 2.1 Build 303.4009 (314)
Installation and uninstallation:Installation under XP goes without problems, at startup, the learning mode is turned on.
ZoneLabs ZoneAlarm Pro with Web Filtering 4.5.594.000 - Personal Firewall
Installation and uninstallation:During the installation, XP hung up during an attempt to start after installation. After a reboot everything worked fine.
AtGuard 3.22>
Installation and uninstallation:Installation and uninstallation does not cause any problems
Advantages:
- Firewall is small in size, it has an interesting solution in terms of interface - it is made in the form of a panel placed at the top of the screen
Disadvantages and features:
- Vulnerable in learning mode - from the moment a request to create a rule is issued until it is created, it passes packets in both directions
- The interface is a little glitchy when redrawing windows
Overall rating:
Simple firewall, but quite functional
Kerio Personal Firewall 4
Installation and uninstallation:The installation goes without problems, the removal is "clean" - no problems were noticed after uninstallation.
Norton internet security 2004 (NIS)
Installation and uninstallation: Installation does not cause problems, but of all the analyzed installers, it is the heaviest.Internet Connection Firewall, ICF - built-in windows firewall XP
Installation and uninstallation: Installation is not required, is regular means xp. Enabled in settings network adapter. By default, ICF operates in maximum security mode and (this is the result of my observation) the principle of its operation is as follows - application requests are released to the outside, and only packets that came in response to my requests are received outside (request-response correspondence is clearly maintained in the form of a dynamic table). Thus, when scanning ports on a computer with ICF enabled, there is not a single open port (this is logical - port scanner packets will not be skipped, because no one requested them). The situation is similar with various kinds of "nukes" based on sending non-standard packets.Internet Connection Firewall, ICF - built-in firewall in Windows XP SP2
Installation and uninstallation: Installation is not required, it is a regular XP tool (included in the SP2 update package for XP). Enabling is done in the settings of the network adapter. It should be noted that when installing SP2 or when installing XP with integrated SP2, in addition to the Firewall, a security center appears in the system, which can show ICF settingsSygate Personal Firewall Pro 5.5 build 2525
Installation and uninstallation:ISS BlackIce 3.6.cci
Installation and uninstallation: The installation of the program and its uninstallation occur without problems, but during the installation an error occurs in the ikernel library. The same error occurred during uninstallation. The occurrence of this error does not affect the process of installing and uninstalling the program. The installer did not require a system restart, which is unusual for FirewallVisnetic Firewall 2.2
Installation and uninstallation: Installation of the program and its uninstallation proceeds without problems. A reboot is required after installation.Look n stop personal firewall 2.05
Installation and uninstallation: Installation of the program and its uninstallation proceeds without problems. A reboot is required after installation. It installs its own driver to work.Kaspersky AntiHacker 1.5
Installation and uninstallation: Installation of the program and its uninstallation proceeds without problems. A reboot is required after installation.Tiny Personal Firewall Pro 6.0
Installation and uninstallation:Installation of the program and its uninstallation proceeds without problems. A reboot is required after installation.
McAfee Personal Firewall Plus 6.0 Build 6014
Installation and uninstallation:Installation of the program and its uninstallation proceeds without problems. A reboot is required after installation.
R-Firewall 1.0 Build 43
Installation and uninstallation:Installation of the program and its uninstallation proceeds without problems. The size of the distribution kit is small (3.8 MB), you can customize the composition of the product. The work is quite stable, no obvious failures and freezes were noticed on the reference PC
General conclusions and conclusion
So, let's sum up the results of the tests. In fact, the tests confirmed my theoretical ideas about the state of the problem:- Firewall needs to be configured. All tested Firewalls worked well, but only after configuration (training, creating settings manually - it doesn't matter). Using an unconfigured Firewall can do more harm than good (it will let through dangerous packets and, vice versa, interfere with useful programs);
- After firewall settings and IDS needs to be tested- This is also a fairly obvious conclusion, but nevertheless it is important. The first step I took to create a tester was the APS utility. There are two more - a Trojan simulator (i.e. a utility that will perform safe attempts for the user to "break" the Firewall from the inside (of course, the attacks will be described by the database and will be carried out at the command of the user under his control), which will allow you to observe the reaction Firewall and IDS) and a utility for express port scanning and basic attacks (in fact, APS is exactly the opposite - they can have a common port base). I am already developing these utilities - their presence in the user's arsenal will allow for some kind of "instrumental control".
- The Personal Firewall is vulnerable to malware running from the context of useful ones. Conclusion - at least down with different "left" panels and other BHO from the browser and e-mail!! Before installing any plugin, panel, extension utility, etc. you need to think ten times about their necessity, because. they are not separate operating system processes and run from the context of the parent program. The Trojan is easily detected by the personal Firewall - it "sees" that some process (say, bo2k.exe) is trying to start listening on port xxxxx or communicating with a certain host - a request for validity is issued, the user begins to figure out what kind of "bo2k.exe" is and Backdoor is caught. But if the Trojan program worked from the context of the browser, then almost certainly no one would pay attention to the browser's access to the Internet. Such Trojans exist, the closest example is TrojanDownloader.IstBar - it is installed exactly as an IE bar (naturally, it is not in processes, nor in the autorun list);
- Many personal firewalls are visible as operating system processes and can be stopped by a virus. Conclusion - the work of the Firewall must be monitored and its sudden termination can serve as a signal that a virus has penetrated the PC;
- Some Firewalls (such as Kerio) allow remote control- function remote control must be either disabled or password protected.
Firewall (firewall) Windows settings are crucial in ensuring the security of the computer and the data stored on it when working on the Internet and local networks. The firewall configuration operation is carried out using standard Windows methods and does not require special computer knowledge.
Instruction
Comparative testing of 21 popular firewalls for the quality of protection against attacks coming from within the system. In the test, protection was tested on 64 test utilities specially developed for it, which checked the protection of processes from termination, protection against standard internal attacks, protection against non-standard leaks, and protection against non-standard kernel mode penetration techniques.
Along with antivirus, firewall is one of the main components of computer security. However, unlike antiviruses, objective tests of firewall performance are rarely carried out. We tried to close this gap by conducting a test of firewalls in 2011 and 2012 to protect against internal attacks and a test of personal IDS / IPS to protect against attacks on vulnerable applications. This year, we decided to expand the list of methods used and repeat the firewall test for protection against internal attacks in order to see how the results of popular products have changed over the past time in this criterion.
What is this test aimed at or what functions does the firewall perform? According to the definition of the Internet standard [RFC3511] (2003), a firewall is a system that implements the functions of filtering network packets in accordance with specified rules in order to differentiate traffic between network segments. However, with the increasing complexity of malware and hacker attacks, the original tasks of the firewall were supplemented with new functional modules. It is virtually impossible to imagine a full-fledged firewall without the HIPS module (system event monitoring, system integrity control, etc.).
The main task of a modern firewall is to block unauthorized network communications (hereinafter referred to as attacks), which are divided into internal and external. These include:
External attacks on a firewall-protected system:
- initiated by hackers;
- initiated by malicious code.
- initiated by untrusted applications (malicious code);
- initiated by applications whose network activity is explicitly prohibited by the rules.
In addition, products that could be classified as pure personal firewalls in the classic 2003 formulation have virtually disappeared from the market. They were replaced by complex personal computer protection products, which necessarily include a firewall component.
The firewall test for protection against external attacks includes the study of the quality of protection against attacks coming from within the system. The test was conducted in the following areas:
- Checking the protection of processes from termination.
- Protection against standard internal attacks.
- Testing protection against non-standard leaks.
- Testing protection against non-standard kernel-mode penetration techniques.
Compared to the previous test, the number of attacks used has increased significantly - from 40 to 64. The operating system that the tested products must protect has also changed. In the previous test it was Windows XP, and in this test it was Windows 7 x32. A similar test for the Windows 7 x64 operating system is also planned for the end of the year.
Introduction
21 took part in testing popular program comprehensive protection(of the Internet Security class, if there is no such product in the line, then a purely firewall was chosen) from various manufacturers up-to-date as of the start date of product testing (May 2013) and running on Windows platform 7 x32 :
- Avast! Internet Security (8.0.1488).
- AVG Internet Security (2013.0.3272).
- Avira Internet Security (13.0.0.3499).
- Bitdefender Internet Security (16.29.0.1830).
- Comodo Internet Security (6.1.276867.2813).
- Dr.Web Security Space (8.0).
- Eset smart security (6.0.316.0).
- F-Secure Internet Security (1.77 build 243).
- G DATA Internet Security (1.0.13113.239).
- Jetico Personal Firewall (2.0).
- Kaspersky Internet Security (13.0.1.4190(g).
- McAfee Internet Security (11.6.507).
- Kingsoft Internet Security (2009.05.07.70).
- Microsoft Security Essentials (4.2.223.0) + Windows Firewall.
- Norton Internet Security (20.3.0.36).
- Online Armor Premium Firewall (6.0.0.1736).
- Outpost Security Suite Pro (8.0 (4164.639.1856).
- Panda Internet Security (18.01.01).
- PC Tools Internet Security (9.1.0.2900).
- Trend Micro Titanium Internet Security (6.0.1215).
- TrustPort Internet Security (2013 (13.0.9.5102).
Before the start of the test, the test environment was prepared. To do this, Windows 7 Enterprise SP1 x86 was installed on a clean computer with all updates available at that time, as well as additional software required for the test.
Testing was carried out on two types of settings: standard recommended by the manufacturer (default settings) and maximum. In the first case, the default settings recommended by the manufacturers were used and all actions recommended by the program were performed.
In the second case, in addition, all settings that were disabled in the "default" mode, but at the same time could affect the outcome of testing, were enabled and/or brought to the maximum position (the most stringent settings). In other words, setting the maximum settings means transferring all available from GUI the user's settings for all modules related to the detection of malicious file or network activity to the most stringent option.
The firewall test was conducted on the following groups of internal attacks, broken down into difficulty levels for clarity:
1. Basic difficulty level (56 attack options):
1. checking the protection of processes from termination (41 types of attacks);
2. protection against standard internal attacks (15 types of attacks).
2. Increased difficulty level (8 attack options):
1. testing protection against non-standard leaks (3 types of attacks);
2. testing of protection against non-standard techniques for penetrating into the kernel mode (5 types of attacks).
You can find a detailed description of all attack methods used in the test in the testing methodology.
Checking firewalls for protection against internal attacks
Recall that according to the used reward scheme, 1 point (+) was awarded if the attack was blocked automatically, the protective functionality of the program under test was not violated. 0.5 points (or +/-) - if the attack is blocked only under special circumstances (for example, when right choice user of the desired action at the request of the program under test). And, finally, if the attack was successful in whole or in part with the failure of the protection functionality, then no points were awarded. The maximum possible number of points scored in this test was 64.
Table 1-2 and Figure 1-2 present the results of testing firewalls separately on standard and maximum settings. For clarity, the results for each firewall are divided into two groups: protection against attacks of a basic level of complexity and protection against attacks advanced level difficulties.
Table 1: Firewall test results for stdbutcustom settings
| Tested Product | Total points (max. 64) | Total % |
||||||
| Points | % | % from the sum | Points | % | % from the sum | |||
| Comodo | 53 | 95% | 82,8% | 6 | 75% | 9,4% | 59 | 92% |
| Online Armor | 50 | 89% | 78,1% | 7,5 | 94% | 11,7% | 57,5 | 90% |
| Norton | 45 | 80% | 70,3% | 6 | 75% | 9,4% | 51 | 80% |
| Jetico | 46 | 82% | 71,9% | 4,5 | 56% | 7,0% | 50,5 | 79% |
| Outpost | 45 | 80% | 70,3% | 2,5 | 31% | 3,9% | 47,5 | 74% |
| Trend Micro | 42 | 75% | 65,6% | 3 | 38% | 4,7% | 45 | 70% |
| Kaspersky | 42 | 75% | 65,6% | 2,5 | 31% | 3,9% | 44,5 | 70% |
| Dr. Web | 42,5 | 76% | 66,4% | 2 | 25% | 3,1% | 44,5 | 70% |
| TrustPort | 43 | 77% | 67,2% | 0,5 | 6% | 0,8% | 43,5 | 68% |
| G DATA | 42 | 75% | 65,6% | 1 | 13% | 1,6% | 43 | 67% |
| Avast | 41 | 73% | 64,1% | 1 | 13% | 1,6% | 42 | 66% |
| Eset | 41 | 73% | 64,1% | 1 | 13% | 1,6% | 42 | 66% |
| Bitdefender | 41 | 73% | 64,1% | 1 | 13% | 1,6% | 42 | 66% |
| AVG | 41 | 73% | 64,1% | 0 | 0% | 0,0% | 41 | 64% |
| McAfee | 41 | 73% | 64,1% | 0 | 0% | 0,0% | 41 | 64% |
| PC Tools | 41 | 73% | 64,1% | 0 | 0% | 0,0% | 41 | 64% |
| Avira | 40 | 71% | 62,5% | 0 | 0% | 0,0% | 40 | 63% |
| Microsoft | 40 | 71% | 62,5% | 0 | 0% | 0,0% | 40 | 63% |
| F-Secure | 31,5 | 56% | 49,2% | 1 | 13% | 1,6% | 32,5 | 51% |
| Panda | 30 | 54% | 46,9% | 0 | 0% | 0,0% | 30 | 47% |
| king soft | 27 | 48% | 42,2% | 1 | 13% | 1,6% | 28 | 44% |
Figure 1: Firewall test results on standard settings
Protection against internal attacks at the settings recommended by the manufacturer leaves much to be desired. Only three firewalls were able to overcome the 80% threshold on standard settings - these are Comodo, Online Armor and Norton. Quite close to them are Jetico (79%) and Outpost (74%). The results of other firewalls were significantly worse.
Compared to the results of the previous test, all leaders confirmed their high results, there were only small movements within the leading group, for example, Outpost and Jetico switched positions. The only surprise was Norton product, which in the last test showed a result of 45% and was at the bottom of the table, and in this test with 80% took the third position.
The results obtained are due to the fact that many manufacturers set the default settings in such a way as to reduce the number of messages that the user must respond to. This is confirmed by the results of the test - at standard settings, firewalls asked users questions only in 5.4% of attacks, and at maximum settings - in 9.2% of attacks. However, this affects the quality of protection, which will remain silent in a situation where a malicious program imitates/performs completely legitimate actions in the system.
You should also pay attention to two regularities. First, the percentage of prevention of complex types of attacks is generally noticeably worse than attacks of the basic level of complexity. More than half of these attacks were rejected by only four products - Comodo, Online Armor, Norton and Jetico. Four more products entered the boundary group, rejecting from 25% to 38% of such attacks - these are Outpost, Trend Micro, Kaspersky and Dr.Web. All other products deflected no more than one complex attack. Secondly, the indicators of repelling basic attacks have improved. If in the previous test 11 (50%) products rejected less than 50% of attacks, in this test there are only 3 (14%) such products.
Table 2: Firewall test results at maximum settings
| Tested Product | Attacks of the basic level of complexity (max. 56 points) | Advanced attacks (max. 8 points) | Total points (max. 64) | Total % |
||||
| Points | % | % from the sum | Points | % | % from the sum | |||
| Comodo | 56 | 100% | 87,5% | 8 | 100% | 12,5% | 64 | 100% |
| Bitdefender | 56 | 100% | 87,5% | 8 | 100% | 12,5% | 64 | 100% |
| Online Armor | 53 | 95% | 82,8% | 8 | 100% | 12,5% | 61 | 95% |
| Kaspersky | 53 | 95% | 82,8% | 7 | 88% | 10,9% | 60 | 94% |
| Norton | 50,5 | 90% | 78,9% | 8 | 100% | 12,5% | 58,5 | 91% |
| PC Tools | 49,5 | 88% | 77,3% | 5,5 | 69% | 8,6% | 55 | 86% |
| Outpost | 49 | 88% | 76,6% | 5,5 | 69% | 8,6% | 54,5 | 85% |
| Eset | 49 | 88% | 76,6% | 5,5 | 69% | 8,6% | 54,5 | 85% |
| Dr. Web | 46,5 | 83% | 72,7% | 5 | 63% | 7,8% | 51,5 | 80% |
| Jetico | 46 | 82% | 71,9% | 4,5 | 56% | 7,0% | 50,5 | 79% |
| Trend Micro | 43 | 77% | 67,2% | 3 | 38% | 4,7% | 46 | 72% |
| TrustPort | 43 | 77% | 67,2% | 2,5 | 31% | 3,9% | 45,5 | 71% |
| G DATA | 42 | 75% | 65,6% | 3 | 38% | 4,7% | 45 | 70% |
| Avira | 41,5 | 74% | 64,8% | 2 | 25% | 3,1% | 43,5 | 68% |
| Avast | 41 | 73% | 64,1% | 1,5 | 19% | 2,3% | 42,5 | 66% |
| AVG | 41 | 73% | 64,1% | 0 | 0% | 0,0% | 41 | 64% |
| McAfee | 41 | 73% | 64,1% | 0 | 0% | 0,0% | 41 | 64% |
| Microsoft | 40 | 71% | 62,5% | 0 | 0% | 0,0% | 40 | 63% |
| F-Secure | 31,5 | 56% | 49,2% | 1 | 13% | 1,6% | 32,5 | 51% |
| Panda | 30 | 54% | 46,9% | 0 | 0% | 0,0% | 30 | 47% |
| king soft | 27 | 48% | 42,2% | 1 | 13% | 1,6% | 28 | 44% |
Figure 2: Firewall test results at maximum settings
With the maximum settings turned on, the quality of protection against internal attacks in many tested firewalls improved significantly. This is especially noticeable in strong middle peasants. All the leaders of the previous test in this test also showed high results. Of the changes, it is worth noting the Bitdefender product, which, along with Comodo, showed a 100% result and the Norton product, which moved to the leading group.
The results of a number of products at standard and maximum settings turned out to be the same. This is due to the fact that these products do not have settings that can affect the results of our test.
Comparison of protection quality at standard and maximum settings
In accordance with the logic of this test, we will not sum or average the results of the same product with various settings. On the contrary, we want to compare them and show significant differences in the quality of protection of the tested products, depending on the settings used.
For clarity, we present the final results of the firewall test with standard and maximum settings in Table 3 and Figure 3.
Table 3: Summary of firewall test results at standard and maximum settings
|
Product |
Standard Settings | Maximum settings |
| Comodo | 92% | 100% |
| Online Armor | 90% | 95% |
| Norton | 80% | 91% |
| Jetico | 79% | 79% |
| Outpost | 74% | 85% |
| Trend Micro | 70% | 72% |
| Kaspersky | 70% | 94% |
| Dr. Web | 70% | 80% |
| TrustPort | 68% | 71% |
| G DATA | 67% | 70% |
| Avast | 66% | 66% |
| Eset | 66% | 85% |
| Bitdefender | 66% | 100% |
| AVG | 64% | 64% |
| McAfee | 64% | 64% |
| PC Tools | 64% | 86% |
| Avira | 63% | 68% |
| Microsoft | 63% | 63% |
| F-Secure | 51% | 51% |
| Panda | 47% | 47% |
| king soft | 44% | 44% |
Figure 3: Summary of firewall test results at standard and maximum settings
Figure 3 very clearly demonstrates the difference in test results depending on the selected settings.
First, only two products - Comodo and Online Armor show close to maximum performance protection, both at standard and at maximum settings.
Secondly, when changing the standard settings proposed by the manufacturer, some products show significantly best level protection. This is most clearly seen in products such as Bitdefender, Kaspersky, Eset, F-Secure and PC Tools.
Thirdly, as noted above, some of the tested products do not have settings at all that could in any way affect the test results. Therefore, their results on all types of settings in this test are the same. This group includes Jetico, Avast, AVG, McAffe, F-Secure, Panda, Kingsoft and Microsoft.
The final score does not take into account situations where an attack was repelled, but there were problems with the user interface of the products. In most cases, the problems consisted in the “flying out” of the interface for a short time (from 2 to 10 seconds) or until the next boot of the operating system. Although the products continued to provide protection in the event of user interface problems, the presence of such problems is subjectively perceived negatively and can affect product preferences. The number of problems with the user interface is presented in Table 3 and Figure 3. The errors that occurred during level 1 attacks were evaluated, the total number of which is 41.
Table 4: Number of user interface issues at standard and maximum settings
| Tested Product | Standard Settings | Maximum settings | ||
| Number of mistakes | % | Number of mistakes | % | |
| McAfee | 34 | 83% | 34 | 83% |
| Microsoft | 33 | 80% | 33 | 80% |
| king soft | 20 | 49% | 20 | 49% |
| F-Secure | 19 | 46% | 19 | 46% |
| Panda | 17 | 41% | 17 | 41% |
| Jetico | 16 | 39% | 16 | 39% |
| PC Tools | 13 | 32% | 13 | 32% |
| Trend Micro | 12 | 29% | 12 | 29% |
| AVG | 10 | 24% | 9 | 22% |
| TrustPort | 9 | 22% | 9 | 22% |
| G DATA | 9 | 22% | 9 | 22% |
| Bitdefender | 8 | 20% | 8 | 20% |
| Norton | 6 | 15% | 6 | 15% |
| Avast | 5 | 12% | 5 | 12% |
| Outpost | 5 | 12% | 5 | 12% |
| Eset | 5 | 12% | 4 | 10% |
| Comodo | 5 | 12% | 0 | 0% |
| Avira | 2 | 5% | 2 | 5% |
| Dr. Web | 2 | 5% | 2 | 5% |
| Kaspersky | 1 | 2% | 1 | 2% |
| Online Armor | 1 | 2% | 1 | 2% |
Figure 4: Number of user interface issues at standard and maximum settings
The results show that McAfee and Microsoft products had user interface problems in most attacks (more than 80%). This can be called an unacceptable level, because. almost any successfully repelled attack will lead to problems. Fairly poor results, ranging from 30% to 50%, are shown by Kingsoft, F-Secure, Panda, Jetico and PC Tools. When using them, every 2-3 attacks will lead to problems with the interface. A number of products show results from 10% to 30%, which can be called satisfactory. Nice results showed products Avira, Dr.Web, Kaspersky and Online Armor, which had problems in the range from 2% to 5% of attacks. The only product that never had any UI issues was Comodo at max settings, which is a great result. However, with standard settings, Comodo's result deteriorates (12%), which indicates that the use of this product requires some knowledge of its configuration.
Final test results and awards
Just like in the previous test, we did not average the results of the same product with different settings, but considered them independently. Thus, each of the tested products can receive two awards, one for each type of setting.
In accordance with the reward scheme, the best firewalls receive awards indicating the settings used, see table 4.
Table 5: Final results of the firewall test at standard and maximum settings
| Tested product | Option settings |
Attack Prevention [%] | Total [%] |
Reward | |
| Base level of difficulty |
Increased difficulty level | ||||
| Comodo | Max | 100% | 100% | 100% | Platinum Firewall Outbound protection award |
| Bitdefender | Max | 100% | 100% | 100% | |
| Online Armor | Max | 95% | 100% | 95% | Gold Firewall Outbound protection award |
| Kaspersky | Max | 95% | 88% | 94% | |
| Comodo | standard | 95% | 75% | 92% | |
| Norton | Max | 90% | 100% | 91% | |
| Online Armor | standard | 89% | 94% | 90% | |
| PC Tools | Max | 88% | 69% | 86% | |
| Outpost | Max | 88% | 69% | 85% | |
| Eset | Max | 88% | 69% | 85% | |
| Norton | standard | 80% | 75% | 80% | |
| Dr. Web | Max | 83% | 63% | 80% | |
| Jetico | Max | 82% | 56% | 79% | Silver Firewall Outbound protection award |
| Jetico | standard | 82% | 56% | 79% | |
| Outpost | standard | 80% | 31% | 74% | |
| Trend Micro | Max | 77% | 38% | 72% | |
| TrustPort | Max | 77% | 31% | 71% | |
| Trend Micro | standard | 75% | 38% | 70% | |
| Kaspersky | standard | 75% | 31% | 70% | |
| Dr. Web | standard | 76% | 25% | 70% | |
| G DATA | Max | 75% | 38% | 70% | |
| TrustPort | standard | 77% | 6% | 68% | Bronze Firewall Outbound protection award |
| Avira | Max | 74% | 25% | 68% | |
| G DATA | standard | 75% | 13% | 67% | |
| Avast | Max | 73% | 19% | 66% | |
| Avast | standard | 73% | 13% | 66% | |
| Eset | standard | 73% | 13% | 66% | |
| Bitdefender | standard | 73% | 13% | 66% | |
| AVG | Max | 73% | 0% | 64% | |
| AVG | standard | 73% | 0% | 64% | |
| McAfee | Max | 73% | 0% | 64% | |
| McAfee | standard | 73% | 0% | 64% | |
| PC Tools | standard | 73% | 0% | 64% | |
| Microsoft | Max | 71% | 0% | 63% | |
| Microsoft | standard | 71% | 0% | 63% | |
| Avira | standard | 71% | 0% | 63% | |
| F-Secure | Max | 56% | 13% | 51% | No reward |
| F-Secure | standard | 56% | 13% | 51% | |
| Panda | Max | 54% | 0% | 47% | |
| Panda | standard | 54% | 0% | 47% | |
| king soft | Max | 48% | 13% | 44% | |
| king soft | standard | 48% | 13% | 44% | |
Comodo and Bitdefender firewalls showed the best results in the test, scoring 100% points at maximum settings. These two products receive an award PlatinumfirewallOutboundProtectionAward.
Award-winning firewalls Online Armor, Kaspersky, Comodo, Norton, PC Tools, Outpost, Eset and Dr.Web also showed very high results in the test (over 80%) GoldfirewallOutboundProtectionAward. It is important to note that Comodo received this award on standard settings, Online Armor and Norton on standard and maximum settings, and all the rest - only on maximum settings.
Next on the list is a group of seven firewalls whose scores fall between 60% and 70%. These are Outpost, Kaspersky and Dr.Web on standard settings; TrustPort and G DATA at maximum settings, as well as Jetico and Trend Micro simultaneously at standard and maximum settings. All of them are rewarded
A sufficiently large group of products that fall into the range from 60% to 70% receives an award. It should be noted products Eset and Bitdefender on standard settings, which at maximum settings were able to repel a significantly larger number of attacks.
You can get acquainted with the detailed test results and make sure that the final calculations are correct by downloading the test results in Microsoft Excel format.
Shabanov Ilya, Managing partner site:
“I was very pleased with the fact that so many manufacturers have significantly improved proactive protection against internal attacks and self-defense in their products. We even had to revise the award scheme in order to raise the bar of requirements. A score of less than 51% is now considered a total failure.
I was pleasantly surprised by Bitdefender, which paranoidly repelled all 100% of attacks, Eset and Dr.Web with results at maximum settings of 85% to 80%, respectively, as well as the newcomer of our TrustPort tests. The "golden group" of products according to the results of this test included Comodo, Norton and Online Armor firewalls, which scored more than 80% on standard and maximum settings. Consistently high results in tests involving proactive defense were demonstrated by Kaspersky, Outpost, PC Tools.
However, in the case of a number of tested products, the logic by which the standard settings are set is not clear. As a result, the level of protection for most users who are accustomed to using protection with standard settings turns out to be significantly underestimated. This primarily applies to Bitdefender, Kaspersky, Eset and PC Tools products.”
Kartavenko Mikhail, head of the test laboratory site:
“Considering this test as a continuation of the previous similar test, we can identify several main trends and problems in the operation of firewalls.
First, on average, most products showed better results than 1.5 years ago, but they did this mainly by repelling the simplest level 1 attacks. More complex attacks "too tough" only for a limited range of products.
Secondly, even if the protection of processes from termination (1 level of attacks) has worked, the user interface crashes in many products. This puts the user in an uncomfortable position in which he does not understand whether the protection is working or not.
Thirdly, there is a fairly large gap in the operation of firewalls at standard and maximum settings. As a result, an acceptable level of protection can often only be obtained by experienced users who know and are able to properly configure firewalls.
Thus, the test revealed the “pain” points of modern firewalls, the solution of which can improve their protection.”
The section is updated daily. Always up-to-date versions of the best free programs for everyday use in the Essential programs section. Almost everything you need is there daily work. Start to phase out pirated versions in favor of more convenient and functional free analogues. If you still do not use our chat, we strongly advise you to get acquainted with it. You will find many new friends there. Moreover, it is the fastest and effective way contact project administrators. The Antivirus Updates section continues to work - always up-to-date free updates for Dr Web and NOD. Didn't have time to read something? The full content of the ticker can be found at this link.
Free Comodo firewall. Testing, conclusions
Comodo Firewall in action
After installing and configuring Comodo hid in the tray and began to annoy me with his questions. On the first day, I played around with all the firewall and proactive defense modes and eventually silenced it. No brakes were found after its appearance in its system. In general, working with Comodo's firewall was quite easy and convenient. The interface of the main window is very simple and informative:
But I had to get used to navigating through the firewall settings and proactive protection - it's not always possible to quickly find the right item. I think it will pass with time.
A few days after installing Comodo Firewall, I decided to test it out a bit.
Test number 1. Online testing
When you click on the "Test" button, the program tries to establish a connection with the site server.
Since Comodo Firewall does not yet know this utility, when it first tried to break into the Internet, an immediate reaction was followed by proactive protection and the firewall:
In both cases, I clicked block and received confirmation that the test passed:
Then I renamed the file FireWallTest.exe in opera.exe and replaced the standard Opera file with it. Thus, I tried to trick the Comodo Firewall, which already knows this browser well and constantly and automatically releases it to the Internet. Comodo reacted to the launch of the “fake” Opera from Total as follows:
Having received my permission for a one-time launch, the firewall warned me about an attempt to access Opera on the Internet:
It turns out that any application for which rules already exist, if the executable file is replaced, will not be able to access the Internet without my knowledge. Everything seems to be fine, but here's the thing: the color of the top part of the warning window depends on the severity of the situation. If Comodo evaluates the event as critical, then the color will be red, if the event is less dangerous - yellow. In my case, Comodo considered the simulated situation not particularly dangerous and lit “yellow”. In addition, instead of the wording "executable file opera.exe not recognized" I would prefer to see that "there was a change in the parameters of the file opera.exe". So they warn similar situations harvesters from Kaspersky and Eset, for example. Moreover, the user sees an alarm window using red color, which immediately makes you pay attention to the situation. And a warning from Comodo can simply be ignored by the user due to insufficient emphasis on the ongoing event.
The substitution of the Opera file was only part of my cunning plan. The next victim was Internet Explorer 6, which is integrated into the operating system, and, therefore, iexplore.exe can be considered a full system file. What was my surprise when, under the complete silence of Comodo, I saw a window about the failure of the test:
Apparently, an extra rule was created, I decided, and got into the firewall and proactive defense policies. After digging around there for about 15 minutes, I made the only right decision - to reinstall Comodo. No sooner said than done. Leaving the default operating modes, I repeated the experiment with the substitution iexplore.exe. When launched from Total, proactive protection worked, as in the case of Opera:
Here we have to make a small lyrical digression. The fact is that when the IE executable file is replaced, the system restores the original one within 4-8 seconds. iexplore.exe. In this regard, the results of my test depended on whether the replaced file had time to knock on the Internet or not.
In the case when I manage to complete all the manipulations before restoring explore.exe, the following happens. With my permission for a one-time run explore.exe, Total launches the FireWallTest utility, I press "Test", Defens + proactive protection issues a warning:
If we allow (as an experiment) - the firewall works:
We have time to click "Block" - the test is passed, the utility did not slip into the Internet. But if iexplore.exe restored before the lock button was pressed - nothing depends on your choice - the utility automatically accesses the Internet at the moment the original file is restored.
The same applies to the work of proactive defense: if you do not have time to command to block before restoring explore.exe- the utility automatically gets access to the Internet.
Having played enough with fake IE, I remembered the very first failure of the test, when Comodo kept silent and released the "left" file on the Internet. After reinstalling Comodo, I put Defense+ and the firewall into learning mode and launched IE. After that, I returned the default modes and repeated the test. Comodo failed it silently again...
Test number 3. Duel
Impressed by the results of the previous test, I searched for additional features to test Comodo and finally found the AWFT utility.
This program emulates the behavior of Trojans and contains a series of six tests that demonstrate various methods of unauthorized access to the network, bypassing firewall protection. Among these tests, there are both old methods of cheating firewalls and more modern methods. For each successfully passed test, the firewall is awarded a certain number of points. If the test is not passed, points are awarded to the AWFT. The maximum number of points is ten.
The utility is shareware, limited to 10 launches. At the top of the program window there are buttons that launch the corresponding tests, at the bottom there is a site where AWFT will break through, and the result of a duel between the firewall and the utility. The Reset Points button is used to reset the accumulated points.
Just in case, I decided to change the site address to my own.
Testing took place with Comodo Firewall and Defense+ turned on, Opera running and Avira's monitor turned off.
In the first test, a trick was used with loading a hidden copy of the browser and patching the memory before launching it.
When I clicked the test button, an error window popped up:
After closing this window, Сomodo responded to the test with a request window, when you clicked the "Block" button, AWFT, after a little thought, gave the first point to the firewall.
According to the developers of the utility, test #2 is an old and well-known trick. Comodo responds with a request window again and gets a point again.
Test #3 also uses an old trick. Comodo just silently blocks it, apparently, the trick is really known.
Test #4 is similar to the first test, running a hidden copy of the browser and patching the memory before launching it. The firewall did not issue any warnings, but after a short pause earned another point.
During the fifth and sixth tests, you need to switch to the browser and surf a bit (I just refreshed the page loaded in the browser).
In test No. 5, the utility performs a heuristic search for authorized software installed on the computer (or on the network) that has Internet access through port 80, then launches a copy of the authorized program and patches the memory occupied by this program (i.e., AWFT launches itself in the memory of the allowed program). Comodo silently passed the test and received a whopping 3 points for it.
Test #6 is similar to the previous fifth test. The same technique is used with a heuristic search for installed software that has the right to go outside through port 80. Only the hacking method has now been changed - a user request is used. Along with this, AWFT is trying to stick a left hidden toolbar to the browser. With Opera open, this window popped up:
At the moment I confirmed this user request, Comodo issued its request, the utility was blocked again, and the firewall received 3 points in credit.
The result of the duel is 10:0 in favor of Comodo. Repeating the tests with Internet Explorer open, I got the same results.
Conclusion
Despite some unpleasant aftertaste in my soul after testing the firewall, I still recommend Comodo Internet Security for home use, but only as a firewall. And do not listen to those smart people who advise disabling proactive defense, in no case! Only with the use of Defense+ does this firewall really keep your computer safe. What you really shouldn't use is Comodo's antivirus. Not only does it decently skip, you will have problems updating it - it has very bulky databases. In addition, it decently affects the performance of the system. It just worked great for me in a pair of Comodo Firewall and Avira Antivir Personal.
I did not find any brakes and glitches in the system during the operation of the firewall. I will keep my thoughts about the results of my testing to myself for now, I would like to listen to your comments.
While writing the final part of this article, I came across the results of a recent firewall test by the Matousec lab. Comodo Internet Security was the only firewall with a 100% score (see firewall forum). Well, I made my choice... And you?
pros (explicit):
free distribution,
the presence of its own database of programs;
the presence of proactive protection (Defense +);
ease of installation and initial configuration;
very informative and convenient summary window;
pros (doubtful):
the presence of several modes of operation;
cons (obvious):
annoying installation mode;
executable file substitution is not defined by proactive defense as a critical event;
cons (doubtful):
frankly unsuccessful antivirus.
This second article describes how to resolve packet filter related issues. Instead of casting finished table in the form of "problem" - "solution" methods of a systematic approach are given to solve the problems that have arisen.
This second article (in a series of three) describes how to troubleshoot packet filter issues. Instead of bringing the finished table in the form of "problem" - "solution", methods of a systematic approach are given to solve the problems that have arisen.
Introduction
The packet filter executes the filtering policy by bypassing the set of rules and, accordingly, either blocks or passes the packets. The chapter explains how to check that the filtering policy is being implemented correctly and how to find errors if it is not.
In general, in the course of this chapter, we will compare the task of writing a set of filtering rules with programming. If you do not have programming skills, then this comparison will seem to you rather complicating the task. But writing rules in and of itself doesn't require a computer science degree or programming experience, does it?
The answer is no, you probably don't need that. The language used for packet filter configuration is made to look like human languages. For example:
block all
pass out all keep state
pass in proto tcp to any port www keep state
In fact, you don't need to have a programmer nearby to understand what this set does or even, using intuition, write a similar filtering policy. There is even a high chance that a set of filtering rules created in this semblance will perform the actions that its author had in mind.
Unfortunately, computers only do what you ask them to do, not what you want them to do. Worse than that, they will not be able to distinguish the desired from the actual, if there is such a difference. So, if the computer doesn't do what you want it to do correctly, even if you think you described the instructions clearly, it's up to you to find the differences and change the instructions. And since this is a common problem in programming, we can see how programmers deal with it. Here it turns out that the skills and methods used to test and debug programs and filtering rules are very similar. And yet here you do not need knowledge of any programming language in order to understand the implications for testing and debugging.
Good filtering policy.
A filter policy is an informal specification of what we want from a firewall. On the other hand, a set of rules, the implementation of a specification, is a set of standard instructions, a program that is executed by a machine. Accordingly, in order to write a program, you must determine what it should do.
Thus, the first step in firewall configuration is to informally specify what needs to be achieved. What connections should be blocked or allowed?
An example would be:
There are three networks that must be separated from each other by a firewall. Any connections from one network to another go through the firewall. The firewall has 3 interfaces, each of which is connected to the corresponding network:
$ext_if - to the external network.
$dmz_if - DMZ with servers.
$lan_if - LAN with workstations.
Hosts on the LAN must freely connect to any hosts in the DMZ or external network.
Servers in the DMZ must freely connect to hosts on the external network. External network hosts can only connect to the following servers in the DMZ:
Web server 10.1.1.1 port 80
Mail server 10.2.2.2 port 25
All other connections should be prohibited (for example, from machines on the external network to machines on the LAN).
This policy is expressed informally so that any reader can understand it. It should be so specific that the reader can easily formulate answers to the question like ‘Should the connection from host X to host Y be passed incoming (or outgoing) on interface Z?’. If you are thinking about the cases where your policy does not meet this requirement, then it is not clearly defined.
"Misty" policies like "allow only essentials" or "block attacks" need to be clarified or you won't be able to enforce or test them. As in software development, poorly formalized tasks rarely lead to justified or correct implementations. (“Why don’t you go and start writing code while I figure out what the customer needs”).
The set of rules that implements the policy
The set of rules is written as a text file containing sentences in a formal language. As well as source is processed and translated into machine code instructions by the compiler, the ruleset "source" is processed by pfctl and the result is interpreted by pf in the kernel.
When the source code violates formal language rules, the parser reports a syntax error and refuses to process the file further. This error is a compile-time error and is usually fixed quickly. When pfctl can't parse your ruleset file, it will print the line that it found an error and a more or less informative message that it couldn't parse. Until the entire file has been processed without single error, pfctl will not change the previous set of rules in the kernel. And because the file contains one or more syntax errors, there won't be a "program" that pf can execute.
The second type of error is called "run-time errors" because it occurs when a syntactically correct program is successfully compiled and executed. IN general case, in programming languages, this can happen when a program performs a division by zero, attempts to access invalid areas of memory, or runs out of memory. But since rule sets only vaguely resemble the functionality of programming languages, most of these errors cannot occur during the application of the rules, for example, the rules cannot cause the so-called. "system crash", as regular programs do. However, a set of rules may cause similar errors, in the form of blocking or vice versa, passing packets that do not match the policy. This is sometimes called a logical error, an error that does not cause an exception and a halt, but simply produces incorrect results.
So, before we can start checking how correctly the firewall implements our security policy, we must first successfully load the rule set.
Analyzer errors
Parser errors occur when trying to load a list of rules using pfctl, for example:
# pfctl-f/etc/pf.conf
/ etc/pf.conf:3:syntaxerror
This message indicates that there is a syntax error on line 3 of the /etc/pf.conf file and pfctl is unable to load the rules. The set in the kernel has not changed, it remains the same as before trying to load a new one.
There are many kinds of errors thrown by pfctl. To get started with pfctl, you just need to read them carefully. It is possible that not all parts of the message will reveal their meaning to you immediately, but it is necessary to read them all, because. later it will make it easier to understand what went wrong. If the message contains a part of the form "filename:number:text", it refers to the line with the corresponding number in the specified file.
The next step is to look at the output line using a text editor (in vi you can jump to line 3 by typing 3G in beep mode), or like this:
# cat -n /etc/pf.conf
1 int_if = "fxp 0"
2 blocks all
3 pass out on $int_if inet all kep state
pass out inet on $int_if all kep state
The problem could be a simple typo, as in this case (“kep” instead of “keep”). After fixing, try reloading the file:
# pfctl-f/etc/pf.conf
/ etc/pf.conf:3:syntaxerror
# head -n 3 /etc/pf.conf | tail -n 1
pass out inet on $int_if all keep state
Now all the keywords are correct, but upon closer inspection, we notice that the placement of the "inet" keyword before "on $int_if" is wrong. This illustrates that the same line can contain more than one error. Pfctl prints out a message about the first error it finds and stops running. If the same line number was generated during the second run, then there are still errors in it, or the previous ones were not correctly eliminated.
Incorrectly placed keywords are also a common mistake. This can be seen by comparing the rule with the reference BNF syntax at the end of the man pf.conf(5) help file, which contains:
pf-rule= action [("in" | "out") ]
[ "log" | "log-all" ] [ "quick" ]
["on" ifspec] [route] [af] [protospec]
hosts [filteropt-list]
ifspec = ([ "!" ] interface-name) | "("interface-list")"
af="inet" | "inet6"
What does it mean that keyword"inet" must follow "on $int_if"
Let's fix it and try again:
# pfctl-f/etc/pf.conf
/ etc/pf.conf:3:syntaxerror
# head -n 3 /etc/pf.conf | tail -n 1
pass out on $int_if inet all keep state
There are no obvious errors now. But we can't see all the related details! The string depends on the $inf_if macro. What can be misunderstood?
# pfctl -vf /etc/pf.conf
int_if = "fxp 0"
block drop all
/etc/pf.conf:3: syntax error
After correcting the typo "fxp 0" to "fxp0" try again:
# pfctl-f/etc/pf.conf
The absence of messages indicates that the file was successfully loaded.
In some cases, pfctl may produce more specific error messages than just "syntax error":
# pfctl -f /etc/pf.conf
/etc/pf.conf:3: port only applies to tcp/udp
/etc/pf.conf:3: skipping rule due to errors
/etc/pf.conf:3: rule expands to no valid combination
# head -n 3 /etc/pf.conf | tail -n 1
pass out on $int_if to port ssh keep state
The first line of the error message is the most informative of the rest. In this case, the problem is that the rule, specifying the port, does not specify the protocol - tcp or udp.
On rare occasions, pfctl gets discouraged by the presence of non-printing characters or unnecessary spaces in a file, such errors are not easy to detect without special file processing:
# pfctl -f /etc/pf.conf
/etc/pf.conf:2: whitespace after \
/etc/pf.conf:2: syntax error
# cat -ent /etc/pf.conf
1 block all$
2 pass out on gem0 from any to any \$
3 ^ Ikeepstate$
The problem here is the space character, after the "backslash" but before the end of the second line, indicated by the "$" sign in the output of cat -e.
After the rule set has been successfully loaded, it's a good idea to look at the result:
$ cat /etc/pf.conf
block all
# pass from any to any \
pass from 10.1.2.3 to any
$ pfctl -f /etc/pf.conf
$ pfctl-sr
blockdropall
The "backslash" at the end of a comment line actually means that the comment line will continue below.
Expanding lists enclosed in curly braces () can produce a result that may surprise you, and at the same time show the set of rules processed by the analyzer:
$ cat /etc/pf.conf
pass from ( !10.1.2.3, !10.2.3.4 ) to any
$ pfctl -nvf /etc/pf.conf
pass inet from ! 10.1.2.3 to any
pass inet from ! 10.2.3.4toany
The catch here is that the expression "( !10.1.2.3, !10.2.3.4 )" will not mean "all addresses except 10.1.2.3 and 10.2.3.4", the expanded expression itself means to match any possible address.
You must reload your rule set after making permanent changes to ensure that pfctl can also load it when the machine is rebooted. On OpenBSD, the startup rc script in /etc/rc first loads a small set of default rules that blocks all traffic except what is needed during the boot phase (such as dhcp or ntp). If the script fails to load the actual rule set from /etc/pf.conf due to syntax errors introduced before the machine was rebooted without checking, then the default rule set will remain active. Fortunately, incoming ssh connections are allowed in this set, so the problem can be solved remotely.
Testing
Since we have an extremely precisely defined policy, and a set of rules that must implement it, then the term testing will mean in our case the compliance of the resulting set with a given policy.
There are only two ways to misapply rules: blocking connections that should be allowed, and vice versa, passing those connections that should be blocked.
Testing in the general case implies a systematic approach to the orderly creation various kinds connections. It is not possible to test all possible source/destination combinations and corresponding ports on the interfaces, as firewall can theoretically collide with huge amount such combinations. Ensuring the initial correctness of a set of rules can only be ensured for very simple cases. In practice, the best solution is to create a list of test connections based on the security policy, such that each policy item is affected. So, for our example policy, the list of tests would be:
Connection from LAN to DMZ (should be skipped)
from LAN to external network (should be skipped)
from DMZ to LAN (should be blocked)
from DMZ to external network (should be skipped)
from the external network in the DMZ to 10.1.1.1 on port 80 (should be skipped)
from external network to DMZ to 10.1.1.1 on port 25 (should be blocked)
from external network to DMZ to 10.2.2.2 on port 80 (should be blocked)
from the external network in the DMZ to 10.2.2.2 on port 25 (should be skipped)
from external network to LAN (should be blocked)
The expected result must be defined in this list prior to testing.
It may sound strange, but the purpose of each test is to find bugs in the implementation of a set of firewall rules, not just to state their absence. And the ultimate goal of the process is to build a set of rules that are error-free, so if you think there are likely to be errors, you'd be better off finding them than missing them. And if you take on the role of a tester, you must adopt a destructive thinking style and try to get around the firewall restrictions. And only the fact that the restrictions cannot be broken will become a reasoned confirmation that the set of rules does not contain errors.
TCP and UDP connections can be checked with nc. nc can be used as both client and server side (using the -l option). And for ICMP requests and responses, best client to check will ping.
To check if the connection is blocked, you can use any means that will try to create connections with the server.
Using tools from the Ports Collection, such as nmap, you can easily scan many ports, even on multiple hosts. If the results are not entirely clear, feel free to take a look at the man page. For example, for a TCP port, the scanner returns 'unfiltered' when nmap receives an RST from pf. Also, pf installed on the same machine as the scanner can affect the correct operation of nmap.
More sophisticated scanning tools may include facilities for generating fragmented or invalid ip packets.
To check that the filter is passing connections specified in the policy, the best method is to check using those applications that will subsequently be used by clients. So, checking the passage of http connections from different client machines of the web server, as well as from different browsers and sample various content would be better than just confirming the establishment of a TCP session to nc acting as a backend. Various factors, such as the host operating system, can also cause errors - issues with TCP window scaling or TCP SACK responses between certain operating systems.
When the next test item is passed, its results may not always be the same. The connection may be interrupted during the connection establishment, in case the firewall returns RST. Establishing a connection can simply time out. The connection can be fully established, work, but hang or break after a while. The connection may hold, but throughput or latency may be different than expected, higher or lower (in case you use AltQ to limit bandwidth).
As expected results, in addition to skipping / blocking the connection, one can also note whether the packets are logged, how they are broadcast, routed, whether the necessary counters are increased, if necessary. If these aspects are important to you, then they should also be included in the testing methodology.
Your policy may include requirements regarding performance, overload response, fault tolerance. And they may require separate tests. If you're setting up a fault-tolerant system using CARP, you'll probably want to know what happens on various kinds of failures.
When you see a result different from what you expected, step by step note your steps during the test, what you expected, why you expected it, the result obtained, and how the result differs from your expectations. Repeat the test to see if the situation is reproducible or different from time to time. Try changing the test input parameters (source/destination address or ports).
From the moment you get a reproducible problem, you need to start debugging to find out why things are not working as you expected and how to "fix" everything. With this setting, you must change the ruleset and retry all tests, including those that did not cause errors, since by changing the rules you could inadvertently affect the operation of correctly working parts of the ruleset.
The same principle applies to other changes to the set. This formal verification procedure will help make the process less prone to introducing errors. It may not be necessary to repeat the entire procedure for small changes, but the sum of several small changes can affect the overall result of processing the set. You can use a version control system such as cvs to manage your configuration file. this will help in investigating the changes that led to the error. If you know the error didn't occur a week ago, but it does now, reviewing all the changes made for last week will help to notice the problem, or at least roll back to the moment of its absence.
Non-trivial rule sets can be thought of as programs, they are rarely perfect in their first version, and it takes time to be sure that they are free of bugs. However, unlike regular programs, which are never considered bug-free by most programmers, rulesets are still simple enough to be close to this definition.
Debugging
The term debugging usually refers to finding and fixing programming errors in computer programs. Or, in the context of firewall rule sets, the term would mean the process of finding the reason why a set does not return the desired result. There are few types of errors that can appear in rules, however, the methods for finding them are similar to programming.
Before you begin to search for the cause of the problem, you must clearly understand the essence of this problem. If you yourself noticed the error during testing, it is very simple. But if another person reports a bug to you, setting a clear goal from an inaccurate bug report can be tricky. The best place to start is by reproducing the error yourself.
Not always network problems can be caused by a packet filter. Before you focus on debugging the pf configuration, you need to make sure that the problem is caused by a packet filter. This is easy to do and will also save you time troubleshooting elsewhere. Just shut down pf with pfctl -d and see if the problem reappears. If so, enable pf with pfctl -e and see what happens. This method will fail in some cases, for example if pf does not translate properly network addresses(NAT), then turning off pf will obviously not get rid of the error. But whenever possible, try to make sure that the packet filter is the culprit.
Accordingly, if the problem is in the packet filter, the first thing to do is to make sure that pf really works and the correct set of rules has been successfully loaded:
# pfctl -si | grep Status
Status: Enabled for 4 days 13:47:32Debug: Urgent
# pfctl -sr
pass quick on lo0 all
pass quick on enc0 all
Debugging by protocols
The next debugging step is to reflect the problem on specific network connections. If you have a message: "instant messaging in application X does not work", you need to find out what network connections are used. The conclusion may be in the form "host A cannot establish a connection with host B on port C". Sometimes this task is the most difficult, but if you have information about the necessary connections and you know that the firewall will not let them through, all you need to do is change the rules to solve this problem.
There are several ways to find out which protocols or connections an application is using. Tcpdump can display packets arriving or leaving both the real network interface and virtual ones such as pflog and pfsync. You can specify a filtering expression to specify which packets to display and to exclude spurious network "noise". Try to install network connection in the desired application and look at the packets being sent. For example:
# tcpdump -nvvvpi fxp0 tcp and not port ssh and not port smtp
23:55:59.072513 10.1.2.3.65123 > 10.2.3.4.6667:S
4093655771:4093655771(0) win 5840
1039287798 0,nop,wscale 0> (DF)
This is a TCP SYN packet, the first packet of a TCP connection (TCP handshake) to be established.
The sender is 10.1.2.3 port 65123 (looks like a random non-privileged port) and the destination is 10.2.3.4 port 6667. A detailed explanation of the tcpdump output format can be found in the utility's manual pages. Tcpdump is the most important tool for debugging pf related problems and it is very important to get familiar with it.
Another method is to use pf's logging functionality. Assuming you use the 'log' option on all rules with 'block', then all packets blocked by pf will be logged. It is possible to remove the 'log' option from rules that deal with known protocols, i.e. only those packets that go to unknown ports will be logged. Try using an application that can't communicate and look at pflog:
# ifconfig pflog0 up
# tcpdump -nettti pflog0
Nov 26 00:02:26.723219 rule 41/0(match): block in on kue0:
195.234.187.87.34482 > 62.65.145.30.6667: S 3537828346:3537828346(0) win
16384
If you are using pflog, a daemon that constantly listens on pflog0 and saves the information it receives to /var/log/pflog, the saved information can be seen like this:
# tcpdump-nettr /var/log/pflog
When displaying pf-stored packages, you can use additional filtering expressions, such as viewing packages that have been blocked from entering on the wi0 interface:
# tcpdump -netttr /var/log/pflog inbound and action block and on wi0
Some protocols, such as FTP, are not so easy to track down because they do not use fixed port numbers, or they use multiple coexisting connections. It may not be possible to get them through the firewall without opening a wide range of ports. There are solutions similar to ftp-proxy for individual protocols.
Debugging Rules
If your rule set is blocking a particular protocol because you haven't opened the right port, it's more of a design-time problem than a bug in the rules. But what if you see that a connection is being blocked for which you have a rule that allows it to pass?
For example your set contains the rule
block in return-rst on $ext_if proto tcp from any to $ext_if port ssh
But when you try to connect to TCP port 22, the connection is accepted! It looks like the firewall is ignoring your rule. As with jigsaw puzzles, there is a simple logical and usually trivial explanation for these cases, when encountered the first few times.
First, you should check all those steps mentioned earlier. For example, let's say that the firewall is running and contains the rule above. It is unlikely that our previous concerns are true, but it is easy to verify:
# pfctl -si | grep Status
Status: Enabled for 4 days 14:03:13Debug: Urgent
# pfctl -gsr | grep "port=ssh"
@14 block return-rst in on kue0 inet proto tcp from any to 62.65.145.30 port = ssh
The next thing we have is that TCP connections are being accepted on port 22 on kue0. You might think that this is already obvious, but it would be useful to check. Run tcpdump:
# tcpdump -nvvvi kue0 tcp and port 22 and dst 62.65.145.30
Now retry the SSH connection. You should see packets from your connection in the tcpdump output. Maybe you don't see them, and that might be because the connection doesn't actually go through kue0, but goes through a different interface, which explains why the rule doesn't work. Or you may be connecting to a different address. In short, if you don't see ssh packets, then pf won't see them either, and probably can't block them using the rule given in our problem.
But if you see packets with tcpdump, pf will "see" them too and filter them. The next assumption is that the blocking rule should not just be present in the set (which we have already figured out), but be the last matching rule for the required packets. If this is not the last rule, obviously, according to this, a decision is not made to delay packets.
In what cases might a rule not be the last matching rule? There are three possible reasons:
A) the rule does not work, because the viewing of the rules does not reach the desired one.
The previously present rule also fires and causes the execution to be terminated with the 'quick' option;
B) the rule is processed, but the rule does not work due to a mismatch of individual criteria.
C) the rule is processed, the rule fires, but processing continues and subsequent rules also fire for the packet.
To override these three cases, you can visualize the processing of a hypothetical TCP packet arriving at interface kue0 and port 22 by looking at the loaded ruleset. Highlight the block to be debugged. Start bypassing with the first rule. Does it match? If yes, mark it. Does it have a 'quick' option? If so, then we stop the round. If not, continue with next rule. Repeat the test until a match is found with the 'quick' option or the end of the rule set is reached. Which rule matched last? If it's not rule number 14, you've found an explanation for the problem.
Bypassing the rules manually may seem funny, however, with enough experience, it can be done fairly quickly and with a high degree of reliability. If the set is large enough, you can temporarily reduce it. Save a copy of the real list of rules and delete those entries that you think will not affect the result. Download this set and test again. If the connection is now blocked, then rules that seemed unrelated to the packets you are looking for are responsible for cases A or B. Add rules one by one, repeating the test, until you find the right one. If the connection is still allowed after removing rules that do not affect it, repeat the reduced set mental traversal.
Another method is to use pf's logging ability to identify cases A or C. Add 'log' to all rules with 'pass quick' before the 14th rule. Add 'log' to all rules with 'pass' after the 14th rule. Run tcpdump on the pflog0 interface and establish an ssh connection. You will see which rules, besides the 14th, are the last ones to work on your packets. If there is nothing in the log, then case B occurs.
Firewall Connection Tracking
When a connection passes through a firewall, packets arrive on one interface and are transmitted out through the second. Responses come to the second interface, and go to the first. Connections can thus fail in each of these four cases.
First, you must figure out which of the four cases is the problem. If you try to establish a connection, you should see a TCP SYN packet on the first interface using tcpdump. You should also see the same TCP SYN packet coming out of the second interface. If you don't see it, then we conclude that pf is blocking incoming packet on the first interface, or outgoing on the second.
If the SYN is not blocked, you should see a SYN+ACK coming in on the second interface and out on the first. If not, pf blocks SYN+ACK on some interface.
Add the 'log' option to the rules that should allow SYN and SYN+ACK on both interfaces, as well as the rules that should block them. Retry the connection and check pflog. It should clarify in which case the blocking occurs and by what rule.
Debugging connection states
The most common reason for blocking pf packets is that there is an excessive blocking rule in the set. The corresponding rule that fires on the last match can be found by adding the 'log' option to all potentially affecting rules and listening to the pflog interface.
In a smaller number of cases, it happens that pf "silently" drops packets based on non-rules, and here adding 'log' to all rules will not result in dropped packets getting into pflog. Often a package is almost, but not completely, a state entry.
Remember that for each packet it processes, the packet filter scans the state table. If a matching entry is found, the packet is immediately passed without causing the rule set to be processed on itself.
An entry in the state table contains information related to a single connection.
Each entry has a unique key. This key consists of several values that limit constant throughout the lifetime of the connection. Here they are:
- Address type (Ipv4 or Ipv6)
- Source address
- Receiver address
- Protocol (TCP/UDP)
- Source port
- Receiver Port
This key is used for all packets related to the same connection, and packets different compounds will always have different keys.
When the 'keep state' option from a rule creates an entry in the state table, the connection entry is stored using the key of that connection. An important constraint for the state table is that all keys must be unique. Those. there cannot be two records with the same keys.
It may not be immediately obvious that the same two hosts cannot establish multiple coexisting connections using the same addresses, protocols and ports, but it is fundamental property both TCP and UDP. In fact, TCP/IP stacks can only associate individual packets with their sockets by performing a selection based on addresses and ports.
Even if the connection is closed, the same pair of addresses and ports cannot be re-used immediately. Retransmitted packets may later be delivered by network equipment, and if they are mistaken by the receiver's TCP/IP stack for packets from a newly created connection, this will interfere with or even terminate the new connection. For this reason, both hosts must wait a certain amount of time called 2MSL ("twice the maximum segment lifetime") before being able to use the same addresses and ports again for a new connection.
You can observe this property by manually establishing multiple connections to the same host. For example, having a web server running on 10.1.1.1 and port 80, and connecting twice from 10.2.2.2. using nc:
$ nc -v 10.1.1.1 80 & nc -v 10.1.1.1 80
Connection to 10.1.1.1 80 port succeeded!
While connections are open, you can display information about these connections using netstat on the client or server:
$ netstat -n | grep 10.1.1.1.80
tcp 0 0 10.2.2.6.28054 10.1.1.1.80 ESTABLISHED
tcp 0 0 10.2.2.6.43204 10.1.1.1.80 ESTABLISHED
As you can see, the client has chosen two different (random) source ports, so this does not violate the key uniqueness requirements.
You can tell nc to use a specific source port with the -p option:
$ nc -v -p 31234 10.1.1.1 80 & nc -v -p 31234 10.1.1.1 80
Connection to 10.1.1.1 80 port succeeded!
nc: bind failed: Address already in use
The client's TCP/IP stack has prevented key uniqueness violations. Some rare and erroneous implementations of TCP/IP stacks didn't follow this rule, and therefore, as we'll see shortly, pf will block their connections if the uniqueness of the keys is violated.
Let's go back to the point where pf makes a query to the state table, at the time when the packet starts to be filtered. The request has two steps. The first query is made to look for an entry in the entry table with a key corresponding to the protocol, addresses, and port of the packet. The search will be for packets going in any direction. Assume that the package below has created an entry in the state table:
comingTCPfrom 10.2.2.2:28054to 10.1.1.1:80
The table query will find the following entries in the state table:
incoming TCP from 10.2.2.2:28054 to 10.1.1.1:80
outgoing TCP from 10.1.1.1:80 to 10.2.2.2:28054
The entry in the table includes information about the direction (inbound or outbound) of the first packet that created the entry. For example, the following entries will not cause a match:
outgoingTCPfrom 10.2.2.2:28054to 10.1.1.1:80
comingTCPfrom 10.1.1.1:80to 10.2.2.2:28054
The reason for these restrictions is not obvious, but is quite simple. Imagine that you have only one interface at 10.1.1.1, where the web server is listening on port 80. When a client on 10.2.2.2 connects using the randomly selected outgoing port 28054, the first packet of the connection arrives on your interface and all your outgoing responses must will go from 10.1.1.1:80 to 10.2.2.2:28054. You will not pass outgoing packets from 10.2.2.2:28054 to 10.1.1.1:80, since such packets do not make sense.
If your firewall is configured for two interfaces, then by watching the packets passing through it, you will see that every packet that arrives on the first interface goes out and through the second. If you create a state entry in which the initial packet arrives on the first interface, that entry will prevent the same packet from leaving the second interface because it is in the wrong direction.
When an attempt to find a packet among the entries in the state table fails, the list of filter rules is traversed. You must specifically allow the packet to pass out through the second interface with a separate rule. You are probably using 'keep state' in this rule so that the second entry in the state table covers the entire connection on the second interface as well.
You might be wondering how it is possible to create a second record in a table when we just explained that records must have unique keys. The explanation here is that the record also contains information about the direction of the connection, and the combination of this with the rest of the data must be unique.
Now we can also explain the difference between a loose connection and a connection bound to an interface. By default, pf creates entries that are not bound to any interface. So if you allow connections on one interface, the packets related to the connection and matching the table entry (including the packet's direction information!) go through any interface. In simple installations with static routing, these are more theoretical calculations. In principle, you should not see packets from the same connection arriving on multiple interfaces and response packets also leaving on multiple interfaces. However, with dynamic routing, this is possible. You can bind state entries to a particular interface using the 'set state-policy if-bound' global setting or the per-rule option 'keep state (if-bound)'. This ensures that packets will only be matched by entries from the interface that created those entries.
If interfaces for tunnels are used, then the same connection passes through the firewall several times. For example, the first packet of a connection may first go through interface A, then through B, then C, and finally leave us through interface D. Typically, packets will be encapsulated on interfaces A and D and decapsulated on B and C, so pf sees packets of different protocols and you can create 4 different entries in the state table. Without encapsulation, the packet will be unchanged on all four interfaces and you won't be able to use some features like address translation or tcp sequence number modulation, because this will result in conflicting keys appearing in the state table. Until you have a complete installation that includes interfaces with tunneling and debugged errors like "pf: src_tree insert failed", you will not be able to consider your installation successful enough. Let's return to the request to the state table that is made for each packet before checking the rules. The query must return a single record with suitable key or return nothing. If the query returns nothing, the list of rules is bypassed.
If the entry is found, the second step for TCP packets is to check the sequence number before they are treated as belonging to a particular connection and subjected to filtering.
There is a large number of TCP attacks in which the attacker tries to control the connection between two hosts. In most cases, the attacker is not in the path of routes between hosts, and therefore cannot listen to legitimate packets forwarded by hosts. However, he can send packets to any of the hosts that imitate the packets of his interlocutor, by spoofing ("spoofing") - forging the sender's address. The attacker's goal may be to prevent the possibility of establishing connections between hosts or to terminate already established connections (to cause a denial of service) or to create a malicious download on connections.
For a successful attack, the attacker needs to correctly “guess” several connection parameters, such as the address / port of the source and destination. And for widely used protocols, this may not be as difficult as it might seem. If the attacker knows the host addresses and one of the ports (since we are talking about a common service), he will only need to “guess” one port. Even if the client is using a truly random source port (which isn't always true), the attacker only needs to try 65536 ports in a short amount of time. (In most cases, even (65536-1024) ports, i.e. only non-privileged ports - translator's note))
But what's really hard to guess for an attacker is the correct sequence number (and its confirmation). If both hosts choose starting number sequence number randomly (or you use sequence number modulation for hosts that have a “weak” ISN (Initial Sequence Number) generator), then the attacker will not be able to pick up the appropriate value at the right moment of the connection.
During the existence of a valid TCP connection, sequence numbers (and acknowledgments) for individual packets change according to certain rules.
For example, if a host sends some data segment and its receiver acknowledges receipt, there should be no reason why the sender should send the segment data again. But, in fact, an attempt to overwrite parts of information already received by the host is not a violation TCP protocol, although it can be a form of attack.
pf uses rules to determine the smallest range for legitimate sequence numbers. In general, pf can only accurately determine the authenticity of 30,000 out of 4294967296 possible numbers sequence at any time during the connection. Only if the sequence number and acknowledgment is in this window will pf make sure the packet is legitimate and let it through.
If during the query to the state table is found suitable entry, on the next step sequence numbers of packets stored in a table are checked for range possible values. If the comparison fails, pf will generate a "BAD state" message and discard the packet without evaluating the rule set. There are two reasons why a rule match may not occur: it will almost certainly be an error to skip a packet, because if the calculation of the set will lead to hitting the option in the rule "keep state" and pf will not be able to decide and create new record because it will cause conflicting keys to appear in the table.
In order to see and log "BAD state" messages, you need to enable debug mode using the command:
$ pfctl-xm
Debug messages are written to the console by default, and syslogd also writes them to /var/log/messages. Look for posts starting with "pf":
pf:badstate:TCP 192.168.1.10:20 192.168.1.10:20 192.168.1.200:64828
[ lo=1185380879high=1185380879win=33304modulator=0wscale=1]
4:4 A seq=1185380879 ack=1046638749 len=1448 ackskew=0 pkts=940:631
dir=out,fwd
pf: State failure on: 1 |
These messages always come in pairs. The first message shows the entry in the state table at the time the packet was blocked and the sequence number of the packet that resulted in the error. The second entry displays the conditions that were violated.
At the end of the first message, you will see whether a status entry was created for an incoming (dir=in) or outgoing (dir=out) packet, and whether the blocked packet went in the same direction (dir=,fwd) or the opposite direction (dir=,rev ) direction.
The entry in the table contains three addresses: pairs of ports, two of which are always equal to each other, if the connection has not undergone nat, rdr or bnat translation. For outgoing connections, the source is displayed on the left and the destination of the packet is displayed on the right. If outgoing connection enables source address translation, the pair in the middle shows the source after translation. For incoming connections, the source is on the right side of the output, and the destination address is in the middle. If an incoming connection undergoes a destination address translation, the ip/port pair on the left shows the destination after the translation. This format corresponds to the output of pfctl -ss, with the only difference being that pfctl shows the packet's direction using arrows.
In the output, you can see the current sequence numbers of the hosts in square brackets. So the value "4:4" means that the connection is established completely (smaller values are more likely at the stage of establishing a connection, larger ones - by the time the connection is closed)."A" means that the blocked packet had the ACK flag set (same as tcpdump's flag output), followed by the values of the sequence numbers (seq=) and (ack=) in the blocked packets, and the payload length of the packet - the data length (len =). askskew is part of the internal representation of the data in the table, and is only used for non-zero values.
The entry "pkts=930:631" means that it matched 940 packets going in the same direction as the one that caused the creation of this entry, and 631 packets in the opposite direction. These counters will be especially useful when troubleshooting problems during the connection establishment phase, if one of them zero, this would be contrary to your expectation that the given entry matches packets traveling in both directions.
The next message will contain a list of one or more digits. Each digit represents the check on which the error occurred:
- the packet window size exceeds the receiver's maximum size (seq + len > high)
- the packet contains already transferred data (seq< lo - win)
- ackskew is less than the minimum value
- ackskew is greater than the maximum value
- same as (1) but with a difference (seq + len > high + win)
- same as in (2), but (seq< lo - maximum win)
Fortunately, "BAD state" messages do not apply to real everyday traffic, and checking the pf of the sequence number avoids most anomalies. If you see these messages appearing intermittently and don't notice a large number of hanging connections, you can simply ignore them. There are many implementations of TCP/IP running on the Internet, and some of them can sometimes generate erroneous packets.
However, this class of problems can be easily diagnosed by the appearance of "BAD state" messages that appear only in such cases.
Create state recordsTCP over initialSYN to the package.
Ideally, state records should be created when the first SYN packet occurs.
You can force the use of this rule using the principle:
“Use "flags S/SA" options in all "pass proto tcp keep state" rules”
Only (and only) initial SYN packets have the SYN flag set and the ACK collected. When applying the "keep state" option is bound only to initial SYN packets, only those packets will create entries in the state table. Thus, any existing entry in the state table will be derived from the packet's initial SYN.
The reason for creating entries only for initial packets is a TCP extension called "window scaling" defined in RFC1323. The TCP header field used to announce the size of received windows is too small for today's high speed links. Modern implementations of TCP/IP prefer using larger window sizes than can fit in the existing field. Window size scaling means that all window sizes known from the destination host must be multiplied by a specific value specified by the destination, rather than taken on their own. In order for this scheme to work, both hosts must support the extension and indicate to each other their ability to implement it during the connection establishment (“handshake”) using TCP options. These options are present only in the initial SYN and SYN+ACK packets. And only if each of these packets contains an option, the nexus will be successful and the window size of all subsequent packets will be multiplied by a factor.
If pf were "unaware" of the window scaling being used, the provided value would be taken without the factor, and the calculation of window sizes for acceptable sequence number values would be incorrect. Typically, hosts provide small window sizes at the beginning of the connection and increase them during the connection. Unaware of the existence of window resizing factors, pf will start blocking packets at some point, because it will assume that one of the hosts is trying to bypass the maximum window size provided by the "peer". The effects of this may be more or less noticeable. Sometimes, hosts will respond to packet loss by switching to a so-called. “loss recovery mode” and will advertise a smaller window size. After pf retransmits the packets dropped the first time, the window sizes will continue to increase, up to the point where pf starts blocking them again. An external manifestation may be a temporary hangup of connections and poor performance. Possibly also complete freeze or connection reset by timeout.
But pf knows about window scaling and supports it. However, the prerequisite for creating state table entries for the first SYN packets is that pf can associate the first two packets of the connection with the entry in the table. And since full window size factor matching occurs only in these first two packets, there is no reliable way to determine these factor after link negotiation.
In the past, window scaling was not widely used, but this is changing rapidly. Only recently Linux has this option enabled by default. If you're having trouble with hanging connections, especially with some combinations of hosts, and you're seeing "BAD state" messages related to those connections, check that you're actually creating state table entries on the first packets of the connection.
You can determine if pf uses the scaling option for the connection from pfctl output:
$ pfctl -vss
kue0 tcp 10.1.2.3:10604 -> 10.2.3.4:80 ESTABLISHED:ESTABLISHED
wscale 0wscale 1
If there is a "wscale x" entry printed on the second line (even if x is zero), pf knows that the connection is using scaling.
Another simple method for troubleshooting scaling issues is to temporarily turn off scaling support and replay the situation. In OpenBSD, the use of scaling can be controlled by the sysctl option:
$ sysctlnet.inet.tcp.rfc1323
net.inet.tcp.rfc1323=1
$ sysctl-wsysctlnet.inet.tcp.rfc1323=0
net.inet.tcp.rfc1323: 1 -> 0
Similar problems appear when you create state table entries for packets other than the initial SYN and use the "moulate state" option or translation. In both cases, the translation is made at the beginning of the connection. If the first packet is not translated, the forwarding of subsequent packets usually discourages the receiving end and causes the replies sent to be blocked by pf with a "BAD state" message.
