WinPcap  4.1.3
Modules
WinPcap internals

This portion of the manual describes the internal structure and interfaces of WinPcap, starting from the lowest-level module. It is targeted at people that must extend or modify this software, or to the ones interested in how it works. Therefore, developers who just want to use WinPcap in their software don't need to read it.

WinPcap structure

Quoted from the home page of winpcap:

WinPcap is an architecture for packet capture and network analysis for the Win32 platforms. It includes a kernel-level packet filter, a low-level dynamic link library (packet.dll), and a high-level and system-independent library (wpcap.dll).

Why we use the term "architecture" rather than "library"? Because packet capture is a low level mechanism that requires a strict interaction with the network adapter and with the operating system, in particular with its networking implementation, so a simple library is not sufficient.

The following figure shows the various components of WinPcap:

Main components of WinPcap.

First, a capture system needs to bypass the operating systems's protocol stack in order to access the raw data transiting on the network. This requires a portion running inside the kernel of OS, interacting directly with the network interface drivers. This portion is very system dependent, and in our solution it is realized as a device driver, called Netgroup Packet Filter (NPF); we provide different versions of the driver for Windows 95, Windows 98, Windows ME, Windows NT 4, Windows 2000 and Windows XP. These drivers offer both basic features like packet capture and injection, as well as more advanced ones like a programmable filtering system and a monitoring engine. The first one can be used to restrict a capture session to a subset of the network traffic (e.g. it is possible to capture only the ftp traffic generated by a particular host), the second one provides a powerful but simple to use mechanism to obtain statistics on the traffic (e.g. it is possible to obtain the network load or the amount of data exchanged between two hosts).

Second, the capture system must export an interface that user-level applications will use to take advantage of the features provided by the kernel driver. WinPcap provides two different libraries: packet.dll and wpcap.dll

The first one offers a low-level API that can be used to directly access the functions of the driver, with a programming interface independent from the Microsoft OS. 

The second one exports a more powerful set of high level capture primitives that are compatible with libpcap, the well known Unix capture library. These functions enable packet capture in a manner that is independent of the underlying network hardware and operating system.

Throughout this documentation we will refer to the Packet Driver API or packet.dll as the first set of functions, whereas wpcap, wpcap.dll or libpcap will refer to the to the second one.

More...

Modules

 NPF driver internals manual
 

This section documents the internals of the Netgroup Packet Filter (NPF), the kernel portion of WinPcap. Normal users are probably interested in how to use WinPcap and not in its internal structure. Therefore the information present in this module is destined mainly to WinPcap developers and maintainers, or to the people interested in how the driver works. In particular, a good knowledge of OSes, networking and Win32 kernel programming and device drivers development is required to profitably read this section. 

NPF is the WinPcap component that does the hard work, processing the packets that transit on the network and exporting capture, injection and analysis capabilities to user-level.

The following paragraphs will describe the interaction of NPF with the OS and its basic structure.

NPF and NDIS

NDIS (Network Driver Interface Specification) is a standard that defines the communication between a network adapter (or, better, the driver that manages it) and the protocol drivers (that implement for example TCP/IP). Main NDIS purpose is to act as a wrapper that allows protocol drivers to send and receive packets onto a network (LAN or WAN) without caring either the particular adapter or the particular Win32 operating system.

NDIS supports three types of network drivers:

  1. Network interface card or NIC drivers. NIC drivers directly manage network interface cards, referred to as NICs. The NIC drivers interface directly to the hardware at their lower edge and at their upper edge present an interface to allow upper layers to send packets on the network, to handle interrupts, to reset the NIC, to halt the NIC and to query and set the operational characteristics of the driver. NIC drivers can be either miniports or legacy full NIC drivers.
    • Miniport drivers implement only the hardware-specific operations necessary to manage a NIC, including sending and receiving data on the NIC. Operations common to all lowest level NIC drivers, such as synchronization, is provided by NDIS. Miniports do not call operating system routines directly; their interface to the operating system is NDIS.
      A miniport does not keep track of bindings. It merely passes packets up to NDIS and NDIS makes sure that these packets are passed to the correct protocols.
    • Full NIC drivers have been written to perform both hardware-specific operations and all the synchronization and queuing operations usually done by NDIS. Full NIC drivers, for instance, maintain their own binding information for indicating received data. 
  2. Intermediate drivers. Intermediate drivers interface between an upper-level driver such as a protocol driver and a miniport. To the upper-level driver, an intermediate driver looks like a miniport. To a miniport, the intermediate driver looks like a protocol driver. An intermediate protocol driver can layer on top of another intermediate driver although such layering could have a negative effect on system performance. A typical reason for developing an intermediate driver is to perform media translation between an existing legacy protocol driver and a miniport that manages a NIC for a new media type unknown to the protocol driver. For instance, an intermediate driver could translate from LAN protocol to ATM protocol. An intermediate driver cannot communicate with user-mode applications, but only with other NDIS drivers.
  3. Transport drivers or protocol drivers. A protocol driver implements a network protocol stack such as IPX/SPX or TCP/IP, offering its services over one or more network interface cards. A protocol driver services application-layer clients at its upper edge and connects to one or more NIC driver(s) or intermediate NDIS driver(s) at its lower edge.

NPF is implemented as a protocol driver. This is not the best possible choice from the performance point of view, but allows reasonable independence from the MAC layer and as well as complete access to the raw traffic.

Notice that the various Win32 operating systems have different versions of NDIS: NPF is NDIS 5 compliant under Windows 2000 and its derivations (like Windows XP), NDIS 3 compliant on the other Win32 platforms. 

Next figure shows the position of NPF inside the NDIS stack:

Figure 1: NPF inside NDIS.

The interaction with the OS is normally asynchronous. This means that the driver provides a set of callback functions that are invoked by the system when some operation is required to NPF. NPF exports callback functions for all the I/O operations of the applications: open, close, read, write, ioctl, etc.

The interaction with NDIS is asynchronous as well: events like the arrival of a new packet are notified to NPF through a callback function (Packet_tap() in this case). Furthermore, the interaction with NDIS and the NIC driver takes always place by means of non blocking functions: when NPF invokes a NDIS function, the call returns immediately; when the processing ends, NDIS invokes a specific NPF callback to inform that the function has finished. The driver exports a callback for any low-level operation, like sending packets, setting or requesting parameters on the NIC, etc.

NPF structure basics

Next figure shows the structure of WinPcap, with particular reference to the NPF driver.

Figure 2: NPF device driver.

NPF is able to perform a number of different operations: capture, monitoring, dump to disk, packet injection. The following paragraphs will describe shortly each of these operations.

Packet Capture

The most important operation of NPF is packet capture. During a capture, the driver sniffs the packets using a network interface and delivers them intact to the user-level applications. 

The capture process relies on two main components:

  • A packet filter that decides if an incoming packet has to be accepted and copied to the listening application. Most applications using NPF reject far more packets than those accepted, therefore a versatile and efficient packet filter is critical for good over-all performance. A packet filter is a function with boolean output that is applied to a packet. If the value of the function is true the capture driver copies the packet to the application; if it is false the packet is discarded. NPF packet filter is a bit more complex, because it determines not only if the packet should be kept, but also the amount of bytes to keep. The filtering system adopted by NPF derives from the BSD Packet Filter (BPF), a virtual processor able to execute filtering programs expressed in a pseudo-assembler and created at user level. The application takes a user-defined filter (e.g. “pick up all UDP packets”) and, using wpcap.dll, compiles them into a BPF program (e.g. “if the packet is IP and the protocol type field is equal to 17, then return true”). Then, the application uses the BIOCSETF IOCTL to inject the filter in the kernel. At this point, the program is executed for every incoming packet, and only the conformant packets are accepted. Unlike traditional solutions, NPF does not interpret the filters, but it executes them. For performance reasons, before using the filter NPF feeds it to a JIT compiler that translates it into a native 80x86 function. When a packet is captured, NPF calls this native function instead of invoking the filter interpreter, and this makes the process very fast. The concept behind this optimization is very similar to the one of Java jitters.

  • A circular buffer to store the packets and avoid loss. A packet is stored in the buffer with a header that maintains information like the timestamp and the size of the packet. Moreover, an alignment padding is inserted between the packets in order to speed-up the access to their data by the applications. Groups of  packets can be copied with a single operation from the NPF buffer to the applications. This improves performances because it minimizes the number of reads. If the buffer is full when a new packet arrives, the packet is discarded and hence it's lost. Both kernel and user buffer can be changed at runtime for maximum versatility: packet.dll and wpcap.dll provide functions for this purpose.

The size of the user buffer is very important because it determines the maximum amount of data that can be copied from kernel space to user space within a single system call. On the other hand, it can be noticed that also the minimum amount of data that can be copied in a single call is extremely important. In presence of a large value for this variable, the kernel waits for the arrival of several packets before copying the data to the user. This guarantees a low number of system calls, i.e. low processor usage, which is a good setting for applications like sniffers. On the other side, a small value means that the kernel will copy the packets as soon as the application is ready to receive them. This is excellent for real time applications (like, for example, ARP redirectors or bridges) that need the better responsiveness from the kernel. From this point of view, NPF has a configurable behavior, that allows users to choose between best efficiency or best responsiveness (or any intermediate situation). 

The wpcap library includes a couple of system calls that can be used both to set the timeout after which a read expires and the minimum amount of data that can be transferred to the application. By default, the read timeout is 1 second, and the minimum amount of data copied between the kernel and the application is 16K.

Packet injection

NPF allows to write raw packets to the network. To send data, a user-level application performs a WriteFile() system call on the NPF device file. The data is sent to the network as is, without encapsulating it in any protocol, therefore the application will have to build the various headers for each packet. The application usually does not need to generate the FCS because it is calculated by the network adapter hardware and it is attached automatically at the end of a packet before sending it to the network.

In normal situations, the sending rate of the packets to the network is not very high because of the need of a system call for each packet. For this reason, the possibility to send a single packet more than once with a single write system call has been added. The user-level application can set, with an IOCTL call (code pBIOCSWRITEREP), the number of times a single packet will be repeated: for example, if this value is set to 1000, every raw packet written by the application on the driver's device file will be sent 1000 times. This feature can be used to generate high speed traffic for testing purposes: the overload of context switches is no longer present, so performance is remarkably better. 

Network monitoring

WinPcap offers a kernel-level programmable monitoring module, able to calculate simple statistics on the network traffic. The idea behind this module is shown in Figure 2: the statistics can be gathered without the need to copy the packets to the application, that simply receives and displays the results obtained from the monitoring engine. This allows to avoid great part of the capture overhead in terms of memory and CPU clocks.

The monitoring engine is made of a classifier followed by a counter. The packets are classified using the filtering engine of NPF, that provides a configurable way to select a subset of the traffic. The data that pass the filter go to the counter, that keeps some variables like the number of packets and the amount of bytes accepted by the filter and updates them with the data of the incoming packets. These variables are passed to the user-level application at regular intervals whose period can be configured by the user. No buffers are allocated at kernel and user level.

Dump to disk

The dump to disk capability can be used to save the network data to disk directly from kernel mode.

Figure 3: packet capture versus kernel-level dump.

In traditional systems, the path covered by the packets that are saved to disk is the one followed by the black arrows in Figure 3: every packet is copied several times, and normally 4 buffers are allocated: the one of the capture driver, the one in the application that keeps the captured data, the one of the stdio functions (or similar) that are used by the application to write on file, and finally the one of the file system.

When the kernel-level traffic logging feature of NPF is enabled, the capture driver addresses the file system directly, hence the path covered by the packets is the one of the red dotted arrow: only two buffers and a single copy are necessary, the number of system call is drastically reduced, therefore the performance is considerably better.

Current implementation dumps the to disk in the widely used libpcap format. It gives also the possibility to filter the traffic before the dump process in order to select the packet that will go to the disk.

Further reading

The structure of NPF and its filtering engine derive directly from the one of the BSD Packet Filter (BPF), so if you are interested the subject you can read the following papers:

- S. McCanne and V. Jacobson, The BSD Packet Filter: A New Architecture for User-level Packet Capture. Proceedings of the 1993 Winter USENIX Technical Conference (San Diego, CA, Jan. 1993), USENIX. 

- A. Begel, S. McCanne, S.L.Graham, BPF+: Exploiting Global Data-flow Optimization in a Generalized Packet Filter Architecture, Proceedings of ACM SIGCOMM '99, pages 123-134, Conference on Applications, technologies, architectures, and protocols for computer communications, August 30 - September 3, 1999, Cambridge, USA

Note

The code documented in this manual is the one of the Windows NTx version of NPF. The Windows 9x code is very similar, but it is less efficient and lacks advanced features like kernel-mode dump.


 
 How to compile WinPcap
 

This section explains how to compile WinPcap, both the kernel level and the user-level portion, on the various Win32 platforms. The source code can be found on the WinPcap website.

Compiling the driver

Two main NPF source trees are available for compilation:  Windows NTx and Windows 9x. Note that, since the NPF Driver is platform-dependent, it is STRONGLY suggested to compile it for the OS where it will be used, in order to link the correct DDK libraries. For example, if you compile the driver with the Windows NT 4 DDK, it will not work properly on Windows 2000 and vice versa.

Compiling the driver for Windows NT4

Software requirements:

  • Microsoft Driver Developer Kit (DDK) for Windows NT4
  • A recent version of the Microsoft Platform Software Development Kit (SDK) that is compatible with Visual Studio 6 (the latest compatible one is Platform SDK February 2003). This version of the PSDK is available on the Microsoft web site at http://www.microsoft.com/msdownload/platformsdk/sdkupdate/psdk-full.htm. It can be ordered online at http://www.qmedia.ca/launch/psdk.htm, and it's also available to Microsoft MSDN subscribers on the Subscribers Downloads web site.
  • Microsoft Visual C++ 6.0 with Service Pack 5 or 6 (both the service packs are available online on the Microsoft web site).

If your system satisfies these requirements, follow these steps:

  1. From the Windows NT Start menu, select the folder Programs and then Development Kits, then Windows NT4 DDK. From here select the voice Checked Build Environment if you want to build a debug version, or Free Build Environment if you want to build a release version.
  2. A command prompt will be opened. Move to the directory PacketNTx inside the WinPcap source folder and type the command

    CompileDriver

    This script will generate the driver (npf.sys). The binary will be put in one of these folders
    • Free Build Environment: winpcap\PacketNTx\driver\bin\NT4\i386\free
    • Checked Build Environment: winpcap\PacketNTx\driver\bin\NT4\i386\checked

Warning: sometimes, during the compilation of the driver, a lot of 'last line incomplete' errors are generated. Ignore these errors and let the compilation process continue, they are due to bugs in some DDK versions.

Compiling the driver for Windows 2000/XP/2003/Vista/2008/Win7/2008R2 (x86 and x64)

Software requirements:

  • Microsoft Windows Driver Kit (WDK) 6001.18002. As of release 4.1, WinPcap is compiled with WDK 6001.18002.

NOTE: it should be possible to use older DDKs to compile WinPcap, but you might need to manually modify the compilation scripts in order to disable PREfast (PREfast is a static code analysis tool shipped with recent versions of the DDK/WDK).

If your system satisfies these requirements, follow these steps:

  1. From the Windows Start menu, select the folder Programs and then Windows Driver K, then WDK 6001.18002, then Build Environments.
    • x86 driver: Choose Windows 2000 and then Windows 2000 x86 Free Build Environment if you want to build a release version or Windows 2000 x86 Checked Build Environment if you want to build a debug version.
    • x64 driver: Choose Windows Server 2003 and then Windows Server 2003 x64 Free Build Environment if you want to build a release version or Windows Server 2003 x64 Checked Build Environment if you want to build a debug version.
  2. A command prompt will be opened. Move to the directory PacketNTx inside the WinPcap source folder and type the command

    CompileDriver

    This script will generate the driver (npf.sys). The binary will be put in one of these folders
    • x86 driver (both Free and Checked Build): winpcap\PacketNTx\driver\bin\i386
    • x64 driver (both Free and Checked Build): winpcap\PacketNTx\driver\bin\amd64

Compiling the driver on Windows 9x

NOTE: this Windows platform is no longer supported by WinPcap. However, the sources for these operating systems are still available in the sources package.

To compile the driver for Windows 9x you will need:

  • Driver Developer Kit (DDK) for Windows 95/98/ME
  • A recent version of the Microsoft Platform Software Development Kit (SDK) that is compatible with Visual Studio 6 (the latest compatible one is Platform SDK February 2003). This version of the PSDK is available on the Microsoft web site at http://www.microsoft.com/msdownload/platformsdk/sdkupdate/psdk-full.htm. It can be ordered online at http://www.qmedia.ca/launch/psdk.htm, and it's also available to Microsoft MSDN subscribers on the Subscribers Downloads web site.
  • Microsoft Visual C++ 6.0 with Service Pack 5 or 6 (both the service packs are available online on the Microsoft web site).

The steps to follow are:

  1. Open a DOS shell
  2. Go to the VisualC++ BIN directory (for example C:\DEVSTUDIO\VC\BIN) and execute the command

    Vcvars32
  3. Go to the SDK directory (for example C:\MSSDK) and execute the command

    Setenv sdk_path

    where sdk_path is the directory of SDK (for example Setenv C:\MSSDK)
  4. Go to the DDK directory (for example C:\DDK) and execute the command

    Ddkenv 32 net
  5. Move to the directory whit the driver's source code and type the command

    nmake rtl

    to obtain a release version, or

    nmake

    to obtain a debug version.
    The release version of packet.vxd will be placed in the retail directory, the debug version in the debug directory.

Warning: On some systems the NMAKE utility is not able to launch ADRC2VXD, this means that the driver binary is generated correctly, but without the copyright information. We don't know the cause of this problem.

Compiling packet.dll

The source tree for this DLL is located in PacketNTx\dll\.

NOTE: the 9x family of Windows operating systems is no longer supported by WinPcap. However, the sources for these operating systems are still available in the sources package.

Software requirements:

  • Microsoft Visual Studio 2005 SP1. It's theoretically possible to compile the x86 version with Visual Studio 6, but the project files are no longer maintained.
  • The AirPcap developer's pack from http://www.cacetech.com/products/airpcap.html. The AirPcap developer's pack needs to be unzipped in a folder in the same folder where the WinPcap sources have been unzipped.

To compile the PACKET.DLL, load the project packet.sln contained in the directory PacketNTx\dll\project in Visual Studio 2005. There are several project configurations, each of them available for the x86 (Win32) and x64 platforms:

  • Release: standard release configuration
  • Debug: standard debug configuration
  • Release NT4: release configuration able to run on NT4. It does not include Wan and IP helper API support.
  • Debug NT4: debug configuration able to run on NT4. It does not include Wan and IP helper API support.
  • Release No NetMon: release configuration able to run on Vista. It does not include Wan support (with the NetMon API).
  • Debug No NetMon: debug configuration able to run on Vista. It does not include Wan support (with the NetMon API).
  • Release LOG_TO_FILE: standard release configuration with tracing to file enabled.
  • Release NT4 LOG_TO_FILE: release configuration able to run on NT4 with tracing to file enabled. It does not include Wan and IP helper API support.
  • Release No NetMon LOG_TO_FILE: release configuration able to run on Vista with tracing to file enabled. It does not include Wan support (with the NetMon API).

Choose the desired configuration and build the project to obtain the binary files.

Compiling wpcap.dll

wpcap.dll can be compiled for any Win32 platform and the generated dll is system independent.

System Requirements:

  • Microsoft Visual Studio 2005 SP1. It's theoretically possible to compile the x86 version with Visual Studio 6, but the project files are no longer maintained.
  • The AirPcap developer's pack from http://www.cacetech.com/products/airpcap.htm. The AirPcap developer's pack needs to be unzipped in a folder in the same folder where the WinPcap sources have been unzipped.There are eight build project configurations:

To compile the wpcap.dll, load the project wpcap.sln contained in the directory wpcap\PRJ in Visual Studio 2005. There are several project configurations, each of them available for the x86 (Win32) and x64 platforms:

  • Release: standard release configuration
  • Debug: standard debug configuration
  • Release No AirPcap: release configuration without support for AirPcap adapters.
  • Debug No AirPcap: debug configuration without support for AirPcap adapters.

Choose the desired configuration and build the project to obtain the binary files.

Note: wpcap.dll contains the source code of libpcap from www.tcpdump.org, with some modifications for remote capture. You will be able to include and build a different libpcap version simply copying it in the directory winpcap\wpcap\prj of the WinPcap source code distribution, but you must use the "Debug" or "Release" build configurations.
 


 
 Packet.dll – Packet Driver API
  Packet.dll is a dynamic link library that offers a set of low level functions to:
  • install, start and stop the NPF device driver
  • Receive packets from the NPF driver
  • send packets to the NPF driver
  • obtain the list of the available network adapters
  • retrieve various information about an adapter, like the description and the list of addresses and netmasks
  • query and set various low-level parameters of an adapter

There are two versions of packet.dll: the first one runs under Windows 95/98/ME, the second one is for  Windows NT/2000/XP.

Packet.dll was created to provide a layer to access the low level functionalities of WinPcap in a system independent way. This library handles all the system-dependent details (like managing the devices, interacting with the OS to manage the adapters, looking for the information in the registry and so on), and exports an API that is uniform across all Windows OSes. In this way, applications or libraries based on it can run without being recompiled under any Windows operating system.

However, not all of the packet.dll API is totally portable: some advanced features, like kernel-mode dump, are present only in the WinNTx version of WinPcap, while packet.dll for Win9x does not provide them. On the other side, the NTx version is a superset of the 9x one, in other words all the function present in the Win9x version are present in WinNTx too.

The other important feature of this library is its ability to handle NPF driver. Packet.dll transparently installs and starts the driver when an application attempts to access an adapter. This avoids the manual installation of the driver through the control panel.

Important note, read carefully!

The source code of Packet.dll is freely available and completely documented. However, packet.dll should be considered an internal API, because its purpose inside WinPcap is to be a building block for the real public API: wpcap.dll.

As a consequence, since the normal and suggested way for an application to use WinPcap is through wpcap.dll, we don't guarantee that the packet.dll API will not be changed in future releases of winpcap, and we don't provide support for this API. For the same reason, this manual doesn't contain any more the Doxygen-generated documentation of Packet.dll: the user will have to run Doxygen on his own to create it, or read the comments in the source code.


 

Detailed Description

This portion of the manual describes the internal structure and interfaces of WinPcap, starting from the lowest-level module. It is targeted at people that must extend or modify this software, or to the ones interested in how it works. Therefore, developers who just want to use WinPcap in their software don't need to read it.

WinPcap structure

Quoted from the home page of winpcap:

WinPcap is an architecture for packet capture and network analysis for the Win32 platforms. It includes a kernel-level packet filter, a low-level dynamic link library (packet.dll), and a high-level and system-independent library (wpcap.dll).

Why we use the term "architecture" rather than "library"? Because packet capture is a low level mechanism that requires a strict interaction with the network adapter and with the operating system, in particular with its networking implementation, so a simple library is not sufficient.

The following figure shows the various components of WinPcap:

Main components of WinPcap.

First, a capture system needs to bypass the operating systems's protocol stack in order to access the raw data transiting on the network. This requires a portion running inside the kernel of OS, interacting directly with the network interface drivers. This portion is very system dependent, and in our solution it is realized as a device driver, called Netgroup Packet Filter (NPF); we provide different versions of the driver for Windows 95, Windows 98, Windows ME, Windows NT 4, Windows 2000 and Windows XP. These drivers offer both basic features like packet capture and injection, as well as more advanced ones like a programmable filtering system and a monitoring engine. The first one can be used to restrict a capture session to a subset of the network traffic (e.g. it is possible to capture only the ftp traffic generated by a particular host), the second one provides a powerful but simple to use mechanism to obtain statistics on the traffic (e.g. it is possible to obtain the network load or the amount of data exchanged between two hosts).

Second, the capture system must export an interface that user-level applications will use to take advantage of the features provided by the kernel driver. WinPcap provides two different libraries: packet.dll and wpcap.dll

The first one offers a low-level API that can be used to directly access the functions of the driver, with a programming interface independent from the Microsoft OS. 

The second one exports a more powerful set of high level capture primitives that are compatible with libpcap, the well known Unix capture library. These functions enable packet capture in a manner that is independent of the underlying network hardware and operating system.

Throughout this documentation we will refer to the Packet Driver API or packet.dll as the first set of functions, whereas wpcap, wpcap.dll or libpcap will refer to the to the second one.


documentation. Copyright (c) 2002-2005 Politecnico di Torino. Copyright (c) 2005-2010 CACE Technologies. Copyright (c) 2010-2013 Riverbed Technology. All rights reserved.