Fingerprinting is a process of identifying operating system based on fingerprints that system returns during scan. There is 2 types of fingerprinting passive and active both offer pros and cons during scanning.

Active Passive
How it works Uses specially crafted packets Uses sniffing techniques to capture packets coming from a system
Analysis Responses are compared to a database of known responses Responses are analyzed, looking for details of the OS
Chance of detection Very high as it introduces traffic into network Low as sniffing does not introduce traffic into network

Baically all fingerprinting techniques based on analyzing the following:
IP TTL
IP ID
TCP Window size
TCP Options
DHCP Requests
ICMP Requests
HTTP Packets
Services
Open ports patterns

Active Fingerprinting
Lets look at simple example of doing active fingerprinting with nmap. It works by sending TCP, UDP packets at the target and watching response. Ones target responds nmap compares it to database to determine OS.
Example:

#nmap -O 192.168.0.1
Starting Nmap 6.40 ( http://nmap.org ) at 2016-09-22 12:08 EDT
Nmap scan report for 192.168.0.1
Host is up (0.0012s latency).
Not shown: 997 filtered ports
PORT     STATE SERVICE
22/tcp   open  ssh
443/tcp  open  https
4443/tcp open  pharos
MAC Address: B4:0C:25:4B:8A:12 (Palo Alto Networks)
Warning: OSScan results may be unreliable because we could not find at least 1 open and 1 closed port
Device type: general purpose
Running: Linux 2.6.X
OS CPE: cpe:/o:linux:linux_kernel:2.6
OS details: Linux 2.6.32
Network Distance: 1 hop

OS detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 20.61 seconds

Pro
Takes less time then Passive
Con
Active Fingerprinting is noize and can easily be detected with IDS/IPS systems

Passive Fingerprinting

OS IP Initial TTL TCP Window Size
Linux 64 5840
Google custom Linux 64 5720
FreeBSD 64 65535
Windows XP 128 65535
Windows Vista, 7 , Server 2008 128 8192
Cisco Router 255 4128

Pro
Very hard to detect as it does not generate traffic
Con
Takes much more time and is less reliable then active
One of the tools that can be used for this is p0f. Other sniffers can be used also.
Example:

apt-get install p0f
 sudo p0f -i eth0
p0f - passive os fingerprinting utility, version 2.0.8
(C) M. Zalewski , W. Stearns 
p0f: listening (SYN) on 'eth0', 262 sigs (14 generic, cksum 0F1F5CA2), rule: 'all'.
192.168.0.18:39610 - UNKNOWN [S20:64:1:60:M1460,S,T,N,W7:.:?:?] (up: 120 hrs) 
  -> 81.22.36.107:80 (link: ethernet/modem)
192.168.0.18:55938 - UNKNOWN [S20:64:1:60:M1460,S,T,N,W7:.:?:?] (up: 120 hrs) 
  -> 38.99.188.19:9869 (link: ethernet/modem)
192.168.0.18:39612 - UNKNOWN [S20:64:1:60:M1460,S,T,N,W7:.:?:?] (up: 120 hrs) 
  -> 81.22.36.107:80 (link: ethernet/modem)
192.168.0.18:39613 - UNKNOWN [S20:64:1:60:M1460,S,T,N,W7:.:?:?] (up: 120 hrs) 
  -> 81.22.36.107:80 (link: ethernet/modem)
192.168.0.18:39614 - UNKNOWN [S20:64:1:60:M1460,S,T,N,W7:.:?:?] (up: 120 hrs) 
  -> 81.22.36.107:80 (link: ethernet/modem)
192.168.0.18:55942 - UNKNOWN [S20:64:1:60:M1460,S,T,N,W7:.:?:?] (up: 120 hrs) 
  -> 38.99.188.19:9869 (link: ethernet/modem)
192.168.0.18:55943 - UNKNOWN [S20:64:1:60:M1460,S,T,N,W7:.:?:?] (up: 120 hrs) 
  -> 38.99.188.19:9869 (link: ethernet/modem)
192.168.0.18:55944 - UNKNOWN [S20:64:1:60:M1460,S,T,N,W7:.:?:?] (up: 120 hrs) 
  -> 38.99.188.19:9869 (link: ethernet/modem)
192.168.0.18:46744 - UNKNOWN [S20:64:1:60:M1460,S,T,N,W7:.:?:?] (up: 120 hrs) 
  -> 192.168.0.1:80 (link: ethernet/modem)