Lesson Notes
TCP/IP and OSI Model
Module 2: Networking Basics. Pure theory—why models matter, OSI 7 layers (PDU, functions, protocols, attacks), encapsulation, TCP/IP 4-layer stack, TCP/UDP deep dive, TLS in transport (handshake, PFS, AEAD), layered attacks, pentest applications. Builds on Intro to Networks.
Module 2: TCP/IP and OSI Model — Comprehensive Theoretical Guide
Module 2: Networking Basics. Pure theory (no labs or commands). For authorized cybersecurity pros: deep dive into layers for pentesting recon, MITM, and crypto analysis. Includes where TLS fits in the transport layer. Builds on Intro to Networks (Lesson 11).
1. Why Models Matter: Standardizing Network Communication
OSI and TCP/IP provide layered abstractions for how data travels from an application down to the wire and back. Each layer has a clear role, which makes it easier to design protocols, troubleshoot, and—for pentesters—predict attack surfaces. For example: an attacker might poison ARP at Layer 2 to position for a MITM, then exploit an application vulnerability at Layer 7. Crypto (TLS) spans the presentation and transport layers to provide confidentiality for application data. Threat model: layers correspond to attack vectors. Lower layers (physical, data link) are where sniffing and spoofing happen; higher layers (application) are where injection and business-logic flaws live. A large share of real-world exploits involve more than one layer (e.g. a Layer 4 DoS causing a Layer 7 outage). Understanding the stack helps you map and test ethically.
2. OSI Model: Seven Layers (Detailed Functions)
The OSI reference model (ISO 7498) divides network communication into seven layers. The unit of data at each layer has a name: at Layer 7–5 it is often called "data"; at Layer 4 it is a "segment" (TCP/UDP); at Layer 3 a "packet" (IP); at Layer 2 a "frame" (Ethernet); at Layer 1, bits or signals. Layer 7 (Application): User-facing services and APIs—HTTP, DNS, SMTP, FTP. Attacks: injection (SQLi, XSS), service enumeration. Layer 6 (Presentation): Syntax translation, encryption, compression—e.g. character encoding, TLS handshake, image formats. Attacks: format-string bugs, crypto downgrade. Layer 5 (Session): Dialog control, sync, checkpoints—e.g. NetBIOS, RPC. Attacks: session hijacking, session exhaustion DoS. Layer 4 (Transport): End-to-end delivery, flow and error control—TCP (reliable) and UDP (best-effort). Attacks: port scans, SYN floods, UDP floods; TLS termination happens here. Layer 3 (Network): Logical addressing and routing—IP (v4/v6), ICMP; IPsec can encrypt at this layer. Attacks: spoofing, TTL manipulation, route manipulation. Layer 2 (Data Link): Physical (MAC) addressing, error detection—Ethernet, PPP, ARP. Attacks: ARP poisoning, MAC flooding, STP attacks. Layer 1 (Physical): Signal transmission—cables, Wi-Fi radio. Attacks: eavesdropping, jamming, cable taps.
Encapsulation and Decapsulation
When sending, data flows down the stack. Application data is processed at Layer 6 (e.g. encrypted or compressed), session information may be added at Layer 5, then a TCP or UDP segment is created at Layer 4 (with port numbers), then an IP packet at Layer 3 (with IP addresses), then an Ethernet frame at Layer 2 (with MAC addresses and often a CRC), and finally bits or signals at Layer 1. Each layer adds its header (and sometimes a trailer); the previous layer's output becomes the next layer's payload. On receive, the process reverses: headers are stripped as the data moves up. This is encapsulation (down) and decapsulation (up).
3. TCP/IP Model: Practical Four-Layer Stack (Internet Reality)
The TCP/IP (DOD) model is what the internet actually uses. It maps to OSI as follows. TCP/IP Application layer corresponds to OSI 5–7: HTTP/HTTPS, DNS, SSH, and application logic; app-level crypto (e.g. HTTPS) is implemented here from a user perspective. TCP/IP Transport corresponds to OSI 4: TCP, UDP, SCTP; TLS/SSL operates here—it encrypts the payload from the layers above, so confidentiality is provided at the transport layer. TCP/IP Internet corresponds to OSI 3: IPv4, IPv6, ICMP; IPsec provides optional L3 encryption (e.g. VPNs). TCP/IP Network Access (or Link) corresponds to OSI 1–2: Ethernet, Wi-Fi, ARP; there is no standard native crypto at L2, so L3 or L4 encryption (IPsec or TLS) is what protects data.
4. Transport Layer Deep Dive: TCP
TCP (Transmission Control Protocol) is connection-oriented and reliable. Connection setup uses a three-way handshake: the client sends SYN, the server replies with SYN-ACK, the client sends ACK. State is maintained (connection established). TCP provides reliability through sequence numbers, acknowledgments (ACKs), and retransmission of lost segments; flow control (window size) and congestion avoidance manage throughput. Ports identify the application: each connection has a source and destination port (e.g. client 54321 → server 443). Important flags: SYN (initiate connection), ACK (acknowledge), FIN (graceful close), RST (reset/abort), PSH (push data). Attacks: SYN flood (exhaust server with half-open connections by not completing the handshake), and port scanning (e.g. SYN scan to discover open ports without fully connecting).
4. Transport Layer Deep Dive: UDP
UDP (User Datagram Protocol) is connectionless and does not guarantee delivery. There is no handshake and no retransmission; only an optional checksum. It is used when low latency or simplicity matters more than reliability—e.g. DNS, DHCP, VoIP, streaming, gaming. Because UDP does not maintain state, it is easier to spoof (e.g. forged source addresses) and is commonly abused in amplification attacks: the attacker sends a small request with a spoofed victim source IP to a server that sends a much larger response (e.g. DNS or NTP), so the response is reflected and amplified toward the victim, contributing to DDoS.
4. TLS/SSL in the Transport Layer (Crypto Fundamentals)
TLS (Transport Layer Security) spans the transport and presentation layers: the TCP connection is established first, then TLS runs on top of it to encrypt and authenticate. The handshake (TLS 1.3 simplified): (1) ClientHello—client sends supported ciphers and SNI (Server Name Indication, the intended hostname). (2) ServerHello and certificate—server chooses cipher and proves identity with a CA-signed certificate (public key). (3) Key exchange—ephemeral Diffie-Hellman (e.g. ECDHE) generates a shared secret; TLS 1.3 mandates this for Perfect Forward Secrecy (PFS). (4) Finished—both sides confirm with an encrypted message. After the handshake, application data is sent in TLS records using authenticated encryption (AEAD, e.g. AES-GCM): the payload is ciphertext, so a MITM sees gibberish. Plain HTTP over TCP sends readable data; once TLS wraps it, the same data is encrypted inside TLS records over TCP. Pitfalls: downgrade attacks (e.g. SSLStrip forcing HTTP), weak ciphers (e.g. RC4), and missing HSTS (browser may accept HTTP). TLS 1.3 removes insecure options and makes PFS mandatory.
5. Protocol Interactions and Layered Attacks
Attacks often chain across layers. Example: ARP poisoning at Layer 2 positions the attacker on the path; the victim's traffic is sent to the attacker's MAC. The attacker can then spoof IP or manipulate routing (Layer 3), hijack or abuse TCP (Layer 4), and steal credentials from application traffic (Layer 7). If TLS is used correctly, the L7 data is encrypted and the attacker cannot read it; if the connection is downgraded to HTTP or TLS is misconfigured, the chain succeeds. PDU evolution as data moves down: application data (L7) becomes a TCP segment (L4), then an IP packet (L3), then an Ethernet frame (L2). Each step adds the appropriate header.
6. Pentest Applications and Detection
Layer targeting: recon and scanning often start at Layer 3 (ping sweeps) and Layer 4 (port scans with tools like Nmap); MITM tools may operate at Layer 2 (e.g. ARP spoofing). Crypto validation: in authorized tests, check that TLS is used (no HTTP for sensitive actions), that strong ciphers and TLS 1.2+ are enforced, and that certificate pinning or HSTS is used where appropriate. Anomalies: duplicate MAC addresses for the same IP can indicate ARP poisoning; unusual TTL values or routing asymmetry can suggest manipulation. Evolution: HTTP/3 uses QUIC (over UDP) with TLS integrated, offering encryption and faster setup. Pentest mindset: map the target's layers, identify which are weak (unencrypted, misconfigured, or missing controls), exploit only within scope, and recommend TLS 1.3 (or equivalent) everywhere for data in transit.
Key Takeaway for Lesson 12
OSI and TCP/IP provide layered models for network communication; each layer has distinct functions, protocols, and attack surfaces. Transport layer (Layer 4) delivers segments via TCP (reliable) or UDP (best-effort); TLS runs on top of TCP to encrypt application data, so confidentiality in transit depends on TLS (or IPsec at L3). Understanding encapsulation, TCP/UDP behavior, and the TLS handshake helps you interpret traffic, plan recon (e.g. port scans), and assess crypto. In pentesting, map layers, target weak ones ethically, and recommend TLS 1.3 (and HSTS) for all sensitive traffic. Next: ports, protocols, and services (HTTP vs HTTPS in practice).