TCP/IP Notes

Intro / Network Addressing / Network Addressing IPv6 / Packet Headers / Networking and Data Link Layers / Transport Layer Protocols / Application Layer Protocols / IP Routing / ICMP / DHCP / Name Resolution and DNS / Security

APPLICATION LAYER PROTOCOLS

Protocols are the underlying standards upon which services operate. There are some instances where a protocol is represented by a service or application — for example a FTP client — and other cases where a service is not directly encountered by the end user, such as DNS.

The Application Layer protocols and services of the TCP/IP suite are, for the end user, the most important. It is in this layer where the end user interacts with network applications, whether in terms of an intranet (home or business network) or the global Internet. There are hundreds of various protocols and services at this level of the TCP/IP stack, several of which are considered more 'traditional' and generally find the most use.

Application Layer protocols and services (each protocol generally has it's own service or application which is associated with and uses the protocol and/or runs as a front-end) depend upon three factors being met:

1) Specified support underlying the protocol or service.

2) Definition of port address between client and server.

3) Necessary software allowing client and server to implement the requesting and providing of services.

The message structure in Application Layer services is known as a request/reply architecture, in which clients request services using 'request messages,' and servers reply using 'reply messages.' This pattern forms the traditional "client/server" architecture of the Internet, but there are also alternatives to this pattern, and services such as Bittorrent are a good example, in which each host can double as both a client and a server (peer-to-peer).

A second type of network traffic is server-to-server, which might be used as a RAID type of backup storage, DNS replication, load balancing, or for updating multiple machines within a server farm. When servers communicate with each other, the word 'replication' is generally used. When a server acts as a "client" to another server, it acts in the capacity of what it called a pull operation. The opposite (sender initiating the transfer) is called a push operation. Some services require either/or and sometimes both 'push-pull replication' in order to operate.

Below is an introduction to some of the more basic 'traditional' services, as well as some which are newer and gaining traction. A more complete list of Application Layer services can be found at:

https://en.wikipedia.org/wiki/Category:Application_layer_protocols.

File Transfer Protocol
----------------------------

FTP is documented in RFC 959 (https://www.ietf.org/rfc/rfc959.txt) and acts as a method for transferring files over TCP. The main components of a FTP communication are the user interface (command line or GUI), protocol interpreter (PI), FTP commands, data transfer process (DTP), files transferred, TCP transport, command connection, and data transfer connection. The last two elements break up the communication message between FTP commands and the actual data being transferred:

In Windows, FTP is included (ftp.exe). In Unix-like operating systems the ftp command is used. There are numerous client and server graphical user interfaces available on individual and multiple platforms, such as FileZilla (https://filezilla-project.org) and gFTP (https://github.com/masneyb/gftp).

The protocol interpreter (PI) is what reads the user commands and initiates the control connection from the client port to the server port (generally well-known port # 21) and initiates FTP commands and monitors the data transfer process. Often the server interprets the opposite or different versions of user-end commands, e.g. CD vs. CWD, GET vs RETR, etc.

FTP commands follow after the TCP header, and are processed by the receiving PI. For a list of commands, see:

https://en.wikipedia.org/wiki/List_of_FTP_commands.

The data transfer process (DTP) moves the data across the connection. File systems between two connected hosts can include many formats such as ASCII, binary and extended binary coded decimal interchange code (EBCDIC). FTP uses TCP exclusively to establish the connection, monitor the packets and adjust the data transfer window (based on congestion control and avoidance). The command connection is created when a FTP connection is established between hosts (in response to a command from one host to another) using port number 21 on the server end. The client uses a dynamic port number for handling the command connection, which must stay open to allow the DTP to take place via data transfer connection.

FTP includes two different modes: active and passive. In active mode, the client connects from a dynamic port number (above 1023) to the FTP server port number 21. The client will then listen on the port number directly above the one used to connect, while the server port makes another connection from port number 20. For example:

1) Client sends server a command to connect port 1047 to command port 21.

2) Server sends ACK to client's command port 1046.

3) Server starts connection from local data port 20 to client data port 1047.

4) Client sends ACK from data port 1047 to server data port 20.

Because of how this happens, active FTP can have issues going across firewalls (server rather than client initiates a data connection after initial command connection).

To deal with this issue, passive (PASV) FTP was developed. Specifically, in step 3 above firewalls may interpret the server initiating a connection with the client as an unsolicited request. In passive mode, the client initiates both command and data connections by opening two random ports:

1) Client sends PASV command.

2) Server responds by opening random port with an address higher than 1023 and sends port command back to client.

3) Client initiates connection from data port to server temporary data port.

Telnet
--------

Telnet (RFCs 854 & 855: https://tools.ietf.org/html/rfc854 & https://tools.ietf.org/html/rfc855) is a remote, bidirectional computer protocol originally designed for remote terminal access. Telnet commonly uses well-known port number 23 on the server side (although for security this may be changed) and a dynamic port number on the client end. As a connection between two hosts is established, they exchange information about options which are supported (DO, DON'T, WILL, WON'T, etc.). The hosts when using Telnet are referred to as NetworkVirtualTerminals (NVT). Data entered via the keyboard is sent as data from one host to the other.

During the connection establishment process, the hosts negotiate operating parameters based upon codes, e.g. "DO" or "WILL" means a host will accept the option specified. A list of options can be found at: https://www.iana.org/assignments/telnet-options/telnet-options.xhtml. As with FTP, Telnet information immediately follows the TCP header.

In Windows and Unix-like systems, the telnet command brings up a client telnet.

By default Telnet is unencrypted, and because of this SSH has become more preferable.

SSH functionally does much of the same as Telnet, but with strong encryption and public key authentication.

SMTP
-------

The Simple Mail Transfer Protocol is widely used for supporting e-mail across the Internet. SMTP uses a Sender-SMTP and Receiver-SMTP process to transfer e-mail in which the Sender-SMTP issues mail commands and sends the mail messages, and the Receiver-SMTP issues back reply codes.

Simple Mail Transfer Protocol

A Sender-SMTP requests a connection to the Receiver-SMTP on server port 25. Once the connection is established over TCP, the Receiver-SMTP responds with a numerical code telling the client that the sever is ready to accept SMTP commands and data. The client then issues a command to initiate a SMTP session.

RFCs which cover this process include 821, 2821, and 1869 (https://tools.ietf.org/html/rfc821, https://tools.ietf.org/html/rfc2821, & https://tools.ietf.org/html/rfc1869). A summary of basic commands and error codes can be found at: https://web.archive.org/web/20140214114542/http://www.nthelp.com/smtp_commands.htm.

SMTP reply codes indicate to the presence of various encountered errors: https://www.greenend.org.uk/rjk/tech/smtpreplies.html.

Both SMTP commands and reply codes immediately follow the TCP header. The task of storing or forwarding e-mail messages is generally the job of a SMTP server. While servers and other MTAs (mail transfer agents) rely on SMTP to send and receive messages, many client applications use SMTP to send messages to a SMTP server which relays the message to its destination. For the reception of messages, other e-mail based protocols such as POP and IMAP are used.

HTTP
-------

The Hypertext Transfer Protocol provides hypermedia-based distributed information across the World Wide Web, having been in use since 1990. Version 1.1 is documented in RFC 2616 (https://tools.ietf.org/html/rfc2616). HTTP is based upon a request/reply model between client and server. Requests include a Uniform Resource Identifier (URI) specifying the web resource to be received. The URI is a combination of a Uniform Resource Locator (URL) and a Uniform Resource Name (URN). By default, HTTP communications use port 80.

A HTTP client sends to a HTTP server the method/URI and receives reply codes and/or the data requested.

HTTP Methods specify commands that HTTP clients issue to servers. HTTP 1.1 Methods include:

GET - Retrieves information identified by URI.

HEAD - Retrieves meta information related to the URI.

POST - Sends data to HTTP server.

OPTIONS - Determines options/requirements associated with the resource or server.

PUT - Sends data to HTTP server.

DELETE - Deletes resource defined by URI.

TRACE - Invokes remote Application Layer loopbackof request message, allows the client to see what the server receives from the client.

CONNECT - Connects to proxy device (e.g., SSL tunneling).

Once these methods are sent to a HTTP server, the server responds with status codes to indicate success or failure of the request. A list of status codes and definitions can be found at: https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html, and the status code registry with links to RFCs can be found at: https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml.

HTTP commands and status codes follow TCP header.

A secure implementation of HTTP known as HTTPS includes SSL encryption between hosts. TCP port 443 is assigned to HTTP using Transport Layer security over SSL, and UDP port 443 is assigned for HTTP that uses UDP with Transport Layer security over SSL.

Various Instant Messaging Protocols
------------------------------------------------

MSN MESSENGER/WINDOWS LIVE MESSENGER

The original MSN Messenger was superseded by Windows Live Messenger for providing IM across various Microsoft platforms. It is currently considered deprecated; Following the acquisition of Skype, Microsoft announced in November 2012 that Messenger and Skype services would be merged, prompting current MSN client users to switch to Skype. The complete phase-out progressed slowly, as Messenger was still included in the Windows Essentials Suite.

The Microsoft Notification Protocol which formed the basis of MSN Messenger and Windows Live Messenger does not provide encryption; communications can easily be intercepted with the use of packet sniffers. Other security concerns such as phishing attempts and fake/front downloads would need to be taken into account; MSN Messenger had a long history of vulnerabilities but also a widespread user base.

IRC

Internet Relay Chat has been around since 1988 as a client/server multicasting protocol providing interactive text messaging, synchronous group communications and data transfer. IRC generally runs over TCP ports 6660–7000, and has the ability to use Transport Layer security. There are many specifications of the protocol but none are considered official. IRC servers are generally linked together in a tree structure, with messages being routed along branches and network states being sent to each server. This in itself can pose problems as a malfunctioning or malicious server can cause network damage.

IRC uses a command line based structure to send messages to a server, generally prefaced by a "/" (e.g., "/join #channelname").

XMPP

Originally named Jabber, the Extensible Messaging and Presence Protocol is based on XML for message-oriented middleware providing instant messaging, presence indication, and contact list maintenance. As it was designed to be an open standard and extensible, it has and will possibly have many other uses, including remote network management and system control, content syndication, collaboration tools, geolocation, gaming and cloud computing. Currently, two primary RFCs include 6120 & 6121 (https://tools.ietf.org/html/rfc6120 & https://tools.ietf.org/html/rfc6121). The first version of Jabber was released in 1999 and has since been used by Google as a base of instant messaging features in Google Talk as part of the Jingle protocol; AOL has also introduced XMPP compatibility with AIM but it has since been dropped to limited functionality. Limited XMPP support can also be found in Facebook and Skype. Multiple implementations of XMPP standards exist for clients, servers and code libraries.

XMPP uses a decentralized client-server architecture, and has built-in SASL and TLS security making it ideal for company intranets. Unlike AIM or Windows Live Messenger, XMPP does not rely upon a centralized server. Each user on a given network (XMPP server) has their own Jabber ID which is structured as an email address (username@example.com). In addition to running over TCP, there is also a push method implementation for HTTP transport via Bidirectional-streams Over Synchronous HTTP (BOSH). Currently there are also experimental implementations for realtime messaging over WebSocket.

YAHOO! MESSENGER PROTOCOL (YMSG)

The Yahoo! Messenger Protocol is used by the Yahoo! Messenger network to support instant messaging, chat, offline messaging, file transfer, conferencing, voice chat and webcam functionality. It uses a client-server model on TCP port 5050 by default; HTTP can also be used to relay messages. Like MSN Messenger, data sent over a YMSG connection is not encrypted.

SIMPLE (Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions)

SIMPLE is an open standard protocol suite based on the Session Initiation Protocol (SIP). Details can be found in RFC 6914 (https://tools.ietf.org/html/rfc6914).

OSCAR (Open System for Communication in Realtime)

Similar to YMSG, OSCAR is AOL's proprietary standard for AIM and ICQ. In the past there have been attempts to prevent competitors from developing compatible clients (with the exception of Apple's iChat), but in more recent years AOL had opened up various parts of the protocol to allow interconnection between AIM and Google Talk.

A further comparison guide to various IM protocols can be found at: https://en.wikipedia.org/wiki/Comparison_of_instant_messaging_protocols.

VoIP
------

Voice over Internet Protocol is a category for a suite of protocols which routing voice conversations across the Internet. The elements consist of voice/video clients, a Multipoint Control Unit (MCU) for bridging connections, a Gateway device for translating among protocol stacks (ISDN, H.323 digital, etc.), and a Gatekeeper for providing address translation and access controls. The elements are referred to as 'H.323 endpoints' (support callable, bidirectional communications media). VoIP is also known as "internet telephony" to specify that communication services are sent over the internet (packet-switched networking) rather than the traditional public switched telephone network.

H.323 is a protocol suite for providing realtime, streaming multimedia across the Internet. Various subprotocols handle different aspects of H.323, including:

H.245 - Defines control protocol for multimedia communication and procedures for opening and closing channels.

H.235 - Security and encryption.

H.450 - Supplementary services.

H.239 - Relaying communications across different network mediums.

Q.931 - Protocol for ISDN connections.

The H.323 Multipoint Control Unit contains a Multipoint Controller (MC) for call signaling and conference control, and can use an optional Multipoint Processor (MP) for switching endpoints in a conference call. The MCU is also responsible for multiplexing voice, video and data, and some may transcode audio and video streams in realtime.

The H.323 Gatekeeper provides address translation from LAN to IP address. This can be thought of as a virtual network switch.

The H.323 Gateway is optional, but provides protocol translation between different endpoints.

An analog-to-digital converter is used in the end-user hardware device (computer, server, peripheral device, etc.) to convert analog voice signals to digital data (the digital-to-analog converter is the reverse of this). H.323 defines several standards for compression and decompression. Although H.323 was the first set of agreed upon standards (International Telecommunications Union), the IETF's SIP protocol forms another independently developed VoIP standard. Both implementations make use of the Real-Time Protocol (RTP) packet format to deliver audio and video content; both use different signaling protocols. There is also the OpenH.323 specification which supports the interoperability of both. To be considered effective, VoIP implementations should allow guaranteed delivery of multiplexed data streams. VoIP has some advantages over the conventional telephone system, including cost. VoIP can also handle multiple simultaneous connections.

While connecting using H.323, a TPKT header directly follows the TCP header, a form of data chunking which TCP lacks natively (RFCs 1000 & 2126: https://tools.ietf.org/html/rfc1000 & https://tools.ietf.org/html/rfc2126). During this phase other options or subprotocols may be present such as Q.931 and H.255.0. As one host connects to another over the Internet, two separate Gatekeepers are used.

Throughout the past decade there has been a huge success and adoption rate of VoIP-based software such as Skype. VoIP applications can be found on smartphones, and in addition to H.323 and SIP there are other protocols finding use including XMPP (Jingle).

A comparison list of VoIP software including which protocols are in use can be found at: https://en.wikipedia.org/wiki/Comparison_of_VoIP_software.

Other Common TCP/IP Services
------------------------------------------

Echo - Defined in RFC 862 (https://tools.ietf.org/html/rfc862), Echo is somewhat outdated by ICMP Echo. It works by echoing whatever is sent to port 7.

Quote of the Day (QOD) - Sends a message in reply to a requested packet, and can be used to deliver messages from the system administrator. It uses port 17 on both TCP and UDP.

Chargen - This acts as a character generator and used as a debugging tool. It sends a continuous sequence of characters in response to a client request, uses TCP/UDP port 119 and is documented in RFC 864 (https://tools.ietf.org/html/rfc864). In Unix-like systems, chargen is built into the inetd daemon. It can be used in a denial of service attack. For security it is recommended that this service be disabled.

Whois - Provides directory services by looking up domain names to indicate user information, where they are registered and identifies the DNS servers for the DNS database in which the entry resides. Whois is documented in RFC 3912 (https://tools.ietf.org/html/rfc3912) and uses TCP/UDP port 43.

Trivial File Transfer Protocol (TFTP) - File transfer protocol used for automated transfer, generally for configuration or boot files between two hosts in a local area environment. TFTP is limited compared with FTP, however it can use UDP in broadcast to send files to multiple clients. It is implemented in the Preboot Execution Environments (PXE) network boot protocol within the firmware of a network card. It is documented in RFC 1350 (https://tools.ietf.org/html/rfc1350) and is assigned to UDP port 69. Due to lack of security it is not recommended to be used over the Internet.

Finger - User information protocol for obtaining information about hosts or users. Finger is not recommended to be enabled for external access. Finger is documented in RFC 742 & 1288 (https://tools.ietf.org/html/rfc742 & https://tools.ietf.org/html/rfc1288) and assigned to TCP/UDP port 79.

Remote Procedure Call (RPC) - RPC was originally developed at Sun Microsystems for distributed application development. There are different implementations of RPC, not all of which are compatible. A remote procedure call is initiated by a client which sends a request message to a server to execute a procedure. The server sends it's own response to the client and the procedure runs until it ends. The sequence of events is as follows:

1) Client sends a local procedure call (client stub).

2) The client stub takes the parameters of the call and makes a system call to the message (marshalling).

3) The client sends the message from the client machine to the server machine.

4) The server takes the packets to its own stub.

5) The server stub unpacks the message (unmarshalling).

6) Server stub calls the server procedure.

Remote Procedure Call

Various analogue protocols and APIs offering similar functionality exist.

Simple Network Management Protocol (SNMP) - Documented in more than 60 RFCs, this protocol allows remote monitoring and management of devices (routers, switches, servers, workstations, printers, modem racks, etc.).

SNMP is based on "managers" and "agents," in which the managers are tasked with monitoring remote hosts or devices, and the agents run as software components within the hosts, with the data as variables organized hierarchically. The software which runs on the management end is known as a Network Management System (NMS). SNMP agents receive requests on UDP port 161, and the manager may send requests from any available source port.

Notifications (Traps, InformRequests) from the agent are sent to port 162. When used with Transport Layer Security requests are received on port 10161 and traps are sent to port 10162. The seven protocol data units are as follows:

GetRequest - Manager to agent request to retrieve values or list of variables, specified in variable bindings.

SetRequest - Manager to agent request to change the value of a variable or list, specified in the body of the request.

GetNextRequest - Manager to agent request to find available variables.

GetBulkRequest - Manager to agent request for multiple iterations of GetNextRequest.

Response - Returns variable bindings from agent to manager for GetRequest, SetRequest, GetNextRequest, GetBulkRequest and InformRequest.

Trap - Notification from agent to manager. Includes sysUpTime, OID for the type of trap and optional variable bindings.

InformRequest - Acknowledgement notification (originally defined as manager to manager but allows agent to manager).

Further Reading
---------------------

Extended Simple Mail Transfer Protocol
https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#Extended_Simple_Mail_Transfer_Protocol

Basic FTP Commands
https://www.cs.colostate.edu/helpdocs/ftp.html

Hypertext
https://en.wikipedia.org/wiki/Hypertext

HyperCard
https://en.wikipedia.org/wiki/HyperCard

XMPP
https://en.wikipedia.org/wiki/XMPP

WebSocket
https://en.wikipedia.org/wiki/WebSocket

Understanding the Linux and Unix Command: telnet
https://www.lifewire.com/telnet-linux-command-unix-command-4097062

SSH
https://en.wikipedia.org/wiki/Secure_Shell

Getting started with SSH security and configuration
https://developer.ibm.com/articles/au-sshsecurity/

Message-oriented Middleware
https://en.wikipedia.org/wiki/Message-oriented_middleware

OSCAR Protocol
https://en.wikipedia.org/wiki/OSCAR_protocol

Chargen Abuse
https://en.wikipedia.org/wiki/Chargen#Abuse

Whois
https://en.wikipedia.org/wiki/Whois

Finger
https://en.wikipedia.org/wiki/Finger_protocol

H.323
https://en.wikipedia.org/wiki/H.323

RPC Other Analogues
https://en.wikipedia.org/wiki/Remote_procedure_call#Other_RPC_analogues

SNMP
https://en.wikipedia.org/wiki/Simple_network_management_protocol