Skip to content

Commit 35b256f

Browse files
ccie18643ccie18643
committed
Restoring version 1.0
0 parents  commit 35b256f

110 files changed

Lines changed: 12842 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/lint_python.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
name: lint_python
2+
on: [pull_request, push]
3+
jobs:
4+
lint_python:
5+
runs-on: ubuntu-latest
6+
steps:
7+
- uses: actions/checkout@v2
8+
- uses: actions/setup-python@v2
9+
- run: pip install -r requirements.txt
10+
- run: codespell --ignore-words-list="ect,ether,nd,tha" --quiet-level=2 README.md $(find . -name '*.py')
11+
- run: isort -c --profile black $(find . -name '*.py')
12+
- run: black --check $(find . -name '*.py')
13+
- run: flake8 $(find . -name '*.py')
14+
- run: mypy $(find . -name '*.py')

‎LICENSE‎

Lines changed: 622 additions & 0 deletions
Large diffs are not rendered by default.

‎README.md‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
# PyTCP (version 1.0)
2+
3+
PyTCP is an attempt to create fully functional TCP/IP stack in Python. It supports TCP stream based transport with reliable packet delivery based on sliding window mechanism and basic congestion control. It also supports IPv6/ICMPv6 protocols with SLAAC address configuration. It operates as user space program attached to Linux TAP interface. As of today stack is able to send and receive traffic over Internet using IPv4 and IPv6 default gateways for routing. Since goal of this project is purely educational (at least at this point) the clarity of code is preferred over its efficiency. For the same reason security features are not being implemented just yet unless they are integral part of TCP/IP suite protocols specification.
4+
5+
PyTCP version 1.0 is not being developed anymore. Only bug fixes are being occasionally implemented. There is new version 2.0 being currently developed which has been implemented with use of new packet parser and assembler mechanisms to improve stack efficiency.
6+
7+
8+
#### Version 1.0 features:
9+
10+
- Ethernet protocol - *support of Ethernet II standard frame*
11+
- Ethernet protocol - *unicast, IPv4 multicast, IPv6 multicast and broadcast addressing supported*
12+
- ARP protocol - *replies, queries, ARP cache mechanism*
13+
- ARP protocol - *ARP Probe/Announcement IP conflict detection (ACD) mechanism*
14+
- IPv4 protocol - *default routing, stack can talk to hosts over Internet using IPv4 protocol*
15+
- IPv4 protocol - *automatic address configuration using DHCP protocol*
16+
- IPv4 protocol - *inbound and outbound IP fragmentation*
17+
- IPv4 protocol - *IPv4 options accepted but not supported*
18+
- IPv4 protocol - *multiple stack's IPv4 addresses supported, each of them acts as it was assigned to separate VRF*
19+
- ICMPv4 protocol - *echo request, echo reply, port unreachable*
20+
- IPv6 protocol - *default routing, stack can talk to hosts over Internet using IPv6 protocol*
21+
- IPv6 protocol - *automatic Link Local address configuration using EUI64 and Duplicate Address Detection*
22+
- IPv6 protocol - *automatic GUA address configuration using Router Advertisement / EUI64*
23+
- IPv6 protocol - *automatic assignment of Solicited Node Multicast addresses*
24+
- IPv6 protocol - *automatic assignment of IPv6 multicast MAC addresses*
25+
- ICMPv6 protocol - *echo request, echo reply, port unreachable*
26+
- ICMPv6 protocol - *Neighbor Discovery, Duplicate Address Detection*
27+
- ICMPv6 protocol - *Neighbor Discovery cache mechanism*
28+
- ICMPv6 protocol - *Multicast Listener Discovery v2 (MLDv2) protocol implementation (only messages needed by stack)*
29+
- UDP protocol - *full support, stack is able to exchange data with other hosts using UDP protocol*
30+
- UDP sockets - *full support, stack's 'end user' API similar to Berkeley sockets*
31+
- UDP services - *UDP Echo, Discard, Daytime implemented for testing purposes*
32+
- TCP protocol - *full implementation of TCP Finite State Machine, at this point stack is able to exchange bulk data with other hosts over TCP protocol*
33+
- TCP protocol - *TCP option support for: MSS, WSCALE, SACKPERM, TIMESTAMP*
34+
- TCP protocol - *TCP sliding window mechanism with and data retransmission (fast retransmit and time based scenarios)*
35+
- TCP protocol - *TCP backoff mechanism / basic congestion control*
36+
- TCP protocol - *TCP SYN/FIN packet retransmission*
37+
- TCP sockets - *full support, stack's 'end user' API similar to Berkeley sockets*
38+
39+
40+
### Examples:
41+
42+
#### Couple ping packets and two monkeys delivered via TCP over IPv6 protocol
43+
44+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/malpi_00.png)
45+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/malpi_01.png)
46+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/malpi_02.png)
47+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/malpi_03.png)
48+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/malpi_04.png)
49+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/malpi_05.png)
50+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/malpi_06.png)
51+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/malpi_07.png)
52+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/malpi_08.png)
53+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/malpi_09.png)
54+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/malpi_10.png)
55+
56+
57+
#### IPv6 Neighbor Discovery / Duplicate Address Detection / Address Auto Configuration
58+
- stack tries to auto configure it's link local address, it generates it as EUI64 address, as part od DAD process it joins appropriate solicited node multicast group and sends neighbor solicitation for the address it generated
59+
- stack doesn't receive any Neighbor Advertisement for the address it generated so assigns it to its interface
60+
- stack tries to assign preconfigured static address, as part of DAD process it joins appropriate solicited node multicast group and sends neighbor solicitation for the static address
61+
- another host that has the same address already assigned replies with Neighbor Advertisement message, this tells the stack that the address its trying to assign has been already
62+
y assigned by another host so stack cannot us it
63+
- stack sends out Router Solicitation message to check if there are any global prefixes it should use
64+
- router responds with Router Advertisement containing additional prefix
65+
- stack tries to assign address generated based on received prefix and EUI64 host portion, as part of DAD process it joins appropriate solicited node multicast group and sends neighbor solicitation for the static address
66+
- stack doesn't receive any Neighbor Advertisement for the address it generated so assigns it to its interface
67+
- after all addresses are assigned stacks sends out one more Multicast Listener report listing all of the multicast addresses it wants to listen to
68+
69+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ipv6_nd_dad_01.png)
70+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ipv6_nd_dad_02.png)
71+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ipv6_nd_dad_03.png)
72+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ipv6_nd_dad_04.png)
73+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ipv6_nd_dad_05.png)
74+
75+
76+
#### TCP Fast Retransmit in action after lost TX packet
77+
- outgoing packet is 'lost' as result of simulated packet loss mechanism
78+
- peer notices the inconsistence in packet SEQ numbers and sends out 'fast retransmit request'
79+
- stack eceives the request and retransmits lost packet
80+
81+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_tx_fst_ret_01.png)
82+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_tx_fst_ret_02.png)
83+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_tx_fst_ret_03.png)
84+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_tx_fst_ret_04.png)
85+
86+
87+
#### Out of order queue in action during RX packet loss event
88+
- incoming packet is 'lost' as reult of simulated packet loss mechanism
89+
- stack notices that there is an inconsistence in inbound packet's SEQ number and sends out 'fast retransmit' request
90+
- before peer receives the request it already sends multiple packets with higher SEQ than what stack is expecting, stack queues all those packets
91+
- peer retransmits lost packet
92+
- stack receives lost packet, pulls all the packets stored in ooo queue so far and processes them
93+
- stacks sends out ACK packet to acknowledge latest of the packets pulled from queue
94+
95+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_ooo_ret_01.png)
96+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_ooo_ret_02.png)
97+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_ooo_ret_03.png)
98+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_ooo_ret_04.png)
99+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_ooo_ret_05.png)
100+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_ooo_ret_06.png)
101+
102+
103+
#### TCP Finite State Machine - stack is running TCP Echo service
104+
- peer opens connection
105+
- peer sends data
106+
- stack echoes the data back
107+
- peer closes connection
108+
109+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_fsm_srv_01.png)
110+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_fsm_srv_02.png)
111+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_fsm_srv_03.png)
112+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_fsm_srv_04.png)
113+
114+
115+
#### TCP Finite State Machine - stack is running TCP Echo client
116+
- stack opens connection
117+
- stack sends data
118+
- peer echoes the data back
119+
- stack closes connection
120+
121+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_fsm_clt_01.png)
122+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_fsm_clt_02.png)
123+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_fsm_clt_03.png)
124+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/tcp_fsm_clt_04.png)
125+
126+
127+
#### Pre-parse packet sanity check in action
128+
- first screenshot shows stack with sanity check turned off, malformed ICMPv6 packet is able to crash it
129+
- second screenshot shows stack with sanity check turned on, malformed ICMPv6 packet is being discarded before being passed to ICMPv6 protocol parser
130+
- third screenshot shows the malformed packet, number of MA records field has been set to 777 despite packet contains only one record
131+
132+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/pre_sanity_chk_01.png)
133+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/pre_sanity_chk_02.png)
134+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/pre_sanity_chk_03.png)
135+
136+
137+
#### ARP Probe/Announcement mechanism
138+
- stack is using ARP Probes to find any possible conflicts for every IP address that has been configured
139+
- one of IP addresses (192.168.9.102) is already taken so stack gets notified about it and skips it
140+
- rest of IP addresses are free so stack claims them by sending ARP Announcement for each of them
141+
142+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ip_arp_probe_01.png)
143+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ip_arp_probe_02.png)
144+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ip_arp_probe_03.png)
145+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ip_arp_probe_04.png)
146+
147+
148+
#### ARP resolution and handling ping packets
149+
- host 192.168.9.20 tries to ping the stack, to be able to do it it first sends ARP Request packet to find out stack's MAC address
150+
- stack responds by sending ARP Reply packet (stack doesn't need to send out its own request since it already made note of the host's MAC from host's request)
151+
- hosts sends ping packets, stack responds to them
152+
153+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/arp_ping_01.png)
154+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/arp_ping_02.png)
155+
156+
157+
#### IP fragmentation
158+
- host sends 4Kb UDP datagram using three fragmented IP packet (three fragments)
159+
- stack receives packets and assembles them into single piece, then passes it (via UDP protocol handler and UDP Socket) to UDO Echo service
160+
- UDP Echo service picks data up and puts it back into UDP Socket
161+
- UDP datagram is being passed to IP protocol handler which creates IP packet and after checking that it exceedes link MTU fragments it into three separate IP packets
162+
- IP packets are being encapsulated in Ethernet frames and put on TX ring
163+
164+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ip_udp_frag_01.png)
165+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ip_udp_frag_02.png)
166+
![Sample PyTCP log output](https://github.com/ccie18643/PyTCP/blob/PyTCP_1_0/pictures/ip_udp_frag_03.png)
167+

‎arp_cache.py‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
#!/usr/bin/env python3
2+
3+
############################################################################
4+
# #
5+
# PyTCP - Python TCP/IP stack #
6+
# Copyright (C) 2020-2021 Sebastian Majewski #
7+
# #
8+
# This program is free software: you can redistribute it and/or modify #
9+
# it under the terms of the GNU General Public License as published by #
10+
# the Free Software Foundation, either version 3 of the License, or #
11+
# (at your option) any later version. #
12+
# #
13+
# This program is distributed in the hope that it will be useful, #
14+
# but WITHOUT ANY WARRANTY; without even the implied warranty of #
15+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the #
16+
# GNU General Public License for more details. #
17+
# #
18+
# You should have received a copy of the GNU General Public License #
19+
# along with this program. If not, see <https://www.gnu.org/licenses/>. #
20+
# #
21+
# Author's email: ccie18643@gmail.com #
22+
# Github repository: https://github.com/ccie18643/PyTCP #
23+
# #
24+
############################################################################
25+
26+
##############################################################################################
27+
# #
28+
# This program is a work in progress and it changes on daily basis due to new features #
29+
# being implemented, changes being made to already implemented features, bug fixes, etc. #
30+
# Therefore if the current version is not working as expected try to clone it again the #
31+
# next day or shoot me an email describing the problem. Any input is appreciated. Also #
32+
# keep in mind that some features may be implemented only partially (as needed for stack #
33+
# operation) or they may be implemented in sub-optimal or not 100% RFC compliant way (due #
34+
# to lack of time) or last but not least they may contain bug(s) that i didn't notice yet. #
35+
# #
36+
##############################################################################################
37+
38+
39+
#
40+
# arp_cache.py - module contains class supporting ARP cache
41+
#
42+
43+
44+
import time
45+
46+
import loguru
47+
48+
import ps_arp
49+
import stack
50+
from ipv4_address import IPv4Address
51+
52+
ARP_ENTRY_MAX_AGE = 3600
53+
ARP_ENTRY_REFRESH_TIME = 300
54+
55+
56+
class ArpCache:
57+
"""Support for ARP cache operations"""
58+
59+
class CacheEntry:
60+
"""Container class for cache entries"""
61+
62+
def __init__(self, mac_address, permanent=False):
63+
self.mac_address = mac_address
64+
self.permanent = permanent
65+
self.creation_time = time.time()
66+
self.hit_count = 0
67+
68+
def __init__(self, packet_handler):
69+
"""Class constructor"""
70+
71+
self.packet_handler = packet_handler
72+
73+
self.arp_cache = {}
74+
75+
if __debug__:
76+
self._logger = loguru.logger.bind(object_name="arp_cache.")
77+
78+
# Setup timer to execute ARP Cache maintainer every second
79+
stack.timer.register_method(method=self.maintain_cache, delay=1000)
80+
81+
if __debug__:
82+
self._logger.debug("Started ARP cache")
83+
84+
def maintain_cache(self):
85+
"""Method responsible for maintaining ARP cache entries"""
86+
87+
for ip4_address in list(self.arp_cache):
88+
89+
# Skip permanent entries
90+
if self.arp_cache[ip4_address].permanent:
91+
continue
92+
93+
# If entry age is over maximum age then discard the entry
94+
if time.time() - self.arp_cache[ip4_address].creation_time > ARP_ENTRY_MAX_AGE:
95+
mac_address = self.arp_cache.pop(ip4_address).mac_address
96+
if __debug__:
97+
self._logger.debug(f"Discarded expired ARP cache entry - {ip4_address} -> {mac_address}")
98+
99+
# If entry age is close to maximum age but the entry has been used since last refresh then send out request in attempt to refresh it
100+
elif (time.time() - self.arp_cache[ip4_address].creation_time > ARP_ENTRY_MAX_AGE - ARP_ENTRY_REFRESH_TIME) and self.arp_cache[
101+
ip4_address
102+
].hit_count:
103+
self.arp_cache[ip4_address].hit_count = 0
104+
self.__send_arp_request(ip4_address)
105+
if __debug__:
106+
self._logger.debug(f"Trying to refresh expiring ARP cache entry for {ip4_address} -> {self.arp_cache[ip4_address].mac_address}")
107+
108+
def add_entry(self, ip4_address, mac_address):
109+
"""Add / refresh entry in cache"""
110+
111+
self.arp_cache[ip4_address] = self.CacheEntry(mac_address)
112+
113+
def find_entry(self, ip4_address):
114+
"""Find entry in cache and return MAC address"""
115+
116+
if arp_entry := self.arp_cache.get(ip4_address, None):
117+
arp_entry.hit_count += 1
118+
if __debug__:
119+
self._logger.debug(
120+
f"Found {ip4_address} -> {arp_entry.mac_address} entry, age {time.time() - arp_entry.creation_time:.0f}s, hit_count {arp_entry.hit_count}"
121+
)
122+
return arp_entry.mac_address
123+
124+
if __debug__:
125+
self._logger.debug(f"Unable to find entry for {ip4_address}, sending ARP request")
126+
self.__send_arp_request(ip4_address)
127+
return None
128+
129+
def __send_arp_request(self, arp_tpa):
130+
"""Enqueue ARP request packet with TX ring"""
131+
132+
self.packet_handler.phtx_arp(
133+
ether_src=self.packet_handler.mac_unicast,
134+
ether_dst="ff:ff:ff:ff:ff:ff",
135+
arp_oper=ps_arp.ARP_OP_REQUEST,
136+
arp_sha=self.packet_handler.mac_unicast,
137+
arp_spa=self.packet_handler.ip4_unicast[0] if self.packet_handler.ip4_unicast else IPv4Address("0.0.0.0"),
138+
arp_tha="00:00:00:00:00:00",
139+
arp_tpa=arp_tpa,
140+
)

0 commit comments

Comments
 (0)