Back to: About the Maidsafe Network
3. Architecture and Implementation
3.1 Place in OSI Reference Model
Layers 0-4 will provide the basis for communication, without modification. Internet/Routing (Layer 3) will be based on IPv4 and/or IPv6. Host-to-host Transport (Layer 4) will provide QUIC transport, which is based on UDP (a connection-less service). SAFE Network will not modify anything below and including Layer 3. SAFE Network will add to Layer 4, and functions in Layers 5, 6, and 7:
LAYER 7 – Application: Vaults, SAFE Browser, Authenticator, SDNS, (not sure about SFTP, SNTP, SIMAP) (tentatively placed here until verified)
LAYER 6 – Presentation: “Self-Encryption” (tentatively placed here until verified)
LAYER 5 – Session: Session ID and network entry point selection, Self-Authentication (tentative)
LAYER 4 – Routing: SAFE Network PARSEC Consensus, and Kademlia Distributed Hash Table-based Routing, using QUIC/UDP
(CRUST was replaced by QUIC.)
3.2 SAFE Network Overlay Network
The SAFE Network is a virtual overlay network on the Internet, consisting of SAFE Network Nodes, which correspond to an instance of SAFE Network “Vault” software in the real world. Nodes are dynamically organized by the network as Vaults come, gain trust over time, and leave the network.
SAFE Network Address Space
SAFE Network Nodes exist in the virtual overlay network in a multi-dimensional space called the XOR Space. There is no correlation between geographical distance (or IP address distance) between physical nodes in the real world, and distance between the same nodes in XOR Space. This is the result of how Node IDs are allocated by the SAFE Network. People running a Vault have no control over this process, which is an important security feature.
In XOR space, the distance between Nodes is the XOR of the IDs of the two Nodes, which is read as an unsigned integer. Distance is used for routing in this space and for organizing Nodes into groups, as well as for allocating responsibilities over other entities in SAFE Network XOR space such as “Clients” and units of data (“Chunks”), which also have an ID using which distances to Nodes can be calculated in exactly the same way.
Clients need to connect to Nodes to access the SAFE Network and the appropriate node is decided based on distance. Chunks are managed by groups of Nodes, which are also determined by distance. Chunks of data derive their ID from a hash of the contents of the chunk.
- Entities in XOR Space: Vault Nodes, Clients, Chunks (other?)
- XOR Space addresses length: 256 bits
Organization of Objects and Entities
The XOR space is dynamically subdivided by a sectioning algorithm into a growing number of dis-joint XOR sub-spaces as the number of Nodes grows. The sub-spaces are called “Sections”. Nearby Sections of Nodes watch over each other, and within each Section a hierarchy of Nodes exists based on Age, which determines the level of trust, responsibility of a Node, and the power it has over other Nodes.
Sections run PARSEC consensus algorithms to make decisions, and sign decisions using a group signature so that neighboring Sections can verify them. Decisions (and events) are stored in Data Chains.
- Algorithms: Sectioning, … (other TBD)
- Section group size and split threshold: 8 (typical number which may be changed in the future)
- Consensus threshold: 5 out of 8 (62.5% (of “quorum size”?))
- Information held by Sections: Data Chains (of a Section, or a neighboring Section), Routing Table
Object and Entity Types (2019/outdated)
- Users accessing the network (clients)
- Passports (ID linked to multiple sub-IDs)
- Anonymous ID (MA-ID) – for anonymous network access
- Public IDs (MP-ID) – for public network access
- Share IDs (MS-ID) – can be associated with multiple Passports – for access to private shares
- Proxy IDs (PM-ID) – for providing own resources
- People providing resources to the network: vaults (farmers)
- Vaults (instances of a vault node)
- ClientManager vault persona – part of a group but independently relaying traffic between client and the correct DataManagers
- DataManager vault persona – part of a group but independently forwarding requests to the correct DataHolders
- DataHolderManager vault persona – part of a group but independently selecting and observing DataHolders
- DataHolder vault persona – part of a group but independently judging validity of requests
- (what is the 5th persona, or is the 5th not a persona but the client?)
Object and Entity Types
Safe Network Records (2024)
- 5Mb size limit
Types of Records
3.3 Basic Facilities
- Storage of ALL information is in a network wide DHT
- ALL nodes are connected to at least four neighbors
- ALL inter-node communication is authenticated and encrypted (PKI)
3.4 Transport Protocol for P2P
3.4.1 IETF QUIC Introduction
QUIC, originally standing for “Quick UDP Internet Connections,” is a new Internet transport protocol that intends to replace TCP and TLS. It relies on UDP datagrams over IP. This is a good introduction. Standard IETF QUIC started as Google’s QUIC or qQUIC, but has significantly diverged and improved.
Multiplexing of multiple streams reduces the number of QUIC connections and handshakes and makes streams less sensitive to congestion and faster. Authentication and encryption are not delegated to a higher layer protocol like TLS, resulting in higher speed and better security. A typical handshake takes only one round trip. However, UDP protocols are susceptible to reflection attacks and to avoid this, the server can cryptographically confirm if requestor and destination are the same, at the expense of doubling the handshake duration.
There are some early adopter issues, such as worse handling of NAT by the typical router designed with only TCP in mind. Second, connections are uniquely identified and tracked by connection ID because ports and even IP address of endpoints may change, which is a feature of QUIC. However, conventional routers may not deliver the UDP packets correctly to the right end-point after such a QUIC endpoint change. Finally, performance may suffer compared to TCP because of a general lack of off-loading capabilities of processing to OS or network interface hardware.
3.4.2 MDSF Lib quic-p2p
MaidSafe [[https://crates.io/crates/quic-p2p|quic-p2p crate]] ([[https://docs.rs/quic-p2p/0.2.0/quic_p2p//|0.2.0 docs]],[[https://github.com/maidsafe/quic-p2p/projects/2|to do]]) is a library implementing QUIC using [[https://www.cloudflare.com/learning-resources/tls-1-3/|TLS 1.3]].
[[https://github.com/maidsafe/quic-p2p/network|Watch]] the master branch.
== Notes ==
The library provides 2 connection types in P2P mode: Bi-directional and uni-directional.
Uni-directional type is used to force to prove a node’s ability to accept incoming connections and create outgoing connections, which is a necessary requirement to be accepted as a network peer worker node.
The library provides 3 connection types of different identity certification levels:
- Agreed CA certificates required
- Private CA certificates allowed
- No Identity certification required
The most recent 200 endpoints that are not behind NAT, and their certificates are cached and used for re-joining of nodes.
3.5 Routing
3.5.1 Kademlia Routing Introduction
Kademlia routing (it’s just a name) was first proposed in 2002. A Kademlia route is a shortest sequence of nodes to query to locate a node with a given address in a network of nodes with distributed routing tables. For a network with n node addresses the number of nodes to query is Order log(n).
Nodes build their individual routing tables from queries they receive during the routing process. Routing tables contain IP addresses and Port numbers for a limited number of certain other nodes. Communication between nodes during and after this process continues to be physically routed using conventional routing.
Nodes know exponentially less about nodes that are farther away, and are experts about nodes in their closest vicinity. Kademlia routes converge to an increasingly common route for each given destination address, as that destination is approached. This exponentially homing-in behavior from originating node towards given destination node is thanks to a special kind of mathematical distance that is used to decide which node to query next.
Kademlia’s creators not only proposed an algorithm but also a “novel” distance metric, the XOR distance. It satisfies all requirements of a proper metric:
- Zero distance between two nodes always means that those nodes must be one and the same.
- Distances between two nodes are the same no matter which is the origin, and are always positive.
- The triangle inequality applies; for any three nodes there always is a distance between one pair that is equal or less than the sum of the distances between the two other possible pairs of nodes.
However, XOR distance is non-Euclidean, i.e. Euclid’s 5th “parallel” postulate does not hold. This results in some very interesting and hard to grasp properties:
- Nodes observed from any particular node, all have a unique distance to that observing node.
- Closeness (e.g. k-nearest nodes) is uni-directional or asymmetric, which means that a node n1 in the set of k-nearest nodes to node n0, does not need to be in the set of k-nearest nodes to n1.
This includes an answer explaining most of the above by @fraser in other better words. This is a long but useful write-up by @dirvine to help understand things better intuitively and pointing out many peculiarities of XOR distance. Finally, there is this Medium post by MaidSafe on this topic.
Article of 2015 on Kademlia applications.
3.5.2 XOR Distance
The following table shows XOR distances in a 3-bit space, or 0..7, to help you grasp XOR distances better. Horizontally the table shows source node number, and the vertical axis is decimal increment to obtain the destination node number modulo 8. For example, distance from node 2 (on top), to node 4 (on skip 1 row), is 6.
The 2-nearest and 2-farthest distances are indicated to verify k-nearest relations. For example, note that the 2 closest nodes to n0 are {n1, n2}. The nearest two nodes to n1 are {n2, n0}. However, the nearest two nodes to n2 are {n3, n0}, where n1 is notably not included.
Source Node xor | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | dist. | 000 | 001 | 010 | 011 | 100 | 101 | 110 | 111 | skip 0 | [1] | 3 | [1] | -7- | [1] | 3 | [1] | -7- | skip 1 | [2] | [2] | -6- | -6- | [2] | [2] | -6- | -6- | skip 2 | 3 | 5 | -7- | 5 | 3 | 5 | -7- | 5 | skip 3 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | 4 | skip 4 | 5 | -7- | 5 | 3 | 5 | -7- | 5 | 3 | skip 5 | -6- | -6- | [2] | [2] | -6- | -6- | [2] | [2] | skip 6 | -7- | [1] | 3 | [1] | -7- | [1] | 3 | [1] |
- In the case of 3 bit address space, it is still possible to place the 8 decimal values (modulo 8) in decimal value order on a circle, and consistently read the relative distances off by measuring the distances along the circle.
- In the case of 4 bit address space, the circle representation works in most cases, but the distance may be wrong by 1 unit of relative distance depending on node pair between which distance is measured. (Is there a parametric curve or surface that would still work? How many dimensions would one need?)
Post by @polpolrene. And Podcast on XOR by @fergish. Erick’s page with some related information.
3.5.3 MDSF Lib routing
MaidSafe [[https://crates.io/crates/routing|routing crate]] ([[https://docs.rs/routing/0.37.0/routing/|0.37.0 docs]], [[https://github.com/maidsafe/routing/projects?query=is%3Aopen|projects]]) implements “Kademlia-like” implementation of Distributed Hash Tables (DHT).
[[https://github.com/maidsafe/routing/network|Watch]] the fleming branch.
3.6 Gossip Protocol
3.6.1 Randomized Rumor Spreading
Randomized Rumor Spreading [[http://zoo.cs.yale.edu/classes/cs426/2013/bib/karp00randomized.pdf|Karp et al.]] (FOCS 2000)
3.6.2 MDSF Lib safe_gossip
MaidSafe safe_gossip (0.1.0 docs, projects) implements Randomized Rumor Spreading.
3.7 Parsec
3.7.1 Byzantine Agreement
PARSEC stands for Protocol for Asynchronous Reliable Secure and Efficient Consensus and is described in this [[http://docs.maidsafe.net/Whitepapers/pdf/PARSEC.pdf|whitepaper]] by MaidSafe.
3.7.2 MDSF parsec
MaidSafe [[https://crates.io/crates/parsec|parsec crate]] ([[https://docs.rs/parsec/0.5.0/parsec/|0.5.0 docs]], projects) is a library for reaching consensus on network events.
[[https://github.com/maidsafe/parsec/network|Watch]] the master branch.
3.8 Node (ex. Vault)
Node
- Multiple nodes may run on a single compute but require each unique port numbers.
- 4096 chunks max. per node; 2GB.
- 4092 records max. per node; soft-cap
MDSF application safe_vault (old)
MaidSafe safe_vault crate (0.18.0 docs, projects) implements the SAFE Network node capable of data storage, publishing, sharing, (computation, and Safecoin transfers.) Watch the master branch.
3.9 CashNote (ex. DBC)
updated: 20240300