Chord Module

class py2p.chord.ChordConnection(*args, **kwargs)[source]

The class for chord connection abstraction. This inherits from py2p.mesh.MeshConnection

Inheritance diagram of py2p.chord.ChordConnection
__init__(*args, **kwargs)

Sets up a connection to another peer-to-peer socket

Parameters:
  • sock – The connected socket object
  • server – A reference to your peer-to-peer socket
  • outgoing – Whether this connection is outgoing (default: False)
leeching
id_10

Returns the nodes ID as an integer

active
addr
buffer
collect_incoming_data(data)

Collects incoming data

Parameters:data – The most recently received bytes
Returns:True if the data collection was successful, False if the connection was closed
compression
expected
fileno()

Mirror for the fileno() method of the connection’s underlying socket

find_terminator()

Returns whether the defined return sequences is found

found_terminator()

Processes received messages

handle_renegotiate(packets)

The handler for connection renegotiations

This is to deal with connection maintenance. For instance, it could be that a compression method fails to decode on the other end, and a node will need to renegotiate which methods it is using. Hence the name of the flag associated with it, “renegotiate”.

Parameters:packets – A tuple containing the packets received in this message
Returns:True if an action was taken, False if not
handle_waterfall(msg, packets)

This method determines whether this message has been previously received or not.

If it has been previously received, this method returns True.

If it is older than a preset limit, this method returns True.

Otherwise this method returns False, and forwards the message appropriately.

Parameters:
  • msg – The message in question
  • packets – The message’s packets
Returns:

Either True or False
id
last_sent
outgoing
protocol

Returns server.protocol

send(msg_type, *args, **kargs)

Sends a message through its connection.

Parameters:
  • msg_type – Message type, corresponds to the header in a InternalMessage object
  • *args – A list of bytes-like objects, which correspond to the packets to send to you
  • **kargs – There are two available keywords:
  • id – The ID this message should appear to be sent from (default: your ID)
  • time – The time this message should appear to be sent from (default: now in UTC)
Returns:

the IntenalMessage object you just sent, or None if the sending was unsuccessful
send_InternalMessage(msg)

Sends a preconstructed message

Parameters:msg – The IntenalMessage you wish to send
Returns:the IntenalMessage object you just sent, or None if the sending was unsuccessful
server
sock
time
class py2p.chord.ChordDaemon(*args, **kwargs)[source]

The class for chord daemon. This inherits from py2p.mesh.MeshDaemon

Inheritance diagram of py2p.chord.ChordDaemon
__init__(*args, **kwargs)

Sets up a daemon process for your peer-to-peer socket

Parameters:
  • addr – The address you wish to bind to
  • port – The port you wish to bind to
  • server – A reference to the peer-to-peer socket
Raises:
  • socket.error – The address you wanted is already in use
  • ValueError – If your peer-to-peer socket is set up with an unknown encryption method
conn_type
handle_accept()[source]

Handle an incoming connection

alive
daemon
exceptions
kill_old_nodes(handler)

Cleans out connections which never finish a message

main_thread
mainloop()

Daemon thread which handles all incoming data and connections

process_data(handler)

Collects incoming data from nodes

protocol

Returns server.protocol

server
sock
class py2p.chord.ChordSocket(*args, **kwargs)[source]

The class for chord socket abstraction. This inherits from py2p.mesh.MeshSocket

Inheritance diagram of py2p.chord.ChordSocket

Added Events:

Event ‘delete’(conn, key)

This event is triggered when a key is deleted from your distributed dictionary.

Parameters:
__init__(*args, **kwargs)

Initialize a chord socket

Parameters:
  • addr – The address you wish to bind to (ie: “192.168.1.1”)
  • port – The port you wish to bind to (ie: 44565)
  • prot – The Protocol you wish to operate over, defined by a py2p.base.Protocol object
  • out_addr – Your outward facing address. Only needed if you’re connecting over the internet. If you use ‘0.0.0.0’ for the addr argument, this will automatically be set to your LAN address.
  • debug_level – The verbosity you want this socket to use when printing event data
Raises:

SocketException – The address you wanted could not be bound, or is otherwise used
daemon
id_10
data
leeching
addr

An alternate binding for self.out_addr, in order to better handle self-references in pathfinding

data_storing
disconnect_least_efficient()[source]

Disconnects the node which provides the least value.

This is determined by finding the node which is the closest to its neighbors, using the modulus distance metric

Returns:A bool that describes whether a node was disconnected
_handle_peers(msg, handler)[source]

This callback is used to deal with peer signals. Its primary jobs is to connect to the given peers, if this does not exceed py2p.chord.max_outgoing

Parameters:
Returns:

Either True or None
dump_data(start, end)[source]
Parameters:
  • start – An int which indicates the start of the desired key range. 0 will get all data.
  • end – An int which indicates the end of the desired key range. None will get all data.
Returns:

A nested dict containing your data from start to end
__getitem__(key)[source]

Looks up the value at a given key. Under the covers, this actually checks five different hash tables, and returns the most common value given.

Parameters:

key – The key that you wish to check. Must be a str or bytes-like object
Returns:

The value at said key
Raises:
  • socket.timeout – If the request goes partly-unanswered for >=timeout seconds
  • KeyError – If the request is made for a key with no agreed-upon value

Note

It’s probably much better to use get()

getSync(key, ifError=None, timeout=10)[source]

Looks up the value at a given key. Under the covers, this actually checks five different hash tables, and returns the most common value given.

Parameters:
  • key – The key that you wish to check. Must be a str or bytes-like object
  • ifError – The value you wish to return on exception (default: None)
  • timeout – The longest you would like to await a value (default: 10s)
Returns:

The value at said key, or the value at ifError if there’s an Exception

Note

It’s probably much better to use get()

get(key, ifError=None, timeout=10)[source]

Looks up the value at a given key. Under the covers, this actually checks five different hash tables, and returns the most common value given.

Parameters:
  • key – The key that you wish to check. Must be a str or bytes-like object
  • ifError – The value you wish to return on exception (default: None)
  • timeout – The longest you would like to await a value (default: 10s)
Returns:

A Promise of the value at said key, or the value at ifError if there’s an Exception
__setitem__(key, value)[source]

Updates the value at a given key. Under the covers, this actually uses five different hash tables, and updates the value in all of them.

Parameters:
  • key – The key that you wish to update. Must be a str or bytes-like object
  • value – The value you wish to put at this key.
Raises:

TypeError – If your key is not bytes -like OR if your value is not serializable. This means your value must be one of the following:
set(key, value)[source]

Updates the value at a given key. Under the covers, this actually uses five different hash tables, and updates the value in all of them.

Parameters:
  • key – The key that you wish to update. Must be a str or bytes-like object
  • value – The value you wish to put at this key.
Raises:

TypeError – If your key is not bytes -like OR if your value is not serializable. This means your value must be one of the following:
apply_delta(key, delta)[source]

Updates a stored mapping with the given delta. This allows for more graceful handling of conflicting changes

Parameters:
  • key – The key you wish to apply a delta to. Must be a str or bytes-like object
  • delta – A mapping which contains the keys you wish to update, and the values you wish to store
Returns:

A Promise which yields the resulting data, or rejects with a TypeError if the updated key does not store a mapping already.
Raises:

TypeError – If the updated key does not store a mapping already.
update(update_dict)[source]

Equivalent to dict.update()

This calls ChordSocket.store() for each key/value pair in the given dictionary.

Parameters:update_dict – A dict-like object to extract key/value pairs from. Key and value be a str or bytes-like object
find(key)[source]

Finds the node which is responsible for a certain value. This does not necessarily mean that they are supposed to store that value, just that they are along your path to said node.

Parameters:key – The key that you wish to check. Must be a int or long

Returns: A ChordConnection or this socket

find_prev(key)[source]

Finds the node which is farthest from a certain value. This is used to find a node’s “predecessor”; the node it is supposed to delegate to in the event of a disconnections.

Parameters:key – The key that you wish to check. Must be a int or long

Returns: A ChordConnection or this socket

next

The connection that is your nearest neighbor ahead on the hash table ring

prev

The connection that is your nearest neighbor behind on the hash table ring

_send_meta(handler)[source]

Shortcut method for sending a chord-specific data to a given handler

Parameters:handler – A ChordConnection
join()[source]

Tells the node to start seeding the chord table

unjoin()[source]

Tells the node to stop seeding the chord table

connect(*args, **kwargs)[source]

This function connects you to a specific node in the overall network. Connecting to one node should connect you to the rest of the network, however if you connect to the wrong subnet, the handshake failure involved is silent. You can check this by looking at the truthiness of this objects routing table. Example:

>>> conn = mesh.MeshSocket('localhost', 4444)
>>> conn.connect('localhost', 5555)
>>> # do some other setup for your program
>>> if not conn.routing_table:
...     conn.connect('localhost', 6666)  # any fallback address
Parameters:
  • addr – A string address
  • port – A positive, integral port
  • id – A string-like object which represents the expected ID of this node
keys()[source]

Returns an iterator of the underlying dict’s keys

__iter__()[source]

Returns an iterator of the underlying dict’s keys

values()[source]
Returns:

an iterator of the underlying dict’s values
Raises:
items()[source]
Returns:

an iterator of the underlying dict’s items
Raises:
pop(key, *args)[source]

Returns a value, with the side effect of deleting that association

Parameters:
  • Key – The key you wish to look up. Must be a str or bytes-like object
  • ifError – The value you wish to return on Exception (default: raise an Exception )
Returns:

The value of the supplied key, or ifError
Raises:
popitem()[source]

Returns an association, with the side effect of deleting that association

Returns:

An arbitrary association
Raises:
_BaseSocket__closed
_BaseSocket__handlers
_ChordSocket__connect(addr, port, id=None)

Private API method for connecting and handshaking

Parameters:
  • addr – the address you want to connect to/handshake
  • port – the port you want to connect to/handshake
_ChordSocket__delta(method, key, delta)

Updates the value at a given key, using the supplied delta. This method deals with just one of the underlying hash tables.

Parameters:
  • method – The hash table that you wish to check. Must be a str or bytes-like object
  • key – The key that you wish to check. Must be a int or long
  • delta – The delta you wish to apply at this key.
_ChordSocket__getitem(key, timeout=10)

Looks up the value at a given key. Under the covers, this actually checks five different hash tables, and returns the most common value given.

Parameters:
  • key – The key that you wish to check. Must be a str or bytes-like object
  • timeout – The longest you would like to await a value (default: 10s)
Returns:

The value at said key
Raises:
  • socket.timeout – If the request goes partly-unanswered for >=timeout seconds
  • KeyError – If the request is made for a key with no agreed-upon value

Note

It’s probably much better to use get()

_ChordSocket__handle_delta(msg, handler)

This callback is used to deal with delta storage signals. Its primary job is:

  • update the mapping in a given key
Args:
msg: A Message handler: A ChordConnection
Returns:
Either True or None
_ChordSocket__handle_key(msg, handler)

This callback is used to deal with new key entries. Its primary job is:

  • Ensure keylist syncronization
Parameters:
Returns:

Either True or None
_ChordSocket__handle_meta(msg, handler)

This callback is used to deal with chord specific metadata. Its primary job is:

  • set connection state
Parameters:
Returns:

Either True or None
_ChordSocket__handle_retrieve(msg, handler)

This callback is used to deal with data retrieval signals. Its two primary jobs are:

  • respond with data you possess
  • if you don’t possess it, make a request with your closest peer to
    that key
Parameters:
Returns:

Either True or None
_ChordSocket__handle_retrieved(msg, handler)

This callback is used to deal with response signals. Its two primary jobs are:

  • if it was your request, send the deferred message
  • if it was someone else’s request, relay the information
Parameters:
Returns:

Either True or None
_ChordSocket__handle_store(msg, handler)

This callback is used to deal with data storage signals. Its two primary jobs are:

  • store data in keys you’re responsible for
  • if you aren’t responsible, make a request with your closest peer to
    that key
Parameters:
Returns:

Either True or None
_ChordSocket__keys
_ChordSocket__lookup(method, key, handler=None)

Looks up the value at a given hash function and key. This method deals with just one of the underlying hash tables.

Parameters:
  • method – The hash table that you wish to check. Must be a str or bytes-like object
  • key – The key that you wish to check. Must be a int or long
Returns:

The value at said key in an py2p.utils.awaiting_value object, which either contains or will eventually contain its result
_ChordSocket__store(method, key, value)

Updates the value at a given key. This method deals with just one of the underlying hash tables.

Parameters:
  • method – The hash table that you wish to check. Must be a str or bytes-like object
  • key – The key that you wish to check. Must be a int or long
  • value – The value you wish to put at this key. Must be a str or bytes-like object
_MeshSocket__clean_waterfalls()

This function cleans the set of recently relayed messages based on the following heuristics:

  • Delete all older than 60 seconds
_MeshSocket__handle_handshake(msg, handler)

This callback is used to deal with handshake signals. Its three primary jobs are:

  • reject connections seeking a different network
  • set connection state
  • deal with connection conflicts
Parameters:
Returns:

Either True or None
_MeshSocket__handle_request(msg, handler)

This callback is used to deal with request signals. Its three primary jobs are:

  • respond with a peers signal if packets[1] is '*'
  • if you know the ID requested, respond to it
  • if you don’t, make a request with your peers
Parameters:
Returns:

Either True or None
_MeshSocket__handle_response(msg, handler)

This callback is used to deal with response signals. Its two primary jobs are:

  • if it was your request, send the deferred message
  • if it was someone else’s request, relay the information
Parameters:
Returns:

Either True or None
_MeshSocket__resolve_connection_conflict(handler, h_id)

Sometimes in trying to recover a network a race condition is created. This function applies a heuristic to try and organize the fallout from that race condition. While it isn’t perfect, it seems to have increased connection recovery rate from ~20% to ~75%. This statistic is from memory on past tests. Much improvement can be made here, but this statistic can likely never be brought to 100%.

In the failure condition, the overall network is unaffected for large networks. In small networks this failure condition causes a fork, usually where an individual node is kicked out.

Parameters:
  • handler – The handler with whom you have a connection conflict
  • h_id – The id of this handler
_get_peer_list()

This function is used to generate a list-formatted group of your peers. It goes in format [ ((addr, port), ID), ...]

_logger
_send_handshake(handler)

Shortcut method for sending a handshake to a given handler

Parameters:handler – A MeshConnection
_send_peers(handler)

Shortcut method to send a handshake response. This method is extracted from __handle_handshake() in order to allow cleaner inheritence from py2p.sync.SyncSocket

awaiting_ids
close()

If the socket is not closed, close the socket

Raises:RuntimeError – The socket was already closed
copy()[source]

Returns a dict copy of this DHT

Warning

This is a very slow operation. It’s a far better idea to use items(), as this produces an iterator. That should even out lag times

debug_level
disconnect(handler)

Closes a given connection, and removes it from your routing tables

Parameters:handler – the connection you would like to close
emit(event, *args, **kwargs)

Emit event, passing *args and **kwargs to each attached function. Returns True if any functions are attached to event; otherwise returns False.

Example:

ee.emit('data', '00101001')

Assuming data is an attached function, this will call data('00101001')'.

For coroutine event handlers, calling emit is non-blocking. In other words, you do not have to await any results from emit, and the coroutine is scheduled in a fire-and-forget fashion.

handle_msg(msg, conn)

Decides how to handle various message types, allowing some to be handled automatically

Parameters:
Returns:

True if an action was taken, None if not.
id
incoming

IDs of incoming connections

listeners(event)

Returns the list of all listeners registered to the event.

on(event, f=None)

Registers the function (or optionally an asyncio coroutine function) f to the event name event.

If f isn’t provided, this method returns a function that takes f as a callback; in other words, you can use this method as a decorator, like so:

@ee.on('data')
def data_handler(data):
    print(data)

As mentioned, this method can also take an asyncio coroutine function:

@ee.on('data')
async def data_handler(data)
    await do_async_thing(data)

This will automatically schedule the coroutine using the configured scheduling function (defaults to asyncio.ensure_future) and the configured event loop (defaults to asyncio.get_event_loop()).

once(event, f=None)

The same as ee.on, except that the listener is automatically removed after being called.

out_addr
outgoing

IDs of outgoing connections

protocol
queue
recv(quantity=1)

This function has two behaviors depending on whether quantity is left as default.

If quantity is given, it will return a list of Message objects up to length quantity.

If quantity is left alone, it will return either a single Message object, or None

Parameters:quantity – The maximum number of Message s you would like to pull (default: 1)
Returns:A list of Message s, an empty list, a single Message , or None
register_handler(method)

Register a handler for incoming method.

Parameters:method – A function with two given arguments. Its signature should be of the form handler(msg, handler), where msg is a py2p.base.Message object, and handler is a py2p.base.BaseConnection object. It should return True if it performed an action, to reduce the number of handlers checked.
Raises:ValueError – If the method signature doesn’t parse correctly
remove_all_listeners(event=None)

Remove all listeners attached to event. If event is None, remove all listeners on all events.

remove_listener(event, f)

Removes the function f from event.

request_peers()

Requests your peers’ routing tables

requests
routing_table
send(*args, **kargs)

This sends a message to all of your peers. If you use default values it will send it to everyone on the network

Parameters:
  • *args – A list of objects you want your peers to receive
  • **kargs – There are two keywords available:
  • flag – A string or bytes-like object which defines your flag. In other words, this defines packet 0.
  • type – A string or bytes-like object which defines your message type. Changing this from default can have adverse effects.
Raises:

TypeError – If any of the arguments are not serializable. This means your objects must be one of the following:

Warning

If you change the type attribute from default values, bad things could happen. It MUST be a value from py2p.base.flags, and more specifically, it MUST be either broadcast or whisper. The only other valid flags are waterfall and renegotiate, but these are RESERVED and must NOT be used.

status

The status of the socket.

Returns:"Nominal" if all is going well, or a list of unexpected (Exception, traceback) tuples if not
waterfall(msg)

This function handles message relays. Its return value is based on whether it took an action or not.

Parameters:msg – The Message in question
Returns:True if the message was then forwarded. False if not.
waterfalls
py2p.chord.distance(a, b, limit=None)[source]

This is a clockwise ring distance function. It depends on a globally defined k, the key size. The largest possible node id is limit (or 2**384).

py2p.chord.get_hashes(key)[source]

Returns the (adjusted) hashes for a given key. This is in the order of:

  • SHA1 (shifted 224 bits left)
  • SHA224 (shifted 160 bits left)
  • SHA256 (shifted 128 bits left)
  • SHA384 (unadjusted)
  • SHA512 (unadjusted)

The adjustment is made to allow better load balancing between nodes, which assign responisbility for a value based on their SHA384-assigned ID.