1.ApplicationNumber: US-94295410-A
1.PublishNumber: US-8406153-B2
2.Date Publish: 20130326
3.Inventor: VASSEUR JEAN-PHILIPPE
SHAFFER SHMUEL
SHETTY SANDEEP

4.Inventor Harmonized: VASSEUR JEAN-PHILIPPE(FR)
SHAFFER SHMUEL(US)
SHETTY SANDEEP(US)

5.Country: US
6.Claims:
(en)In one embodiment, a node may request to join a parent node in a directed acyclic graph (DAG) in a computer network, and may also notify the parent node of a load associated with the request and whether the node has any other parent node options. The requesting node may then receive a response from the parent node that is either an acceptance or a denial. While the node may join the parent node in response to an acceptance, if a denial is received, the node may divide the load into first and second portions, and may re-request to join the parent node with the load of the first portion. In this manner, by partitioning the load, a load balancing mode of operation across multiple is parents in a DAG is provided.

7.Description:
(en)RELATED APPLICATION
The present disclosure is related to commonly-owned, U.S. patent application Ser. No. 12/942,949, filed Nov. 9, 2010, now U.S. Publication No. 2012-0117213, on May 10, 2012, entitled “NEGOTIATED PARENT JOINING IN DIRECTED ACYCLIC GRAPHS (DAGS),” by Shaffer et al.
TECHNICAL FIELD
The present disclosure relates generally to computer networks, and, more particularly, to directed acyclic graph (DAG) routing and management, e.g., for Low power and Lossy Networks (LLNs).
BACKGROUND
Low power and Lossy Networks (LLNs), e.g., sensor networks, have a myriad of applications, such as Smart Grid and Smart Cities. Various challenges are presented with LLNs, such as lossy links, low bandwidth, battery operation, low memory and/or processing capability, etc. One example routing solution to LLN challenges is a protocol called Routing Protocol for LLNs or “RPL,” which is a distance vector routing protocol that builds a Destination Oriented Directed Acyclic Graph (DODAG, or simply DAG) in addition to a set of features to bound the control traffic, support local (and slow) repair, etc. The RPL architecture provides a flexible method by which each node performs DODAG discovery, construction, and maintenance.
Currently, when a node wishes to join a DAG, it may do so without requiring permission from another node chosen as the entry point into the DAG. That is, the “child” nodes adopt their “parent” nodes without any input from the parent. Though this may not generally cause a problem, when the child node brings a heavy load with it (e.g., 1000 children of its own, or very high traffic volumes, etc.), then this autonomy can result in over-subscription and network congestion. For example, assume that a node N has two options for parent nodes, P 1 and P 2 . At first, P 1 may be selected as a better choice, and node N may bring a few orders of its own children with it to the DAG through P 1 . Due to the high number of children, P 1 may then become congested and traffic congestion and/or collisions may begin to occur. As such, P 2 may look like a better choice, and the node N may decide to use P 2 instead. However, P 2 would then become congested, and node N, under current schemes, would flip/flop between P 1 and P 2 , thus causing network instability in addition to the congestion.



BRIEF DESCRIPTION OF THE DRAWINGS
The embodiments herein may be better understood by referring to the following description in conjunction with the accompanying drawings in which like reference numerals indicate identically or functionally similar elements, of which:
FIG. 1 illustrates an example computer network;
FIG. 2 illustrates an example network device/node;
FIG. 3 illustrates an example message;
FIG. 4A illustrates an example directed acyclic graph (DAG);
FIG. 4B illustrates another example DAG;
FIG. 5A illustrates an example negotiation message exchange in the network;
FIG. 5B illustrates an example of load partitioning in the network;
FIG. 5C illustrates another example of load partitioning in the network;
FIG. 6 illustrates an example shed load request exchange in the network;
FIG. 7A illustrates an example set of negotiation message exchanges in the network;
FIG. 7B illustrates another example shed load request exchange in the network;
FIG. 7C illustrate another example of load shedding in the network; and
FIG. 8 illustrates an example simplified procedure for negotiated DAG joining and load partitioning.



DESCRIPTION OF EXAMPLE EMBODIMENTS
Overview
According to one or more embodiments of the disclosure, a node may request to join a parent node in a directed acyclic graph (DAG) in a computer network, and may also notify the parent node of a load associated with the request, and whether the node has any other parent node options. The requesting node may then receive a response from the parent node that is either an acceptance or a denial. While the node may join the parent node in response to an acceptance, if a denial is received, the node may divide the load into first and second portions, and may re-request to join the parent node with the load of the first portion. In this manner, by partitioning the load, a load balancing mode of operation across multiple parents in a DAG is provided.
Description
A computer network is a geographically distributed collection of nodes interconnected by communication links and segments for transporting data between end nodes, such as personal computers and workstations, or other devices, such as sensors, etc. Many types of networks are available, with the types ranging from local area networks (LANs) to wide area networks (WANs). LANs typically connect the nodes over dedicated private communications links located in the same general physical location, such as a building or campus. WANs, on the other hand, typically connect geographically dispersed nodes over long-distance communications links, such as common carrier telephone lines, optical lightpaths, synchronous optical networks (SONET), synchronous digital hierarchy (SDH) links, or Powerline Communications (PLC) such as IEEE 61334, CPL G3, WPC and others. In addition, a Mobile Ad-Hoc Network (MANET) is a kind of wireless ad-hoc network, which is generally considered a self-configuring network of mobile routes (and associated hosts) connected by wireless links, the union of which forms an arbitrary topology.
Smart object networks, such as sensor networks, in particular, are a specific type of network having spatially distributed autonomous devices such as sensors, actuators, etc., that cooperatively monitor physical or environmental conditions at different locations, such as, e.g., energy/power consumption, resource consumption (e.g., water/gas/etc.) temperature, pressure, vibration, sound, radiation, motion, pollutants, etc. Other types of smart objects include actuators, e.g., responsible for turning on/off an engine or perform any other actions. Sensor networks, a type of smart object network, are typically wireless networks, though wired connections are also available. That is, in addition to one or more sensors, each sensor device (node) in a sensor network may generally be equipped with a radio transceiver or other communication port, a microcontroller, and an energy source, such as a battery. Generally, size and cost constraints on sensor nodes result in corresponding constraints on resources such as energy, memory, computational speed and bandwidth. Correspondingly, a reactive routing protocol may, though need not, be used in place of a proactive routing protocol for sensor networks.
In certain configurations, the sensors in a sensor network transmit their data, along with routing/relaying data from other sensors, to one or more centralized or distributed database management nodes that obtain the data for use with one or more associated applications. Alternatively (or in addition), certain sensor networks provide for mechanisms by which an interested subscriber (e.g., “sink”) may specifically request data from devices in the network. In a “push mode,” the sensors transmit their data to the sensor sink/subscriber without prompting, e.g., at a regular interval/frequency or in response to external triggers, such as alarm messages. Conversely, in a “pull mode,” the sensor sink may specifically request that the sensors (e.g., specific sensors or all sensors) transmit their current data (or take a measurement, and transmit that result) to the sensor sink. (Those skilled in the art will appreciate the benefits and shortcomings of each mode, and both apply to the techniques described herein.)
FIG. 1 is a schematic block diagram of an example computer network 100 illustratively comprising nodes/devices 200 (e.g., labeled as shown, “Root,” “11,” “12,” . . . “44,” “4N”) interconnected by various methods of communication. For instance, the links 105 may be wired links or may comprise a wireless communication medium, where certain nodes 200 , such as, e.g., routers, sensors, computers, etc., may be in communication with other nodes 200 , e.g., based on distance, signal strength, current operational status, location, etc. Those skilled in the art will understand that any number of nodes, devices, links, etc. may be used in the computer network, and that the view shown herein is for simplicity. Also, while the embodiments are shown herein with reference to a generally “tree” shaped network, the description herein is not so limited, and may be applied to networks that have branches emitting to all directions from with the root node generally centralized among a plurality of surrounding nodes
Illustratively, certain devices in the network may be more capable than others, such as those devices having larger memories, sustainable non-battery power supplies, etc., versus those devices having minimal memory, battery power, etc. For instance certain devices 200 may have no or limited memory capability. Also, one or more of the devices 200 may be considered “root nodes/devices” (or root capable devices), also referred to as LLN border routers (LBRs), while one or more of the devices may also be considered “destination nodes/devices.”
Data packets 140 (e.g., traffic and/or messages sent between the devices/nodes) may be exchanged among the nodes/devices of the computer network 100 using predefined network communication protocols such as the Transmission Control Protocol/Internet Protocol (TCP/IP), User Datagram Protocol (UDP), Multi-Protocol Label Switching (MPLS), various proprietary protocols, etc. In this context, a protocol consists of a set of rules defining how the nodes interact with each other. In addition, packets within the network 100 may be transmitted in a different manner depending upon device capabilities, such as source routed packets.
FIG. 2 is a schematic block diagram of an example node/device 200 that may be used with one or more embodiments described herein, e.g., as a root node or other node (e.g., sensor) in the network. The device may comprise one or more network interfaces 210 , one or more sensor components 215 (e.g., sensors, actuators, etc.), a processor 220 (e.g., an 8-64 bit microcontroller), and a memory 240 interconnected by a system bus 250 , as well as a power supply 260 (e.g., battery, plug-in, etc.).
The network interface(s) 210 contain the mechanical, electrical, and signaling circuitry for communicating data over physical and/or wireless links coupled to the network 100 . The network interfaces may be configured to transmit and/or receive data using a variety of different communication protocols, including, inter alia, TCP/IP, UDP, wireless protocols (e.g., IEEE Std. 802.15.4, WiFi, Bluetooth®), Ethernet, powerline communication (PLC) protocols, etc. Note that the root may have two different types of network connections 210 . Namely, one or more interfaces may be used to communicate with the mesh network (into the mesh cell), i.e., the other nodes shown in FIG. 1 , while another interface is used as a WAN uplink network interface between the root node and, to for example, a head-end device located through the WAN.
The memory 240 comprises a plurality of storage locations that are addressable by the processor(s) 220 and the network interfaces 210 for storing software programs and data structures associated with the embodiments described herein. As noted above, certain devices may have limited memory or no memory (e.g., no memory for storage other than for programs/processes operating on the device). The processors 220 may comprise necessary elements or logic adapted to execute the software programs and manipulate the data structures, such as routes or prefixes 245 (notably on capable devices only). An operating system 242 , portions of which are typically resident in memory 240 and executed by the processor(s), functionally organizes the device by, inter alia, invoking operations in support of software processes and/or services executing on the device. These software processes and/or services may comprise routing process/services 244 , which may include an illustrative directed acyclic graph (DAG) process 246 . Also, for root devices (or other management devices), a topology management process 248 and associated stored topologies 249 may also be present in memory 240 , for use as described herein. It will be apparent to those skilled in the art that other processor and memory types, including various computer-readable media, may be used to store and execute program instructions pertaining to the techniques described herein. Also, while the description illustrates various processes, it is expressly contemplated that various processes may be embodied as modules configured to operate in accordance with the techniques herein (e.g., according to the functionality of a similar process).
Routing process (services) 244 contains computer executable instructions executed by the processor 220 to perform functions provided by one or more routing protocols, such as proactive or reactive routing protocols as will be understood by those skilled in the art. These functions may, on capable devices, be configured to manage a routing/forwarding table 245 containing, e.g., data used to make routing/forwarding decisions. In particular, in proactive routing, connectivity is discovered and known prior to computing routes to any destination in the network, e.g., link state routing such as Open Shortest Path First (OSPF), or Intermediate-System-to-Intermediate-System (ISIS), or Optimized Link State Routing (OLSR). Reactive routing, on the other hand, discovers neighbors (i.e., does not have an a priori knowledge of network topology), and in response to a needed route to a destination, sends a route request into the network to determine which neighboring node may be used to reach the desired destination. Example reactive routing protocols may comprise Ad-hoc On-demand Distance Vector (AODV), Dynamic Source Routing (DSR), DYnamic MANET On-demand Routing is (DYMO), etc. Notably, on devices not capable or configured to store routing entries, routing process 244 may consist solely of providing mechanisms necessary for source routing techniques. That is, for source routing, other devices in the network can tell the less capable devices exactly where to send the packets, and the less capable devices simply forward the packets as directed.
Low power and Lossy Networks (LLNs), e.g., certain sensor networks, may be used in a myriad of applications such as for “Smart Grid” and “Smart Cities.” A number of challenges in LLNs have been presented, such as:
1) Links are generally lossy, such that a Packet Delivery Rate/Ratio (PDR) can dramatically vary due to various sources of interferences, e.g., considerably affecting the bit error rate (BER);
2) Links are generally low bandwidth, such that control plane traffic must generally be bounded and negligible compared to the low rate data traffic;
3) There are a number of use cases that require specifying a set of link and node metrics, some of them being dynamic, thus requiring specific smoothing functions to avoid routing instability, considerably draining bandwidth and energy;
4) Constraint-routing may be required by some applications, e.g., to establish routing paths that will avoid non-encrypted links, nodes running low on energy, etc.;
5) Scale of the networks may become very large, e.g., on the order of several thousands to millions of nodes; and
6) Nodes may be constrained with a low memory, a reduced processing capability, a low power supply (e.g., battery).
In other words, LLNs are a class of network in which both the routers and their interconnect are constrained: LLN routers typically operate with constraints, e.g., processing power, memory, and/or energy (battery), and their interconnects are characterized by, illustratively, high loss rates, low data rates, and/or instability. LLNs are comprised of anything from a few dozen and up to thousands or even millions of LLN routers, and support point-to-point traffic (between devices inside the LLN), point-to-multipoint traffic (from a central control point to a subset of devices inside the LLN) and multipoint-to-point traffic (from devices inside the LLN towards a central control point).
An example protocol specified in an Internet Engineering Task Force (IETF) Internet Draft, entitled “RPL: IPv6 Routing Protocol for Low Power and Lossy Networks” by Winter, at al. (Oct. 25, 2010 version), provides a mechanism that supports multipoint-to-point (MP2P) traffic from devices inside the LLN towards a central control point (e.g., LLN Border Routers (LBRs) or “root nodes/devices” generally), as well as point-to-multipoint (P2MP) traffic from the central control point to the devices inside the LLN (and also point-to-point, or “P2P” traffic). RPL (pronounced “ripple”) may generally be described as a distance vector routing protocol that builds a Directed Acyclic Graph (DAG) for use in routing traffic/packets 140 , in addition to defining a set of features to bound the control traffic, support repair, etc.
A DAG is a directed graph having the property that all edges are oriented in such a way that no cycles (loops) are supposed to exist. All edges are contained in paths oriented toward and terminating at one or more root nodes (e.g., “clusterheads or “sinks”), often to interconnect the devices of the DAG with a larger infrastructure, such as the Internet, a wide area network, or other domain. In addition, a Destination Oriented DAG (DODAG) is a DAG rooted at a single destination, i.e., at a single DAG root with no outgoing edges. A “parent” of a particular node within a DAG is an immediate successor of the particular node on a path towards the DAG root, such that the parent has a lower “rank” than the particular node itself, where the rank of a node identifies the node's position with respect to a DAG root (e.g., the farther away a node is from a root, the higher is the rank of that node). Further, a sibling of a node within a DAG may be defined as any neighboring node which is located at the same rank within a DAG. Note that siblings do not necessarily share a common parent, and routes between siblings are generally not part of a DAG since there is no forward progress (their rank is the same). Note also that a tree is a kind of DAG, where each device/node in the DAG generally has one parent or one preferred parent.
DAGs may generally be built based on an Objective Function (OF). The role of the Objective Function is generally to specify rules on how to build the DAG (e.g. is number of parents, backup parents, etc.).
In addition, one or more metrics/constraints may be advertised by the routing protocol to optimize the DAG against. Also, the routing protocol allows for including an optional set of constraints to compute a constrained path, such as where if a link or a node does not satisfy a required constraint, it is “pruned” from the candidate list when computing the best path. (Alternatively, the constraints and metrics may be separated from the OF.) Additionally, the routing protocol may include a “goal” that defines a host or set of hosts, such as a host serving as a data collection point, or a gateway providing connectivity to an external infrastructure, where a DAG's primary objective is to have the devices within the DAG be able to reach the goal. In the case where a node is unable to comply with an objective function or does not understand or support the advertised metric, it may be configured to join a DAG as a leaf node. As used herein, the various metrics, constraints, policies, etc., are considered “DAG parameters.”
Illustratively, example metrics used to select paths (e.g., preferred parents) may comprise cost, delay, latency, bandwidth, estimated transmission count (ETX), etc., while so example constraints that may be placed on the route selection may comprise various reliability thresholds, restrictions on battery operation, multipath diversity, bandwidth requirements, transmission types (e.g., wired, wireless, etc.). The OF may provide rules defining the load balancing requirements, such as a number of selected parents (e.g., single parent trees or multi-parent DAGs). Notably, an example for how routing metrics and constraints may be obtained may be found in an IETF Internet Draft, entitled “Routing Metrics used for Path Calculation in Low Power and Lossy Networks” by Vasseur, et al. (Oct. 23, 2010 version). Further, an example OF (e.g., a default OF) may be found in an IETF Internet Draft, entitled “RPL Objective Function 0” by Thubert (Jul. 29, 2010 version).
Building a DAG may utilize a discovery mechanism to build a logical representation of the network, and route dissemination to establish state within the network so that routers know how to forward packets toward their ultimate destination. Note that a “router” refers to a device that can forward as well as generate traffic, while a “host” refers to a device that can generate but does not forward traffic. Also, a “leaf” may be used to generally describe a non-router that is connected to a DAG by one or more routers, but cannot itself forward traffic received on the DAG to another router on the DAG. Control messages may be transmitted among the devices within the network for discovery and route dissemination when building a DAG.
According to the illustrative RPL protocol, a DODAG Information Object (DIO) is a type of DAG discovery message that carries information that allows a node to discover a RPL Instance, learn its configuration parameters, select a DODAG parent set, and maintain the upward routing topology. In addition, a Destination Advertisement Object (DAO) is a type of DAG discovery reply message that conveys destination information upwards along the DODAG so that a DODAG root (and other intermediate nodes) can provision downward routes. A DAO message includes prefix information to identify destinations, a capability to record routes in support of source routing, and information to determine the freshness of a particular advertisement. Notably, “upward” or “up” paths are routes that lead in the direction from leaf nodes towards DAG roots, e.g., following the orientation of the edges within the DAG. Conversely, “downward” or “down” paths are routes that lead in the direction from DAG roots towards leaf nodes, e.g., generally going in the opposite direction to the upward messages within the DAG.
Generally, a DAG discovery request (e.g., DIO) message is transmitted from the root device(s) of the DAG downward toward the leaves, informing each successive receiving device how to reach the root device (that is, from where the request is received is generally the direction of the root). Accordingly, a DAG is created in the upward direction toward the root device. The DAG discovery reply (e.g., DAO) may then be returned from the leaves to the root device(s) (unless unnecessary, such as for UP flows only), informing each successive receiving device in the other direction how to reach the leaves for downward routes. Nodes that are capable of maintaining routing state may aggregate routes from DAO messages that they receive before transmitting a DAO message. Nodes that are not capable of maintaining routing state, however, may attach a next-hop parent address. The DAO message is then sent directly to the DODAG root that can in turn build the topology and locally compute downward routes to all nodes in the DODAG. Such nodes are then reachable using source routing techniques over regions of the DAG that are incapable of storing downward routing state.
FIG. 3 illustrates an example simplified control message format 300 that may be used for discovery and route dissemination when building a DAG, e.g., as a DIO or DAO. Message 300 illustratively comprises a header 310 with one or more fields 312 that identify the type of message (e.g., a RPL control message), and a specific code indicating the specific type of message, e.g., a DIO or a DAO (or a DAG Information Solicitation). Within the body/payload 320 of the message may be a plurality of fields used to relay the pertinent information. In particular, the fields may comprise various flags/bits 321 , a sequence number 322 , a rank value 323 , an instance ID 324 , a DAG ID 325 , and other fields, each as may be appreciated in more detail by those skilled in the art. Further, for DAO messages, additional fields for destination prefixes 326 and a transit information field 327 may also be included, among others (e.g., DAO_Sequence used for ACKs, etc.). For either DIOs or DAOs, one or more additional sub-option fields 328 may be used to supply additional or custom information within the message 300 . For instance, an objective code point (OCP) sub-option field may be used within a DID to carry codes specifying a particular objective function (OF) to be used for building the associated DAG. Alternatively, sub-option fields may be used to carry certain information within a message 300 , as may be described herein.
As noted above, when a node wishes to join a DAG in current systems, it may do so without requiring permission from another node chosen as the entry point into the DAG, such that “child” nodes adopt their “parent” nodes without any input from the parent. Though this may not generally cause a problem, when the child node brings a heavy load with it (e.g., 1000 children of its own, or very high traffic volumes, etc.), then this autonomy can result in over-subscription and network congestion.
For example, as the RPL protocol builds its routing table, it may use a measurement which reflects the path cost of the path from each node towards the root. Once a node finds a path towards the root it may continuously monitor the quality of the path towards the root for example by maintaining a measurement of the ratio between packets which failed to be sent successfully and the number of packets which were sent successfully upstream towards the root. For instance, although other metrics could be used to optimize the path (e.g. latency) the ETX is a common metric used to compute the link and path reliability. The more packets a node fails to deliver upstream the higher is its ETX measurement, indicating that the path is less desired.
Assume, as shown in FIG. 4A , that a node 32 , which can route its packets towards the root either via parent node 21 or 22 having path cost (quality in this example) ETXi and ETXj respectfully. Assuming that ETXiETXj. After some time, then, node 32 may decide to act upon this information and optimize its route towards the root. Given that the path through 22 is now better than the path through 21 , node 32 may switch parents and associates itself with node 22 for the DAG, as shown in FIG. 4B .
By switching to 22 , however, node 32 directs its uplink traffic through this node, thus increasing the congestion though 22 and reducing the traffic and congestion through node 21 . This new flow will therefore alter the relative ETX measurements of nodes 21 and 22 , resulting in ETXi Even in the absence of oscillations, when a node such as 32 selects node 21 without permission, this may lead to a dramatic increase of the load on some links along the path to the root crossing a certain traffic threshold (e.g., becoming congested), resulting in sub-optimal network operation.
Negotiated Parent Adoption Through Load Partitioning
According to one or more embodiments of the disclosure, therefore, when a node desires to join a DAG at a potential parent node, that node requests permission or negotiates the join (parent adoption), such as by informing the potential parent node of the “load” or burden associated with adding the requesting node. If the load is acceptable, then the node may join the DAG at the desired parent. If the load is unacceptable, however, then the node may be requested to “shed” some of the load, such as by partitioning some of its own children (child nodes of the requesting node) into different groups, or partitioning actual traffic flows into different groups, as described in detail herein. For example, if node 21 tells node 32 that it can only accept two of node 32 's children (more like on the order of hundreds or thousands), and node 22 tells node 32 that it also can only accept two of node 32 's children, then node 32 may partition its children into two groups, and join the DAG at node 21 for the first group, and at node 22 for the second group (and forwards traffic accordingly). Accordingly, a type of Routing Admission Control (RAC) is provided.
Illustratively, the techniques described herein may be performed by hardware, software, and/or firmware, such as in accordance with DAG process 246 , which may contain computer executable instructions executed by the processor 220 to perform functions relating to the novel techniques described herein, e.g., in conjunction with routing process 244 . For example, the techniques herein may be treated as extensions to conventional protocols, such as the RPL protocol, and as such, would be processed by similar components understood in the art that execute the RPL protocol, accordingly.
Operationally, the techniques herein extend the sub-options field 328 of message 300 (e.g., an additional type-length-value (TLV) in DAO messages) so each node may provide its prospective parent with a potential load/burden of adding the particular node to the DAG, e.g., based on the node itself and any of its child nodes (e.g., “grandchildren” to the prospective parent node). For example, the actual “load” may indicate various attributes, such as a number of child nodes that are already associated with the requesting node (e.g., nodes 41 - 4 N for requesting node 32 ), an amount of traffic being handled by the requesting (e.g., bandwidth throughput, expected/planned traffic like reservations or based on network statistics), a frequency of traffic (e.g., the rate at which the node is expected to transmit and/or relay data), etc.
As shown in FIG. 5A , for example, node 32 may send a request 520 a to join a prospective parent node 21 for DAG 410 (and a request 520 b to prospective parent node 22 , as described below). Within this request is the notification of the load associated with adding the requesting node 32 , as noted above. In addition, another sub-option field 328 may be used to notify the potential parent node as to whether the requesting node has any other parent node options (e.g., node 22 ). For instance, if there are other options, this notification may also include a quality associated with each other parent node option. In other words, each node provides its prospective parent with information whether it has other viable options or is the prospective parent the only option the requesting node has. Similarly, in accordance with a specific embodiment, a node may notify its prospective parent whether its children may have other options to split their traffic accordingly. Also, the requesting node may provide some metrics, such as ETX, delay, etc., related to the relative quality of each prospective parent.
Note that in one or more embodiments, a node may query all of its potential parents first; at this stage it does not know (and thus does not say) if it has alternative parents. Then it could start to interrogate all nodes while indicating if it has alternatives parents. In this example, if node 22 simply rejects the request, if could interrogate node 21 indicating that it has no other alternative parent.
A potential parent node may receive the request and associated notifications, and may determine an impact associated with allowing the child node to join the particular node in the DAG based on the load. That is, a parent node receiving a request may evaluate the load it is already experiencing, such as a number of children it already has, the traffic it already has to handle, etc., as well as the impact the prospective child (and its children) may have on the overall traffic the parent would need to handle. Note that when available, the node may evaluate the impact of that new child joining not only on its local link to its own parent but upper links along the path to the DAG root. That is, the parent node may also have information about the load experienced in non-directly attached links (e.g., links along its path to the root) if metric recording is enabled. An example of metric recording for the RPL protocol is defined in the IETF Internet Draft, entitled “Routing Metrics used for Path Calculation in Low Power and Lossy Networks” by Vasseur, at al. (Oct. 23, 2010 version).
If the new traffic is under a predetermined threshold level, the new child may be accepted, and the parent node may transmit an acceptance response 530 . As such, the requesting/child node may receive the acceptance response, and may join the parent node in the DAG in response.
Conversely, a denial response 530 may be transmitted to the requesting node in response to the determined impact being above the threshold level. Notably, in accordance with one or more embodiments herein, this denial need not simply reject the request from the child/requesting node (a “straight denial”), but instead the parent may counter the request by asking the child node to shed some of its load (e.g., children, traffic, etc.) before asking to be associated with the parent node. Note that such a threshold may be dynamic based on various network statistics, which may be observed in real-time. For instance, the threshold may be based on the number of observed dropped packet due to collisions occurring higher up in the DAG (e.g., between node 11 and root node, etc.), or based on other statistics such as available bandwidth, utilized bandwidth, delay, etc.
In response to a denial 530 , the requesting node may either simply attempt another parent node option, or, in accordance with one or more embodiments herein, may attempt to divide/partition the load into more granular (e.g., more manageable) portions. For instance, the requesting node 32 may divide its load into at least first and second portions or subgroups, depending upon the measure of the load and the number of the available parents (e.g., n portions for n parent options).
For example, when the load is determined as a number of child nodes, then the dividing may partition the number of child nodes into a first and second group of child nodes. Illustratively, FIG. 5B shows an example where requesting node 32 has partitioned its child-based load into two partitions ( 1 and 2 ), where partition 1 includes child nodes 41 and 42 , while partition 2 includes child nodes 43 - 4 N. The actual partitioning boundaries, e.g., the number of child nodes per group, may be defined based on a number of factors. For instance, in one embodiment, the denial comprises a counter request to shed load to a particular threshold value (e.g., half the requested load, a specific number of children, etc.). As such, the dividing may create the first portion based on the particular threshold value (i.e., the feedback from the parent). Alternatively, the division may be based on having sent out multiple requests 520 a/b , and determining an optimal division based on responses from numerous parents (e.g., if two parents can each handle three new grandchildren, then the options would be to use one parent for three grandchildren and one for the fourth, or to use each parent for two grandchildren . . . ). As still another alternative, the requesting node may simply divide the load at will, and continue attempting to join the parent node with new partitions.
In particular, as a node partitions its load, it can compare the updated load of each partition to the threshold learned from the prospective parent node. When a node manages to reduce its load (e.g., the number of its children per request of the prospective parent), then the node may re-issue a request (or “re-request”) to the prospective parent to join the DAG with just a particular portion, this time bringing less load (less children and/or traffic) to the prospective parent. Alternatively, the node may simply join the parent without issuing another request, e.g., in order to avoid additional control plane traffic (e.g., in trust mode).
In the example in FIG. 5B , therefore, nodes 41 and 42 are presented to node 21 as grandchildren, while nodes 43 to 4 N may be presented to an alternate parent, node 22 . Note that in this instance, the DAG 410 may comprise a particular portion 410 b that is contrary to typical DAG operation, but manageable given certain safeguards. In particular, node 32 has been partitioned, and, as described below, may participate in two separate branches of the DAG according to proper forwarding separation techniques (e.g., ensuring that traffic from the correct children nodes is transmitted on the correct DAG branch). Note that in FIG. 5B , it is shown that node 22 may have had to adjust its own parent in order to accommodate the new load from the requesting node 32 . For example, if the reason node 21 couldn't take all of the traffic from the grandchildren due to congestion of the link between node 11 and the root node, then simply sending half of the traffic through parent node 22 would not solve that problem on its own, since it would be remerged at the congested link. Accordingly, node 22 may find an alternate parent node (node 12 ) through which the new increased load may be relayed toward the root.
As shown in FIG. 5C , the load may also be considered as an amount of traffic. As such, the dividing load into groups may comprise configuring flow control of the requesting node to partition the traffic into a first and second group of flows. For example, assuming that there is a way to distinguish flows, as may be appreciated by those skilled in the art (e.g., priority, class, type, source address, destination address, etc.), the requesting node may determine based on various traffic metrics how to divide the load in a manner that allows for a certain portion of the load to be forwarded to one parent node (e.g., to node 21 on DAG 410 ), and another portion to be forwarded to another parent node (e.g., to node 22 on DAG portion 410 b ). Note further that flows need not be child specific, and a particular child, e.g., node 43 , may contribute multiple flows to the requesting node 32 .
According to these and other embodiments, it is important to note that it may be necessary or otherwise desirable to record the traffic path of the DAG to ensure that any new divided traffic does not end up through the same congested path since nodes are often afforded only partial visibility since they use a distance-vector protocol. By using recorded path DIO messages ( 300 ), therefore, when attempting to divide the traffic, links may be examined to determine whether the new traffic would bypass a link which is experiencing congestion.
In order to ensure proper traffic forwarding at the requesting node (e.g., node 32 ), in accordance with one or more embodiments herein, the forwarding behavior of the requesting node may need to be updated such that traffic corresponding to each portion is forwarded to the proper joined parent node for that portion. For instance, assume that a certain portion “1” of the load is to be forwarded to a first parent node (e.g., node 21 ), and a second portion “2” of the load is to be forwarded to a second parent node (e.g., node 22 ). Accordingly, all downlink messages to node 32 as well as to the first partition (e.g., nodes 41 and 42 as in FIG. 5B ) will be handled by node 21 , while all downlink messages to the second partition (e.g., nodes 43 - 4 N) will be handled by node 22 . The forwarding of node 32 (e.g., routing process 244 and/or DAG process 246 ) should reflect this handling of the traffic so as to not confuse the two partitions, and vice-versa (in the UP direction). In other words, traffic/packets may be forwarded according to their origin, and not just their destination, and such information should be stored in a corresponding routing table (e.g., of routing process 244 ).
Note that the actual determination as to which traffic is to be forwarded as the first portion and the second portion may be made in one embodiment strictly at the requesting node 32 on a case-by-case basis. For instance, with added intelligence, it may be possible to base the forwarding on one or more network metrics and one or more attributes of the traffic. For example, if the partition was in response to a bandwidth constraint, the requesting node 32 may simply prohibit an excess amount of bandwidth from utilizing a particular parent node, whether that traffic originates at the same grandchild or not (under certain network configurations, where the source location is unimportant to lower-ranked devices, that is).
In addition to real-time dynamic partitioning, however, a more simple technique herein simply assigns particular load partitions (e.g., child groups and/or flows) to a particular grandparent, and leaves that assignment alone until necessary to do otherwise. At the same time, however, this primarily stable assignment may be based on the requesting node's own decisions, or, alternatively, may be based on requests by child nodes from which the traffic is received to have the corresponding traffic be handled as a particular portion of the traffic of the first and second portions. For example, assume that there is a choice between three paths to three different parents, and assume that there are three associated path costs 1 , 2 , and 3 , respectively. In one embodiment, it may be beneficial to forward the most important traffic over the lowest cost path, while intermediate traffic can take the next lowest cost path. Further, while the third path may be acceptable, the node may inform certain of its children (or flows) in a least important group to try to find a better value through another parent option. As an alternative arrangement, these three paths may be configured with certain allowances, and the child nodes may mark their traffic with certain tags (e.g., quality of service or “QoS” indicators, such as priority, etc.), and the requesting node 32 may forward traffic on the best available path based on the requested handling of the traffic.
In accordance with one or more embodiments of the disclosure, in response to a denial 530 that has a counter request to shed load, and if the requesting node is unable to adequately partition its load to join the parent node with just a partition, then the requesting node may examine its children and ask the children who had other options to seek association with the other prospective parent. For example, as shown in FIG. 6 , node 32 may propagate a shed load request 640 to one or more of its child nodes, such as to all child nodes (e.g., 41 - 4 N) or only those with other parent node options (e.g., 43 - 4 N). The shed load request 640 , such as a message 300 with sub-options field 328 , may request that the one or more child nodes attempt to shed their load. Specifically, the “shed load” request may request that the child nodes join any other parent node option, or to attempt to further shed load themselves. For instance, though not shown, each of the child nodes 41 - 4 N may also have further child nodes. The request to shed load may thus ask the child nodes to split the traffic it gets from its children (grandchildren of the requesting node 32 ) into two or more groups, and by doing so reduce the effective traffic to its prospective parent, or, alternatively, the child nodes may ask the grandchildren to find alternative parents (other than nodes 41 - 4 N), etc. Note that a level of exploration may be limited, e.g., 2-3 rank levels, before ceasing the query. If this approach in FIG. 6 is unsuccessful, then the requesting node may simply re-request to the parent, and indicate that there are no alternative options, e.g., and that no additional load may be shed, at which time, the sole option parent must generally accept the requesting node.
In addition, in accordance with one or more embodiments herein, prior to transmitting a denial response to a requesting node, particularly in response to the child (requesting) node having no other parent node options, the parent node that received the request may propagate a shed load request to one or more of its own child nodes. For instance, as shown in FIG. 7A , prospective parent node 21 may send a shed load request 640 a to its child node 31 , requesting that the child nodes attempt to join any other parent node option, or else (e.g., if unable to), to attempt to shed some of its load (e.g., shedding grandchildren, partitioning loads, etc.).
As shown in FIG. 7B , in response to shedding load at the prospective parent (node 21 ) such that the addition of the join request load from node 32 results in an impact below the threshold, the prospective parent node 21 may transmit an acceptance response to the child node 32 to join the DAG. For example, as shown in FIG. 7B , the child node 31 of node 21 may have been able to switch parents to node 22 , thus leaving room at node 21 for node 32 to join the DAG (e.g., assuming that the load of node 32 was un-partitionable, or other preference setting of action to be performed in this situation). Note also, that node 22 , which was originally joined to the DAG via parent node 11 , may have had to switch its own parent node (e.g., to node 12 ) in order to accommodate the new additional child node's load.
According to one or more embodiments of the disclosure, prior to transmitting an acceptance response, a prospective parent (e.g., node 21 ) may first transmit its own “join request” to its own parent in the DAG 410 . For instance, as shown in FIG. 7C , this join request 520 c may request acceptance of a load increase based on the load of the received join request, since the addition at “lower” ranks/levels of a DAG may produce a cumulative effect the closer to the root the traffic is. As shown, this load increase request may propagate up the DAG toward the root (e.g., all the way to the root or up for a few ranks) for each node (e.g., messages 520 c , 520 d , etc.). Note that in this essence, the requesting node may already be joined with the parent node (e.g., node 21 to node 11 ), and the request to “join” the parent is in response to a load increase due to a join request received at the requesting node (e.g., at node 21 from node 32 ). Note that this recursive mode of operation is particularly useful if the requesting node does not have information about the load of the upper links in the DAG: in contrast, if such information is available via metric recording as specified above, this recursive request may not be needed.
If there is a problem along the way toward the root, such as where the cumulative effect is too great at any node/link in the DAG, then a denial response may be generated, potentially producing a greater number of shed load requests, or simply a straight denial. Note that according to the techniques herein, each shed load request encountered may result in a node, which received the shed load request, attempting to split its traffic amongst various prospective parents. On the other hand, however, if there are no encountered problems, then acceptance responses 530 (c and d) may be returned toward the potential parent node 21 , which may then inform the requesting/child node of the overall acceptance into the DAG.
FIG. 8 illustrates an example simplified procedure for negotiated parent joining in accordance with one or more embodiments described herein, e.g., from the perspective of the requesting/child node. The procedure 800 starts at step 805 , and continues to step 810 , where a node (e.g., 32 ) requests to join a parent node (e.g., 21 ) in a DAG. As described above, the request 520 may notify the parent node of a load associated with the request, as well as whether there are any other parent node options (e.g., and their quality). In step 815 , the requesting node may receive a response 530 to the request from the parent node, which is either an acceptance or denial (step 820 ).
If the response indicates that the join is acceptable, then in step 825 , the requesting node may join the parent node in the DAG, along with its associated load. Note that as described above, and as relevant below (if partitioned loads), in step 830 forwarding of the requesting node may be configured according to partitions, ensuring that correct traffic (e.g., from certain child nodes or for certain flows) is forwarded to the correct parent node. Note also that the routing table would govern the routing of downlink messages as well, as described above.
If the response in step 820 is a denial, then in step 835 the requesting node may attempt another parent node option, which ends the procedure (step 860 ) to produce a new request for a new parent node. Alternatively, the requesting node may instead determine whether the denial response is a counter request to shed load in step 840 . If it is not, then the requesting node may try another parent node (assuming, that is, that there are other parent nodes available), or, in either event of step 840 , may attempt to divide the load in step 845 , e.g., into at least first and second portions. As described above, if the partitioning results in adequate load shedding in step 850 , then the requesting node may (optionally) re-request to join the parent node in step 855 with the load of the first portion, only (ending the procedure in step 860 to send a request again in step 810 ). If this is accepted in step 825 , then again it is noted that it is important to properly configure the forwarding of traffic in step 830 according to the partitions. For example, in accordance with the techniques described above, traffic forwarding may take advantage of the partitioned loads to load balance traffic effectively through the DAG. Further, if is insufficient load can be shed in step 850 , then an alternative parent may be attempted in step 835 .
Note that while the procedure 800 above is described with reference to the requesting node, the parent node may also perform certain actions in order to attempt to place the requesting node onto the DAG. For instance, as described above, the acceptance of the requesting node may first need to be confirmed up the DAG toward the root node. Also, if the parent node is unable to accept for any reason, it may send out its own shed load requests to its own child nodes, or may partition its load in other ways. The example procedure 800 above, then, is not meant to be limiting to the scope of the embodiments herein, and is merely illustrative.
The novel techniques described herein provide for a negotiation between child and parent nodes regarding acceptance of the DAG relationship. In particular, previous techniques simply allowed any child node to join any parent node, without any say or even feedback from the parent node. However, the techniques herein provide the parent node with the authority to push back on child nodes, and request that they lighten their loads if necessary (e.g., which may be completed by partitioning the load, as described above). For instance, the techniques herein perform the negotiation in advance of the relationship, and may help to prevent over-subscription in the network and to minimize metric flopping or simply poor metrics in general (e.g., congestion in the DAG), particularly before simply forming the relationships and realizing that something is broken, as in conventional systems. Also, the dynamic load partitioning techniques above provide functionality that would be difficult, if not impossible, to perform manually. Notably, contrary to conventional load balancing techniques, the embodiments herein may actually change the topology of the network in response to the traffic.
While there have been shown and described illustrative embodiments that manage DAGs in a computer network, it is to be understood that various other adaptations and modifications may be made within the spirit and scope of the embodiments herein. For example, the embodiments have been shown and described herein with relation to LLNs, and more particular, to the RPL protocol. However, the embodiments in their broader sense are not so limited, and may, in fact, be used with other types of networks and/or protocols utilizing DAG routing (e.g., distance vector protocols). Also, while the techniques described above reference partitioning load, these concepts may also be used in conjunction with load shedding, such as asking child nodes to find other parent nodes, in order to reduce the load of the requesting node to an acceptable level (e.g., as found in the related application noted above) or asking a child to divide its traffic into two or more sub-flows and direct divide these flows amongst various parents. For example, certain loads may be partitioned, while certain loads may be shed.
The foregoing description has been directed to specific embodiments. It will be apparent, however, that other variations and modifications may be made to the described embodiments, with the attainment of some or all of their advantages. For instance, it is expressly contemplated that the components and/or elements described herein can be implemented as software being stored on a tangible (non-transitory) computer-readable medium (e.g., disks/CDs/etc.) having program instructions executing on a computer, hardware, firmware, or a combination thereof. Accordingly this description is to be taken only by way of example and not to otherwise limit the scope of the embodiments herein. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the embodiments herein.