DETNET                                                          Q. Xiong
Internet-Draft                                           ZTE Corporation
Intended status: Informational                                     Z. Du
Expires: 27 April 2023                                      China Mobile
                                                                 J. Zhao
                                                                   CAICT
                                                                 D. Yang
                                             Beijing Jiaotong University
                                                         24 October 2022


 DetNet Data Plane Enhancements for Large-Scale Deterministic Networks
             draft-xiong-detnet-large-scale-enhancements-01

Abstract

   From charter and milestones, the enhanced Deterministic Networking
   (DetNet) is required to provide the enhancement of flow
   identification and packet treatment for data plane to achieve the
   DetNet QoS in large-scale networks.

   This document analyzes the gaps of the existing technologies
   especially applying the DetNet data plane as per RFC8938 and proposes
   the enhancement of packet treatment to support the functions and
   metadata for enhanced DetNet data plane.  It describes related
   enhanced controller plane considerations as well.

Status of This Memo

   This Internet-Draft is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 27 April 2023.

Copyright Notice

   Copyright (c) 2022 IETF Trust and the persons identified as the
   document authors.  All rights reserved.



Xiong, et al.             Expires 27 April 2023                 [Page 1]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info) in effect on the date of publication of this document.
   Please review these documents carefully, as they describe your rights
   and restrictions with respect to this document.  Code Components
   extracted from this document must include Revised BSD License text as
   described in Section 4.e of the Trust Legal Provisions and are
   provided without warranty as described in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   3
   2.  Conventions used in this document . . . . . . . . . . . . . .   4
     2.1.  Terminology . . . . . . . . . . . . . . . . . . . . . . .   4
     2.2.  Requirements Language . . . . . . . . . . . . . . . . . .   4
   3.  Gap Analysis of DetNet Data Plane . . . . . . . . . . . . . .   4
     3.1.  Service Requirements of Large-Scale Deterministic
           Networks  . . . . . . . . . . . . . . . . . . . . . . . .   4
       3.1.1.  Support the Differentiated DetNet QoS of Multiple
               Services  . . . . . . . . . . . . . . . . . . . . . .   4
       3.1.2.  Support the Utilization of Network Resources  . . . .   6
     3.2.  Characteristics of Large-Scale Deterministic Networks . .   6
       3.2.1.  Large-scale Dynamic Flows . . . . . . . . . . . . . .   6
       3.2.2.  Large-scale Network Topology  . . . . . . . . . . . .   7
     3.3.  Gap Analysis of Large-Scale Deterministic Networks  . . .   7
       3.3.1.  Gap Analysis of Providing Aggregated Flows
               Identification  . . . . . . . . . . . . . . . . . . .   7
       3.3.2.  Gap Analysis of Providing Deterministic Latency . . .   8
         3.3.2.1.  Gap Analysis of Explicit Routes . . . . . . . . .   8
         3.3.2.2.  Gap Analysis of Resources Allocation  . . . . . .   9
         3.3.2.3.  Gap Analysis of Queuing Mechanisms  . . . . . . .   9
   4.  Enhancements of DetNet Data Plane . . . . . . . . . . . . . .  10
     4.1.  Enhancements of Packet Treatment  . . . . . . . . . . . .  10
       4.1.1.  Flow Identification . . . . . . . . . . . . . . . . .  10
       4.1.2.  Deterministic Routes  . . . . . . . . . . . . . . . .  11
         4.1.2.1.  Deterministic Links . . . . . . . . . . . . . . .  11
         4.1.2.2.  Inter-domain Deterministic Routes . . . . . . . .  12
       4.1.3.  Deterministic Resources . . . . . . . . . . . . . . .  12
       4.1.4.  Queuing treatment . . . . . . . . . . . . . . . . . .  13
     4.2.  Enhancements of DetNet-Specific Metadata  . . . . . . . .  13
     4.3.  Enhancements of DetNet IP/MPLS/SRv6 Data Plane  . . . . .  14
   5.  Controller Plane (Management and Control) Considerations  . .  14
     5.1.  Management and Scheduling of Multiple Queuing
           Mechanisms  . . . . . . . . . . . . . . . . . . . . . . .  14
     5.2.  Distributed Deterministic Path  . . . . . . . . . . . . .  14
     5.3.  Inter-domain Deterministic Path . . . . . . . . . . . . .  14
     5.4.  Deterministic Path Computation  . . . . . . . . . . . . .  15
     5.5.  Configuration of Flow Mapping . . . . . . . . . . . . . .  15



Xiong, et al.             Expires 27 April 2023                 [Page 2]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   6.  Security Considerations . . . . . . . . . . . . . . . . . . .  15
   7.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  15
   8.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  15
   9.  Normative References  . . . . . . . . . . . . . . . . . . . .  15
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  18

1.  Introduction

   According to [RFC8655], Deterministic Networking (DetNet) operates at
   the IP layer and delivers service which provides extremely low data
   loss rates and bounded latency within a network domain.  The
   framework of DetNet data planes has been specified in RFC8938.  The
   IP and MPLS DetNet data plane has been defined respectively in
   RFC8939 and RFC8964.  The DetNet IP data plane primarily uses 6-
   tuple-based flow identification.  And the DetNet MPLS data plane
   leverages existing pseudowire (PW) encapsulations and MPLS Traffic
   Engineering (MPLS-TE) encapsulations.

   The applications in 5G networks demand much more deterministic and
   precise properties in large-scale networks.  The existing
   deterministic technologies are facing large-scale number of nodes and
   long-distance transmission, traffic scheduling, dynamic flows, and
   other controversial issues in large-scale networks.  The enhanced
   DetNet Data plane is required to support a data plane method of flow
   identification and packet treatment.
   [I-D.liu-detnet-large-scale-requirements] has described the
   enhancement requirements for DetNet data plane.

   The enhanced DetNet data plane aims to describe how to use IP and/or
   MPLS, and related OAM, to support a data plane method of flow
   identification and packet treatment over Layer 3.  The enhanced QoS-
   related functions and metadata should be provided in large-scale
   networks.  For example, as described in
   [I-D.ietf-detnet-bounded-latency], the end-to-end bounded latency
   depends on the value of queuing delay bound along with the queuing
   mechanisms.  Mutiple queuingmechanisms can be used to guarantee the
   bounded latency in DetNet.  New DetNet-specific metadata should be
   carried in enhanced DetNet IP/MPLS/SRv6 Data Plane.

   This document analyzes the gaps of the existing technologies
   especially applying the DetNet data plane as per RFC8938 and proposes
   the enhancement of packet treatment to support the functions and
   metadata for enhanced DetNet data plane.  It describes related
   enhanced controller plane considerations as well.







Xiong, et al.             Expires 27 April 2023                 [Page 3]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


2.  Conventions used in this document

2.1.  Terminology

   The terminology is defined as [RFC8655] and [RFC8938].

2.2.  Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in BCP
   14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

3.  Gap Analysis of DetNet Data Plane

3.1.  Service Requirements of Large-Scale Deterministic Networks

3.1.1.  Support the Differentiated DetNet QoS of Multiple Services

   5G network is oriented to the internet of everything.  It need to
   supports the Ultra-reliable Low Latency Communications (uRLLC)
   services.  The uRLLC services demand SLA guarantees such as low
   latency and high reliability and other deterministic and precise
   properties especially in Wide Area Network (WAN) applications.The
   uRLLC services should be provided in large-scale networks which cover
   the industries such as intelligent electrical network, intelligent
   factory, internet of vehicles, industry automation and other
   industrial internet scenarios.  The industrial internet is the key
   infrastructure that coordinate various units of work over various
   system components, e.g. people, machines and things in the industrial
   environment including big data, cloud computing, Internet of Things
   (IOT), Augment Reality (AR), industrial robots, Artificial
   Intelligence (AI) and other basic technologies.  For the intelligent
   electrical network, there are deterministic requirements for
   communication delay, jitter and packet loss rate.  For example, in
   the electrical current difference model, a delay of 3~10ms and a
   jitter variation is no more than 100us are required.  For the
   automation control, it is one of the basic application and the the
   core is closed-loop control system.  The control process cycle is as
   low as millisecond level, so the system communication delay needs to
   reach millisecond level or even lower to ensure the realization of
   precise control.  There are three levels of real-time requirements
   for industrial interconnection: factory level is about 1s, and
   process level is 10~100ms, and the highest real-time requirement is
   motion control, which requires less than 1ms.  So the deterministic
   latency requirements are different with varying services and network
   scenarios.



Xiong, et al.             Expires 27 April 2023                 [Page 4]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   As defined in [RFC8655], the DetNet QoS can be expressed in terms of
   : Minimum and maximum end-to-end latency, bounded jitter (packet
   delay variation), packet loss ratio and an upper bound on out-of-
   order packet delivery.  As described in [RFC8578], DetNet
   applications differ in their network topologies and specific desired
   behavior and different services requires differentiated DetNet QoS.
   In the large-scale networks, multiple services with differentiated
   DetNet QoS is co-existed in the same DetNet network.  The
   classification of the deterministic flows within different levels is
   should be taken into considerations.  It is required to provide
   Latency, bounded jitter and packet loss dynamically and flexibly in
   all scenarios for each characterized flow.

   As the Figure 1 shows, the services can be divided into 5 levels and
   level 2~5 is the DetNet flows and level-1 is non-DetNet flow.  DetNet
   applications and DetNet QoS is differentiated within each level.



   +-------------+-----------+----------+----------+----------+-----------+
   | Item        | Level-1   | Level-2  | Level-3  | Level-4  |  Level-5  |
   +-------------+-----------+----------+----------+----------+-----------+
   | Applications| Broadcast |  Voice   | Audio and| AR/VR    | Industrial|
   | Examples    |           |          | Video    |          |           |
   +-------------+-----------+----------+----------+----------+-----------+
   | DetNet QoS  | Bandwidth | Jitter   | Latency  | Low      | Ultra-low |
   |             | Guarantee | Guarantee| Guarantee| latency  |latency and|
   |             |           |          |          |and jitter| jitter    |
   +-------------+-----------+----------+----------+----------+-----------+


          Figure 1: The classification of multiple services


   From the perspective of deterministic service requirements,
   deterministic Quality of Service (QoS) in the network can be divided
   into five types or levels:

   Level-1: bandwidth guarantee.  The indicator requirements include
   basic bandwidth guarantee and certain packet loss tolerance.  There
   is no requirement for the upper bound of the latency, and no
   requirement for the jitter.  Typical services include download and
   FTP services.

   Level-2: jitter guarantee.  The indicator requirements include:
   jitter 50ms, delay 300ms.  Typical services include synchronous voice
   services, such as voice call.




Xiong, et al.             Expires 27 April 2023                 [Page 5]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   Level-3: Latency guarantee.  The indicator requirements include:
   delay 50ms, jitter 50ms.  Typical services include real-time
   communication services, such as video, production monitoring, and
   communication services.

   Level-4: low delay and low jitter guarantee.  The indicator
   requirements include: delay 20ms, jitter 5ms.  Typical services
   include video interaction services, such as AR/VR, holographic
   communication, cloud video and cloud games.

   Level-5: ultra-low delay and jitter guarantee.  The indicator
   requirements include: delay 10ms, jitter 100us.  Typical services
   include production control services, such as power protection and
   remote control.

   Moreover, different DetNet services is required to tolerate different
   percentage of packet loss ratio such as 99.9%, 99.99%, 99.999%, and
   so on.

3.1.2.  Support the Utilization of Network Resources

   Traditional Ethernet, IP and MPLS networks which is based on
   statistical multiplexing provides best-effort packet service and
   offers no delivery and SLA guarantee.  As described in [RFC8655], the
   primary technique by which DetNet achieves its QoS is to allocate
   sufficient resources.  But it can not be achieved by not sufficient
   resource which can be allocated due to practical and cost reason.  So
   it is required to achieve the high-efficiency of resources
   utilization when provide the DetNet service.

3.2.  Characteristics of Large-Scale Deterministic Networks

3.2.1.  Large-scale Dynamic Flows

   As described in [RFC8557], deterministic forwarding can only apply to
   flows with such well-defined characteristics as periodicity and
   burstiness.  As defined in DetNet architecture [RFC8655], the traffic
   characteristics of an App-flow can be CBR (constant bit rate) or VBR
   (variable bit rate) of L1, L2 and L3 layers (VBR takes the maximum
   value when reserving resources).  But the current scenarios and
   technical solutions only consider CBR flow, without considering the
   coexistence of VBR and CBR, the burst and aperiodicity of flows.  The
   operations such as shaping or scheduling have not been specified.
   Even TSN mechanisms are based on a constant and forecastable traffic
   characteristics.






Xiong, et al.             Expires 27 April 2023                 [Page 6]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   It will be more complicated in a large-scale network where much more
   flows coexist and the traffic characteristics is more dynamic.  A
   huge number of flows with different DetNet QoS requirements is
   dynamically concurrent and the state of each flow cannot be
   maintained.  It is required to offer reliable delivery and SLA
   guarantee for dynamic flows.  For example, periodic flow and
   aperiodic flow (including micro burst flow, etc.), CBR and VBR flow,
   flow with different periods or phases, etc.  When the network needs
   to forward these deterministic flows at the same time, it must solve
   the problems of time micro bursts, queue processing and aggregation
   of multiple flows.

3.2.2.  Large-scale Network Topology

   In large-scale applications, the network topology may consists of a
   large number of nodes and links which leads to difficulty with
   controlling the end-to-end delay and jitter.  High speed, long-
   distance transmission and asymmetric links may also co-exists and
   affects the bounded latency such as increasing transmission latency,
   jitter and packet loss in large-scale networks.

   The network topology in a large-scale network may across multiple
   domains within a single administrative control or a closed group of
   administrative control as per [RFC8655].  Moreover, DetNet domains or
   nodes may be interconnected with different sub-network technologies
   such as FlexE tunnels, TSN sub-network, IP/MPLS/SRv6 tunnels and so
   on.  It is required to support the inter-domain deterministic metric
   and routes to achieve the end-to-end bounded latency.

3.3.  Gap Analysis of Large-Scale Deterministic Networks

   As defined in [RFC8938], the DetNet data plane describes how
   application flows, or App-flows are carried over DetNet networks and
   it is provided by the DetNet service and forwarding sub-layers with
   DetNet-related data plane functions and mechanisms.  This section
   analyzes the DetNet technical gaps when applying the DetNet data
   plane as per RFC8938 in large-scale networks.

3.3.1.  Gap Analysis of Providing Aggregated Flows Identification

   In [RFC8938], the DetNet data plane can provide the DetNet-Specific
   Metadata such as Flow-ID for both the service and forwarding sub-
   layers.  The flow-based state information is required to be
   maintained for per-flow processing rules.  For example, the resource
   reservation configuration is required for each flow.  DetNet as per
   [RFC8938] provides the capability to aggregate the individual flows
   to downscale the operations of flow states.  However, it still
   requires large amount of control signaling to establish and maintain



Xiong, et al.             Expires 27 April 2023                 [Page 7]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   DetNet flows.  It may be challenging for network operations with a
   large number of deterministic flows and network nodes in large-scale
   networks.

3.3.2.  Gap Analysis of Providing Deterministic Latency

   As described in [RFC8655], the primary goals are to achieve the
   DetNet QoS to provide minimum and maximum end-to-end latency and
   bounded jitter, low packet loss ratio and an upper bound on out-of-
   order packet delivery.  But the data plane [RFC8938] particularly
   focuses on the DetNet service sub-layer which provides a set of
   Packet Replication, Elimination, and Ordering Functions (PREOF)
   functions to provide end-to-end service assurance.  It mainly
   provides the capabilities for DetNet to guarantee the reliability.

   The DetNet forwarding sub-layer provides corresponding forwarding
   assurance with IETF existing functions using resource allocations and
   explicit routes.  But these functions can not provide the
   deterministic latency (bounded latency, low packet loss and in-order
   delivery) assurance in large-scale networks.  The following sections
   mainly discuss the gap analysis for the forwarding sub-layer
   functions to provide deterministic latency assurance.

3.3.2.1.  Gap Analysis of Explicit Routes

   Traditional routes only have reachability.  As per [RFC8938],
   explicit optimized paths with allocation of resources should be
   provided to achieve the DetNet QoS.  But the deterministic
   requirements such as end-to-end delay and jitter are only used as
   path computation constraints.  Multiple network metrics which are
   measured and distributed by the routing system should be taken into
   consideration.

   In large-scale networks, it may be challenging to compute the best
   path to meet all of the requirements.  In multi-domain scenarios, the
   inter-domain deterministic routes need to be established and
   provisioned.  Especially when interconnecting with sub-networks, the
   selection of intra-domain paths acrossing cooperating domains should
   consider the bounded latency in each domain and the stitching of the
   paths.

   Moreover, the paths vary with the real-time change of the network
   topology.  On the basic of the resources, the steering path and
   routes for deterministic flows should be programmed before the flows
   coming and able to provide SLA capability.  And the routes should be
   considered to be established in distributed and centralized control
   Plane.




Xiong, et al.             Expires 27 April 2023                 [Page 8]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   As described in [RFC8557], the packet replication and elimination
   service protection should be provided to achieve the low packet loss
   ratio.  It will copy the flows and spread the data over multiple
   disjoint forwarding paths.  The bounded latency and jitter of each
   path should be meet service deterministic requirement.  And the
   difference of latency within these paths should be limited.  So the
   replication and elimination deterministic routes with configured
   latency and jitter policy should be taken into consideration.  It is
   required to generate two disjoint paths with very close delay to form
   1+1 protection and perform concurrent transmission and dual
   reception, and make replication and elimination on the egress PE.

3.3.2.2.  Gap Analysis of Resources Allocation

   As per [RFC8938], the forwarding sub-layer uses buffer resources for
   packet queuing, as well as reservation and allocation of bandwidth
   capacity resources.  In large-scale networks, the bandwidth, buffer
   and scheduling resources are combined with queuing mechanisms to
   guarantee the deterministic latency.  The reservation and allocation
   of queuing related resources or deterministic latency resources
   should be taken into consideration in DetNet data plane.

3.3.2.3.  Gap Analysis of Queuing Mechanisms

   As per [RFC8938], the forwarding sub-layer provides the QoS-related
   functions needed by the DetNet flow including the use of queuing
   techniques.  But the queuing techniques which are defined in existing
   IETF technologies can not guarantee the bounded latency such as
   Active Queue Management(AQM).  And the queuing mechanisms which are
   defined in IEEE802.1 TSN can not be directly applied in large-scale
   networks such Time Aware Shaping [IIEEE802.1Qbv] and Cyclic Queuing
   and Forwarding [IEEE802.1Qch] with time synchronization.

   Enhancement of queuing mechanisms have been discussed in DetNet such
   as cyclic-scheduling queuing mechanism
   [I-D.dang-queuing-with-multiple-cyclic-buffers], deadline-scheduling
   queuing mechanism [I-D.stein-srtsn] and
   [I-D.peng-detnet-deadline-based-forwarding], and asynchronous queuing
   mechanism [I-D.joung-detnet-asynch-detnet-framework].  The function
   of multiple queuing mechanisms and related DetNet-Specific Metadata
   has not been defined in DetNet data plane.










Xiong, et al.             Expires 27 April 2023                 [Page 9]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


4.  Enhancements of DetNet Data Plane

   As defined in [RFC8938], the DetNet data plane describes how
   application flows, or App-flows are carried over DetNet networks and
   it is provided by the DetNet service and forwarding sub-layers with
   DetNet-related data plane functions and mechanisms.  From charter and
   milestones, the enhanced DetNet data plane is required to provide the
   enhancemant of flow identification and packet treatment including the
   enhanced QoS-related functions and metadata in large-scale networks.

4.1.  Enhancements of Packet Treatment

   This section proposes the enhancement for the DetNet Data Plane
   Protocol Stack as shown in Figure 1 and the enhanced DetNet-related
   data plane functions and mechanisms should be provided by the DetNet
   service and forwarding sub-layers.

                 |  packets going  |            ^  packets coming   ^
                 v down the stack  v            |   up the stack    |
           +-----------------------------+   +----------------------------+
           |           Source            |   |        Destination         |
           +-----------------------------+   +----------------------------+
           |Service sub-layer:           |   |Service sub-layer:          |
           |  Flow Identification        |   |  Flow Identification       |
           +-----------------------------+   +----------------------------+
           |Forwarding sub-layer:        |   |Forwarding sub-layer:       |
           |  Deterministic Routes       |   |  Deterministic Routes      |
           |  Deterministic Resources    |   |  Deterministic Resources   |
           |  Queuing treatment          |   |  Queuing treatment         |
           +-----------------------------+   +----------------------------+
           |       Lower layers          |   |       Lower layers         |
           +-----------------------------+   +----------------------------+
                             v                           ^
                              \_________________________/



   Figure 2: Enhanced Functions in DetNet Data Plane Protocol Stack


4.1.1.  Flow Identification

   From the perspective of differentiated services requirements in
   section 3.1.1, a large-scale network needs to provide the
   deterministic service for various applications.  And the
   deterministic service may demand different DetNet QoS levels
   according to different application scenarios.  The DetNet data plane
   should support the identification of multiple flows and the



Xiong, et al.             Expires 27 April 2023                [Page 10]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   differentiated deterministic QoS for each DetNet flow.

   According to the gap described in section 3.3.1, this document
   proposes the enhanced DetNet data plane to support flow
   identification of DetNet differentiated services with service-level
   identification.  It may downscale the network operations with a large
   number of deterministic flows and network nodes in large-scale
   networks.

4.1.2.  Deterministic Routes

   As discussed in section 3.3.2.1, it may be challenging to compute the
   best path to meet all of the requirements and the the paths vary with
   the real-time change of the network topology in large-scale networks.
   The explicit routes may be not appropriate for large-scale networks.
   This document propose the deterministic routes which can be strict
   explicit paths or loose routes.  The former is applicable to
   centralized scenarios with controllers, and the latter is applicable
   to distributed scenarios.

4.1.2.1.  Deterministic Links

   As discussed in section 3.3.2.1, it may be challenging to compute the
   best path to meet all of the requirements within a large-scale
   network topology pool including multiple network metrics.  This
   document proposes the deterministic links to provide a one-
   dimensional deterministic metric to guarantee for the deterministic
   forwarding capabilities at different levels.

   The computing end-to-end delay bounds is defined in
   [I-D.ietf-detnet-bounded-latency].  It is the sum of non-queuing
   delay bound and queuing delay bound in DetNet bounded latency model.
   The upper bounds of queuing delay depends on the queuing mechanisms
   deployed along the path.  For example, a link with a queuing
   mechanism that does not guarantee a bounded delay a non-determinisitc
   link and a link with a queuing mechanism that can provide
   deterministic delay is called a deterministic link.  The delay of a a
   deterministic link is consist of the propagation delay of the packet
   on the link and the queuing delay of the packet at the node.  A
   deterministic link can be a sub-network that provides deterministic
   transmission or a Point-to-Point (P2P) link.  The deterministic links
   could be distributed by IGP protocol as per
   [I-D.peng-lsr-flex-algo-deterministic-routing].








Xiong, et al.             Expires 27 April 2023                [Page 11]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


4.1.2.2.  Inter-domain Deterministic Routes

   As discussed in section 3.3.2.1, the inter-domain deterministic
   routes need to be established and provisioned in multi-domain
   scenarios.  The stitching of the intra-domain paths should be
   considered in DetNet data plane.

   In the centralized scenario, when the source and destination PEs of a
   deterministic service are located at the two ends with a limited
   physical range, one controller (single domain) or multiple
   controllers (cross domains) compute one or more paths with
   deterministic SLA according to the typical Traffic Specification
   (T-SPEC) based on the collected deterministic resources, or compute
   dynamically according to the service T-SPEC as required by the
   services.

   In the distributed scenario, deterministic loose routes are computed
   on the device through routing protocols.  Interior Gateway Protocol
   (IGP) is used to compute deterministic routes based on deterministic-
   delay inside a domain, and Border Gateway Protocol (BGP) is used to
   compute deterministic routes based on accurate delay/jitter across
   domains.

4.1.3.  Deterministic Resources

   As discussed in section 3.3.2.2, the reservation and allocation of
   queuing related resources or deterministic latency resources should
   be taken into consideration in DetNet data plane.  The networks need
   to shield the differences between network capabilities.
   Deterministic resource is the basis for providing deterministic
   network services.  It refers to the resources that meet the
   deterministic indicators of a node and link processing as well as the
   corresponding resource processing mechanisms (such as link bandwidth,
   queues, and scheduling algorithms).  It is required to make unified
   modeling for all the deterministic resources.  The deterministic
   links are provided and distributed to support the deterministic
   resource and forwarding capabilities.

   As discussed in section 3.1.2, it is necessary to make overall
   resource planning and scheduling for the network to achieve the high-
   efficiency of resources utilization when provide multiple DetNet
   services.  The admission control policy of a flow should take into
   account the deterministic resource.








Xiong, et al.             Expires 27 April 2023                [Page 12]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


4.1.4.  Queuing treatment

   As dicussed in section 3.3.2.3, it is required to support the
   enhancement of queuing mechanisms.  Multiple queuingmechanisms can
   provide different levels of latency, jitter and other guarantees.
   The DetNet forwarding sub-layer may provide the function and
   technology such as multiple queuing and traffic treatment for DetNet
   application flows.  The DetNet data plane may also encode the queuing
   related information in packets.  The encapsulation of a DetNet flow
   allows the packets to be sent over an unique queuing technology.  The
   DetNet forwarding nodes along the path can follow the queue
   scheduling carried in the packet to achieve the end-to-end bounded
   latency.

   The DetNet forwarding sub-layer may provide capabilities applying
   existing queuing mechanisms or traffic treatment.  For example, the
   traffic treatment has been proposed in [draft-du-detnet-layer3-low-
   latency] to decrease the micro-bursts in layer3 network for low-
   latency traffic.  The time-scheduling queuing mechanisms includes the
   Time Aware Shaping [IIEEE802.1Qbv] and priority-scheduling includes
   the Credit-Based Shaper[IEEE802.1Q-2014] with Asynchronous Traffic
   Shaping[IEEE802.1Qcr].  The cyclic-scheduling queuing mechanism has
   been proposed in [IEEE802.1Qch] and improved in
   [I-D.dang-queuing-with-multiple-cyclic-buffers].  The deadline-
   scheduling queuing mechanism has been proposed in [I-D.stein-srtsn]
   and improved in [I-D.peng-detnet-deadline-based-forwarding].  The
   per-flow queuing mechanism includes Guaranteed-Service Integrated
   service (IntServ) [RFC2212].

4.2.  Enhancements of DetNet-Specific Metadata

   1. deterministic latency information

   DetNet forwarding sub-layer may provide the function and technology
   such as multiple queuing and traffic treatment for DetNet application
   flows to guarantee the deterministic latency.  The DetNet data plane
   may also encode the deterministic latency related information in
   packets.

   The information ensuring deterministic latency should be provided for
   enhanced data plane as defined in
   [I-D.xiong-detnet-6man-queuing-option] and
   [I-D.sx-detnet-mpls-queue].  For example, the encapsulation of a
   DetNet flow allows the packets to be sent over an unique queuing
   mechanism.  It is required to carry queuing related information in
   data plane so as to make appropriate packet forwarding and scheduling
   decisions to meet the time bounds.




Xiong, et al.             Expires 27 April 2023                [Page 13]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


4.3.  Enhancements of DetNet IP/MPLS/SRv6 Data Plane

   An IP data plane may operate natively or through the use of an
   encapsulation.  IP encapsulation can satisfy enhanced DetNet
   requirements.  Explicit inclusion of the flow identification, path
   selection, queuing and traffic treatment is possible through the use
   of IP options, IP extension headers or existing IP headers.  For
   example, the queuing information has been carried in IPv6/SRv6
   networks as defined in [I-D.xiong-detnet-6man-queuing-option].

   MPLS provides a service sub-layer for traffic by adding specific flow
   attributes (S-label and d-cw) in packets.  MPLS provides a forwarding
   sub-layer for traffic over implicit and explicit paths such as
   F-Labels.  Explicit inclusion of queuing and traffic treatment is
   possible through the use of MPLS metadata or MPLS TC field as defined
   in [I-D.sx-detnet-mpls-queue] and [I-D.eckert-detnet-mpls-tc-tcqf].

5.  Controller Plane (Management and Control) Considerations

5.1.  Management and Scheduling of Multiple Queuing Mechanisms

   As described in [I-D.liu-detnet-large-scale-requirements] section
   3.6.1, it is required to support the configuration of multiple
   queuing mechanisms.  Different queuing mechanisms may be supported at
   different levels of latency, jitter and other guarantees.  The type
   of queuing mechanism and the related queuing parameters should be
   advertised and configured.  For example, the deterministic links with
   queuing resource could be distributed by IGP protocol as per
   [I-D.peng-lsr-flex-algo-deterministic-routing].  And the queuing
   parameters are carried in deterministic latency information may be
   selected in path computation as per
   [I-D.xiong-pce-detnet-bounded-latency].

5.2.  Distributed Deterministic Path

   The deterministic routes may be loose routes in distributed
   scenarios.  It is required to support the distributed deterministic
   routes which are established by distributed protocols such as IGP as
   defined in [I-D.peng-lsr-flex-algo-deterministic-routing].

5.3.  Inter-domain Deterministic Path

   In large-scale deterministic networks, it may across multiple network
   domains, it is required to support the inter-domain deterministic
   routes to achieve the end-to-end latency, bounded jitter.  And the
   deadline of latency and jitter of each domain and segment should be
   determined and controlled.  The inter-domain mechanism MUST be
   considered at the boundary nodes such as BGP configurations defined



Xiong, et al.             Expires 27 April 2023                [Page 14]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   in [I-D.peng-idr-bgp-metric-credit].

5.4.  Deterministic Path Computation

   As defined in [I-D.xiong-pce-detnet-bounded-latency], the
   deterministic latency constraints can be carried in PCEP extensions
   and the end-to-end deterministic path computation should be achieved
   for DetNet service.

5.5.  Configuration of Flow Mapping

   As defined in [I-D.xiong-idr-detnet-flow-mapping], the BGP flowspec
   can be used for the filtering of the packets that match the DetNet
   networks and the mapping between TSN streams and DetNet flows in the
   control plane.

6.  Security Considerations

   TBA

7.  Acknowledgements

   The authors would like to thank Peng Liu, Bin Tan, Aihua Liu Shaofu
   Peng for their review, suggestions and comments to this document.

8.  IANA Considerations

   TBA

9.  Normative References

   [I-D.dang-queuing-with-multiple-cyclic-buffers]
              Liu, B. and J. Dang, "A Queuing Mechanism with Multiple
              Cyclic Buffers", Work in Progress, Internet-Draft, draft-
              dang-queuing-with-multiple-cyclic-buffers-00, 22 February
              2021, <https://www.ietf.org/archive/id/draft-dang-queuing-
              with-multiple-cyclic-buffers-00.txt>.

   [I-D.eckert-detnet-mpls-tc-tcqf]
              Eckert, T., Bryant, S., and G. Andrew Malis,
              "Deterministic Networking (DetNet) Data Plane - MPLS TC
              Tagging for Cyclic Queuing and Forwarding (MPLS-TC TCQF)",
              Work in Progress, Internet-Draft, draft-eckert-detnet-
              mpls-tc-tcqf-03, 11 July 2022,
              <https://www.ietf.org/archive/id/draft-eckert-detnet-mpls-
              tc-tcqf-03.txt>.





Xiong, et al.             Expires 27 April 2023                [Page 15]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   [I-D.ietf-detnet-bounded-latency]
              Finn, N., Boudec, J. L., Mohammadpour, E., Zhang, J., and
              B. Varga, "DetNet Bounded Latency", Work in Progress,
              Internet-Draft, draft-ietf-detnet-bounded-latency-10, 8
              April 2022, <https://www.ietf.org/archive/id/draft-ietf-
              detnet-bounded-latency-10.txt>.

   [I-D.ietf-detnet-controller-plane-framework]
              Malis, A. G., Geng, X., Chen, M. (., Qin, F., and B.
              Varga, "Deterministic Networking (DetNet) Controller Plane
              Framework", Work in Progress, Internet-Draft, draft-ietf-
              detnet-controller-plane-framework-02, 28 June 2022,
              <https://www.ietf.org/archive/id/draft-ietf-detnet-
              controller-plane-framework-02.txt>.

   [I-D.joung-detnet-asynch-detnet-framework]
              Joung, J., Ryoo, J., Cheung, T., Li, Y., and P. Liu,
              "Asynchronous Deterministic Networking Framework for
              Large-Scale Networks", Work in Progress, Internet-Draft,
              draft-joung-detnet-asynch-detnet-framework-00, 26 June
              2022, <https://www.ietf.org/archive/id/draft-joung-detnet-
              asynch-detnet-framework-00.txt>.

   [I-D.liu-detnet-large-scale-requirements]
              Liu, P., Li, Y., Eckert, T., Xiong, Q., and J. Ryoo,
              "Requirements for Large-Scale Deterministic Networks",
              Work in Progress, Internet-Draft, draft-liu-detnet-large-
              scale-requirements-02, 10 April 2022,
              <https://www.ietf.org/archive/id/draft-liu-detnet-large-
              scale-requirements-02.txt>.

   [I-D.peng-detnet-deadline-based-forwarding]
              Peng, S., Tan, B., and P. Liu, "Deadline Based
              Deterministic Forwarding", Work in Progress, Internet-
              Draft, draft-peng-detnet-deadline-based-forwarding-01, 1
              March 2022, <https://www.ietf.org/archive/id/draft-peng-
              detnet-deadline-based-forwarding-01.txt>.

   [I-D.peng-idr-bgp-metric-credit]
              Peng, S. and B. Tan, "BGP Metric Credit Based Routing",
              Work in Progress, Internet-Draft, draft-peng-idr-bgp-
              metric-credit-00, 28 December 2021,
              <https://www.ietf.org/archive/id/draft-peng-idr-bgp-
              metric-credit-00.txt>.

   [I-D.peng-lsr-flex-algo-deterministic-routing]
              Peng, S. and T. Li, "IGP Flexible Algorithm with
              Deterministic Routing", Work in Progress, Internet-Draft,



Xiong, et al.             Expires 27 April 2023                [Page 16]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


              draft-peng-lsr-flex-algo-deterministic-routing-03, 24
              August 2022, <https://www.ietf.org/archive/id/draft-peng-
              lsr-flex-algo-deterministic-routing-03.txt>.

   [I-D.pthubert-detnet-ipv6-hbh]
              Thubert, P. and F. Yang, "IPv6 Options for DetNet", Work
              in Progress, Internet-Draft, draft-pthubert-detnet-ipv6-
              hbh-07, 22 February 2022,
              <https://www.ietf.org/archive/id/draft-pthubert-detnet-
              ipv6-hbh-07.txt>.

   [I-D.stein-srtsn]
              Stein, Y. (., "Segment Routed Time Sensitive Networking",
              Work in Progress, Internet-Draft, draft-stein-srtsn-01, 29
              August 2021, <https://www.ietf.org/archive/id/draft-stein-
              srtsn-01.txt>.

   [I-D.sx-detnet-mpls-queue]
              Song, X. and Q. Xiong, "DetNet Queue Encapsulation with
              MPLS Data Plane", Work in Progress, Internet-Draft, draft-
              sx-detnet-mpls-queue-00, 24 June 2022,
              <https://www.ietf.org/archive/id/draft-sx-detnet-mpls-
              queue-00.txt>.

   [I-D.xiong-detnet-6man-queuing-option]
              Xiong, Q. and A. Liu, "DetNet Deterministic Latency Option
              for IPv6", Work in Progress, Internet-Draft, draft-xiong-
              detnet-6man-queuing-option-02, 29 September 2022,
              <https://www.ietf.org/archive/id/draft-xiong-detnet-6man-
              queuing-option-02.txt>.

   [I-D.xiong-idr-detnet-flow-mapping]
              Xiong, Q., Wu, H., Zhao, J., and D. Yang, "BGP Flow
              Specification for DetNet and TSN Flow Mapping", Work in
              Progress, Internet-Draft, draft-xiong-idr-detnet-flow-
              mapping-03, 13 September 2022,
              <https://www.ietf.org/archive/id/draft-xiong-idr-detnet-
              flow-mapping-03.txt>.

   [I-D.xiong-pce-detnet-bounded-latency]
              Xiong, Q. and P. Liu, "PCEP Extension for DetNet Bounded
              Latency", Work in Progress, Internet-Draft, draft-xiong-
              pce-detnet-bounded-latency-01, 9 October 2022,
              <https://www.ietf.org/archive/id/draft-xiong-pce-detnet-
              bounded-latency-01.txt>.






Xiong, et al.             Expires 27 April 2023                [Page 17]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC2212]  Shenker, S., Partridge, C., and R. Guerin, "Specification
              of Guaranteed Quality of Service", RFC 2212,
              DOI 10.17487/RFC2212, September 1997,
              <https://www.rfc-editor.org/info/rfc2212>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017, <https://www.rfc-editor.org/info/rfc8174>.

   [RFC8557]  Finn, N. and P. Thubert, "Deterministic Networking Problem
              Statement", RFC 8557, DOI 10.17487/RFC8557, May 2019,
              <https://www.rfc-editor.org/info/rfc8557>.

   [RFC8578]  Grossman, E., Ed., "Deterministic Networking Use Cases",
              RFC 8578, DOI 10.17487/RFC8578, May 2019,
              <https://www.rfc-editor.org/info/rfc8578>.

   [RFC8655]  Finn, N., Thubert, P., Varga, B., and J. Farkas,
              "Deterministic Networking Architecture", RFC 8655,
              DOI 10.17487/RFC8655, October 2019,
              <https://www.rfc-editor.org/info/rfc8655>.

   [RFC8938]  Varga, B., Ed., Farkas, J., Berger, L., Malis, A., and S.
              Bryant, "Deterministic Networking (DetNet) Data Plane
              Framework", RFC 8938, DOI 10.17487/RFC8938, November 2020,
              <https://www.rfc-editor.org/info/rfc8938>.

Authors' Addresses

   Quan Xiong
   ZTE Corporation
   No.6 Huashi Park Rd
   Wuhan
   Hubei, 430223
   China
   Email: xiong.quan@zte.com.cn


   ZongPeng Du
   China Mobile
   Beijing
   China
   Email: duzongpeng@chinamobile.com



Xiong, et al.             Expires 27 April 2023                [Page 18]

Internet-Draft  DetNet Data Plane Enhancements for Large    October 2022


   Junfeng Zhao
   CAICT
   China
   Email: zhaojunfeng@caict.ac.cn


   Dong Yang
   Beijing Jiaotong University
   Beijing
   China
   Email: dyang@bjtu.edu.cn








































Xiong, et al.             Expires 27 April 2023                [Page 19]