Network File System Version 4 M. Zhang
Internet-Draft Huawei Technologies
Intended status: Standards Track S. Bhargo
Expires: 4 December 2024 VMware
R. Parambattu
D. Geng
Y. Du
Huawei Technologies
2 June 2024

Recursively Setting Directories and Subitems
draft-mzhang-nfsv4-recursively-setting-03

Abstract

In recent years, the concept of near-data computing has been widely
recognized in storage architectures. The core idea is to process
data nearby, reduce the overhead of network transmission, and utilize
the computing capability of smart devices (such as intelligent NICs,
smart SSDs, and DPUs). This reduces CPU and memory usage of clients
(computing nodes) and improves data processing efficiency. This
design idea is applied in NFSv4.2 or future NFS verions, such as
Server-Side Copy, in which client sends the control command and the
storage server copies data without passing through the data between
the client and storage server. Compared with traditional copy
operations, data is read from the source storage server and then
written to the target storage server after two network transmissions.
Data transmission on the network is reduced, and bandwidth resources
are greatly released. In addition, the client changes from an
original data copy executor to a data copy controller, and a specific
execution action is executed by the storage server. Therefore, a
large amount of computing resources and memory resources are saved on
the client side.

Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].

Status of This Memo

This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.

Zhang, et al. Expires 4 December 2024 [Page 1]

Internet-Draft Recursively Setting Directoriestle June 2024

Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."

This Internet-Draft will expire on 4 December 2024.

This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.

Table of Contents

1. Problem Statement . . . . . . . . . . . . . . . . . . . . . . 2
2. Protocol Overview . . . . . . . . . . . . . . . . . . . . . . 4
3. Implementation Considerations . . . . . . . . . . . . . . . . 6
4. Recursive Set Operations . . . . . . . . . . . . . . . . . . 9
5. Security Considerations . . . . . . . . . . . . . . . . . . . 13
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 13
7. References . . . . . . . . . . . . . . . . . . . . . . . . . 13
7.1. Normative References . . . . . . . . . . . . . . . . . . 13
7.2. Informative References . . . . . . . . . . . . . . . . . 13
Appendix A. An Appendix . . . . . . . . . . . . . . . . . . . . 13
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 13

1. Problem Statement

In actual storage applications, users often recursively set the
attributes of directories and subitems(their subfiles and
subdirectories). Message interaction between client and server is
complex, and the client consumes a lot of resources, which does not
match the concept of near-data computing. FIG. 1 shows the existing
flowchart of recursively setting the attributes of all files under

Zhang, et al. Expires 4 December 2024 [Page 2]

Internet-Draft Recursively Setting Directoriestle June 2024

dir1.

Step 1: The client sends the readdir command to obtain the list of
all files in dir1.

Step 2: The storage server responds to the readdir operation. If the
directory contains many subitems, the client needs to run the readdir
operation for multiple times.

Step 3: The client obtains the file list based on the response
information, executes the getattr request for each file, and obtains
the file attributes.

Step 4: The storage server responds to the getattr request.

Step 5: The client sends a setattr request.

Step 6: The storage server responds to the setattr request.

Repeat step 1 to step 6 to traverse all files. If the dir1 directory
contains 100,000 files, repeat step 1 to step 6 for 100,000 times to
recursively set the dir1 attribute. It can be learned that the
process consumes CPU resources and memory resources of the client,
and a large number of messages are exchanged between the client and
the storage server. As a result, an end-to-end time for the
attribute setting operation is relatively long.

preamble to the figure.

Client Server
+ +
| |
|------ Readdir ----------------------->|
|<--------------------------------------|
|------ Getattr ----------------------->|
|<--------------------------------------|
|------ Setattr ----------------------->|
|<--------------------------------------|
| .... |
| |

Figure 1: Existing flowchart for recursive set operation

As you can see, this figure doodled and dawdled.

Zhang, et al. Expires 4 December 2024 [Page 3]

Internet-Draft Recursively Setting Directoriestle June 2024

2. Protocol Overview

After adopting the concept of near data calculation, the above
scenario can be optimized.

Step 1: The client identifies that the operation object of the
attribute setting is a directory and the attribute setting is
recursive, and invokes a new operation RECURSIVE_SET in compound
request, e.g,

Compound request:

SEQUENCE

PUTFH (directory filehandle)

RECURSIVE_SET

SETATTR

RECURSIVE_SET_STATUS

Step 2: The storage server recieve the compound request with
RECURSIVE_SET as one operation before SETATTR, server identify the
filehandle is a directory create a recursively set task and start
recursively queries all files in the directory, sets attributes for
each file.In case of the filehandle is a file, server can ignore the
RECURSIVE_SET operation and directly set the file attributes. Task
includes clientid state id and start time. Clientid is fetched from
the SEQUENCE and state id is the sequence number of the task related
to the client instance and initialized from zero and start time is
the time when the recursive set task created.

Step 3: The storage server response the result to client once finish
the recursive set operation. Client can decide to use
synchronization setting or asynchronous setting. If client choose
synchronization setting server must response the result directly to
the client once finished during the timeout or response the error
code NFS4ERR_PENDING with state id and verifier to the client and
client query the result periodly till the task completely finished.
If client choose asynchronous setting, server will immediately return
the error code NFS4ERR_PENDING with state id and verifier to the
client and client will start a observer task to monitor the server.
Server will send callback operation to the client once server finish
the setting task with state id and client will finish the observer
task once after recieve the callback notification from server.

Zhang, et al. Expires 4 December 2024 [Page 4]

Internet-Draft Recursively Setting Directoriestle June 2024

Compared with the original process, this process not only saves the
CPU and memory usage of the client, but also significantly reduces
the number of messages exchanged between the client and server. This
greatly improves the efficiency in recursive attribute setting
scenarios.

Similar to the design of Server-Side Copy, the new operation word
RECURSIVE_SET is used to recursively set the attributes of a
directory and its subitems. This operation can be in synchronous or
asynchronous mode, which is determined by the input parameter:

o If no back channel is created when the client and server establish
a connection, the client can only use the synchronous mode in the
RECURSIVE_SET request. If the client uses the asynchronous mode, the
server returns the error code NFS4ERR_CB_PATH_DOWN.

o If a back channel is created when the client and server establish a
connection, the RECURSIVE_SET request can be in synchronous or
asynchronous mode.

Server reboot

When server reboot, client instance in server will disappear, and
once client send request and get bad_clientid client must cancel the
previous monitor task and retry. In case of client retry and server
denied because of the permission changed client will return deny
error to application.

Client reboot

When client reboot, server will start to cancel the task after lease
expired. In case of this scenario maybe partial of the file's
attribute are successfully set.

Lease Consideration

Recursively setting task is related to the specific client instance,
so if the client instance lease expired the server should cancel the
setting task. In case of there are huge number of files need to be
set attributes server must response client with timeout that must be
lesser than lease time.

Backchannel Consideration

Before client choose asynchronous or synchronous recursively setting
client must check if backchannel exist in the session or not. If
there is no bachchannel client must choose synchronous recursively
setting otherwise client can choose synchronous or asynchronous

Zhang, et al. Expires 4 December 2024 [Page 5]

Internet-Draft Recursively Setting Directoriestle June 2024

recursively setting. If the server want to send a callback operation
over the backchannel of a session and backchannel is down, the
requester of course cannot reconnect because only the client can
associate connections with the backchannel. The server can re-send
the request over another connection that is bound to the same
session's backchannel. If there is no such connection, the server
indicate that the session has no backchannel by setting the
SEQ4_STATUS_CB_PATH_DOWN_SESSION flag bit in the response to the next
SEQUENCE operation from the client. The client then associate a
connection with the session (or destroy the session).

Grace Consideration

Recursively setting operation must hornor the server grace time.
During server grace time server should NFS4ERR_GRACE the client
request to recursively set directory attibutes and accept the request
after grace time.

Position Consideration

Recursively setting operation can not be the first operation of the
compound request and the first operation of compound in case of
client want to do recursively setting or query the status must be
SEQUENCE.

Note to RFC Editor: this section may be removed on publication as an
RFC.

3. Implementation Considerations

A recommended Recursive Set operation in an synchronous mode is shown
in Figure 2.

Step 1: The client sends a RECURSIVE_SET request. In the request,
select the synchronization setting mode and set rsa_sync to true.

Step 2: If the storage server completes the setting within the
timeout period, the storage server returns the request result to the
client. If the setting is not complete within the timeout period,
generate rsr_callback_id and rsr_recursiveverf. In addition, the
rsr_callback_id, rsr_recursiveverf, and error code are sent to the
client.

Step 3: The client sends a RECURSIVE_SET_STATUS query request. The
request contains the information of rss_stateid. The value of
rss_stateid is obtained from the response packet of RECURSIVE_SET
which value is the value rsr_callback_id. If the value of
rss_stateid is the same as the value of rsr_callback_id cached on the

Zhang, et al. Expires 4 December 2024 [Page 6]

Internet-Draft Recursively Setting Directoriestle June 2024

storage server, the storage server returns the setting result.
Storage server return NFS4_OK if the server successfully finish the
recursive setting or NFS4_Pending if the setting is till on going, or
else set failure error code. If the value of rss_stateid in the
request is different from the value cached on the server, the storage
server returns the error code NFS4ERR_BAD_STATEID.

Step 4: After receiving the RECURSIVE_SET_STATUS response, the client
delays sending the RECURSIVE_SET_STATUS request if the error code is
NFS4_PENDING. The delay setting request must contain rss_stateid.
If the error code returned by the server is NFS4_OK, the recursive
attribute setting is successful. If the error code is the error code
of setattr defined in NFSv4.2, the recursive attribute setting fails.
In this case, the client returns a response to the application.

preamble to the figure.

Client Server
+ +
| |
|------ RECURSIVE_SET(rsa_sync = 1) ------------------------> |
| |
|<-----Response(rsr_callback_id = 0, rsr_recursiveverf = 0)---| within the timeout period
| |
| |
|<----Response(rsr_callback_id = 1, rsr_recursiveverf = 1)----| beyond the timeout period
| |
| |
| |
|-------RECURSIVE_SET_STATUS(rss_stateid = 1)---------------> |
| |
|<------Response--------------------------------------------- |
| |
| |

Figure 2: A synchronous Recursive Set

As you can see, this figure doodled and dawdled.

An alternative Recursive Set operation in an synchronous mode is also
given in Figure 3.

Step 1: The client sends a RECURSIVE_SET request. In the request,
the asynchronous setting mode is rsa_sync=false.

Zhang, et al. Expires 4 December 2024 [Page 7]

Internet-Draft Recursively Setting Directoriestle June 2024

Step 2: The storage server needs to generate rsr_callback_id and
rsr_recursiveverf, and set the error code to NFS4_OK. In addition,
the rsr_callback_id, rsr_recursiveverf, and error code are sent to
the client. The storage server continues the recursive setting
operation.

Step 3: After receiving the response, the client checks the error
code and starts an asynchronous task to receive the callback message
from the server.

Step 4: The client create an asynchronous listening task and matches
rsr_callback_id and rsr_recursiveverf. If the matching succeeds, the
task is successfully executed. If rsr_callback_id can be matched but
rsr_recursiveverf cannot be matched, client skip the message.

Step 5: After recursively setting attributes, the storage server
sends a message to the client through the backchannel of the NFS4 and
notifies the client of the setting result.

Step 6: If the client does not receive the asynchronous message, the
started task is forcibly terminated when the session is destroyed.
If an error occurs when the storage server recursively sets subitem
attributes, the storage server terminates the task and returns the
error code to the client. All possible errors are subject to the
error codes defined by setattr.

preamble to the figure.

Client Server
+ +
| |
|------ RECURSIVE_SET(rsa_sync = 0) ------------------------->|
| |
|<------Response(rsr_callback_id = 1, rsr_recursiveverf = 1)--|
| |
| |
|<------CB_RECURSIVE_SET--------------------------------------|
| |
| |
| |

Figure 3: An asynchronous Recursive Set

As you can see, this figure doodled and dawdled.

Zhang, et al. Expires 4 December 2024 [Page 8]

Internet-Draft Recursively Setting Directoriestle June 2024

4. Recursive Set Operations

4.1 Operation TBD1: RECURSIVE_SET â€“ Recursively sets the attributes
of a directory and its subitems

ARGUMENT

Struct RECURSIVE_SET4args {

bool rsa_sync;

};

RESULT

struct recursive_set_response4 {

stateid4 rsr_callback_id;

verifier4 rsr_recursiversr_recursiveverf;

};

union RECURSIVE_SET4res (nfsstat4 rsr_status) {

case NFS4_OK:

recursive_set_response4 rsr_resok4;

default:

void;

};

DESCRIPTION

The RECURSIVE_SET operation is used by the client to recursively set
the attributes of a directory and all its subitems. The operation
should be placed after setattr in the existing setattr

Zhang, et al. Expires 4 December 2024 [Page 9]

Internet-Draft Recursively Setting Directoriestle June 2024

operation.After the storage server receives the setattr combination
operation, if the setattr operation does not followed by
RECURSIVE_SET, the original process remains unchanged. If the
setattr operator is followed by the RECURSIVE_SET operation, the
storage server considers the attributes of the directory and its
subitems to initiate recursive set mode.

If the storage is successfully executed, the values of
rsr_callback_id and rsr_recursiversr_recursiveverf are 0.

If the storage is not complete, the values of rsr_callback_id and
rsr_recursiversr_recursiveverf are generated.

If rsa_sync is set to true, there are two processing policies:

o According to the NFSv4 protocol, the client must wait for the
response from the server. Therefore, the client can wait for the
processing result from the server. A problem in this mode is that
the current request occupies a slot in a session, resulting in a
decrease in the number of available slots. If multiple tasks of the
same type are being executed, no slot is available on the client in
severe cases.

o The storage server determines the implementation duration. If the
implementation duration is too long, the storage server may return
non-zero values of rsr_callback_id and
rsr_recursiversr_recursiveverf.

After the client receives the request, the client waits for a period
of time and executes RECURSIVE_SET_STATUS to query the execution
progress of the current task. If the server does not finish the
execution , NFS4ERR_PENDING is returned. After receiving the error
code, the client retryes the query after a period of time. If the
execution is complete, NFS4_OK is returned.

4.2 Operation TBD2: RECURSIVE_SET_STATUS â€“ Query the result of the
recursively setting the attributes of directories and their subitems

ARGUMENT

struct RECURSIVE_SET_STATUS4args {

stateid4 rssa_stateid;

};

Zhang, et al. Expires 4 December 2024 [Page 10]

Internet-Draft Recursively Setting Directoriestle June 2024

RESULT

#define NFS4ERR_PENDING 10090

struct RECURSIVE_SET_STATUS4res {

nfsstat4 rssr_status;

};

DESCRIPTION

The RECURSIVE_SET_STATUS operation is used by the client to query the
status of a recursively set task (attributes of a directory and its
subitems). If the task on the storage server is complete, NFS4_OK is
returned. If any error occurs during task execution, a response
error code is returned and the error code is not extended or modified
in this case so the error code is the same as the error code that may
occur during the setattr operation. If the current setting task is
not complete, NFS4_PENDING is returned.

4.3 Operation TBD3: RECURSIVE_SET_CANCEL â€“ Canceling a Running Task
on the Client

ARGUMENT

//The following operation is used to cancel the recursive setting
task that is being executed.

struct RECURSIVE_SET_CANCEL4args {

stateid4 rsca_stateid;

};

RESULT

Zhang, et al. Expires 4 December 2024 [Page 11]

Internet-Draft Recursively Setting Directoriestle June 2024

struct RECURSIVE_SET_CANCEL4res {

nfsstat4 rscr_status;

};

DESCRIPTION

RECURSIVE_SET_CANCEL is used to cancel the task that is being
executed. The request packet contains rsca_stateid. The value of
rsca_stateid is obtained from the response of RECURSIVE_SET. If the
storage server fails to cancel the task, NFS4ERR_DELAY is returned.
When receiving the message, the client delays the retry. If the
current task is complete, NFS4_OK is returned.

4.4 Operation TBD4: CB_RECURSIVE_SET_NOTIFY â€“ Notify the recursively
setting result to client

ARGUMENT

struct CB_RECURSIVE_SET_NOTIFY4args {

nfs_fh4 crsna_fh;

stateid4 crsna_stateid;

verifier4 crsna_recursiveverf;

nfsstat4 crsna_status;

};

RESULT

struct CB_RECURSIVE_SET_NOTIFY4res {

nfsstat4 crsnr_status;

};

Zhang, et al. Expires 4 December 2024 [Page 12]

Internet-Draft Recursively Setting Directoriestle June 2024

DESCRIPTION

CB_RECURSIVE_SET_NOTIFY is used to send the server callback to client
to notify the client of the result of the task of recursively setting
the attributes of directories and their subitems. Client check the
crsna_stateid and crsna_recursiveverf and client will finish the wait
task if they are matched the value received from previous
RECURSIVE_SET response or will skip the notification in case of not
match and return NFS4ERR_BAD_STATEID to server.

5. Security Considerations

TBD

6. IANA Considerations

TBD

7. References

7.1. Normative References

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.

[RFC7862] Haynes, T., "Network File System (NFS) Version 4 Minor
Version 2 Protocol", RFC 7862, DOI 10.17487/RFC7862,
November 2016, <https://www.rfc-editor.org/info/rfc7862>.

7.2. Informative References

[InfRef] "", 2004.

Appendix A. An Appendix

Authors' Addresses

Minqian Zhang
Huawei Technologies
1899 Xiyuan
Chengdu
High-tech West District, 611731
China

Zhang, et al. Expires 4 December 2024 [Page 13]

Internet-Draft Recursively Setting Directoriestle June 2024

Phone: +86-13547833949
Email: zhangmingqian.zhang@huawei.com

Sunil Kumar Bhargo
VMware
Phone: +
Email: marx_bhargav@yahoo.com

Rijesh Kunhi Parambattu
Huawei Technologies
Email: rijesh.kunhi.parambattu1@huawei.com

Dongyu Geng
Huawei Technologies
Email: gengdongyu@huawei.com

Yunfei Du
Huawei Technologies
Email: duyunfei1@huawei.com

Zhang, et al. Expires 4 December 2024 [Page 14]