SPECstorage™ Solution 2020_eda_blended Result: Microsoft and NetApp Inc. - Azure NetApp Files large volume breakthrough mode scale

Microsoft and NetApp Inc.	SPECstorage Solution 2020_eda_blended = 17280 Job_Sets
Azure NetApp Files large volume breakthrough mode scale	Overall Response Time = 0.60 msec

Azure NetApp Files large volume breakthrough mode scale
Tested by	Microsoft and NetApp Inc.
Hardware Available	May 2024
Software Available	May 2024
Date Tested	January 2026
License Number	33
Licensee Locations	San Jose, CA USA

Azure NetApp Files is an Azure native, first-party, enterprise-class, high-performance file storage service. It provides Volumes as a service, which you can create within a NetApp account and a capacity pool, and share to clients using SMB and NFS. You can also select service and performance levels and manage data protection. You can create and manage high-performance, highly available, and scalable file shares by using the same protocols and tools that you're familiar with and rely on on-premises.

Solution Under Test Bill of Materials

Item No	Qty	Type	Vendor	Model/Name	Description
1	6	Storage Service	Microsoft	Azure NetApp Files large volume breakthrough mode	Azure NetApp Files large volumes can support from 50 TiB to 2 PiB in size. Breakthrough mode enables individual volumes to scale beyond 12.5 GiB/s. Volumes can be resized up or down on demand, and throughput can be adjusted automatically (based on volume size) or manually depending on the capacity pool QoS type.
2	108	Azure Virtual Machine	Microsoft	Standard_D16_v5	(Workload clients) Red Hat Enterprise Linux running on Azure D16s_v5 Virtual Machines (16 vCPU, 64 GB Memory, 12.5 Gbps Networking). The Dsv5-series virtual machines offer a combination of vCPUs and memory to meet the requirements associated with most enterprise workloads
3	1	Azure Virtual Machine	Microsoft	Standard_D2_v5	(Prime Client) Red Hat Enterprise Linux running on Azure D2s_v5 Virtual Machines (2 vCPU, 8 GB Memory, 12.5 Gbps Networking).

Configuration Diagrams

Component Software

Hardware Configuration and Tuning - Virtual

Software Configuration and Tuning - Virtual

Item No	Component	Type	Name and Version	Description
1	RHEL 9.5	Operating System	RHEL 9.5 (Kernel 5.14.0-503.38.1.el9_5.x86_64)	Operating System (OS) for the workload clients

Clients
Parameter Name	Value	Description
rsize,wsize	262144	NFS mount options for data block size
protocol	tcp	NFS mount options for protocol
nfsvers	3	NFS mount options for NFS version
nconnect	8	NFS mount options for multiple TCP connections
actimeo	600	NFS mount option to modify the timeouts for attribute caching
nocto	present (boolean)	NFS mount option to turn off close-to-open consistency
noatime	present (boolean)	NFS mount option to turn off access time updates
nofile	102400	Maximum number of open files per user
nproc	10240	Maximum number of processes per user
sunrpc.tcp_slot_table_entries	128	Sets the number of (TCP) RPC entries to pre-allocate for in-flight RPC requests
net.core.wmem_max	16777216	Maximum size of the socket send buffer
net.core.rmem_max	16777216	Maximum size of the socket receive buffer
net.core.wmem_default	1048576	Default setting in bytes of the socket send buffer
net.core.rmem_default	1048576	Default setting in bytes of the socket receive buffer
net.ipv4.tcp_rmem	1048576 8388608 33554432	Minimum, default and maximum size of the TCP receive buffer
net.ipv4.tcp_wmem	1048576 8388608 33554432	Minimum, default and maximum size of the TCP send buffer
net.core.optmem_max	4194304	Maximum ancillary buffer size allowed per socket
net.core.somaxconn	65535	Maximum tcp backlog an application can request
net.ipv4.tcp_mem	4096 89600 8388608	Maximum memory in 4096-byte pages across all TCP applications. Contains minimum, pressure and maximum.
net.ipv4.tcp_window_scaling	1	Enable TCP window scaling
net.ipv4.tcp_timestamps	0	Turn off timestamps to reduce performance spikes related to timestamp generation
net.ipv4.tcp_no_metrics_save	1	Prevent TCP from caching connection metrics on closing connections
net.ipv4.route.flush	1	Flush the routing cache
net.ipv4.tcp_low_latency	1	Allows TCP to make decisions to prefer lower latency instead of maximizing network throughput
net.ipv4.ip_local_port_range	1024 65000	Defines the local port range that is used by TCP and UDP traffic to choose the local port.
net.ipv4.tcp_slow_start_after_idle	0	Congestion window will not be timed out after an idle period
net.core.netdev_max_backlog	300000	Sets maximum number of packets, queued on the input side, when the interface receives packets faster than kernel can process
net.ipv4.tcp_sack	0	Disable TCP selective acknowledgements
net.ipv4.tcp_dsack	0	Disable duplicate SACKs
net.ipv4.tcp_fack	0	Disable forward acknowledgement
vm.dirty_expire_centisecs	30000	Defines when dirty data is old enough to be eligible for writeout by the kernel flusher threads. Unit is 100ths of a second.
vm.dirty_writeback_centisecs	30000	Defines a time interval between periodic wake-ups of the kernel threads responsible for writing dirty data to hard-disk. Unit is 100ths of a second.

Software Configuration and Tuning Notes

Tuned the necessary client parameters as shown above, for communication between clients and storage over Azure Virtual Networking, to optimize data transfer and minimize overhead.

Service SLA Notes

Service Level Agreement (SLA) for Azure NetApp Files

Storage and Filesystems

Item No	Description	Data Protection	Stable Storage	Qty
1	Azure NetApp Files large volume, Flexible Service Level, 50 TiB, 21504 MiB/s	Azure NetApp Files Flexible, Standard, Premium and Ultra service levels are built on a fault-tolerant bare-metal fleet powered by ONTAP, delivering enterprise-grade resilience, and uses RAID-DP (Double Parity RAID) to safeguard data against disk failures. This mechanism distributes parity across multiple disks, enabling seamless data recovery even if two disks fail simultaneously. RAID-DP has a long-standing presence in the enterprise storage industry and is recognized for its proven reliability and fault tolerance.	Stable Storage	6

Number of Filesystems	6
Total Capacity	300 TiB
Filesystem Type	Azure NetApp Files large volume breakthrough mode

Filesystem Creation Notes

Large volumes were created via the public Azure API using the azure cli tool. Creation commands are available here: https://learn.microsoft.com/en-us/cli/azure/netappfiles/volume?view=azure-cli-latest#az-netappfiles-volume-create
Creating the Azure NetApp Files Account: az netappfiles account create --account-name [account-name] --resource-group [resource-group] --location [location]
Creating the Azure NetApp Files Capacity Pool: az netappfiles pool create --account-name [account-name] --resource-group [resource-group] --location [location] --pool-name [pool-name] --service-level Flexible --size 54975581388800 --CustomThroughputMibps 21504
Creating the Azure NetApp Files Volume: az netappfiles volume create --resource-group [resource-group] --account-name [account-name] --location [location] --pool-name [pool-name] --name [volume-name] --usage-threshold 51200 --file-path [mount-point] --protocol-types NFSv3 --vnet [vnet-id] --zones 1 --throughput-mibps 21504 --breakthrough-mode true

Storage and Filesystem Notes

n/a

Transport Configuration - Virtual

Item No	Transport Type	Number of Ports Used	Notes
1	12.5 Gbps Virtual NIC	108	Each Workload Linux Virtual Machine has a single Network Adapter with Accelerated Networking enabled.
2	12.5 Gbps Virtual NIC	1	The Prime client has a single Network Adapter with Accelerated Networking enabled.

Transport Configuration Notes

Azure Virtual machines allocated bandwidth limits egress (outbound) traffic from the virtual machines. The virtual machines ingress bandwidth rates may exceed 12.5 Gbps depending on other resources available to the virtual machine (https://learn.microsoft.com/en-us/azure/virtual-network/virtual-machine-network-throughput)

Switches - Virtual

Item No	Switch Name	Switch Type	Total Port Count	Used Port Count	Notes
1	Azure Virtual Network - Brazil South	Virtual Network	24	24	Each Azure virtual network had 6 connection for the Azure NetApp Files storage endpoints and 18 (1 per) RHEL client. Azure virtual networks allow up to 65,536 Network interface cards and Private IP addresses per virtual network. This Azure VNet was Peered to the VNet in Canada Central.
2	Azure Virtual Network - Southeast Asia	Virtual Network	24	24	This Azure VNet was Peered to the VNet in Canada Central.
3	Azure Virtual Network - West Europe	Virtual Network	24	24	This Azure VNet was Peered to the VNet in Canada Central.
4	Azure Virtual Network - West US	Virtual Network	24	24	This Azure VNet was Peered to the VNet in Canada Central.
5	Azure Virtual Network - West US3	Virtual Network	24	24	This Azure VNet was Peered to the VNet in Canada Central.
6	Azure Virtual Network - Germany West Central	Virtual Network	24	24	This Azure VNet was Peered to the VNet in Canada Central.
7	Azure Virtual Network - Canada Central	Virtual Network	1	1	This Azure VNet was Peered to all other VNets to allow Prime client communication between itself and the workload clients

Processing Elements - Virtual

Memory - Virtual

Stable Storage

Solution Under Test Configuration Notes

Item No	Qty	Type	Location	Description	Processing Function
1	1728	vCPU	Azure Cloud	Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz (16 cores allocated to each Workload VM)	Client Workload Generator
2	2	vCPU	Azure Cloud	Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz (2 cores allocated to Prime VM)	Prime Client

Description	Size in GiB	Number of Instances	Nonvolatile	Total GiB
Client Workload Generator	64	108	V	6912
Prime Client	8	1	V	8
Grand Total Memory Gibibytes	6920

Clients each accessed one storage endpoint (They were round robin assigned to one of the six endpoints) for the Azure NetApp Files large volume within the same availability zone/virtual network.
Unlike a general-purpose operating system, Azure NetApp Files does not provide mechanisms for customers to run third-party code (https://learn.microsoft.com/en-us/security/benchmark/azure/baselines/azure-netapp-files-security-baseline?toc=/azure/azure-netapp-files/TOC.json#security-profile). Azure Resource Manager allows only an allow-listed set of operations to be executed via the Azure APIs (https://learn.microsoft.com/en-us/azure/azure-netapp-files/control-plane-security).
Underlying Azure infrastructure was patched for Spectre/Meltdown on or prior to January 2018. (https://azure.microsoft.com/en-us/blog/securing-azure-customers-from-cpu-vulnerability/ and https://learn.microsoft.com/en-us/azure/virtual-machines/mitigate-se).

Other Solution Notes

Dataflow

108 clients were used to generate the workload with 1 client in Canada Central was the Prime client. There is 18 workload clients for each of the 6 different regions as follows:

Brazil South: 18 Workload Generators -> 1 large volume breakthrough mode
Southeast Asia: 18 Workload Generators -> 1 large volume breakthrough mode
West Europe: 18 Workload Generators -> 1 large volume breakthrough mode
West US: 18 Workload Generators -> 1 large volume breakthrough mode
West US3: 18 Workload Generators -> 1 large volume breakthrough mode
Germany West Central: 18 Workload Generators -> 1 large volume breakthrough mode
Canada Central: Prime VM <-> Workload Generators (Orchestration Traffic)

The only cross-region traffic is between the Prime client in Canada Central and the workload clients in the other 6 regions for workload orchestration.
All data/storage traffic between clients and storage was contained within each region/availability zone for the test. The Prime client does not participate in the data path.

Each workload client used one 12.5 Gbps virtual network adapter, through a single vnet connected to one Azure NetApp Files endpoint. Each client mounted one ANF large volume as an NFSv3 filesystem.

With Azure NetApp Files breakthrough mode, each large volume is given six dedicated, logical storage endpoints distributed across bare-metal infrastructure. Additional endpoints provide separate access paths to the same filesystem/namespace, leveraging additional network interfaces. This architecture allows the large volume to scale performance by utilizing networking resources from multiple controllers, rather than being limited by the constraints of a single endpoint/controller.
The clients were round robin assigned to one of the six endpoints (3 clients per endpoint, per region).
Round robin assignment was handled client-side by statically assigning clients to endpoints within the same region.

The Prime client communicated with all workload clients outside of the virtual network it was connected to using virtual network peering.

Other Notes

Each of the six filesystems are independent; they are not a single distributed filesystem across regions.

There is 1 mount per client each to /mnt/eda. Each region's clients statically mounted a filesystem locally within the region to the same local path. Example mount commands from one server in each region are shown below. /etc/fstab entry:
[Southeast Asia] 10.254.171.4:/southeast-asia-vol /mnt/eda nfs hard,proto=tcp,vers=3,rsize=262144,wsize=262144,nconnect=8,nocto,noatime,actimeo=600 0 0

[Brazil South] 10.254.161.4:/south-brazil-vol /mnt/eda nfs hard,proto=tcp,vers=3,rsize=262144,wsize=262144,nconnect=8,nocto,noatime,actimeo=600 0 0

[West Europe] 10.254.151.4:/west-europe-vol /mnt/eda nfs hard,proto=tcp,vers=3,rsize=262144,wsize=262144,nconnect=8,nocto,noatime,actimeo=600 0 0

[West US] 10.254.141.4:/west-us-vol /mnt/eda nfs hard,proto=tcp,vers=3,rsize=262144,wsize=262144,nconnect=8,nocto,noatime,actimeo=600 0 0

[West US3] 10.254.131.4:/west-us3-vol /mnt/eda nfs hard,proto=tcp,vers=3,rsize=262144,wsize=262144,nconnect=8,nocto,noatime,actimeo=600 0 0

[Germany West Central] 10.254.121.4:/germany-west-central-vol /mnt/eda nfs hard,proto=tcp,vers=3,rsize=262144,wsize=262144,nconnect=8,nocto,noatime,actimeo=600 0 0

Other Report Notes

SPECstorage™ Solution 2020_eda_blended Result

Performance

Product and Test Information