> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# GPFS Overview
GPFS is a shared file storage service for high-performance computing and AI workloads. You can mount a GPFS file system to CES instances or CKS cluster nodes to store and share datasets, model weights, training outputs, and log files.
This document helps you decide whether to use GPFS and understand the storage types, capacity metering, and usage limits that you need to consider before creating a file system.
## Use cases
GPFS is suitable for scenarios that require shared data access through file system semantics, such as:
* Multiple CES instances or CKS cluster nodes need to access the same data.
* Training jobs need to read datasets and write model weights, checkpoints, or training outputs.
* Inference, batch processing, or data processing jobs need to share input files and output results.
If your workloads mainly read and write data through object APIs, OSS may be a better fit.
## Storage types
GPFS provides two storage types: Performance and Capacity.
| Storage type | Performance | Capacity |
| --------------------------------- | -------------------------------------------- | --------------------------------------------- |
| Read throughput | 500 MB/s/TiB | 10 MB/s/TiB |
| Write throughput | 300 MB/s/TiB | 5 MB/s/TiB |
| Maximum bandwidth per file system | 500 GB/s | 20 GB/s |
| Minimum capacity | 11.5 TiB | 40 TiB |
| Maximum capacity | 2 PiB | 100 PiB |
| Scaling increment | 11.5 TiB | 40 TiB |
| Number of inodes | `min( * 5 million/TiB, 1 billion)` | `min( * 3 million/TiB, 10 billion)` |
If a Performance file system requires more than 500 GB/s of bandwidth, or if you need to increase the number of inodes, contact support for evaluation.
## Storage capacity and billing
GPFS file systems support two billing methods:
* Pay-as-you-go: billed based on the billing duration and provisioned capacity.
* Subscription: prepaid for a fixed period.
The capacities in the GPFS storage type table are usable capacities. GPFS file systems may use replicas, erasure coding, or other mechanisms to protect data at the underlying layer. Because the underlying configuration may vary, the capacity and usage that you see on compute nodes by running `df` or `du` may not exactly match the usable capacity and usage displayed in the console. For example, in CES and CKS scenarios, the file system may use a replica configuration. If you create an 11.5 TiB file system, `df` may show approximately 23 TiB. After you write a 1 GiB file, `du` may show that the file occupies 2 GiB.
## Quotas and limits
GPFS file systems use thick provisioning. You specify the capacity when you create a file system. Later scale-up or scale-down operations adjust the target capacity of the file system.
The minimum capacity and scaling increment depend on the storage type. The same storage type may vary across regions and storage clusters.
| Limit | Description |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Number of file systems | A tenant can create up to 3 file systems in a region. |
| Number of client clusters | A tenant can create up to 5 client clusters in a region. |
| Mounts on a single node | A CES instance or CKS cluster node can mount multiple file systems that belong to the same tenant. It can also mount subdirectories in a file system. |
| Number of mounted nodes per file system | We recommend that you mount a file system to no more than 256 CES instances or CKS cluster nodes. If you need to exceed this number, contact support for evaluation. |
| Inode limit | Performance: `min( * 5 million/TiB, 1 billion)`. Capacity: `min( * 3 million/TiB, 10 billion)`. To increase the limit, contact support. |
| Maximum capacity | The maximum capacity of a file system depends on the storage type and regional inventory. The maximum capacity is 2 PiB for Performance and 100 PiB for Capacity. If a single file system needs to exceed 200 TiB, contact support in advance to confirm inventory. |
## Performance recommendations
For stable performance, we recommend that you keep file system usage below 95%.
This usage includes:
* Usable space usage.
* Inode usage.
If file system usage is too high, I/O performance may decrease.
## Data protection
GPFS protects data at the underlying layer by using replicas, erasure coding, or other mechanisms. To further improve data security, you still need to back up important data by yourself.
After you delete a file system or delete data from a file system, the data cannot be recovered. Before you delete data or a file system, make sure that the data you want to keep has been backed up or migrated.
## Runtime environment restrictions
The following restrictions apply to the runtime environment of CES instances or CKS cluster nodes after mounting. If you need to modify these configurations, contact support first.
| Type | Limit |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Hostname | After CES instances or CKS cluster nodes join an XStor cluster, do not modify hostnames or configure duplicate hostnames, including host entries in `/etc/hosts`. |
| Number of XStor master nodes | CES instances or CKS cluster nodes must have at least 3 XStor master nodes. If there are fewer than 3 master nodes, the cluster cannot handle split-brain scenarios such as network partitions, and XStor may not work properly. |
| XStor master nodes | Do not restart XStor master nodes at the same time. |
| XStor components | Do not delete, modify, or stop XStor binaries, libraries, configuration files, or services. |
| Assistant | To use GPFS mount, unmount, and client management capabilities, keep Assistant components running properly. Do not delete, modify, or stop related components. |
| GPFS components | Do not delete, modify, or stop GPFS binaries, libraries, configuration files, or services. |
| OFED components | Do not delete, modify, or stop OFED binaries, libraries, configuration files, or services. |
| Operating system and kernel | Do not upgrade or reinstall the operating system or kernel version on CES instances or CKS cluster nodes. |
| System disk usage | Keep the system disk usage of CES instances or CKS cluster nodes below 80%. |
| Root SSH configuration | You can change the login passwords of CES instances or CKS cluster nodes, but do not modify the root SSH configuration from XStor master nodes to other nodes, including `/root/.ssh/authorized_keys` and `/root/.ssh/known_hosts`. |
## Port requirements
Keep the following ports and protocols used by GPFS available:
| Type | Port or protocol |
| --------------------------- | -------------------------------------- |
| Protocol | ICMP |
| Management and access ports | 22, 1191, 4739, 8889, 9085, 9980, 9981 |
| Dynamic port range | 60000\~61000 |
If the network port of an IP address used by GPFS changes, for example from `eth0` to `br0`, restart the GPFS client on the node.
## API rate limits
If you manage GPFS file systems by using APIs, follow the API rate limits.
Storage management APIs:
| Feature | API | Rate limit |
| ----------------------- | ------------------------------- | --------------- |
| Create a file system | `/api/fscm/filesystem` | 1 per 60 sec |
| Query file systems | `/api/fscm/filesystem/describe` | 1000 per 60 sec |
| Create a client cluster | `/api/fscm/cluster/client` | 1 per 60 sec |
| Delete a client cluster | `/api/fscm/cluster/client` | 1 per 60 sec |
| Delete a file system | `/api/fscm/filesystem` | 10 per 60 sec |
CES management service APIs:
| Feature | API | Rate limit |
| --------------------- | -------------------------- | -------------- |
| Mount a file system | `/api/v1/xstor/mnt/mount` | 100 per 60 sec |
| Unmount a file system | `/api/v1/xstor/mnt/umount` | 100 per 60 sec |