> ## Documentation Index > Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt > Use this file to discover all available pages before exploring further. # Best practices for ephemeral storage > Learn what consumes Pod ephemeral storage, how to check usage and configure alerts, and how to clean up or redirect writes to avoid workload eviction. Development environments, training jobs, and services in Siverse run as Pods. This guide explains how to monitor, alert on, clean up, and relocate Pod ephemeral storage data to reduce the risk of eviction when storage fills up. > [!WARNING] > > If ephemeral storage fills up, the system may evict the Pod. Training may fail, a development environment may be interrupted, or a service instance may become unhealthy. Write data that must be retained, reused, or allowed to grow to a mounted Volume. ## What is ephemeral storage In Kubernetes, Pod ephemeral storage is node-local space for data generated while a Pod runs, including files written under paths such as `/tmp`, `/root`, and `/usr`. Its lifecycle follows the Pod: when the Pod is deleted, recreated, moved, or evicted, its associated ephemeral storage is released. Data written to ephemeral storage is not persistent. Do not keep datasets, experiment results, checkpoints, model files, business logs, or long-lived caches solely on local container paths if you need them after the workload ends. Common sources of ephemeral storage usage include: | Source | Description | | - | - | | Container file system | Files created, downloaded, or generated in the container, including data written to `/tmp`, `/root`, `/usr`, or `$HOME`. | | Container logs | Program output written to standard output or standard error, such as `stdout` and `stderr` logs. | | Temporary directory volumes | Empty directories mounted for the current run, such as `emptyDir`. | Ephemeral storage limits can vary by cluster, instance specification, and workload type. Check the usage, quota, utilization, and status displayed on the workload details page to assess the available space for your workload. ## Check current usage Development environments, training jobs, LLM inference services, and general services all show Pod ephemeral storage on their details pages. Open the **Storage** tab for the relevant object and check the amount used, quota, utilization, and status. For example, the training job details page shows storage information: ![Storage tab on a training job details page](https://mintcdn.com/scitix-adfc65c2/ZqwdmfntBUDyGPkJ/siverse/best-practices/media/training-task-storage.png?fit=max\&auto=format\&n=ZqwdmfntBUDyGPkJ\&q=85\&s=04f5b8bbb31cf47bfab127711f0ea830) If ephemeral storage usage keeps rising, check whether your program writes logs, caches, checkpoints, intermediate files, or outputs to a local container path. ## Configure disk alerts For long-running training jobs, development environments, or services, configure disk alerts so that you receive notifications when Pod ephemeral storage or node disk usage rises. 1. Under **Alarm > Notification Management**, prepare a notification target or contact group and make sure it is enabled. See [Manage Alert Notification Targets](/siverse/alerts/manage-alert-notification-targets.md). Without an enabled notification recipient, the platform cannot deliver a triggered alert. 2. Under **Alarm > Alarm Subscription**, create a subscription. See [Create an Alarm Subscription](/siverse/alerts/create-alert-subscriptions.md). When creating the subscription, expand the **Volume** category. To receive both critical and warning alerts for Pod ephemeral storage and node disk utilization, select these alert items: * `pod_disk_watermark-critical` * `pod_disk_watermark-warning` * `node_disk_watermark-critical` * `node_disk_watermark-warning` After creating the subscription, review alert records and delivery results in the alarm overview. Alert notifications generally include a direct link to the affected workload for troubleshooting. ## Clean up high-usage paths If the console shows high ephemeral storage usage, first stop writing more data to local paths. Decide which data to keep and move it to a mounted Volume. Delete only data that you have confirmed can be discarded. Common paths to inspect include: | Path | Typical contents | Recommended action | | - | - | - | | `$HOME/.cache` | Downloads cached by `pip`, Hugging Face, `torch`, `uv`, and similar tools. | These caches can consume substantial space. After confirming that no current process needs them, remove caches you no longer need. | | `$HOME/.local` | Packages installed with `pip install --user`. | Remove packages only after confirming they are no longer required. | | Other directories under `$HOME` | Files downloaded or uploaded by users, or intermediate and output files generated by scripts. | Move files you need to retain, then remove files that can be regenerated. | | Conda `pkgs` directory | Cached Conda packages. | Use Conda's cache-cleaning mechanism to free space. | | `/tmp`, `/var/tmp` | Runtime temporary files. | Check the contents and modification times before deleting files you have confirmed can be discarded. | | Custom log paths | Logs written by application scripts, such as `~/xxx.log`. | Limit log size or write logs to a Volume or logging system. | If the Pod has already been evicted, data in ephemeral storage may no longer be recoverable. Before rerunning the workload, change its write paths to avoid the same problem. ## Write caches and outputs to a Volume To avoid filling ephemeral storage, direct growing data and data you need to retain or reuse to a mounted Volume instead of a local container directory. Examples include dependency and model caches, training checkpoints, job outputs, business logs, and large downloads. Mount paths typically look like `/volume/`. Set cache directories explicitly in a startup command, job script, or `~/.bashrc`. Replace the example path components with your actual mounted path and workspace directory before running these commands: | Environment variable | Suggested use | | - | - | | `PIP_CACHE_DIR` | `pip` download cache. | | `HF_HOME` | Hugging Face cache. | | `TORCH_HOME` | `torch hub` weights and caches. | | `CONDA_PKGS_DIRS` | Conda package cache. | | `TMPDIR` | Program temporary files. | For example: ```bash theme={null} # Replace both path components with values for your mounted Volume. VOLUME_ROOT=/volume/REPLACE_WITH_RESOURCE_POOL/REPLACE_WITH_WORKSPACE export PIP_CACHE_DIR="$VOLUME_ROOT/cache/pip" export HF_HOME="$VOLUME_ROOT/cache/huggingface" export TORCH_HOME="$VOLUME_ROOT/cache/torch" export CONDA_PKGS_DIRS="$VOLUME_ROOT/cache/conda-pkgs" export TMPDIR="$VOLUME_ROOT/tmp" ``` Use the same approach for other tools. If a development or agent tool writes configuration and caches to `$HOME` by default, first check whether it supports an environment variable for changing the directory. For example, Claude Code normally writes configuration and caches under `$HOME`. To direct future writes to a mounted Volume, set `CLAUDE_CONFIG_DIR`: ```bash theme={null} # Replace both path components with values for your mounted Volume. export CLAUDE_CONFIG_DIR=/volume/REPLACE_WITH_RESOURCE_POOL/REPLACE_WITH_WORKSPACE/claude ``` When configuring these paths: 1. Use absolute paths, not relative paths. 2. Set environment variables before starting the tool, for example in `~/.bashrc` or the startup command. Putting a path only in the tool's configuration file usually does not redirect its writes. 3. Restart the tool or process after changing the variables. 4. Claude Code does not support `--config-dir` for changing its configuration directory; use `CLAUDE_CONFIG_DIR`. 5. Environment variables generally affect only future writes. They do not automatically move existing files, the program itself, project files, or other tools' caches. ## Check whether full ephemeral storage caused eviction If a training job fails, a development environment stops, or a service instance becomes unhealthy, check **Event** on its details page. Eviction, disk pressure, or insufficient ephemeral storage events may indicate a Pod storage or node disk issue. Check the following information together: | Information | What to check | | - | - | | **Storage** | Pod ephemeral storage usage, quota, utilization, and status. | | **Event** | Eviction, scheduling failures, and disk pressure events. | | **Log** | Whether the program keeps writing large files, caches, logs, or outputs. | | **Job or service configuration** | Whether output and cache paths and Volume mount paths are set as intended. | After confirming that ephemeral storage is involved, change the write path or remove disposable data before restarting the development environment, resubmitting the training job, or redeploying the service. ## FAQ > **Why does disk space reported in the container differ from the console?** > > Running `df -h /` in a container usually reports disk space from the node's or container's perspective, not the current Pod's ephemeral storage quota. To determine whether a Pod is near its limit, check usage, quota, and status for **Pod Ephemeral Storage** on the console's **Storage** tab. > **Which files should I clean up first?** > > Use paths, timestamps, and console usage trends to prioritize cleanup. New files under `/tmp`, `/root`, `/usr`, or `$HOME` may raise ephemeral storage usage; image-bundled files are usually lower priority. Before deleting a file, check whether a process still uses it or it needs moving to a Volume. > **Can I recover data after ephemeral storage fills up?** > > Not always. Eviction, recreation, or migration may release a Pod's ephemeral storage. Write code, datasets, checkpoints, models, experiment results, caches, and long-lived logs to a Volume while it runs if you need to retain, reuse, or grow them. Do not wait for a Pod failure to try to recover them. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.