> ## Documentation Index
> Fetch the complete documentation index at: https://docs.scitix.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Results, Recovery, and Cleanup

> Handle inconsistent statistics, underutilized jobs, training recovery, and checks before deleting images, datasets, Volumes, and services.

This page is used to handle post-execution assessment and cleanup issues. Before analyzing statistical discrepancies, ensure that the query criteria are consistent. Before stopping, resuming, or deleting, confirm task progress and object dependencies.

## Inconsistent Numbers in the Dashboard, Resource Details, and Task Details

Different pages may use different filter scopes, time granularities, statistical units, and refresh times. First standardize the query criteria, then determine whether any anomalies exist.

1. Align the query scope across regions, clusters, project groups, resource pools, instance specifications, workload types, and other dimensions.
2. Use the same time window, confirm whether you are comparing point-in-time values, interval aggregates, or trend data, and verify the statistical units for GPU, CPU, instance counts, duration, and other metrics.
3. Refresh the relevant pages and account for time lags caused by data collection, aggregation, and cache. If the discrepancy persists, record the filter criteria, time range, values, and screenshots from each page, then submit a support request.

> [!TIP]
>
> The Resource Dashboard and Task Dashboard are suitable for viewing trends and summaries. For recently completed operations, real-time utilization, and actual runtime status, prioritize task or service details, and use events and monitoring for assessment.

For queries by resource dimension, see [View Resource Usage](/siverse/resources/view-resource-usage.md); for queries by task dimension, see [View Task Resource Usage](/siverse/resources/view-task-resource-usage.md).

## Low Utilization: Can the Task Be Stopped or Deleted Directly?

Do not judge based solely on GPU or CPU utilization at a single point in time. Low utilization may occur during data loading, communication waits, debugging, or warm-up stages.

**Determine Whether the Task Is Truly Idle**

1. Confirm the task’s purpose, owner, and current progress.
2. Compare utilization, training steps, logs, and output files over a period of time; do not rely on a single sampling point.

**Decide Whether to Stop or Delete**

1. Confirm that checkpoints, models, and other artifacts have been written to persistent storage.
2. Confirm that the task or its artifacts will not affect dependent tasks or the scaling or rollback of related services.
3. After confirming the business impact with the task owner, proceed with stopping or deleting the task.

The Task Dashboard is used only to identify tasks that may require attention. To confirm actual runtime status, go to the corresponding task details. For information on analyzing resource usage, see [View Task Resource Usage](/siverse/resources/view-task-resource-usage.md).

## How to Resume Training After a Task Interruption

Whether training can be resumed also depends on the platform recovery configuration and the checkpointing capability of the training program.

1. Confirm that the job is configured with a recovery policy and that the training program can save and load checkpoints.
2. Review the original job’s **Fault Tolerance Detail** and associated jobs to identify the current latest attempt and determine whether the platform performed an in-place restart or resubmitted the job.
3. Review the logs and training metrics to confirm that the latest attempt has loaded the checkpoint, training steps continue to progress, and outputs can be written to persistent storage.
4. When the job is manually stopped, the retry budget is exhausted, or a fast-fail policy is triggered, the job may no longer recover automatically. Address the corresponding termination cause before deciding whether to manually resubmit it.
5. Before manually resubmitting, confirm that no other jobs for the same training are still running to avoid duplicate training.

For the applicable scope, configuration, and validation methods of automatic fault tolerance, see [Recover Training Tasks with Automatic Fault Tolerance](../training/recover-training-tasks-with-auto-fault-tolerance).

## Can I delete images, datasets, Volumes, or services?

Objects that are no longer needed can be deleted, but their dependencies are independent of one another. Before deletion, assess the dependencies separately and prepare and verify replacement versions or backups.

| Object | Pre-deletion checks | Related documentation |
| - | - | - |
| Image | Check whether it is still used by development environments, training jobs, job retries, service scaling, or historical rollbacks; check whether an alternative version is available for pulling. | [Manage Custom Images](/siverse/images/manage-custom-images.md) |
| Dataset | Check whether it is still mounted by development environments or training jobs; check whether the underlying data needs to be retained or backed up. | [Mount Datasets in Development Environments and Training Jobs](/siverse/datasets/mount-datasets-in-workloads.md) |
| Volume | Check whether it is still mounted by workloads; check whether important data has been backed up or migrated; confirm the impact of deletion with users. | [Create and Use Storage Volumes](/siverse/resources/create-and-use-storage-volumes.md) |
| Service | Check whether it is still required by callers, released versions, or historical rollbacks. | [Manage General Services](/siverse/inference/manage-general-inference-services.md), [Manage LLM Inference Services](/siverse/inference/manage-llm-inference-services.md) |

> [!NOTE]
>
> A running instance being healthy does not prove that the image or storage it depends on can be safely deleted.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.