# File Processing & Sync ## Overview File Processing & Sync provides background processing and synchronization capabilities for data files within the Nimble portal. This system tracks the lifecycle of file operations through distinct processing stages and manages background synchronization jobs that move data between environments. ## File Processing ### Processing Stages When a data file is uploaded through the Data Collection system, it moves through a series of processing stages. Each stage is tracked with a history record that captures the outcome and any messages. **File Upload** — The initial stage where the file is received and validated. The system checks file format, headers, and minimum row requirements before accepting the upload. **File Staging** — After successful upload, the file data is staged into intermediate storage. This stage handles data transformation and prepares records for loading into the data warehouse. **Cube Processing** — The final stage where staged data is processed into the reporting cube. Once complete, the data becomes available in reports and dashboards. ### Process History Every processing operation creates a history record that captures: * **File Request** — Which file upload triggered the processing * **Stage** — Which processing stage the record represents * **Status** — Whether the stage succeeded or encountered an error * **Messages** — Detailed information about the processing outcome * **Timestamp** — When the processing occurred This audit trail allows administrators to trace the complete lifecycle of any file from upload through final processing. ## File Synchronization ### Background Sync Jobs File synchronization runs as background jobs that transfer data files between environments (for example, from a staging environment to production). Each sync job tracks its overall status and the results for individual files. ### Job Statuses Sync jobs progress through the following statuses: * **Queued** — The job has been created and is waiting to be picked up by the background processor * **In Progress** — The job is actively running and processing files * **Completed** — All files in the job have been successfully synchronized * **Failed** — The job encountered an error that prevented completion ### File-Level Tracking Within each sync job, individual files are tracked with their own status: * **In Progress** — The file is currently being synchronized * **Completed** — The file has been successfully synchronized * **Failed** — The file encountered an error during synchronization When a job fails, the exception details are captured for troubleshooting. File-level results are stored as structured data within the job record, allowing administrators to identify exactly which files succeeded and which encountered problems. ## Related Features * [**Data Collection — File Upload**](/guides/data-collection-file-upload.md) — File processing is triggered by uploads from the data collection system * [**Migrate Configuration**](/guides/migrate-configuration.md) — Configuration migration also uses background sync jobs for transferring data between environments --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://nimble.docs.otised.com/guides/file-processing-sync.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.