# Data Collection — File Upload

The File Upload feature lets agencies submit data files to fulfill collection requests from administrators. Administrators create file requests for specific agencies and collection terms; agency users then upload CSV files against those requests. The system validates each file, stores it securely, and stages the data for downstream processing.

## Table of Contents

* [Overview](#overview)
* [Key Concepts](#key-concepts)
* [Uploading a File (Agency Users)](#uploading-a-file-agency-users)
* [Working with File Requests (Administrators)](#working-with-file-requests-administrators)
* [Filtering the Upload List](#filtering-the-upload-list)
* [Previewing an Uploaded File](#previewing-an-uploaded-file)
* [Downloading an Uploaded File](#downloading-an-uploaded-file)
* [File Processing History](#file-processing-history)
* [File Type Configuration](#file-type-configuration)
* [File Request Statuses and Stages](#file-request-statuses-and-stages)
* [Permissions](#permissions)
* [Troubleshooting](#troubleshooting)

***

## Overview

Data collection through file upload works as a two-sided workflow:

1. An administrator creates a **file request** that tells a specific agency which file type to submit and for which collection term.
2. An agency user sees that request in the **Upload Agency Files** list and uploads the required CSV file.
3. The system validates the file against the configured rules for that file type, stores the file, stages the data into the data lake, and records the outcome in the request's processing history.

***

## Key Concepts

**File Request** — A record linking an agency, a file type, and a collection term. It tracks the current status of the submission and all historical processing events.

**File Type** — A named definition that describes what kind of data a file should contain. File types carry optional validation rules (required column headers, minimum row counts) and belong to a category and group for organizational purposes.

**File Type Category** — A broad classification used to group related file types (for example, "Student Data" or "Finance").

**File Type Group** — A secondary classification that further organizes file types within a category.

**Term** — The academic year or reporting period to which a file request belongs. The upload action is only available for the current active term; requests for past terms are read-only.

**Processing Stage** — Each upload moves through up to three stages: File Upload, File Staging, and Cube Processing. Each stage can succeed or fail independently, and each outcome is recorded in the processing history.

***

## Uploading a File (Agency Users)

### Before you begin

* You must belong to the organization unit that corresponds to the agency on the file request. If you do not have the necessary organization membership, the row shows a ban icon and the upload action is unavailable.
* Uploads are only permitted for the current active collection term. Requests for prior terms display without an upload action.
* Files must be in CSV format.

### Steps

1. Navigate to the **Upload Agency Files** page from the Data Collection section of the portal.
2. Use the filters at the top of the page to select the academic year and, optionally, narrow by agency or status. The list updates to show matching file requests.
3. Find the file request you want to fulfill. The row shows the file type name, your organization name and code, the current status, and any previous upload details.
4. Click the **Upload** icon (arrow pointing upward) in the Actions column for that row.
5. The **File Upload** dialog opens. It confirms the organization name, file type, and — if applicable — the file type category and group.
6. Select your file using one of two methods:
   * Click **Choose file to upload** to open a file browser and select a CSV file from your computer.
   * Drag and drop a CSV file directly onto the dashed drop zone. The selected file name and size are displayed below the drop zone.
7. Click **Upload File**. A spinner appears while the upload is in progress. Do not close the dialog during this time.
8. When the upload completes, the dialog displays a result summary:
   * **Success** — Shows the file name, the number of data rows counted, the upload date and time, and the user who uploaded it.
   * **Error** — Shows the file name and an error message describing what went wrong.
9. Click **Close** to dismiss the dialog. The file request list refreshes to reflect the updated status.

***

## Working with File Requests (Administrators)

Administrators can create, update, and delete file requests. These actions require the appropriate permissions described in the [Permissions](#permissions) section.

### Creating a file request

1. Navigate to the file requests management page.
2. Click the button to create a new request.
3. Fill in the required fields:
   * **Name** — A descriptive label for the request.
   * **Agency** — The agency that must fulfill this request.
   * **File Type** — The type of file the agency should submit.
   * **Term** — The academic year or reporting period.
   * **Description** — Optional additional context for the agency.
4. Save the request. It becomes visible to users in the selected agency immediately.

### Editing a file request

Open an existing request and update any of its fields. Changes take effect immediately and are visible to agency users.

### Deleting a file request

Deleting a file request removes it from the list for both administrators and agency users. This action cannot be undone. Processing history associated with the request is also removed.

***

## Filtering the Upload List

The filters panel at the top-right of the Upload Agency Files page includes:

* **Academic Year** — Narrows the list to requests for a specific term. Defaults to the current active year.
* **Agency** — Limits results to a specific agency. Select "ALL" to view requests across all agencies you have access to.
* **Status** — Filters by the last known status of each request (Success, Error, or no filter to see all).

Filter selections are preserved between sessions so the page reloads with your last-used filter values.

***

## Previewing an Uploaded File

After a file has been successfully uploaded, a **Preview** icon appears in the Actions column for that row.

1. Click the **Preview** icon.
2. The **File Preview** dialog opens and displays up to the first 50 rows of the file in a table.
3. If the file contains more than 50 rows, a message at the top of the preview indicates the total row count.
4. If the preview encounters parse errors in the file content, they are listed at the top of the dialog.
5. Click **Cancel** to close the preview.

The preview shows the data exactly as it was uploaded. It does not modify the file or the request status.

***

## Downloading an Uploaded File

After a file has been successfully uploaded, a **Download** icon appears in the Actions column for that row.

1. Click the **Download** icon.
2. The browser immediately downloads the CSV file to your default downloads folder.

Downloaded files are the original files as submitted by the agency.

***

## File Processing History

For any file request that has processing history, a **History** icon appears in the Actions column.

1. Click the **History** icon.
2. The **File Process History** dialog opens with a table of processing events for that request.

Each row in the history table shows:

| Column         | Description                                                                                       |
| -------------- | ------------------------------------------------------------------------------------------------- |
| Stage          | The processing stage at which the event occurred (File Upload, File Staging, or Cube Processing). |
| Process Status | Whether the stage succeeded or failed.                                                            |
| Messages       | The system message recorded at that stage, including success confirmations and error details.     |
| Date Created   | The date and time the event was recorded.                                                         |

3. Click **Cancel** to close the history dialog.

The history is cumulative — every upload attempt, success, and failure is recorded, giving a complete audit trail for each request.

***

## File Type Configuration

File types define the validation rules applied to every upload for that type. This configuration is managed by system administrators.

### File type fields

| Field             | Description                                                           |
| ----------------- | --------------------------------------------------------------------- |
| Code              | A short identifier used in system processes and file naming.          |
| Name              | The display name shown to agency users in the upload list and dialog. |
| Description       | Optional notes about the file type's purpose or content requirements. |
| Category          | Groups the file type by broad subject area (managed separately).      |
| Group             | Further organizes file types within a category (managed separately).  |
| Custom Properties | JSON-formatted validation rules applied during upload (see below).    |
| Target Table      | The database table into which the staged data is loaded.              |

### Validation rules via custom properties

Each file type can carry custom validation rules that are enforced at upload time:

* **Minimum row count** — The file must contain at least this many data rows (excluding the header row if one is required). If the file has fewer rows, the upload is rejected with an error message stating the minimum required.
* **Required headers** — A comma-separated list of exact column names that must appear in the file's first row, in the specified order. If the header row does not match — whether due to different names, extra columns, or missing columns — the upload is rejected.

### Managing categories and groups

Categories and groups are standalone reference lists managed in their own administration pages. Each has a name and an optional description. File types reference a category and group by selection; changing a category or group name updates it everywhere it is referenced.

***

## File Request Statuses and Stages

### Statuses

| Status  | Meaning                                                                |
| ------- | ---------------------------------------------------------------------- |
| Success | The most recent upload completed all processing stages without errors. |
| Error   | The most recent upload encountered a problem at one or more stages.    |
| (blank) | No file has been uploaded for this request yet.                        |

### Processing stages

Uploads proceed through stages in sequence. A failure at any stage stops processing and records an Error status on the request.

| Stage           | What happens                                                                                |
| --------------- | ------------------------------------------------------------------------------------------- |
| File Upload     | The file is validated (format, headers, row count) and transferred to secure cloud storage. |
| File Staging    | The uploaded file is read from storage and bulk-inserted into the data lake staging table.  |
| Cube Processing | Downstream processing of the staged data (runs outside the upload workflow).                |

When an upload fails, the Last Stage column in the list identifies which stage encountered the problem, and the Last Message column shows the specific error. Full details are available in the [processing history](#file-processing-history).

***

## Permissions

Access to file upload features is controlled by the following permission groups:

| Permission                              | Who needs it                    | What it enables                                                |
| --------------------------------------- | ------------------------------- | -------------------------------------------------------------- |
| DcFileRequests — Default                | Agency users and administrators | View the file upload list and download/preview uploaded files. |
| DcFileRequests — Create                 | Administrators                  | Create new file requests.                                      |
| DcFileRequests — Edit                   | Administrators                  | Update existing file requests.                                 |
| DcFileRequests — Delete                 | Administrators                  | Delete file requests.                                          |
| DcFileTypes — Default                   | Administrators                  | View file type definitions.                                    |
| DcFileTypes — Create                    | Administrators                  | Create new file types.                                         |
| DcFileTypes — Edit                      | Administrators                  | Update file type definitions.                                  |
| DcFileTypes — Delete                    | Administrators                  | Delete file types.                                             |
| DcFileTypeCategories / DcFileTypeGroups | Administrators                  | Manage category and group reference data.                      |

In addition to role permissions, agency users must be members of the organization unit linked to the agency. Without organization membership, the upload action is not available for that agency's requests.

***

## Troubleshooting

### The Upload icon does not appear for a row

* Confirm you are viewing the current active academic year. Uploads are only enabled for the current term; switch the year filter to the current year if needed.
* Check that you are a member of the organization unit for that agency. If you see a ban icon instead of the upload icon, you do not have organization-level access. Contact your system administrator to have your organization membership reviewed.

### The upload fails with "File header does not match configuration"

The file type requires specific column headers in a specific order. Open your file in a spreadsheet application and verify that:

* The first row contains column names exactly as expected (capitalization is ignored).
* There are no extra columns before, between, or after the required columns.
* There are no leading or trailing spaces in the header names.

Contact your administrator to confirm the exact expected headers for this file type.

### The upload fails with "File must contain at least N rows"

The file type requires a minimum number of data rows. If your file has fewer rows than required, it cannot be accepted. Verify that the file is complete and contains all expected data before retrying.

### The upload fails with "Please select a file to upload"

No file was selected before clicking Upload File. Use the Choose file button or drag a file into the drop zone, then click Upload File again.

### The upload appears stuck (spinner does not go away)

Large files may take longer to upload and stage. Wait for the dialog to display a result. Do not close the browser tab during an upload in progress. If the dialog remains in a loading state for an extended period, refresh the page and check the file request list. If the status shows Error, open the processing history for details and contact support with the message ID if provided.

### The upload shows "Error processing data, please contact support and provide Message ID"

An unexpected system error occurred during staging. Note the Message ID displayed in the error message and contact your system administrator or support team. The message ID allows the support team to locate the detailed error in the system logs.

### Download or Preview icons do not appear

These actions are only available after a successful upload (status shows Success). If the last upload ended in an Error status, resolve the upload issue first and re-upload the file successfully.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://nimble.docs.otised.com/guides/data-collection-file-upload.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.