Survey Data Management and Schema Definition

Overview

Surveys in OCTAVIUS represent collections of astronomical data organized as dynamic tables rendered in the user interface. Each survey is associated with an organization and requires a schema definition that specifies the data structure, field types, and auxiliary resources (such as images) for comprehensive data visualization.

Survey Definition

A survey comprises the following components:

  • Name: Human-readable display name (e.g., “ComCam Survey”, “Legacy Survey”)
  • Collection Code: URL slug and database identifier for the survey data page (e.g., legacy-survey)
  • Image Sources (Optional): External image templates for rendering thumbnail or full-resolution images alongside data records

Workflow: Creating and Configuring a Survey

Step 1: Prepare Dataset

Format your astronomical data as a comma-separated values (CSV) file. The file should include:

  • Column headers that define field names
  • Data rows with values corresponding to each field
  • Consistent data types within each column

Example CSV structure:

_ID,RA,DEC,FLUX,SURVEY
XCS_J001_1,10.5,20.3,1234.5,LegacySurvey
XCS_J001_2,10.6,20.4,2345.6,LegacySurvey
XCS_J001_3,10.7,20.5,3456.7,LegacySurvey

Step 2: Define Schema and Upload Data

  1. Navigate to Admin → Upload (route: /upload)
  2. Enter a survey name (e.g., “Legacy Survey”)
  3. Upload your prepared CSV file
  4. The system automatically detects and suggests column data types:
    • String: Text values
    • Number: Numeric values (integers and floating-point)
    • Boolean: Binary true/false values
    • Array: Comma-separated or JSON array values
  5. Review and adjust type assignments as necessary

Step 3: Configure Image Resources (Optional)

Image resources enable visualization of auxiliary data (cutouts, thumbnails, finding charts) alongside tabular data:

  1. In the schema upload interface, click Add Image Source
  2. Specify the image resource name (e.g., “LegacySurvey”, “DES_Cutout”)

  3. Define the image URL template using the following syntax:

    https://example.com/imaging/${_ID}_${size}-${ext}.png

    Template Variables:

    • ${datum.FIELD} — Substitutes the value of any CSV column (e.g., ${datum.RA}, ${datum._ID})
    • ${size} — Image dimensions requested (width/height in pixels)
    • ${ext} — Image file extension (e.g., “png”, “fits”, “jpg”)
  4. Complete the configuration and proceed to schema save

Example Image Template:

https://legacysurvey.org/viewer/cutout?ra=${RA}&dec=${DEC}&pixscale=0.27&size=${size}

Step 4: Persist Schema and Enable Visualization

  1. Click Save Schema
  2. The system creates a PageSchema document in MongoDB and initializes the corresponding data collection
  3. Survey data becomes accessible through the web interface

Step 5: Associate Survey with Organization

  1. Navigate to Admin → Organizations
  2. Create a new organization or select an existing organization for modification
  3. In the organization configuration, select the survey from the list of available surveys
  4. Save the organization configuration
  5. The survey is now visible to all organization members on the contents page (/contents)

PageSchema Data Model

When a schema is created and persisted, the system generates a MongoDB document conforming to the following structure:

{
  "page": "Legacy",
  "fields": [
    { "name": "_ID", "type": "String" },
    { "name": "RA", "type": "Number" },
    { "name": "DEC", "type": "Number" },
    { "name": "FLUX", "type": "Number" },
    { "name": "SURVEY", "type": "String" }
  ],
  "Images": [
    { "LegacySurvey": "https://example.com/${_ID}_${size}-${ext}.png" }
  ],
  "virtuals": [
    {
      "name": "classifications",
      "ref": "classification",
      "localField": "_id",
      "foreignField": "surveyTarget"
    }
  ],
  "options": {
    "timestamps": true,
    "toObject": { "getters": true, "virtuals": true }
  }
}

Schema Components:

  • page — Survey identifier (must be unique)
  • fields — Column definitions with type information
  • Images — Image source template definitions for auxiliary data visualization
  • virtuals — MongoDB virtual references linking survey records to associated classifications
  • options — Database options including timestamp generation and object transformation settings

Data Visualization Workflow

User Workflow

  1. User authenticates and navigates to the home page (/contents)
  2. The system displays organizations and their associated surveys as interactive cards
  3. User selects a survey, which navigates to the survey-specific data page
  4. The DynamicPage component renders survey data as an interactive table:
    • Column headers correspond to schema field definitions
    • Each row represents a survey target or astronomical object
    • Optional image columns display auxiliary resources when configured
  5. Users classify objects by interacting with table rows and submitting classifications to the backend

Data Rendering Pipeline

  1. Schema Retrieval: The frontend retrieves the PageSchema document to determine column definitions
  2. Data Fetching: Survey data is fetched from the corresponding MongoDB collection
  3. Column Rendering: The table component dynamically constructs columns based on field definitions in the schema
  4. URL Templating: Image URLs are constructed by substituting placeholders with values from the current data row
  5. Display: The table renders with configured columns plus any image columns, enabling direct visual inspection of data

API Endpoints for Data Management

The following endpoints provide programmatic access to survey and schema functionality:

Method Endpoint Purpose
GET /api/v1/surveys Retrieve all surveys in the system
POST /api/v1/schemas/create Create and persist a new survey schema
GET /api/v1/organisations Retrieve all organizations with associated surveys
GET /api/v1/organisations/:id Retrieve specific organization configuration
POST /api/v1/organisations Create a new organization
PUT /api/v1/organisations/:id Update organization configuration
GET /api/v1/:surveyId/data Retrieve survey data for a specific survey
POST /api/v1/classifications Submit new classification records

Additional endpoints for user management, authentication, and administrative functions are documented in the Architecture section.

Image Visualization and Rendering

URL Template Processing

The system processes image URLs at render time, enabling dynamic image selection based on current data row and client-specified parameters:

  1. Data Substitution: Template variables like ${RA} are replaced with values from the current table row
  2. Size Parameter: The ${size} variable accommodates responsive image sizing (e.g., 128, 256, 512 pixels)
  3. Format Parameter: The ${ext} variable allows clients to request different image formats (PNG, FITS, JPEG)
  4. URL Assembly: The complete URL is constructed and rendered as an image element

Image Interaction

Users interact with images through:

  • Thumbnails: Small images displayed inline within the table for quick reference
  • Full-Resolution Viewer: Clicking an image opens a separate viewer or external application for detailed examination
  • Multiple Image Sources: A single record may have multiple associated images (e.g., optical and xray data)

Best Practices and Recommendations

  1. Data Validation: Ensure CSV data is consistent and free of malformed entries before upload
  2. Column Naming: Use descriptive, standardized column names (e.g., RA, DEC for astronomical coordinates)
  3. Image Template Design: Test image templates with sample data to verify URL construction is correct
  4. Organization Structure: Design organization hierarchies to reflect institutional or survey-team structures
  5. Schema Documentation: Document domain-specific field meanings and expected value ranges for end users

Tips

  • CSV Naming: Use consistent, descriptive column names (e.g., _ID, RA, DEC)
  • Schema Names: Use the same page name as the survey name for clarity
  • Image URLs: Test templates with sample data before saving
  • Types: If a column looks like a number but is actually a code, explicitly format it as String in the CSV (e.g., '001' instead of 001)
This site uses Just the Docs, a documentation theme for Jekyll.