Data Flows Overview

This section documents how data moves through the Sartiq platform, from initial product upload through generation to final delivery. Understanding these flows is essential for developers working on integrations and debugging issues.

---

Flow Summary

flowchart LR
    subgraph Ingestion["1. Ingestion"]
        Upload[Product Upload]
    end

    subgraph Generation["2. Generation"]
        Config[Configuration]
        AI[AI Processing]
    end

    subgraph Delivery["3. Delivery"]
        Review[Review]
        Export[Export]
    end

    Upload --> Config
    Config --> AI
    AI --> Review
    Review --> Export

The platform has three primary data flows:

Flow	Description	Documentation
Product Ingestion	Upload → Validation → Storage → Catalog	Product onboarding
Generation Pipeline	Config → Compute → AI → Result	Image creation
Delivery & Export	Review → Approve → Export	Final delivery
MediaResource Lifecycle	Ingestion → Dedup → Canonical Path → Resilient Cleanup	Media management

---

End-to-End Flow

This diagram shows the complete journey from brand product to delivered image.

sequenceDiagram
    participant Brand
    participant Webapp
    participant Backend
    participant Compute
    participant AI
    participant Storage
    participant External

    Note over Brand,External: Phase 1: Product Ingestion
    Brand->>Webapp: Upload products
    Webapp->>Backend: POST /products
    Backend->>Storage: Save images
    Backend-->>Webapp: Products ready

    Note over Brand,External: Phase 2: Configuration
    Brand->>Webapp: Create shooting
    Webapp->>Backend: POST /shootings
    Brand->>Webapp: Configure looks
    Webapp->>Backend: POST /shooting-looks

    Note over Brand,External: Phase 3: Generation
    Brand->>Webapp: Start generation
    Webapp->>Backend: POST /generations
    Backend->>Compute: POST /tasks
    Compute-->>Backend: Task ID
    Compute->>AI: Generate images
    AI-->>Compute: Results
    Compute->>Storage: Save images
    Compute-->>Backend: Redis event
    Backend->>Compute: GET /tasks/{id}/result
    Backend-->>Webapp: WebSocket
    Webapp-->>Brand: View results

    Note over Brand,External: Phase 4: Delivery
    Brand->>Webapp: Approve images
    Webapp->>Backend: PUT /shots/{id}/approve
    Brand->>Webapp: Export
    Webapp->>Backend: POST /exports
    Backend->>External: Deliver images

---

Data Types

Content

Content flows through the system in several forms:

Type	Format	Storage Path	CDN
All new media	WebP, PNG, JPG	media/{resource_id}/file.{ext} (canonical)	Yes
Product Images	WebP, PNG, JPG	images/products/{product_id}/ (legacy)	Yes
Subject References	PNG, WebP	images/subjects/ (legacy)	Internal
Generated Content	WebP	images/generations/{generation_id}/ (legacy)	Yes
Style References	WebP, PNG	images/styles/{style_id}/ (legacy)	Yes
Compute Results	WebP	compute/{task_type}s/{task_id}/	Transient

Metadata

Metadata accompanies images through the pipeline:

Stage	Metadata
Upload	Filename, MIME type, dimensions
Product	SKU, category, brand, attributes
Generation	Prompt, seed, model, timing
Prediction	Quality score, confidence

---

Storage Architecture

flowchart TB
    subgraph Input["Input Storage"]
        Products[images/products/]
        Subjects[images/subjects/]
        Styles[images/styles/]
    end

    subgraph Staging["Staging"]
        Temp[temp/]
        Compute[compute/]
    end

    subgraph Output["Output Storage"]
        Generations[images/generations/]
        Looks[shooting_looks/]
    end

    subgraph CDN["CDN"]
        Public[Public URLs]
    end

    Products --> Compute
    Subjects --> Compute
    Temp -->|relocate| Products
    Compute -->|relocate| Generations
    Generations --> Public
    Products --> Public

Path Conventions

Content is stored in object storage (see Infrastructure for provider details):

{R2_BUCKET_NAME}/
├── media/{resource_id}/                           # Canonical (all new files)
│   └── file.{ext}
├── temp/                                          # Presigned upload staging (24h TTL)
│   └── {file_id}_{timestamp}_{safe_filename}
├── images/                                        # LEGACY: pre-existing data only
│   ├── products/{product_id}/
│   ├── subjects/
│   ├── generations/{generation_id}/
│   ├── styles/{style_id}/
│   └── shots/{shot_id}/revisions/
├── shooting_looks/{look_id}/                      # LEGACY: pre-existing data only
└── compute/{task_type}s/{task_id}/                # Compute server workspace
    └── result_{index}.webp

---

Event Flow

Events propagate through the system to enable real-time updates.

flowchart LR
    subgraph Sources["Event Sources"]
        Backend[Backend]
        Compute[Compute]
    end

    subgraph Bus["Event Bus"]
        Redis[(Redis Event Stream)]
    end

    subgraph Consumers["Event Consumers"]
        WS[WebSocket Manager]
        Workers[Background Workers]
        Analytics[Analytics]
    end

    Backend -->|publish| Redis
    Compute -->|emit| Redis
    Redis --> Backend
    Redis --> WS
    Redis --> Workers
    Redis --> Analytics

Event Types

Event	Source	Consumers
product.created	Backend	Analytics
shooting.started	Backend	WebSocket
generation.started	Backend	WebSocket
task.progress	Compute	Backend → WebSocket
task.completed	Compute	Backend → WebSocket, Workers
task.failed	Compute	Backend → WebSocket, Workers
export.completed	Backend	WebSocket

---

Error Handling

Each flow has specific error handling patterns.

Ingestion Errors

Error	Handling
Invalid file format	Reject with error message
File too large	Reject with size limit info
Duplicate SKU	Warn and offer merge

Generation Errors

Error	Handling
Provider timeout	Retry with backoff
Provider error	Fallback to alternate provider
Invalid configuration	Fail fast with validation error

Export Errors

Error	Handling
External system unavailable	Queue for retry
Authentication failure	Alert and pause
Partial failure	Report partial success

---

Performance Considerations

Bottlenecks

Stage	Bottleneck	Mitigation
Upload	Network bandwidth	Chunked uploads, compression
Storage	Storage latency	Parallel uploads, CDN
Generation	AI provider capacity	Queue management, priority
Export	External system limits	Rate limiting, batching

Monitoring Points

Key metrics to monitor across flows:

• Ingestion: Upload success rate, processing time
• Generation: Queue depth, completion rate, latency
• Delivery: Export success rate, delivery latency

---

Detailed Flows

Continue to the detailed documentation for each flow:

1. Product Ingestion - How products enter the system
2. Generation Pipeline - How images are created
3. Delivery & Export - How images reach their destination

---

Related Documentation

• Components Overview - System components
• Infrastructure - Storage and networking details
• API Reference - Endpoint documentation