squad/implementation.md

# Implementation Architecture

## Executive Summary
Software Development Insight & Release Management Platform with modular microservices architecture, containerized deployment, and modern web frontend.

## System Architecture Overview

```mermaid
graph TB
    subgraph "Frontend Layer"
        UI[React TypeScript Frontend]
        UI --> SG[SignalR Client]
    end
    
    subgraph "API Gateway"
        GW[API Gateway Service<br/>Authentication & Routing]
        HUB[SignalR Hub<br/>Real-time Updates]
    end
    
    subgraph "Core Services"
        PS[Project Service<br/>CC.NET Integration]
        MS[Manuscript Service<br/>FogBugz Integration]
        DS[Document Service<br/>PDF Generation]
        SS[Storage Service<br/>Cloud Abstraction]
        HS[Help Center Service<br/>Article Updates]
        PUB[Publishing Service<br/>Workflow Orchestration]
    end
    
    subgraph "Background Processing"
        HF[Hangfire<br/>Background Jobs]
        HFD[Hangfire Dashboard]
    end
    
    subgraph "Data Layer"
        DB[(PostgreSQL<br/>Database)]
    end
    
    subgraph "External Integrations"
        CC[CruiseControl.NET]
        FB[FogBugz/Manuscript]
        S3[AWS S3]
        BOX[Box.com]
        FTP[FTP Server]
        ZD[Zendesk]
        SF[Salesforce Service]
    end
    
    UI --> GW
    SG --> HUB
    GW --> PS
    GW --> MS
    GW --> DS
    GW --> SS
    GW --> HS
    GW --> PUB
    
    PS --> CC
    MS --> FB
    SS --> S3
    SS --> BOX
    SS --> FTP
    HS --> ZD
    HS --> SF
    
    PUB --> HF
    HF --> DS
    HF --> SS
    HF --> HS
    
    PS --> DB
    MS --> DB
    DS --> DB
    SS --> DB
    HS --> DB
    PUB --> DB
    HF --> DB
    
    classDef frontend fill:#e1f5fe
    classDef gateway fill:#fff3e0
    classDef service fill:#f3e5f5
    classDef background fill:#e8f5e8
    classDef database fill:#fce4ec
    classDef external fill:#f5f5f5
    
    class UI,SG frontend
    class GW,HUB gateway
    class PS,MS,DS,SS,HS,PUB service
    class HF,HFD background
    class DB database
    class CC,FB,S3,BOX,FTP,ZD,SF external
```

## Technology Stack

### Backend Services (.NET 8)
- **API Gateway**: ASP.NET Core Web API with SignalR hubs
- **Database**: PostgreSQL with Entity Framework Core
- **Background Jobs**: Hangfire for document generation and publishing workflows
- **Logging**: Serilog with structured logging
- **Documentation**: OpenAPI/Swagger

### Frontend (React/TypeScript)
- **Framework**: React 18+ with TypeScript
- **State Management**: React Query + Context API
- **UI Library**: Material-UI or Ant Design
- **Real-time**: SignalR client for live updates
- **Build**: Vite for fast development

### Infrastructure
- **Containerization**: Docker Compose for development, Docker for production
- **Reverse Proxy**: Nginx for production
- **Service Discovery**: Docker network with service names

## Service Architecture

### Core Services

#### 1. API Gateway Service (`api-gateway`)
- **Purpose**: Single entry point, authentication, SignalR hub
- **Technology**: ASP.NET Core Web API + SignalR
- **Responsibilities**:
  - User authentication/authorization (JWT)
  - Route requests to appropriate services
  - Real-time notifications via SignalR
  - CORS configuration for React frontend

#### 2. Project Service (`project-service`)
- **Purpose**: CruiseControl.NET integration and project management
- **Technology**: ASP.NET Core Web API
- **Integration Method**: TCP Remoting (`tcp://{server}:21234/CruiseManager.rem`)
- **Responsibilities**:
  - Connect to CC.NET via ThoughtWorks.CruiseControl.Remote library
  - Parse XML build logs and extract commit information
  - Regex pattern matching for FogBugz case ID extraction
  - Cache build history and status with modified file tracking
  - Project configuration management
  - Build status monitoring and real-time updates

#### 3. Manuscript Service (`manuscript-service`)
- **Purpose**: FogBugz/Manuscript integration
- **Technology**: ASP.NET Core Web API
- **Integration Method**: HTTP API (`http://{server}/api.asp`)
- **Authentication**: Token-based authentication system
- **Responsibilities**:
  - Case/bug retrieval with full event history
  - Status tracking (Opened, Resolved, Closed, Reactivated)
  - Release note extraction from case events
  - Zendesk ticket number linking
  - Event history processing and caching
  - Developer comment extraction and aggregation

#### 4. Document Service (`document-service`)
- **Purpose**: PDF generation and document management
- **Technology**: ASP.NET Core + QuestPDF (modern, container-friendly)
- **Responsibilities**:
  - Release note PDF generation with FogBugz data integration
  - Template variable replacement ({{VERSION}}, {{DATE}}, {{PROJECT}})
  - Document versioning and storage
  - Multi-format document support (PDF primary)
  - Template management and customization

#### 5. Storage Service (`storage-service`)
- **Purpose**: Cloud storage abstraction layer
- **Technology**: ASP.NET Core Web API
- **Storage Providers**:
  - **AWS S3**: AWS SDK for .NET with CloudFront CDN integration
  - **Box.com**: Box .NET SDK
  - **FTP**: FluentFTP library
- **Pattern**: Strategy pattern for provider abstraction
- **Key Features**:
  - File compression and decompression (ZIP support)
  - Folder cleanup and old version management
  - Upload progress tracking with real-time updates
  - CDN integration for fast content delivery
  - Multi-file batch upload operations

#### 6. Help Center Service (`helpcenter-service`)
- **Purpose**: Support article updates with template processing
- **Technology**: ASP.NET Core Web API
- **Integrations**:
  - **Zendesk**: Zendesk API v2 with ZendeskApi_v2 library
  - **Salesforce Service**: Salesforce REST API
- **Pattern**: Strategy pattern for provider abstraction
- **Key Features**:
  - Template variable replacement ({{VERSION}}, {{SWURL}}, {{PDFURL}})
  - Multi-locale support (default: en-us)
  - Article versioning and rollback capabilities
  - Automated article publishing workflows
  - Link generation for software downloads and documentation

#### 7. Publishing Service (`publishing-service`)
- **Purpose**: Orchestrate complete publishing workflows
- **Technology**: ASP.NET Core + Hangfire
- **Responsibilities**:
  - Multi-step workflow orchestration with progress tracking
  - Coordinate document generation with FogBugz data
  - Manage cloud uploads with compression and cleanup
  - Update help center articles with template processing
  - Workflow status tracking and real-time notifications
  - Error handling and retry mechanisms
  - Package configuration validation
  - Publishing history and audit trail

### Database Design

```mermaid
erDiagram
    Users {
        int Id PK
        string Username
        string PasswordHash
        string Role
        datetime CreatedAt
        datetime LastLogin
    }
    
    Packages {
        int Id PK
        string Title
        string Version
        string Description
        int ProjectId FK
        int SourceBuildId FK
        string Status
        datetime PublishDate
        datetime CreatedAt
        datetime UpdatedAt
    }
    
    PackageConfigurations {
        int Id PK
        int PackageId FK
        string BuildFolder
        boolean ZipContents
        boolean DeleteOldPublishedBuilds
        string ReleaseNoteTemplate
        json StorageSettings
        json HelpCenterSettings
        datetime CreatedAt
        datetime UpdatedAt
    }
    
    Projects {
        int Id PK
        string Name
        string Description
        string CCNetProjectName
        string Status
        datetime CreatedAt
        datetime UpdatedAt
    }
    
    Builds {
        int Id PK
        int ProjectId FK
        string BuildNumber
        string Status
        datetime StartTime
        datetime EndTime
        string LogPath
        string ArtifactPath
    }
    
    BuildCommits {
        int Id PK
        int BuildId FK
        string CommitHash
        string Comment
        string User
        datetime CommitDate
        string FogBugzCaseId
        json ModifiedFiles
        string ReleaseNote
    }
    
    FogBugzCases {
        int Id PK
        int CaseId
        string Title
        string Project
        string Area
        string Status
        datetime LastUpdated
        boolean IsOpen
    }
    
    FogBugzEvents {
        int Id PK
        int CaseId FK
        string EventType
        string User
        string Comment
        string StatusString
        datetime EventDate
        string ReleaseNote
        int ZendeskNumber
    }
    
    Publications {
        int Id PK
        int PackageId FK
        string Status
        datetime PublishedAt
        string ReleaseNotesPath
        json PublicationDetails
    }
    
    PublishingSteps {
        int Id PK
        int PublicationId FK
        string StepName
        string Status
        string Details
        datetime StartedAt
        datetime CompletedAt
        int ProgressPercent
    }
    
    StorageProviders {
        int Id PK
        string Name
        string Type
        json Configuration
        boolean IsActive
    }
    
    HelpCenterProviders {
        int Id PK
        string Name
        string Type
        json Configuration
        boolean IsActive
    }
    
    Projects ||--o{ Builds : "produces builds"
    Projects ||--o{ Packages : "user creates packages for"
    Builds ||--o{ Packages : "can be referenced by"
    Builds ||--o{ BuildCommits : "contains commits"
    Packages ||--o{ Publications : "has publications"
    Packages ||--o{ PackageConfigurations : "has configuration"
    Publications ||--o{ PublishingSteps : "tracked by steps"
    BuildCommits ||--o| FogBugzCases : "may reference"
    FogBugzCases ||--o{ FogBugzEvents : "has event history"
```

#### Core Entities
```sql
-- Packages: User-defined software release configurations
Packages (Id, Title, Version, Description, ProjectId, SourceBuildId, Status, PublishDate, CreatedAt, UpdatedAt)

-- Package-specific configurations
PackageConfigurations (Id, PackageId, BuildFolder, ZipContents, DeleteOldPublishedBuilds, ReleaseNoteTemplate, StorageSettings, HelpCenterSettings, CreatedAt, UpdatedAt)

-- Projects from CruiseControl.NET
Projects (Id, Name, Description, CCNetProjectName, Status, CreatedAt, UpdatedAt)

-- Build information
Builds (Id, ProjectId, BuildNumber, Status, StartTime, EndTime, LogPath, ArtifactPath)

-- Individual commit information from build logs
BuildCommits (Id, BuildId, CommitHash, Comment, User, CommitDate, FogBugzCaseId, ModifiedFiles, ReleaseNote)

-- FogBugz case information
FogBugzCases (Id, CaseId, Title, Project, Area, Status, LastUpdated, IsOpen)

-- FogBugz case event history
FogBugzEvents (Id, CaseId, EventType, User, Comment, StatusString, EventDate, ReleaseNote, ZendeskNumber)

-- Publishing history
Publications (Id, PackageId, Status, PublishedAt, ReleaseNotesPath, PublicationDetails)

-- Publishing workflow step tracking
PublishingSteps (Id, PublicationId, StepName, Status, Details, StartedAt, CompletedAt, ProgressPercent)

-- User management
Users (Id, Username, PasswordHash, Role, CreatedAt, LastLogin)

-- Provider configurations
StorageProviders (Id, Name, Type, Configuration, IsActive)
HelpCenterProviders (Id, Name, Type, Configuration, IsActive)
```

## Interface Abstractions

### Cloud Storage Interface
```csharp
public interface ICloudStorageProvider
{
    Task<string> UploadAsync(string containerName, string fileName, Stream content, IProgress<UploadProgress> progress = null);
    Task<string> UploadFileAsync(string containerName, string localFilePath, IProgress<UploadProgress> progress = null);
    Task<bool> DeleteAsync(string containerName, string fileName);
    Task<Stream> DownloadAsync(string containerName, string fileName);
    Task<bool> DeleteFolderContentsAsync(string containerName, string folderPath, List<string> excludeFiles);
    Task<List<string>> UploadFolderAsync(string containerName, string localFolderPath, bool zipContents, IProgress<UploadProgress> progress = null);
    Task<string> GetPublicUrlAsync(string containerName, string fileName);
}

public class UploadProgress
{
    public string FileName { get; set; }
    public long BytesTransferred { get; set; }
    public long TotalBytes { get; set; }
    public int PercentComplete { get; set; }
}
```

### Help Center Interface
```csharp
public interface IHelpCenterProvider
{
    Task<bool> UpdateArticleAsync(string articleId, string content, string title = null);
    Task<bool> UpdateArticleWithTemplateAsync(string articleId, string templateContent, Dictionary<string, string> variables);
    Task<string> CreateArticleAsync(string content, string title, string categoryId);
    Task<bool> PublishArticleAsync(string articleId);
    Task<ArticleInfo> GetArticleAsync(string articleId);
    Task<bool> TestConnectionAsync();
}

public class ArticleInfo
{
    public string Id { get; set; }
    public string Title { get; set; }
    public string Content { get; set; }
    public string Locale { get; set; }
    public DateTime LastUpdated { get; set; }
    public bool IsPublished { get; set; }
}
```

### CruiseControl.NET Integration Interface
```csharp
public interface ICCNetClient
{
    Task<List<ProjectStatus>> GetProjectStatusAsync();
    Task<List<string>> GetBuildNamesAsync(string projectName);
    Task<string> GetBuildLogAsync(string projectName, string buildName);
    Task<bool> TestConnectionAsync();
}

public class ProjectStatus
{
    public string Name { get; set; }
    public IntegrationStatus BuildStatus { get; set; }
    public string LastSuccessfulBuildLabel { get; set; }
    public DateTime LastBuildDate { get; set; }
}

public class BuildCommitInfo
{
    public string ProjectName { get; set; }
    public string BuildNumber { get; set; }
    public string Comment { get; set; }
    public string User { get; set; }
    public DateTime Date { get; set; }
    public string FogBugzCaseId { get; set; }
    public List<string> ModifiedFiles { get; set; }
}
```

### FogBugz Integration Interface
```csharp
public interface IManuscriptClient
{
    Task<bool> LogOnAsync();
    Task<FogBugzCase> GetCaseAsync(int caseId);
    Task<List<FogBugzEvent>> GetCaseEventsAsync(int caseId);
    Task<string> GetReleaseNoteAsync(int caseId);
    Task<FogBugzStatus> GetCaseStatusAsync(int caseId);
    Task<bool> TestConnectionAsync();
}

public class FogBugzCase
{
    public int Id { get; set; }
    public string Title { get; set; }
    public string Project { get; set; }
    public string Area { get; set; }
    public string Status { get; set; }
    public DateTime LastUpdated { get; set; }
    public bool IsOpen { get; set; }
    public List<FogBugzEvent> Events { get; set; }
}

public class FogBugzEvent
{
    public string User { get; set; }
    public string Comment { get; set; }
    public string StatusString { get; set; }
    public DateTime Timestamp { get; set; }
    public string ReleaseNote { get; set; }
    public int ZendeskNumber { get; set; }
    public FogBugzStatus Status { get; set; }
}

public enum FogBugzStatus
{
    Unknown,
    Opened,
    Resolved,
    Closed,
    Reactivated
}
```

## Development Environment

### Docker Compose Structure
```yaml
services:
  postgres:
    image: postgres:15
    
  api-gateway:
    build: ./src/ApiGateway
    ports: ["5000:80"]
    
  project-service:
    build: ./src/ProjectService
    
  manuscript-service:
    build: ./src/ManuscriptService
    
  document-service:
    build: ./src/DocumentService
    
  storage-service:
    build: ./src/StorageService
    
  helpcenter-service:
    build: ./src/HelpCenterService
    
  publishing-service:
    build: ./src/PublishingService
    
  react-frontend:
    build: ./src/Frontend
    ports: ["3000:80"]
    
  hangfire-dashboard:
    ports: ["5001:80"]
```

### Project Structure
```mermaid
graph TD
    subgraph "Solution Structure"
        ROOT[Solution Root]
        
        subgraph "Backend Services"
            API[ApiGateway/]
            PROJ[ProjectService/]
            MAN[ManuscriptService/]
            DOC[DocumentService/]
            STOR[StorageService/]
            HELP[HelpCenterService/]
            PUB[PublishingService/]
        end
        
        subgraph "Shared Components"
            SHARED[Shared/]
            DB[Database/]
        end
        
        subgraph "Frontend"
            UI[Frontend/]
        end
        
        subgraph "Infrastructure"
            DOCKER[docker-compose.yml]
            ENV[.env files]
        end
    end
    
    ROOT --> API
    ROOT --> PROJ
    ROOT --> MAN
    ROOT --> DOC
    ROOT --> STOR
    ROOT --> HELP
    ROOT --> PUB
    ROOT --> SHARED
    ROOT --> DB
    ROOT --> UI
    ROOT --> DOCKER
    ROOT --> ENV
    
    classDef service fill:#e3f2fd
    classDef shared fill:#f3e5f5
    classDef frontend fill:#e8f5e8
    classDef infra fill:#fff3e0
    
    class API,PROJ,MAN,DOC,STOR,HELP,PUB service
    class SHARED,DB shared
    class UI frontend
    class DOCKER,ENV infra
```

```
src/
├── ApiGateway/          # Main API + SignalR hub
├── ProjectService/      # CC.NET integration
├── ManuscriptService/   # FogBugz integration  
├── DocumentService/     # PDF generation
├── StorageService/      # Cloud storage abstraction
├── HelpCenterService/   # Article update service
├── PublishingService/   # Workflow orchestration
├── Shared/              # Common models, interfaces
├── Frontend/            # React TypeScript app
└── Database/            # EF migrations, seed data
```

## Key Architectural Decisions

### 1. Microservices with Service Mesh
- **Rationale**: Clean separation of concerns, independent scaling
- **Communication**: HTTP REST + SignalR for real-time updates
- **Service Discovery**: Docker network DNS resolution

### 2. Hangfire for Background Processing
- **Benefits**: .NET native, web dashboard, persistent queues
- **Use Cases**: Document generation, cloud uploads, batch processing
- **Storage**: PostgreSQL for job persistence

### 3. Strategy Pattern for External Integrations
- **Cloud Storage**: Pluggable providers (S3, Box, FTP)
- **Help Centers**: Extensible article update system
- **Benefits**: Easy to add new providers without core changes

### 4. SignalR for Real-time Updates
- **Use Cases**: Build status, publishing progress, job completion
- **Frontend Integration**: React SignalR client with auto-reconnection

### 5. Entity Framework with Repository Pattern
- **Database**: Code-first migrations
- **Testing**: In-memory provider for unit tests
- **Performance**: Query optimization with projection

## Security Considerations

### Authentication & Authorization
- JWT tokens with refresh mechanism
- Role-based access control (Admin, User, ReadOnly)
- Secure password hashing (bcrypt)

### API Security
- HTTPS only in production
- CORS properly configured
- API rate limiting
- Input validation and sanitization

### Secrets Management
- Docker secrets for production
- Environment variables for configuration
- No hardcoded credentials

## Deployment Strategy

### Development
- Docker Compose with hot-reload volumes
- Shared PostgreSQL instance
- Frontend dev server proxy to API

### Production
- Individual Docker containers
- Nginx reverse proxy
- PostgreSQL with persistent volumes
- Health checks and restart policies

## External Service Integration Details

### CruiseControl.NET Integration
- **Connection Method**: TCP Remoting (`tcp://{server}:21234/CruiseManager.rem`)
- **Library**: ThoughtWorks.CruiseControl.Remote
- **Log Format**: XML with embedded modification details
- **FogBugz Case Extraction**: Regex pattern matching on commit comments
- **Supported Patterns**: 
  - Comments starting with case number
  - "fb{number}" or "FB{number}" patterns
  - "asp?{number}" patterns  
  - Full URL patterns with "cases/{number}"

### FogBugz/Manuscript Integration
- **Connection Method**: HTTP API (`http://{server}/api.asp`)
- **Authentication**: Token-based with session management
- **Event Processing**: Full case history with status tracking
- **Release Note Extraction**: From "ReleaseNoted" events with `<IGNORE>` filtering
- **Status Mapping**: Opened, Resolved, Closed, Reactivated
- **Zendesk Linking**: Custom field parsing for ticket numbers

### AWS S3 Integration
- **CDN**: CloudFront integration for fast content delivery
- **Features**: Folder cleanup, progress tracking, public URL generation
- **Compression**: ZIP file creation with SharpSevenZip
- **Cleanup**: Automatic deletion of old versions based on configuration

### Zendesk Integration
- **API Version**: Zendesk API v2 with ZendeskApi_v2 library
- **Template Variables**: `{{VERSION}}`, `{{SWURL}}`, `{{PDFURL}}`
- **URL Generation**: Automatic CloudFront URL generation for downloads
- **Locale Support**: Multi-locale article updates (default: en-us)

## Publishing Workflow

```mermaid
sequenceDiagram
    participant User
    participant UI as React Frontend
    participant GW as API Gateway
    participant PUB as Publishing Service
    participant PROJ as Project Service
    participant MAN as Manuscript Service
    participant DOC as Document Service
    participant STOR as Storage Service
    participant HELP as Help Center Service
    participant HF as Hangfire
    participant SH as SignalR Hub
    participant DB as Database
    
    Note over User,UI: Package Management Phase
    User->>UI: Create/Edit Package Configuration
    UI->>GW: POST /api/packages
    Note over UI: Package List View, Package Details Form
    UI->>GW: GET /api/packages (List View)
    UI->>GW: GET /api/packages/{id} (Details)
    
    Note over User,UI: Publishing Phase
    User->>UI: Select Package & Build Version
    UI->>GW: POST /api/publish/package/{id}/build/{buildId}
    GW->>PUB: Queue Publishing Job
    PUB->>HF: Enqueue Multi-Step Background Job
    PUB->>SH: Notify Job Started
    SH->>UI: Show Publishing Progress UI
    
    Note over HF: Step 1: Validate Package Configuration
    HF->>DB: Create Publication Record
    HF->>PUB: Validate Package Settings
    PUB->>SH: Progress Update (10%)
    
    Note over HF: Step 2: Collect Build Data
    HF->>PROJ: Get Build Commits
    PROJ->>DB: Query BuildCommits for Build
    PUB->>SH: Progress Update (20%)
    
    Note over HF: Step 3: Query FogBugz for Release Notes
    HF->>MAN: Get Case Details for Commits
    MAN->>DB: Cache FogBugz Events
    PUB->>SH: Progress Update (30%)
    
    Note over HF: Step 4: Generate Release Notes PDF
    HF->>DOC: Generate PDF with FogBugz Data
    DOC->>DB: Store Document Path
    PUB->>SH: Progress Update (50%)
    
    Note over HF: Step 5: Compress Files (if configured)
    HF->>PUB: Compress Build Artifacts
    PUB->>SH: Progress Update (60%)
    
    Note over HF: Step 6: Upload to Cloud Storage
    HF->>STOR: Upload Package + PDF
    STOR->>PUB: Upload Progress Events
    PUB->>SH: Progress Update (75%)
    
    Note over HF: Step 7: Clean Up Old Versions
    HF->>STOR: Delete Old Published Builds
    PUB->>SH: Progress Update (80%)
    
    Note over HF: Step 8: Update Help Center Articles
    HF->>HELP: Update with Template Variables
    HELP->>PUB: Article Update Complete
    PUB->>SH: Progress Update (90%)
    
    Note over HF: Step 9: Update Package Status
    HF->>DB: Update Package PublishDate
    HF->>DB: Complete Publication Record
    
    Note over HF: Step 10: Send Completion Notification
    HF->>SH: Notify Complete (100%)
    SH->>UI: Show Success with Download Links
    UI->>User: Display Publication Summary
```

## Data Flow Architecture

```mermaid
flowchart TD
    subgraph "External Sources"
        CC[CruiseControl.NET<br/>Build Logs]
        FB[FogBugz/Manuscript<br/>Developer Data]
    end
    
    subgraph "Data Ingestion"
        PS[Project Service<br/>Log Parser]
        MS[Manuscript Service<br/>API Client]
    end
    
    subgraph "Core Data Model"
        PROJ[Projects<br/>CC.NET Projects]
        BUILDS[Builds<br/>From Projects]
        PKG[Packages<br/>User-Defined Configs]
        PUB_DATA[Publications<br/>Publishing History]
    end
    
    subgraph "Processing Layer"
        DS[Document Service<br/>PDF Generator]
        PUB[Publishing Service<br/>Workflow Orchestrator]
    end
    
    subgraph "Output Channels"
        STOR[Storage Service<br/>Multi-Provider]
        HELP[Help Center Service<br/>Multi-Platform]
    end
    
    subgraph "User Interface"
        UI[React Frontend<br/>Package Management Console]
    end
    
    CC --> PS
    FB --> MS
    PS --> PROJ
    MS --> PROJ
    PROJ --> BUILDS
    
    UI --> PKG
    PKG --> PROJ
    PKG --> BUILDS
    PKG --> PUB
    PUB --> DS
    PUB --> STOR
    PUB --> HELP
    PUB --> PUB_DATA
    
    classDef external fill:#ffebee
    classDef ingestion fill:#e3f2fd
    classDef coredata fill:#e8f5e8
    classDef processing fill:#fff3e0
    classDef output fill:#f3e5f5
    classDef ui fill:#fce4ec
    
    class CC,FB external
    class PS,MS ingestion
    class PROJ,BUILDS,PKG,PUB_DATA coredata
    class DS,PUB processing
    class STOR,HELP output
    class UI ui
```

## Frontend Views

### Package Management Interface
- **Package List View**: Display all packages with columns for Title, Project, Build, Status, Publish Date
  - Search and filtering capabilities
  - Real-time status updates via SignalR
  - Color coding for build status (Success/Failure/In Progress)
  
- **Package Details/Create Form**: 
  - **Basic Info**: Title, Description, Version
  - **Project Association**: Dropdown from available CC.NET projects  
  - **Build Selection**: Dropdown from successful builds with date/version info
  - **Configuration Tab**:
    - Build folder path
    - Compression settings (ZIP contents toggle)
    - Cleanup settings (delete old versions toggle)
    - Release note template selection
  - **Storage Tab**: 
    - Cloud storage provider selection
    - Folder/bucket configuration
    - CDN settings
  - **Help Center Tab**:
    - Article ID association
    - Template content with variable placeholders
    - Multi-locale support
  - Status tracking with progress indicators

### Project Dashboard
- **Project List View**: CC.NET projects with real-time build status
  - Build history and success/failure rates
  - Last successful build information
  - Project status monitoring
  
### Build Management
- **Build History View**: Detailed commit information
  - Commit comments and modified files
  - FogBugz case associations  
  - User attribution and timestamps
  - Release note preview from FogBugz cases
  - Search and filtering by user, date range, FogBugz status

### Publishing Dashboard
- **Active Publications View**: Real-time publishing progress
  - Step-by-step progress indicators
  - Detailed status for each publishing step
  - Error handling and retry capabilities
  - Download links upon completion
  
- **Publication History**: Audit trail of all publications
  - Success/failure status
  - Detailed logs and timing information
  - Download links to generated documents and packages

### Real-time Features
- **SignalR Integration**: Live updates for:
  - Build status changes
  - Publishing progress (0-100%)
  - Step completion notifications
  - Error alerts and notifications

## Detailed Publishing Workflow Steps

Based on legacy analysis, the publishing workflow consists of these detailed steps:

### Step 1: Package Configuration Validation (10%)
- Verify package configuration exists and is complete
- Validate build folder paths and permissions
- Check storage provider credentials and connectivity
- Verify help center article IDs and templates
- Create publication record in database

### Step 2: Build Data Collection (20%)
- Query CC.NET for build information
- Parse XML build logs to extract commit data
- Apply regex patterns to identify FogBugz case references
- Store build commits with user and file modification data
- Track progress with real-time updates

### Step 3: FogBugz Data Integration (30%)
- Query FogBugz API for case details referenced in commits
- Process case event history for status and release notes
- Filter release notes (exclude `<IGNORE>` entries)
- Cache FogBugz data to reduce API calls
- Update build commits with release note content

### Step 4: Document Generation (50%)
- Generate PDF using QuestPDF with modern templates
- Integrate FogBugz release notes and case information
- Include build version, date, and project information
- Store generated document with version tracking
- Support for multiple document formats if needed

### Step 5: File Processing (60%)
- Compress build artifacts if configured (ZIP format)
- Prepare local staging directory
- Copy necessary files for upload
- Apply naming conventions based on package configuration
- Progress tracking for large file operations

### Step 6: Cloud Storage Upload (75%)
- Upload compressed package and documentation
- Generate CDN URLs for fast content delivery
- Support progress tracking with real-time updates
- Handle upload failures with retry mechanisms
- Set appropriate permissions and metadata

### Step 7: Version Cleanup (80%)
- Delete old published versions if configured
- Maintain specified number of historical versions
- Clean up temporary local files
- Update storage usage tracking
- Log cleanup operations for audit trail

### Step 8: Help Center Updates (90%)
- Process template variables in article content
- Replace `{{VERSION}}`, `{{SWURL}}`, `{{PDFURL}}` placeholders
- Generate download URLs with CloudFront integration
- Update articles via Zendesk API
- Support multi-locale article updates

### Step 9: Database Updates (95%)
- Update package publish date and version
- Complete publication record with success details
- Update package configuration if needed
- Create audit log entries
- Store download URLs and document paths

### Step 10: Notification and Completion (100%)
- Send real-time completion notification via SignalR
- Generate publication summary with download links
- Update UI with success status and metrics
- Trigger any configured post-publication webhooks
- Clean up background job resources

## Next Steps

1. **Phase 1: Core Infrastructure**
   - Set up development environment with Docker Compose
   - Create solution structure with all identified services
   - Implement enhanced database models with EF Core migrations
   - Build API Gateway with JWT authentication and SignalR hubs

2. **Phase 2: External Service Integration**
   - Implement CC.NET client with TCP remoting
   - Build FogBugz client with HTTP API integration
   - Create cloud storage abstraction with S3/CloudFront support
   - Develop Zendesk client with template processing

3. **Phase 3: Core Services Development**
   - Project Service with XML log parsing and regex case extraction
   - Manuscript Service with event processing and caching
   - Document Service with QuestPDF integration
   - Storage Service with compression and cleanup capabilities
   - Help Center Service with template variable replacement

4. **Phase 4: Publishing Workflow**
   - Implement 10-step publishing workflow with Hangfire
   - Build progress tracking and real-time notifications
   - Create error handling and retry mechanisms
   - Develop workflow orchestration and coordination

5. **Phase 5: Frontend Development**
   - React frontend with advanced package management
   - Real-time publishing dashboard with progress tracking
   - Build history and FogBugz integration views
   - Publication history and audit trail interface

6. **Phase 6: Testing and Deployment**
   - Comprehensive integration testing with external services
   - Performance testing for large file uploads and processing
   - Security testing for credential management and API access
   - Production deployment with monitoring and logging