From 85117c75a83414b241bec14d3d7d52f0fadd5815 Mon Sep 17 00:00:00 2001 From: noisedestroyers Date: Tue, 22 Jul 2025 12:21:54 -0400 Subject: [PATCH] [claudesquad] update from 'analyze legacy' on 22 Jul 25 12:21 EDT (paused) --- implementation.md | 525 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 470 insertions(+), 55 deletions(-) diff --git a/implementation.md b/implementation.md index 2749596..65646da 100644 --- a/implementation.md +++ b/implementation.md @@ -127,53 +127,79 @@ graph TB #### 2. Project Service (`project-service`) - **Purpose**: CruiseControl.NET integration and project management - **Technology**: ASP.NET Core Web API +- **Integration Method**: TCP Remoting (`tcp://{server}:21234/CruiseManager.rem`) - **Responsibilities**: - - Parse CC.NET log files - - Cache build history and status + - Connect to CC.NET via ThoughtWorks.CruiseControl.Remote library + - Parse XML build logs and extract commit information + - Regex pattern matching for FogBugz case ID extraction + - Cache build history and status with modified file tracking - Project configuration management - - Build status monitoring + - Build status monitoring and real-time updates #### 3. Manuscript Service (`manuscript-service`) - **Purpose**: FogBugz/Manuscript integration - **Technology**: ASP.NET Core Web API +- **Integration Method**: HTTP API (`http://{server}/api.asp`) +- **Authentication**: Token-based authentication system - **Responsibilities**: - - Developer comment extraction - - Feature timing data aggregation - - Release note data compilation + - Case/bug retrieval with full event history + - Status tracking (Opened, Resolved, Closed, Reactivated) + - Release note extraction from case events + - Zendesk ticket number linking + - Event history processing and caching + - Developer comment extraction and aggregation #### 4. Document Service (`document-service`) - **Purpose**: PDF generation and document management -- **Technology**: ASP.NET Core + iText7 or QuestPDF +- **Technology**: ASP.NET Core + QuestPDF (modern, container-friendly) - **Responsibilities**: - - Release note PDF generation - - Template management - - Document versioning + - Release note PDF generation with FogBugz data integration + - Template variable replacement ({{VERSION}}, {{DATE}}, {{PROJECT}}) + - Document versioning and storage + - Multi-format document support (PDF primary) + - Template management and customization #### 5. Storage Service (`storage-service`) - **Purpose**: Cloud storage abstraction layer - **Technology**: ASP.NET Core Web API - **Storage Providers**: - - **AWS S3**: AWS SDK for .NET + - **AWS S3**: AWS SDK for .NET with CloudFront CDN integration - **Box.com**: Box .NET SDK - **FTP**: FluentFTP library - **Pattern**: Strategy pattern for provider abstraction +- **Key Features**: + - File compression and decompression (ZIP support) + - Folder cleanup and old version management + - Upload progress tracking with real-time updates + - CDN integration for fast content delivery + - Multi-file batch upload operations #### 6. Help Center Service (`helpcenter-service`) -- **Purpose**: Support article updates +- **Purpose**: Support article updates with template processing - **Technology**: ASP.NET Core Web API - **Integrations**: - - **Zendesk**: Zendesk API v2 + - **Zendesk**: Zendesk API v2 with ZendeskApi_v2 library - **Salesforce Service**: Salesforce REST API - **Pattern**: Strategy pattern for provider abstraction +- **Key Features**: + - Template variable replacement ({{VERSION}}, {{SWURL}}, {{PDFURL}}) + - Multi-locale support (default: en-us) + - Article versioning and rollback capabilities + - Automated article publishing workflows + - Link generation for software downloads and documentation #### 7. Publishing Service (`publishing-service`) - **Purpose**: Orchestrate complete publishing workflows - **Technology**: ASP.NET Core + Hangfire - **Responsibilities**: - - Coordinate document generation - - Manage cloud uploads - - Update help center articles - - Workflow status tracking + - Multi-step workflow orchestration with progress tracking + - Coordinate document generation with FogBugz data + - Manage cloud uploads with compression and cleanup + - Update help center articles with template processing + - Workflow status tracking and real-time notifications + - Error handling and retry mechanisms + - Package configuration validation + - Publishing history and audit trail ### Database Design @@ -195,14 +221,25 @@ erDiagram string Description int ProjectId FK int SourceBuildId FK - json CloudStorageConfig - json HelpCenterArticles string Status datetime PublishDate datetime CreatedAt datetime UpdatedAt } + PackageConfigurations { + int Id PK + int PackageId FK + string BuildFolder + boolean ZipContents + boolean DeleteOldPublishedBuilds + string ReleaseNoteTemplate + json StorageSettings + json HelpCenterSettings + datetime CreatedAt + datetime UpdatedAt + } + Projects { int Id PK string Name @@ -224,6 +261,41 @@ erDiagram string ArtifactPath } + BuildCommits { + int Id PK + int BuildId FK + string CommitHash + string Comment + string User + datetime CommitDate + string FogBugzCaseId + json ModifiedFiles + string ReleaseNote + } + + FogBugzCases { + int Id PK + int CaseId + string Title + string Project + string Area + string Status + datetime LastUpdated + boolean IsOpen + } + + FogBugzEvents { + int Id PK + int CaseId FK + string EventType + string User + string Comment + string StatusString + datetime EventDate + string ReleaseNote + int ZendeskNumber + } + Publications { int Id PK int PackageId FK @@ -233,6 +305,17 @@ erDiagram json PublicationDetails } + PublishingSteps { + int Id PK + int PublicationId FK + string StepName + string Status + string Details + datetime StartedAt + datetime CompletedAt + int ProgressPercent + } + StorageProviders { int Id PK string Name @@ -252,13 +335,21 @@ erDiagram Projects ||--o{ Builds : "produces builds" Projects ||--o{ Packages : "user creates packages for" Builds ||--o{ Packages : "can be referenced by" + Builds ||--o{ BuildCommits : "contains commits" Packages ||--o{ Publications : "has publications" + Packages ||--o{ PackageConfigurations : "has configuration" + Publications ||--o{ PublishingSteps : "tracked by steps" + BuildCommits ||--o| FogBugzCases : "may reference" + FogBugzCases ||--o{ FogBugzEvents : "has event history" ``` #### Core Entities ```sql -- Packages: User-defined software release configurations -Packages (Id, Title, Version, Description, ProjectId, SourceBuildId, CloudStorageConfig, HelpCenterArticles, Status, PublishDate, CreatedAt, UpdatedAt) +Packages (Id, Title, Version, Description, ProjectId, SourceBuildId, Status, PublishDate, CreatedAt, UpdatedAt) + +-- Package-specific configurations +PackageConfigurations (Id, PackageId, BuildFolder, ZipContents, DeleteOldPublishedBuilds, ReleaseNoteTemplate, StorageSettings, HelpCenterSettings, CreatedAt, UpdatedAt) -- Projects from CruiseControl.NET Projects (Id, Name, Description, CCNetProjectName, Status, CreatedAt, UpdatedAt) @@ -266,9 +357,21 @@ Projects (Id, Name, Description, CCNetProjectName, Status, CreatedAt, UpdatedAt) -- Build information Builds (Id, ProjectId, BuildNumber, Status, StartTime, EndTime, LogPath, ArtifactPath) +-- Individual commit information from build logs +BuildCommits (Id, BuildId, CommitHash, Comment, User, CommitDate, FogBugzCaseId, ModifiedFiles, ReleaseNote) + +-- FogBugz case information +FogBugzCases (Id, CaseId, Title, Project, Area, Status, LastUpdated, IsOpen) + +-- FogBugz case event history +FogBugzEvents (Id, CaseId, EventType, User, Comment, StatusString, EventDate, ReleaseNote, ZendeskNumber) + -- Publishing history Publications (Id, PackageId, Status, PublishedAt, ReleaseNotesPath, PublicationDetails) +-- Publishing workflow step tracking +PublishingSteps (Id, PublicationId, StepName, Status, Details, StartedAt, CompletedAt, ProgressPercent) + -- User management Users (Id, Username, PasswordHash, Role, CreatedAt, LastLogin) @@ -283,9 +386,21 @@ HelpCenterProviders (Id, Name, Type, Configuration, IsActive) ```csharp public interface ICloudStorageProvider { - Task UploadAsync(string containerName, string fileName, Stream content); + Task UploadAsync(string containerName, string fileName, Stream content, IProgress progress = null); + Task UploadFileAsync(string containerName, string localFilePath, IProgress progress = null); Task DeleteAsync(string containerName, string fileName); Task DownloadAsync(string containerName, string fileName); + Task DeleteFolderContentsAsync(string containerName, string folderPath, List excludeFiles); + Task> UploadFolderAsync(string containerName, string localFolderPath, bool zipContents, IProgress progress = null); + Task GetPublicUrlAsync(string containerName, string fileName); +} + +public class UploadProgress +{ + public string FileName { get; set; } + public long BytesTransferred { get; set; } + public long TotalBytes { get; set; } + public int PercentComplete { get; set; } } ``` @@ -293,9 +408,97 @@ public interface ICloudStorageProvider ```csharp public interface IHelpCenterProvider { - Task UpdateArticleAsync(string articleId, string content, string title); + Task UpdateArticleAsync(string articleId, string content, string title = null); + Task UpdateArticleWithTemplateAsync(string articleId, string templateContent, Dictionary variables); Task CreateArticleAsync(string content, string title, string categoryId); Task PublishArticleAsync(string articleId); + Task GetArticleAsync(string articleId); + Task TestConnectionAsync(); +} + +public class ArticleInfo +{ + public string Id { get; set; } + public string Title { get; set; } + public string Content { get; set; } + public string Locale { get; set; } + public DateTime LastUpdated { get; set; } + public bool IsPublished { get; set; } +} +``` + +### CruiseControl.NET Integration Interface +```csharp +public interface ICCNetClient +{ + Task> GetProjectStatusAsync(); + Task> GetBuildNamesAsync(string projectName); + Task GetBuildLogAsync(string projectName, string buildName); + Task TestConnectionAsync(); +} + +public class ProjectStatus +{ + public string Name { get; set; } + public IntegrationStatus BuildStatus { get; set; } + public string LastSuccessfulBuildLabel { get; set; } + public DateTime LastBuildDate { get; set; } +} + +public class BuildCommitInfo +{ + public string ProjectName { get; set; } + public string BuildNumber { get; set; } + public string Comment { get; set; } + public string User { get; set; } + public DateTime Date { get; set; } + public string FogBugzCaseId { get; set; } + public List ModifiedFiles { get; set; } +} +``` + +### FogBugz Integration Interface +```csharp +public interface IManuscriptClient +{ + Task LogOnAsync(); + Task GetCaseAsync(int caseId); + Task> GetCaseEventsAsync(int caseId); + Task GetReleaseNoteAsync(int caseId); + Task GetCaseStatusAsync(int caseId); + Task TestConnectionAsync(); +} + +public class FogBugzCase +{ + public int Id { get; set; } + public string Title { get; set; } + public string Project { get; set; } + public string Area { get; set; } + public string Status { get; set; } + public DateTime LastUpdated { get; set; } + public bool IsOpen { get; set; } + public List Events { get; set; } +} + +public class FogBugzEvent +{ + public string User { get; set; } + public string Comment { get; set; } + public string StatusString { get; set; } + public DateTime Timestamp { get; set; } + public string ReleaseNote { get; set; } + public int ZendeskNumber { get; set; } + public FogBugzStatus Status { get; set; } +} + +public enum FogBugzStatus +{ + Unknown, + Opened, + Resolved, + Closed, + Reactivated } ``` @@ -463,13 +666,38 @@ src/ - PostgreSQL with persistent volumes - Health checks and restart policies -## Questions for Next Phase +## External Service Integration Details -1. **CC.NET Integration**: What format are the log files? XML, plain text, or proprietary? -2. **Manuscript API**: Is there existing API documentation or will we need to reverse-engineer? -3. **Document Templates**: Do you have existing release note templates or should we create standard ones? -4. **Cloud Storage**: Any specific bucket/container naming conventions? -5. **Help Center**: Do you have existing article templates or categories to work with? +### CruiseControl.NET Integration +- **Connection Method**: TCP Remoting (`tcp://{server}:21234/CruiseManager.rem`) +- **Library**: ThoughtWorks.CruiseControl.Remote +- **Log Format**: XML with embedded modification details +- **FogBugz Case Extraction**: Regex pattern matching on commit comments +- **Supported Patterns**: + - Comments starting with case number + - "fb{number}" or "FB{number}" patterns + - "asp?{number}" patterns + - Full URL patterns with "cases/{number}" + +### FogBugz/Manuscript Integration +- **Connection Method**: HTTP API (`http://{server}/api.asp`) +- **Authentication**: Token-based with session management +- **Event Processing**: Full case history with status tracking +- **Release Note Extraction**: From "ReleaseNoted" events with `` filtering +- **Status Mapping**: Opened, Resolved, Closed, Reactivated +- **Zendesk Linking**: Custom field parsing for ticket numbers + +### AWS S3 Integration +- **CDN**: CloudFront integration for fast content delivery +- **Features**: Folder cleanup, progress tracking, public URL generation +- **Compression**: ZIP file creation with SharpSevenZip +- **Cleanup**: Automatic deletion of old versions based on configuration + +### Zendesk Integration +- **API Version**: Zendesk API v2 with ZendeskApi_v2 library +- **Template Variables**: `{{VERSION}}`, `{{SWURL}}`, `{{PDFURL}}` +- **URL Generation**: Automatic CloudFront URL generation for downloads +- **Locale Support**: Multi-locale article updates (default: en-us) ## Publishing Workflow @@ -479,11 +707,14 @@ sequenceDiagram participant UI as React Frontend participant GW as API Gateway participant PUB as Publishing Service + participant PROJ as Project Service + participant MAN as Manuscript Service participant DOC as Document Service participant STOR as Storage Service participant HELP as Help Center Service participant HF as Hangfire participant SH as SignalR Hub + participant DB as Database Note over User,UI: Package Management Phase User->>UI: Create/Edit Package Configuration @@ -493,25 +724,59 @@ sequenceDiagram UI->>GW: GET /api/packages/{id} (Details) Note over User,UI: Publishing Phase - User->>UI: Select Package & Initiate Publishing - UI->>GW: POST /api/publish/package/{id} + User->>UI: Select Package & Build Version + UI->>GW: POST /api/publish/package/{id}/build/{buildId} GW->>PUB: Queue Publishing Job - PUB->>HF: Enqueue Background Job + PUB->>HF: Enqueue Multi-Step Background Job + PUB->>SH: Notify Job Started + SH->>UI: Show Publishing Progress UI - Note over HF: Background Processing Starts + Note over HF: Step 1: Validate Package Configuration + HF->>DB: Create Publication Record + HF->>PUB: Validate Package Settings + PUB->>SH: Progress Update (10%) - HF->>DOC: Generate Release Notes PDF - DOC-->>HF: PDF Generated + Note over HF: Step 2: Collect Build Data + HF->>PROJ: Get Build Commits + PROJ->>DB: Query BuildCommits for Build + PUB->>SH: Progress Update (20%) - HF->>STOR: Upload Package + PDF to Cloud - STOR-->>HF: Upload Complete + Note over HF: Step 3: Query FogBugz for Release Notes + HF->>MAN: Get Case Details for Commits + MAN->>DB: Cache FogBugz Events + PUB->>SH: Progress Update (30%) - HF->>HELP: Update Help Center Articles - HELP-->>HF: Articles Updated + Note over HF: Step 4: Generate Release Notes PDF + HF->>DOC: Generate PDF with FogBugz Data + DOC->>DB: Store Document Path + PUB->>SH: Progress Update (50%) - HF->>SH: Notify Completion - SH->>UI: Real-time Publication Status - UI->>User: Show Completion Status + Note over HF: Step 5: Compress Files (if configured) + HF->>PUB: Compress Build Artifacts + PUB->>SH: Progress Update (60%) + + Note over HF: Step 6: Upload to Cloud Storage + HF->>STOR: Upload Package + PDF + STOR->>PUB: Upload Progress Events + PUB->>SH: Progress Update (75%) + + Note over HF: Step 7: Clean Up Old Versions + HF->>STOR: Delete Old Published Builds + PUB->>SH: Progress Update (80%) + + Note over HF: Step 8: Update Help Center Articles + HF->>HELP: Update with Template Variables + HELP->>PUB: Article Update Complete + PUB->>SH: Progress Update (90%) + + Note over HF: Step 9: Update Package Status + HF->>DB: Update Package PublishDate + HF->>DB: Complete Publication Record + + Note over HF: Step 10: Send Completion Notification + HF->>SH: Notify Complete (100%) + SH->>UI: Show Success with Download Links + UI->>User: Display Publication Summary ``` ## Data Flow Architecture @@ -583,21 +848,171 @@ flowchart TD ### Package Management Interface - **Package List View**: Display all packages with columns for Title, Project, Build, Status, Publish Date + - Search and filtering capabilities + - Real-time status updates via SignalR + - Color coding for build status (Success/Failure/In Progress) + - **Package Details/Create Form**: - - Title (user input) - - Project (dropdown from available projects) - - Build (dropdown from project's successful builds) - - Publish Date (date picker) - - Help Center Articles (multi-select/tags) - - Cloud Storage Configuration - - Status tracking + - **Basic Info**: Title, Description, Version + - **Project Association**: Dropdown from available CC.NET projects + - **Build Selection**: Dropdown from successful builds with date/version info + - **Configuration Tab**: + - Build folder path + - Compression settings (ZIP contents toggle) + - Cleanup settings (delete old versions toggle) + - Release note template selection + - **Storage Tab**: + - Cloud storage provider selection + - Folder/bucket configuration + - CDN settings + - **Help Center Tab**: + - Article ID association + - Template content with variable placeholders + - Multi-locale support + - Status tracking with progress indicators + +### Project Dashboard +- **Project List View**: CC.NET projects with real-time build status + - Build history and success/failure rates + - Last successful build information + - Project status monitoring + +### Build Management +- **Build History View**: Detailed commit information + - Commit comments and modified files + - FogBugz case associations + - User attribution and timestamps + - Release note preview from FogBugz cases + - Search and filtering by user, date range, FogBugz status + +### Publishing Dashboard +- **Active Publications View**: Real-time publishing progress + - Step-by-step progress indicators + - Detailed status for each publishing step + - Error handling and retry capabilities + - Download links upon completion + +- **Publication History**: Audit trail of all publications + - Success/failure status + - Detailed logs and timing information + - Download links to generated documents and packages + +### Real-time Features +- **SignalR Integration**: Live updates for: + - Build status changes + - Publishing progress (0-100%) + - Step completion notifications + - Error alerts and notifications + +## Detailed Publishing Workflow Steps + +Based on legacy analysis, the publishing workflow consists of these detailed steps: + +### Step 1: Package Configuration Validation (10%) +- Verify package configuration exists and is complete +- Validate build folder paths and permissions +- Check storage provider credentials and connectivity +- Verify help center article IDs and templates +- Create publication record in database + +### Step 2: Build Data Collection (20%) +- Query CC.NET for build information +- Parse XML build logs to extract commit data +- Apply regex patterns to identify FogBugz case references +- Store build commits with user and file modification data +- Track progress with real-time updates + +### Step 3: FogBugz Data Integration (30%) +- Query FogBugz API for case details referenced in commits +- Process case event history for status and release notes +- Filter release notes (exclude `` entries) +- Cache FogBugz data to reduce API calls +- Update build commits with release note content + +### Step 4: Document Generation (50%) +- Generate PDF using QuestPDF with modern templates +- Integrate FogBugz release notes and case information +- Include build version, date, and project information +- Store generated document with version tracking +- Support for multiple document formats if needed + +### Step 5: File Processing (60%) +- Compress build artifacts if configured (ZIP format) +- Prepare local staging directory +- Copy necessary files for upload +- Apply naming conventions based on package configuration +- Progress tracking for large file operations + +### Step 6: Cloud Storage Upload (75%) +- Upload compressed package and documentation +- Generate CDN URLs for fast content delivery +- Support progress tracking with real-time updates +- Handle upload failures with retry mechanisms +- Set appropriate permissions and metadata + +### Step 7: Version Cleanup (80%) +- Delete old published versions if configured +- Maintain specified number of historical versions +- Clean up temporary local files +- Update storage usage tracking +- Log cleanup operations for audit trail + +### Step 8: Help Center Updates (90%) +- Process template variables in article content +- Replace `{{VERSION}}`, `{{SWURL}}`, `{{PDFURL}}` placeholders +- Generate download URLs with CloudFront integration +- Update articles via Zendesk API +- Support multi-locale article updates + +### Step 9: Database Updates (95%) +- Update package publish date and version +- Complete publication record with success details +- Update package configuration if needed +- Create audit log entries +- Store download URLs and document paths + +### Step 10: Notification and Completion (100%) +- Send real-time completion notification via SignalR +- Generate publication summary with download links +- Update UI with success status and metrics +- Trigger any configured post-publication webhooks +- Clean up background job resources ## Next Steps -1. Set up development environment with Docker Compose -2. Create solution structure and shared contracts -3. Implement core database models and Entity Framework setup -4. Build API Gateway with authentication -5. Develop individual services in parallel -6. Create React frontend with Package Management views and SignalR integration -7. Integration testing and deployment scripts \ No newline at end of file +1. **Phase 1: Core Infrastructure** + - Set up development environment with Docker Compose + - Create solution structure with all identified services + - Implement enhanced database models with EF Core migrations + - Build API Gateway with JWT authentication and SignalR hubs + +2. **Phase 2: External Service Integration** + - Implement CC.NET client with TCP remoting + - Build FogBugz client with HTTP API integration + - Create cloud storage abstraction with S3/CloudFront support + - Develop Zendesk client with template processing + +3. **Phase 3: Core Services Development** + - Project Service with XML log parsing and regex case extraction + - Manuscript Service with event processing and caching + - Document Service with QuestPDF integration + - Storage Service with compression and cleanup capabilities + - Help Center Service with template variable replacement + +4. **Phase 4: Publishing Workflow** + - Implement 10-step publishing workflow with Hangfire + - Build progress tracking and real-time notifications + - Create error handling and retry mechanisms + - Develop workflow orchestration and coordination + +5. **Phase 5: Frontend Development** + - React frontend with advanced package management + - Real-time publishing dashboard with progress tracking + - Build history and FogBugz integration views + - Publication history and audit trail interface + +6. **Phase 6: Testing and Deployment** + - Comprehensive integration testing with external services + - Performance testing for large file uploads and processing + - Security testing for credential management and API access + - Production deployment with monitoring and logging \ No newline at end of file