[claudesquad] update from 'analyze legacy' on 22 Jul 25 12:21 EDT (paused)

This commit is contained in:
2025-07-22 12:21:54 -04:00
parent 482343b6d9
commit 85117c75a8

View File

@@ -127,53 +127,79 @@ graph TB
#### 2. Project Service (`project-service`)
- **Purpose**: CruiseControl.NET integration and project management
- **Technology**: ASP.NET Core Web API
- **Integration Method**: TCP Remoting (`tcp://{server}:21234/CruiseManager.rem`)
- **Responsibilities**:
- Parse CC.NET log files
- Cache build history and status
- Connect to CC.NET via ThoughtWorks.CruiseControl.Remote library
- Parse XML build logs and extract commit information
- Regex pattern matching for FogBugz case ID extraction
- Cache build history and status with modified file tracking
- Project configuration management
- Build status monitoring
- Build status monitoring and real-time updates
#### 3. Manuscript Service (`manuscript-service`)
- **Purpose**: FogBugz/Manuscript integration
- **Technology**: ASP.NET Core Web API
- **Integration Method**: HTTP API (`http://{server}/api.asp`)
- **Authentication**: Token-based authentication system
- **Responsibilities**:
- Developer comment extraction
- Feature timing data aggregation
- Release note data compilation
- Case/bug retrieval with full event history
- Status tracking (Opened, Resolved, Closed, Reactivated)
- Release note extraction from case events
- Zendesk ticket number linking
- Event history processing and caching
- Developer comment extraction and aggregation
#### 4. Document Service (`document-service`)
- **Purpose**: PDF generation and document management
- **Technology**: ASP.NET Core + iText7 or QuestPDF
- **Technology**: ASP.NET Core + QuestPDF (modern, container-friendly)
- **Responsibilities**:
- Release note PDF generation
- Template management
- Document versioning
- Release note PDF generation with FogBugz data integration
- Template variable replacement ({{VERSION}}, {{DATE}}, {{PROJECT}})
- Document versioning and storage
- Multi-format document support (PDF primary)
- Template management and customization
#### 5. Storage Service (`storage-service`)
- **Purpose**: Cloud storage abstraction layer
- **Technology**: ASP.NET Core Web API
- **Storage Providers**:
- **AWS S3**: AWS SDK for .NET
- **AWS S3**: AWS SDK for .NET with CloudFront CDN integration
- **Box.com**: Box .NET SDK
- **FTP**: FluentFTP library
- **Pattern**: Strategy pattern for provider abstraction
- **Key Features**:
- File compression and decompression (ZIP support)
- Folder cleanup and old version management
- Upload progress tracking with real-time updates
- CDN integration for fast content delivery
- Multi-file batch upload operations
#### 6. Help Center Service (`helpcenter-service`)
- **Purpose**: Support article updates
- **Purpose**: Support article updates with template processing
- **Technology**: ASP.NET Core Web API
- **Integrations**:
- **Zendesk**: Zendesk API v2
- **Zendesk**: Zendesk API v2 with ZendeskApi_v2 library
- **Salesforce Service**: Salesforce REST API
- **Pattern**: Strategy pattern for provider abstraction
- **Key Features**:
- Template variable replacement ({{VERSION}}, {{SWURL}}, {{PDFURL}})
- Multi-locale support (default: en-us)
- Article versioning and rollback capabilities
- Automated article publishing workflows
- Link generation for software downloads and documentation
#### 7. Publishing Service (`publishing-service`)
- **Purpose**: Orchestrate complete publishing workflows
- **Technology**: ASP.NET Core + Hangfire
- **Responsibilities**:
- Coordinate document generation
- Manage cloud uploads
- Update help center articles
- Workflow status tracking
- Multi-step workflow orchestration with progress tracking
- Coordinate document generation with FogBugz data
- Manage cloud uploads with compression and cleanup
- Update help center articles with template processing
- Workflow status tracking and real-time notifications
- Error handling and retry mechanisms
- Package configuration validation
- Publishing history and audit trail
### Database Design
@@ -195,14 +221,25 @@ erDiagram
string Description
int ProjectId FK
int SourceBuildId FK
json CloudStorageConfig
json HelpCenterArticles
string Status
datetime PublishDate
datetime CreatedAt
datetime UpdatedAt
}
PackageConfigurations {
int Id PK
int PackageId FK
string BuildFolder
boolean ZipContents
boolean DeleteOldPublishedBuilds
string ReleaseNoteTemplate
json StorageSettings
json HelpCenterSettings
datetime CreatedAt
datetime UpdatedAt
}
Projects {
int Id PK
string Name
@@ -224,6 +261,41 @@ erDiagram
string ArtifactPath
}
BuildCommits {
int Id PK
int BuildId FK
string CommitHash
string Comment
string User
datetime CommitDate
string FogBugzCaseId
json ModifiedFiles
string ReleaseNote
}
FogBugzCases {
int Id PK
int CaseId
string Title
string Project
string Area
string Status
datetime LastUpdated
boolean IsOpen
}
FogBugzEvents {
int Id PK
int CaseId FK
string EventType
string User
string Comment
string StatusString
datetime EventDate
string ReleaseNote
int ZendeskNumber
}
Publications {
int Id PK
int PackageId FK
@@ -233,6 +305,17 @@ erDiagram
json PublicationDetails
}
PublishingSteps {
int Id PK
int PublicationId FK
string StepName
string Status
string Details
datetime StartedAt
datetime CompletedAt
int ProgressPercent
}
StorageProviders {
int Id PK
string Name
@@ -252,13 +335,21 @@ erDiagram
Projects ||--o{ Builds : "produces builds"
Projects ||--o{ Packages : "user creates packages for"
Builds ||--o{ Packages : "can be referenced by"
Builds ||--o{ BuildCommits : "contains commits"
Packages ||--o{ Publications : "has publications"
Packages ||--o{ PackageConfigurations : "has configuration"
Publications ||--o{ PublishingSteps : "tracked by steps"
BuildCommits ||--o| FogBugzCases : "may reference"
FogBugzCases ||--o{ FogBugzEvents : "has event history"
```
#### Core Entities
```sql
-- Packages: User-defined software release configurations
Packages (Id, Title, Version, Description, ProjectId, SourceBuildId, CloudStorageConfig, HelpCenterArticles, Status, PublishDate, CreatedAt, UpdatedAt)
Packages (Id, Title, Version, Description, ProjectId, SourceBuildId, Status, PublishDate, CreatedAt, UpdatedAt)
-- Package-specific configurations
PackageConfigurations (Id, PackageId, BuildFolder, ZipContents, DeleteOldPublishedBuilds, ReleaseNoteTemplate, StorageSettings, HelpCenterSettings, CreatedAt, UpdatedAt)
-- Projects from CruiseControl.NET
Projects (Id, Name, Description, CCNetProjectName, Status, CreatedAt, UpdatedAt)
@@ -266,9 +357,21 @@ Projects (Id, Name, Description, CCNetProjectName, Status, CreatedAt, UpdatedAt)
-- Build information
Builds (Id, ProjectId, BuildNumber, Status, StartTime, EndTime, LogPath, ArtifactPath)
-- Individual commit information from build logs
BuildCommits (Id, BuildId, CommitHash, Comment, User, CommitDate, FogBugzCaseId, ModifiedFiles, ReleaseNote)
-- FogBugz case information
FogBugzCases (Id, CaseId, Title, Project, Area, Status, LastUpdated, IsOpen)
-- FogBugz case event history
FogBugzEvents (Id, CaseId, EventType, User, Comment, StatusString, EventDate, ReleaseNote, ZendeskNumber)
-- Publishing history
Publications (Id, PackageId, Status, PublishedAt, ReleaseNotesPath, PublicationDetails)
-- Publishing workflow step tracking
PublishingSteps (Id, PublicationId, StepName, Status, Details, StartedAt, CompletedAt, ProgressPercent)
-- User management
Users (Id, Username, PasswordHash, Role, CreatedAt, LastLogin)
@@ -283,9 +386,21 @@ HelpCenterProviders (Id, Name, Type, Configuration, IsActive)
```csharp
public interface ICloudStorageProvider
{
Task<string> UploadAsync(string containerName, string fileName, Stream content);
Task<string> UploadAsync(string containerName, string fileName, Stream content, IProgress<UploadProgress> progress = null);
Task<string> UploadFileAsync(string containerName, string localFilePath, IProgress<UploadProgress> progress = null);
Task<bool> DeleteAsync(string containerName, string fileName);
Task<Stream> DownloadAsync(string containerName, string fileName);
Task<bool> DeleteFolderContentsAsync(string containerName, string folderPath, List<string> excludeFiles);
Task<List<string>> UploadFolderAsync(string containerName, string localFolderPath, bool zipContents, IProgress<UploadProgress> progress = null);
Task<string> GetPublicUrlAsync(string containerName, string fileName);
}
public class UploadProgress
{
public string FileName { get; set; }
public long BytesTransferred { get; set; }
public long TotalBytes { get; set; }
public int PercentComplete { get; set; }
}
```
@@ -293,9 +408,97 @@ public interface ICloudStorageProvider
```csharp
public interface IHelpCenterProvider
{
Task<bool> UpdateArticleAsync(string articleId, string content, string title);
Task<bool> UpdateArticleAsync(string articleId, string content, string title = null);
Task<bool> UpdateArticleWithTemplateAsync(string articleId, string templateContent, Dictionary<string, string> variables);
Task<string> CreateArticleAsync(string content, string title, string categoryId);
Task<bool> PublishArticleAsync(string articleId);
Task<ArticleInfo> GetArticleAsync(string articleId);
Task<bool> TestConnectionAsync();
}
public class ArticleInfo
{
public string Id { get; set; }
public string Title { get; set; }
public string Content { get; set; }
public string Locale { get; set; }
public DateTime LastUpdated { get; set; }
public bool IsPublished { get; set; }
}
```
### CruiseControl.NET Integration Interface
```csharp
public interface ICCNetClient
{
Task<List<ProjectStatus>> GetProjectStatusAsync();
Task<List<string>> GetBuildNamesAsync(string projectName);
Task<string> GetBuildLogAsync(string projectName, string buildName);
Task<bool> TestConnectionAsync();
}
public class ProjectStatus
{
public string Name { get; set; }
public IntegrationStatus BuildStatus { get; set; }
public string LastSuccessfulBuildLabel { get; set; }
public DateTime LastBuildDate { get; set; }
}
public class BuildCommitInfo
{
public string ProjectName { get; set; }
public string BuildNumber { get; set; }
public string Comment { get; set; }
public string User { get; set; }
public DateTime Date { get; set; }
public string FogBugzCaseId { get; set; }
public List<string> ModifiedFiles { get; set; }
}
```
### FogBugz Integration Interface
```csharp
public interface IManuscriptClient
{
Task<bool> LogOnAsync();
Task<FogBugzCase> GetCaseAsync(int caseId);
Task<List<FogBugzEvent>> GetCaseEventsAsync(int caseId);
Task<string> GetReleaseNoteAsync(int caseId);
Task<FogBugzStatus> GetCaseStatusAsync(int caseId);
Task<bool> TestConnectionAsync();
}
public class FogBugzCase
{
public int Id { get; set; }
public string Title { get; set; }
public string Project { get; set; }
public string Area { get; set; }
public string Status { get; set; }
public DateTime LastUpdated { get; set; }
public bool IsOpen { get; set; }
public List<FogBugzEvent> Events { get; set; }
}
public class FogBugzEvent
{
public string User { get; set; }
public string Comment { get; set; }
public string StatusString { get; set; }
public DateTime Timestamp { get; set; }
public string ReleaseNote { get; set; }
public int ZendeskNumber { get; set; }
public FogBugzStatus Status { get; set; }
}
public enum FogBugzStatus
{
Unknown,
Opened,
Resolved,
Closed,
Reactivated
}
```
@@ -463,13 +666,38 @@ src/
- PostgreSQL with persistent volumes
- Health checks and restart policies
## Questions for Next Phase
## External Service Integration Details
1. **CC.NET Integration**: What format are the log files? XML, plain text, or proprietary?
2. **Manuscript API**: Is there existing API documentation or will we need to reverse-engineer?
3. **Document Templates**: Do you have existing release note templates or should we create standard ones?
4. **Cloud Storage**: Any specific bucket/container naming conventions?
5. **Help Center**: Do you have existing article templates or categories to work with?
### CruiseControl.NET Integration
- **Connection Method**: TCP Remoting (`tcp://{server}:21234/CruiseManager.rem`)
- **Library**: ThoughtWorks.CruiseControl.Remote
- **Log Format**: XML with embedded modification details
- **FogBugz Case Extraction**: Regex pattern matching on commit comments
- **Supported Patterns**:
- Comments starting with case number
- "fb{number}" or "FB{number}" patterns
- "asp?{number}" patterns
- Full URL patterns with "cases/{number}"
### FogBugz/Manuscript Integration
- **Connection Method**: HTTP API (`http://{server}/api.asp`)
- **Authentication**: Token-based with session management
- **Event Processing**: Full case history with status tracking
- **Release Note Extraction**: From "ReleaseNoted" events with `<IGNORE>` filtering
- **Status Mapping**: Opened, Resolved, Closed, Reactivated
- **Zendesk Linking**: Custom field parsing for ticket numbers
### AWS S3 Integration
- **CDN**: CloudFront integration for fast content delivery
- **Features**: Folder cleanup, progress tracking, public URL generation
- **Compression**: ZIP file creation with SharpSevenZip
- **Cleanup**: Automatic deletion of old versions based on configuration
### Zendesk Integration
- **API Version**: Zendesk API v2 with ZendeskApi_v2 library
- **Template Variables**: `{{VERSION}}`, `{{SWURL}}`, `{{PDFURL}}`
- **URL Generation**: Automatic CloudFront URL generation for downloads
- **Locale Support**: Multi-locale article updates (default: en-us)
## Publishing Workflow
@@ -479,11 +707,14 @@ sequenceDiagram
participant UI as React Frontend
participant GW as API Gateway
participant PUB as Publishing Service
participant PROJ as Project Service
participant MAN as Manuscript Service
participant DOC as Document Service
participant STOR as Storage Service
participant HELP as Help Center Service
participant HF as Hangfire
participant SH as SignalR Hub
participant DB as Database
Note over User,UI: Package Management Phase
User->>UI: Create/Edit Package Configuration
@@ -493,25 +724,59 @@ sequenceDiagram
UI->>GW: GET /api/packages/{id} (Details)
Note over User,UI: Publishing Phase
User->>UI: Select Package & Initiate Publishing
UI->>GW: POST /api/publish/package/{id}
User->>UI: Select Package & Build Version
UI->>GW: POST /api/publish/package/{id}/build/{buildId}
GW->>PUB: Queue Publishing Job
PUB->>HF: Enqueue Background Job
PUB->>HF: Enqueue Multi-Step Background Job
PUB->>SH: Notify Job Started
SH->>UI: Show Publishing Progress UI
Note over HF: Background Processing Starts
Note over HF: Step 1: Validate Package Configuration
HF->>DB: Create Publication Record
HF->>PUB: Validate Package Settings
PUB->>SH: Progress Update (10%)
HF->>DOC: Generate Release Notes PDF
DOC-->>HF: PDF Generated
Note over HF: Step 2: Collect Build Data
HF->>PROJ: Get Build Commits
PROJ->>DB: Query BuildCommits for Build
PUB->>SH: Progress Update (20%)
HF->>STOR: Upload Package + PDF to Cloud
STOR-->>HF: Upload Complete
Note over HF: Step 3: Query FogBugz for Release Notes
HF->>MAN: Get Case Details for Commits
MAN->>DB: Cache FogBugz Events
PUB->>SH: Progress Update (30%)
HF->>HELP: Update Help Center Articles
HELP-->>HF: Articles Updated
Note over HF: Step 4: Generate Release Notes PDF
HF->>DOC: Generate PDF with FogBugz Data
DOC->>DB: Store Document Path
PUB->>SH: Progress Update (50%)
HF->>SH: Notify Completion
SH->>UI: Real-time Publication Status
UI->>User: Show Completion Status
Note over HF: Step 5: Compress Files (if configured)
HF->>PUB: Compress Build Artifacts
PUB->>SH: Progress Update (60%)
Note over HF: Step 6: Upload to Cloud Storage
HF->>STOR: Upload Package + PDF
STOR->>PUB: Upload Progress Events
PUB->>SH: Progress Update (75%)
Note over HF: Step 7: Clean Up Old Versions
HF->>STOR: Delete Old Published Builds
PUB->>SH: Progress Update (80%)
Note over HF: Step 8: Update Help Center Articles
HF->>HELP: Update with Template Variables
HELP->>PUB: Article Update Complete
PUB->>SH: Progress Update (90%)
Note over HF: Step 9: Update Package Status
HF->>DB: Update Package PublishDate
HF->>DB: Complete Publication Record
Note over HF: Step 10: Send Completion Notification
HF->>SH: Notify Complete (100%)
SH->>UI: Show Success with Download Links
UI->>User: Display Publication Summary
```
## Data Flow Architecture
@@ -583,21 +848,171 @@ flowchart TD
### Package Management Interface
- **Package List View**: Display all packages with columns for Title, Project, Build, Status, Publish Date
- Search and filtering capabilities
- Real-time status updates via SignalR
- Color coding for build status (Success/Failure/In Progress)
- **Package Details/Create Form**:
- Title (user input)
- Project (dropdown from available projects)
- Build (dropdown from project's successful builds)
- Publish Date (date picker)
- Help Center Articles (multi-select/tags)
- Cloud Storage Configuration
- Status tracking
- **Basic Info**: Title, Description, Version
- **Project Association**: Dropdown from available CC.NET projects
- **Build Selection**: Dropdown from successful builds with date/version info
- **Configuration Tab**:
- Build folder path
- Compression settings (ZIP contents toggle)
- Cleanup settings (delete old versions toggle)
- Release note template selection
- **Storage Tab**:
- Cloud storage provider selection
- Folder/bucket configuration
- CDN settings
- **Help Center Tab**:
- Article ID association
- Template content with variable placeholders
- Multi-locale support
- Status tracking with progress indicators
### Project Dashboard
- **Project List View**: CC.NET projects with real-time build status
- Build history and success/failure rates
- Last successful build information
- Project status monitoring
### Build Management
- **Build History View**: Detailed commit information
- Commit comments and modified files
- FogBugz case associations
- User attribution and timestamps
- Release note preview from FogBugz cases
- Search and filtering by user, date range, FogBugz status
### Publishing Dashboard
- **Active Publications View**: Real-time publishing progress
- Step-by-step progress indicators
- Detailed status for each publishing step
- Error handling and retry capabilities
- Download links upon completion
- **Publication History**: Audit trail of all publications
- Success/failure status
- Detailed logs and timing information
- Download links to generated documents and packages
### Real-time Features
- **SignalR Integration**: Live updates for:
- Build status changes
- Publishing progress (0-100%)
- Step completion notifications
- Error alerts and notifications
## Detailed Publishing Workflow Steps
Based on legacy analysis, the publishing workflow consists of these detailed steps:
### Step 1: Package Configuration Validation (10%)
- Verify package configuration exists and is complete
- Validate build folder paths and permissions
- Check storage provider credentials and connectivity
- Verify help center article IDs and templates
- Create publication record in database
### Step 2: Build Data Collection (20%)
- Query CC.NET for build information
- Parse XML build logs to extract commit data
- Apply regex patterns to identify FogBugz case references
- Store build commits with user and file modification data
- Track progress with real-time updates
### Step 3: FogBugz Data Integration (30%)
- Query FogBugz API for case details referenced in commits
- Process case event history for status and release notes
- Filter release notes (exclude `<IGNORE>` entries)
- Cache FogBugz data to reduce API calls
- Update build commits with release note content
### Step 4: Document Generation (50%)
- Generate PDF using QuestPDF with modern templates
- Integrate FogBugz release notes and case information
- Include build version, date, and project information
- Store generated document with version tracking
- Support for multiple document formats if needed
### Step 5: File Processing (60%)
- Compress build artifacts if configured (ZIP format)
- Prepare local staging directory
- Copy necessary files for upload
- Apply naming conventions based on package configuration
- Progress tracking for large file operations
### Step 6: Cloud Storage Upload (75%)
- Upload compressed package and documentation
- Generate CDN URLs for fast content delivery
- Support progress tracking with real-time updates
- Handle upload failures with retry mechanisms
- Set appropriate permissions and metadata
### Step 7: Version Cleanup (80%)
- Delete old published versions if configured
- Maintain specified number of historical versions
- Clean up temporary local files
- Update storage usage tracking
- Log cleanup operations for audit trail
### Step 8: Help Center Updates (90%)
- Process template variables in article content
- Replace `{{VERSION}}`, `{{SWURL}}`, `{{PDFURL}}` placeholders
- Generate download URLs with CloudFront integration
- Update articles via Zendesk API
- Support multi-locale article updates
### Step 9: Database Updates (95%)
- Update package publish date and version
- Complete publication record with success details
- Update package configuration if needed
- Create audit log entries
- Store download URLs and document paths
### Step 10: Notification and Completion (100%)
- Send real-time completion notification via SignalR
- Generate publication summary with download links
- Update UI with success status and metrics
- Trigger any configured post-publication webhooks
- Clean up background job resources
## Next Steps
1. Set up development environment with Docker Compose
2. Create solution structure and shared contracts
3. Implement core database models and Entity Framework setup
4. Build API Gateway with authentication
5. Develop individual services in parallel
6. Create React frontend with Package Management views and SignalR integration
7. Integration testing and deployment scripts
1. **Phase 1: Core Infrastructure**
- Set up development environment with Docker Compose
- Create solution structure with all identified services
- Implement enhanced database models with EF Core migrations
- Build API Gateway with JWT authentication and SignalR hubs
2. **Phase 2: External Service Integration**
- Implement CC.NET client with TCP remoting
- Build FogBugz client with HTTP API integration
- Create cloud storage abstraction with S3/CloudFront support
- Develop Zendesk client with template processing
3. **Phase 3: Core Services Development**
- Project Service with XML log parsing and regex case extraction
- Manuscript Service with event processing and caching
- Document Service with QuestPDF integration
- Storage Service with compression and cleanup capabilities
- Help Center Service with template variable replacement
4. **Phase 4: Publishing Workflow**
- Implement 10-step publishing workflow with Hangfire
- Build progress tracking and real-time notifications
- Create error handling and retry mechanisms
- Develop workflow orchestration and coordination
5. **Phase 5: Frontend Development**
- React frontend with advanced package management
- Real-time publishing dashboard with progress tracking
- Build history and FogBugz integration views
- Publication history and audit trail interface
6. **Phase 6: Testing and Deployment**
- Comprehensive integration testing with external services
- Performance testing for large file uploads and processing
- Security testing for credential management and API access
- Production deployment with monitoring and logging