Project Crisis: How Not to Panic and What to Do

Examples:
- Architecture doesn't scale
- Critical bug in production
- Database can't handle load
- External API integration broke

Examples:
- Key developer quit
- Budget exhausted at 60% of project
- No access to necessary tools
- Team overloaded with other tasks

Examples:
- Client drastically changed requirements
- Conflict between teams
- Lost contact with key stakeholders
- Wrong understanding of tasks

Examples:
- Deadline moved a month earlier
- Scope doubled
- Dependent projects delayed
- Critical path blocked

func HandleCrisis() {
    // DON'T PANIC!
    takeDeepBreath()
    
    // Gather basic facts
    facts := gatherInitialFacts()
    
    // Assess scale
    severity := assessSeverity(facts)
    
    if severity == CRITICAL {
        activateEmergencyProtocol()
    }
}

**Mandatory:**
- Project tech lead
- Person responsible for problem area
- Product Owner/client
- DevOps (if infrastructure issue)

**As needed:**
- Senior developers
- QA lead
- System architect

Subject: URGENT - Project X crisis
Time: Now, 15 minutes
Goal: Situation assessment and action plan

type CrisisResponse struct {
    ImmediateActions []Action
    Workarounds     []Solution
    Communication   []Message
}

func (cr *CrisisResponse) Stabilize() {
    // 1. Stop the bleeding
    cr.stopImmediateDamage()
    
    // 2. Temporary solutions
    cr.implementWorkarounds()
    
    // 3. Notify stakeholders
    cr.notifyStakeholders()
}

Problem: Production is down
Why? Server not responding
Why? Out of memory
Why? Memory leak in new code
Why? Database connections not closing
Why? Forgot to add defer conn.Close()

                    Problem
                       |
    People -------|      |      |------- Processes
                 |      |      |
                 |   CRISIS    |
                 |      |      |
    Technology --|      |      |------- Environment
                        |

type Solution struct {
    Name        string
    TimeToFix   time.Duration
    Cost        int
    Risk        int
    Impact      int
    Probability float64
}

func (s Solution) Score() float64 {
    return float64(s.Impact) * s.Probability / 
           (float64(s.Cost + s.Risk) * s.TimeToFix.Hours())
}

func chooseBestSolution(solutions []Solution) Solution {
    best := solutions[0]
    for _, solution := range solutions {
        if solution.Score() > best.Score() {
            best = solution
        }
    }
    return best
}

# Crisis Response Plan for Project X

## 1. Crisis brief description
- What: Critical bug in payment system
- When: January 15, 2025 2:30 PM
- Impact: 100% users cannot make payments

## 2. Immediate actions (completed)
- [x] Rollback to previous version
- [x] User notification
- [x] Backup payment system activation

## 3. Short-term actions (24 hours)
- [ ] Bug fix in code
- [ ] Testing on staging
- [ ] Hotfix release preparation

## 4. Medium-term actions (1 week)
- [ ] Full payment system testing
- [ ] Monitoring updates
- [ ] Incident documentation

## 5. Long-term actions (1 month)
- [ ] Testing process improvement
- [ ] Additional checks implementation
- [ ] Team training

type TeamReallocation struct {
    CrisisTeam    []Developer // Work only on crisis
    SupportTeam   []Developer // Support current tasks
    BackupTeam    []Developer // Ready to join
}

func (tr *TeamReallocation) OptimizeForCrisis() {
    // Best developers on crisis
    tr.CrisisTeam = selectTopPerformers(allDevelopers)
    
    // Others maintain minimum
    tr.SupportTeam = selectForMaintenance(remainingDevelopers)
    
    // Reserve for escalation
    tr.BackupTeam = getExternalContractors()
}

Subject: CRITICAL - Payment system issue

What happened: Critical bug discovered in payment system at 2:30 PM
Impact: Users cannot make purchases
What we're doing: Team working on fix, ETA - 2 hours
Temporary solution: Backup payment system activated
Next update: in 2 hours

Contact for questions: [your phone]

Crisis update - 4:30 PM

Progress: Bug localized, fix being prepared
Readiness: 80%
New ETA: 6:00 PM
Risks: No critical blockers

What's done:
- Found root cause
- Fix written and tested
- Deployment plan ready

Next steps:
- Final testing (30 min)
- Production deployment (15 min)
- Results monitoring (1 hour)

type StressManagement struct {
    TeamMorale    int
    WorkloadLevel int
    BurnoutRisk   float64
}

func (sm *StressManagement) MaintainTeamHealth() {
    if sm.BurnoutRisk > 0.7 {
        // Rotate people
        rotateTeamMembers()
        
        // Mandatory breaks
        enforceBreaks()
        
        // Additional support
        providePsychologicalSupport()
    }
}

Observe -> Orient -> Decide -> Act -> Repeat

type EarlyWarningSystem struct {
    Metrics     []Metric
    Thresholds  map[string]float64
    Alerts      []Alert
}

func (ews *EarlyWarningSystem) MonitorProject() {
    for _, metric := range ews.Metrics {
        if metric.Value > ews.Thresholds[metric.Name] {
            alert := Alert{
                Level:   WARNING,
                Message: fmt.Sprintf("%s exceeded threshold", metric.Name),
                Action:  "Team attention required",
            }
            ews.sendAlert(alert)
        }
    }
}

# Project Recovery Plan

## Crisis scenarios
1. Key developer loss
2. Critical production bug
3. Client requirement changes
4. Technical debt reaches critical mass

## For each scenario:
- Triggers (when to activate)
- Responsible persons
- Action sequence
- Required resources
- Success criteria

# Post-mortem: Payment system crisis

## Event timeline
2:30 PM - Bug discovered
2:35 PM - Team assembled
2:45 PM - Rollback activated
3:30 PM - Root cause found
5:00 PM - Fix ready
6:00 PM - Deployment completed

## What worked well
- Quick problem detection
- Effective client communication
- Rollback plan availability

## What can be improved
- Automate payment testing
- Improve critical path monitoring
- Create more detailed runbooks

## Prevention actions
- [ ] Add integration tests
- [ ] Set up payment anomaly alerts
- [ ] Conduct team training

type ProcessImprovement struct {
    LessonsLearned []Lesson
    NewProcedures  []Procedure
    Training       []TrainingModule
}

func (pi *ProcessImprovement) ImplementChanges() {
    for _, lesson := range pi.LessonsLearned {
        procedure := createProcedureFromLesson(lesson)
        pi.NewProcedures = append(pi.NewProcedures, procedure)
    }
    
    // Train team on new procedures
    pi.trainTeam()
}

# Additional resources:
- "The Phoenix Project" - Gene Kim
- "Site Reliability Engineering" - Google
- "Incident Response" - PagerDuty
- Crisis Management Framework - PMI