# ============================================================================= # COMPETENCY RECORD - Cedric Ancellin's demonstrated engineering skills and # experience. Third-party technologies and vendors are named as the concrete # skill surface. Professional background / track record. # ============================================================================= # ============================================================ # RELIABILITY & OPERATIONS - COMPETENCY RECORD # ============================================================ # Core Purpose: # Demonstrated experience running a production platform: # incident command, structured root-cause analysis, safe # data remediation at scale, error/alerting pipelines, # environment discipline, and idempotent operational tooling. # ============================================================ summary: > Ran a production platform as an operational discipline - severity-classified incident response with formal postmortems, root-cause analysis across availability/integrity/exposure incidents, idempotent and reversible large-scale data remediation, an environment-gated error/alerting pipeline, and strict multi-environment operational hygiene. # ================================================================== # 1. INCIDENT COMMAND & POSTMORTEM DISCIPLINE # ================================================================== incident_management: description: > Established a formal incident-management practice with a consistent postmortem structure. severity_model: sev1: "Critical, customer-facing outage or confidentiality/privacy breach." sev2: "High, partial unavailability or data-integrity breakage beyond a single record." sev3: "Minor, internal/staging or limited-scope customer-facing issue." sev4: "Minimal, internal disruption, no customer impact." postmortem_fields: - "Severity, status lifecycle (draft → reviewed → actioned), duration, impact, detection." - "Root cause, contributing factors, resolution, and follow-up actions with ownership + due dates." - "Precise timelines, containment/eradication/recovery phases, and lessons learned." # ================================================================== # 2. ROOT-CAUSE ANALYSIS # ================================================================== rca: description: > Performed rigorous root-cause analysis across multiple failure classes, fixing the structural cause rather than the symptom. examples: data_integrity: > Diagnosed multiple writers disagreeing on a canonical identity field; resolved by consolidating to a single-writer model and a clarified identity model, not just repairing the corrupted rows. availability: > Diagnosed a worker-pool exhaustion caused by proxying long-lived connections through a request worker model; remediated the proxy path with follow-ups for a better concurrency model and rate limiting. data_exposure: > Diagnosed authenticated responses becoming edge-cache eligible during a mitigation; restored cache-bypass rules and reviewed application code for per-user non-cacheability. latent_bug: > Diagnosed unsafe relative include paths silently resolving to the wrong files; fixed the path model with follow-ups for absolute includes, environment isolation, and static analysis. # ================================================================== # 3. SAFE DATA REMEDIATION AT SCALE # ================================================================== remediation: description: > Built reusable, safe tooling for repairing production data at scale. principles: - "Dry-run by default; an explicit apply flag required to write." - "Idempotent and reversible via status flags rather than destructive edits." - "Re-derive from the source of truth rather than guessing corrected values." - "Independent verification by someone other than the engineer applying the fix." outcome_shape: "Generalized rebuild/rehome scripts repairing thousands of records across many properties in bounded compute windows, with per-property metrics." # ================================================================== # 4. ERROR HANDLING & ALERTING PIPELINE # ================================================================== alerting: description: > Built an environment-aware error and alerting pipeline. mechanisms: - "A single error-terminating path returning a structured error envelope, never an ad-hoc die/throw." - "Production/preprod errors fan out to chat + email; non-production suppresses unless debug is explicitly enabled." - "Per-datastore failure handlers with environment-gated verbosity." - "Cross-request alert throttling to prevent alert storms." - "A circuit breaker for cache-tier unavailability with a cooldown." # ================================================================== # 5. ENVIRONMENT & DEPLOYMENT DISCIPLINE # ================================================================== ops_hygiene: description: > Enforced strict operational hygiene across environments. practices: - "Production / UAT / staging / sandbox parity for security-relevant configuration." - "UAT validation before any live change; dry-run tooling before production writes." - "A git + publish deployment pipeline with a clean-working-tree guarantee; no direct edits to deployed paths." - "Sandboxing of throwaway/diagnostic scripts away from production code paths." cross_reference: $ref: "https://zc8.com/competencies/12_technical_leadership"