soc/progress-upd...

Project Progress Update

Date: February 13, 2026 Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)

1) Executive Summary

The MVP platform is operational and running end-to-end in the lab environment. Core integrations are in place:

• Detection: Wazuh
• Automation: Shuffle
• Case management: IRIS-web (replacing DFIRTrack)
• Escalation (MVP): PagerDuty Stub
• Orchestration/API layer: soc-integrator

All major containers are currently up, and key health checks are passing.

2) Completed Work

Platform orchestration and operations

• Combined stack runner created and improved (run-combined-stack.sh)
• Added command support for:
  • up, down, logs, status, help
  • per-target control (wazuh, iris, shuffle, pagerduty, integrator)
• Added consolidated health/status script (soc-status.sh)

Integration architecture

• Connected Wazuh, Shuffle, IRIS-web, PagerDuty Stub, and soc-integrator on shared network
• Resolved startup conflicts and runtime issues (port, compose, routing compatibility)

SOC Integrator (MVP)

• Added/validated integration APIs for:
  • Wazuh
  • Shuffle
  • IRIS-web
  • PagerDuty Stub
• Implemented MVP orchestration endpoints:
  • POST /mvp/incidents/ingest
  • POST /mvp/ioc/evaluate
  • POST /mvp/vpn/evaluate
  • GET /mvp/config/policies
  • PUT /mvp/config/policies
  • GET /mvp/health/dependencies
• Added internal API-key protection for mutation endpoints

Persistence layer

• Added PostgreSQL service for soc-integrator (soc-integrator-db)
• Added incident/policy/audit schema and startup initialization
• Enabled deduplication and audit tracking for incident processing

Testing utilities and documentation

• Added Wazuh test-event injection script:
  • scripts/send-wazuh-test-events.sh
• Added root project docs:
  • README.md
• Added root ignore rules:
  • .gitignore

3) Current Live Status (Lab)

Current stack status: UP

Healthy/available components:

• Wazuh manager, indexer, dashboard
• IRIS-web app/nginx/worker/db/rabbitmq
• Shuffle backend/frontend/opensearch/orborus
• PagerDuty Stub
• soc-integrator + soc-integrator-db

Endpoint checks:

• Wazuh Dashboard: OK
• Wazuh API: OK (auth-protected, expected 401 on unauthenticated root)
• IRIS Web: OK
• Shuffle Frontend: OK
• Shuffle Backend: reachable
• Shuffle OpenSearch: reachable (auth-protected)
• PagerDuty Stub: OK
• soc-integrator /health: OK

4) System Architecture Diagram (PlantUML)

@startuml
title FoodProject SOC Platform - System Architecture (MVP)
skinparam componentStyle rectangle

actor "Analyst" as analyst
cloud "External Log Sources" as logs

rectangle "SOC Shared Docker Network" {
  node "Wazuh Stack" as wazuh {
    component "Wazuh Manager" as wazuh_mgr
    component "Wazuh Indexer" as wazuh_idx
    component "Wazuh Dashboard" as wazuh_dash
  }

  node "Shuffle Stack" as shuffle {
    component "Shuffle Frontend" as shuf_fe
    component "Shuffle Backend" as shuf_be
    component "Shuffle Orborus" as shuf_orb
    component "Shuffle OpenSearch" as shuf_os
  }

  node "IRIS-web Stack" as iris {
    component "IRIS Web App" as iris_app
    database "IRIS DB" as iris_db
    component "IRIS RabbitMQ" as iris_mq
  }

  node "SOC Integrator Stack" as integ {
    component "soc-integrator API" as soc_api
    database "soc-integrator-db" as soc_db
  }

  component "PagerDuty Stub" as pd_stub
}

logs --> wazuh_mgr : Security events
wazuh_mgr --> wazuh_idx : Index alerts
analyst --> wazuh_dash : Investigate alerts
wazuh_dash --> wazuh_idx : Query data

wazuh_mgr --> soc_api : Alert/incident input
soc_api --> soc_db : Persist incidents\npolicies\naudit
soc_api --> iris_app : Create/update cases
soc_api --> pd_stub : Escalation (MVP)
soc_api --> shuf_be : Trigger automation

shuf_fe --> shuf_be : UI/API
shuf_be --> shuf_os : Read/write workflow data
shuf_orb --> shuf_be : Execution queue polling
shuf_orb --> shuf_os : Workflow state interactions

iris_app --> iris_db : Case data
iris_app --> iris_mq : Async jobs
@enduml

5) In Progress / Remaining for Customer UAT

1. Detection content tuning
2. Fine-tune Wazuh rules/decoders for customer log patterns and false-positive reduction
3. Use-case calibration
4. Validate risk/severity mapping per approved use cases
5. Tune exception list and threshold logic (especially VPN geo anomaly)
6. UAT evidence package
7. Capture deterministic UAT scenarios and outputs for:
   • IOC flow
   • VPN outside-TH flow
   • IRIS case creation/update
   • PagerDuty Stub escalation path
8. Production hardening items
9. Rotate default/local secrets used in lab config
10. Lock down internal API keys and access boundaries

6) Risks / Notes

• Current escalation target is PagerDuty Stub by design for MVP. Real PagerDuty production integration is the next stage.
• Some Wazuh config certificate directories are root-owned in the local lab clone, which may affect local git add operations if not excluded/fixed.

7) Next Milestone (Proposed)

Next milestone: MVP UAT Completion

Target outputs:

• Approved UAT checklist execution
• Tuned policy thresholds for customer environment
• Signed-off incident lifecycle flow: Wazuh event -> soc-integrator decision -> IRIS case -> PagerDuty Stub escalation

---

Date: February 26, 2026 Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)

Incremental Progress Since February 13, 2026

1) IOC Enrichment and Evaluation

• Added IOC APIs in soc-integrator:
  • POST /ioc/enrich
  • POST /ioc/evaluate
  • GET /ioc/history
  • POST /ioc/upload-file
  • POST /ioc/evaluate-file
  • GET /ioc/analysis/{analysis_id}
• Integrated VirusTotal adapter for domain/hash/file intelligence and analysis lookups.
• Integrated AbuseIPDB adapter for IP reputation checks.
• Added IOC trace persistence (ioc_trace) and repository methods for audit/history.

2) IRIS Integration Enhancements

• Added IRIS ticket APIs in soc-integrator:
  • POST /iris/tickets
  • GET /iris/tickets
• Updated IRIS API key in environment and verified ticket creation path via API.
• Added demo data seeding script:
  • scripts/seed-iris-demo-data.sh

3) Shuffle Workflow Automation

• Created and updated sample Shuffle workflow assets for webhook-driven IRIS ticket creation:
  • shuffle-workflows/sample-webhook-soc-integrator-iris-workflow.json
  • shuffle-workflows/sample-webhook-soc-integrator-iris-workflow.md
• Added workflow update helper script:
  • scripts/update-shuffle-workflow-from-template.sh
• Updated target workflow (07ecad05-ff68-41cb-888d-96d1a8e8db4b) with:
  • webhook trigger
  • HTTP action (http 1.4.0) to call soc-integrator ticket API
  • tested webhook execution path to successful completion

4) Networking and Runtime Fixes

• Resolved Shuffle action DNS failure to soc-integrator by attaching soc-integrator service to Shuffle execution network(s) in:
  • compose-overrides/soc-integrator.yml
• Verified connectivity from Shuffle execution context to:
  • http://soc-integrator:8080/health

5) Security and Repository Hygiene

• Added .env and .env.* to root .gitignore (kept .env.example tracked).
• Removed tracked env files from git cache to prevent secret leakage.
• Updated operational API keys in soc-integrator/.env for Shuffle, IRIS, VirusTotal, and AbuseIPDB.

6) Current Status (Lab)

• soc-integrator health endpoint: reachable.
• IOC enrich/evaluate flows: operational for domain/hash and file submission paths.
• Shuffle webhook-to-IRIS automation: operational after network fix.
• Core stack components remain available for continued UAT and tuning.

7) Simulation Logs Workstream

Completed

• Added FortiGate simulation coverage for multiple models:
  • 40F
  • 60F
  • 80F
  • 501E
• Added endpoint agent simulation coverage for:
  • Windows clients
  • macOS clients
  • Linux clients
• Added continuous run mode (--forever) to simulation scripts for long-running lab traffic generation.
• Extended script set to support realistic event streams for Wazuh ingestion and rule validation.

Operational scripts

• scripts/send-wazuh-test-events.sh

---

Date: March 4, 2026 Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)

Incremental Progress Since Previous Update (March 4, 2026)

1) Production-Profile Simulator Payloads

• Added production payload mode to proposal simulators:
  • scripts/send-wazuh-proposal-required-events.sh
  • scripts/send-wazuh-proposal-appendix-b-events.sh
• New argument:
  • --profile=simulation|production (default remains simulation)
• In production profile, simulator messages omit section/usecase_id/usecase markers and emit production-like key/value fields to support real parser/decoder testing.

2) Wazuh Normalize API Improvements

• Enhanced POST /ingest/wazuh-alert in soc-integrator:
  • returns both legacy normalized shape and SOC normalized event shape.
• Added GET /ingest/wazuh-alert/samples with practical sample request/response cases for:
  • DNS IOC
  • VMware auth
  • Windows Sysmon
  • C1 impossible travel

3) C1 Normalization (Production-First)

• Updated C1 normalization logic in:
  • soc-integrator/app/services/mvp_service.py
• C1 now maps from production characteristics (not only simulator markers):
  • identity/vpn source context
  • successful login/auth indicator
  • user + source IP present
  • geo context present (country and/or src_lat/src_lon)
• Legacy section/usecase_id C1 markers are kept as fallback for backward compatibility.

4) Current Validation Status

• Production-profile simulator events are being sent to Wazuh successfully.
• Some runs still show only base-rule matching during verification due to current Wazuh manager runtime instability in lab (intermittent restart/init issues), which affects deterministic decoder/rule validation windows.
• Next validation step after stable manager window:
  • re-run production-profile A1/B1/B2/B3 batches
  • confirm 110xxx production rules with consistent hit evidence.

5) Mermaid Diagram: C1-C3 Normalization Flow (SOC Integrator)

flowchart LR
    A[Wazuh Raw Alert<br/>full_log + rule + agent] --> B[POST /wazuh/sync-to-mvp<br/>or /ingest/wazuh-alert]
    B --> C[mvp_service.normalize_wazuh_hit]
    C --> D[KV parse from full_log<br/>src_ip user country event_type]
    D --> E[Normalized Event Schema<br/>source event_type timestamp severity<br/>asset network payload risk_context]

    E --> F{C-Detection Evaluate}
    F --> C1[C1 Impossible Travel<br/>geo context + success login + user/src_ip]
    F --> C2[C2 Credential Abuse<br/>off-hours / dormant / service interactive / rapid privilege]
    F --> C3[C3 Lateral Movement<br/>multi-host auth / SMB-RDP burst / internal scan]

    C1 --> G[Persist c_detection_events]
    C2 --> G
    C3 --> G
    G --> H[Optional Incident Pipeline<br/>IRIS case + Shuffle + PagerDuty stub]

• scripts/send-wazuh-endpoint-agent-test-events.sh
• additional simulation scripts under scripts/ for firewall and endpoint scenarios with continuous mode enabled

Detection alignment status

• Simulation work has been aligned to the detection objectives documented in:
  • Security Detection & Threat Intelligence Enhancement Proposal-2.md
• Proposal use-case mapping explicitly covered in simulation:
  • A1. DNS / Firewall (IOC):
  • DNS network communication to malicious domain
  • DNS/Firewall malicious domain IOC detection events
  • A2. FortiGate IPS/IDS & Firewall:
  • allowed RDP from public IP
  • admin password change
  • create/add admin account
  • disable email notification
  • config download
  • multiple critical/high IDS alerts
  • port scanning (public/private source variants)
  • IOC detection and communication to malicious IP
  • A3. FortiGate VPN:
  • authentication success from guest account
  • authentication success from multiple countries
  • brute-force success pattern
  • multiple fail patterns (many accounts from one source)
  • authentication success from outside Thailand
  • A4. Windows / Active Directory:
  • privileged/service account authentication failures
  • password spray and multi-source fail patterns
  • success from public IP / guest account
  • pass-the-hash style success indicators
  • account/group privilege change and account lifecycle events (create/re-enable)
  • AD enumeration behavior indicators
• Endpoint client simulations were added to complement proposal scope for heterogeneous environments:
  • Windows agent events
  • macOS agent events
  • Linux agent events
• Current use is suitable for pipeline and workflow validation (ingest -> detect -> automate -> case creation).
• Remaining work is focused on fine-grained scenario calibration:
  • event frequency tuning
  • field/value realism per source
  • expected alert volume by use case for cleaner UAT evidence

8) API Request/Response Samples

IOC Enrich

Request:

curl -sS -X POST http://localhost:8088/ioc/enrich \
  -H 'Content-Type: application/json' \
  -d '{
    "ioc_type": "domain",
    "ioc_value": "google.com",
    "sources": ["virustotal"]
  }'

Sample response:

{
  "success": true,
  "ioc_type": "domain",
  "ioc_value": "google.com",
  "enrichment": {
    "virustotal": {
      "reputation": 120,
      "last_analysis_stats": {
        "malicious": 0,
        "suspicious": 0,
        "harmless": 90
      }
    }
  }
}

IOC Evaluate

Request:

curl -sS -X POST http://localhost:8088/ioc/evaluate \
  -H 'Content-Type: application/json' \
  -d '{
    "ioc_type": "hash",
    "ioc_value": "44d88612fea8a8f36de82e1278abb02f",
    "sources": ["virustotal"]
  }'

Sample response:

{
  "success": true,
  "matched": true,
  "severity": "high",
  "reason": "VirusTotal marked IOC as malicious",
  "ioc_type": "hash",
  "ioc_value": "44d88612fea8a8f36de82e1278abb02f"
}

Create IRIS Ticket (via soc-integrator)

Request:

curl -sS -X POST http://localhost:8088/iris/tickets \
  -H 'Content-Type: application/json' \
  -d '{
    "title": "Suspicious domain detected",
    "description": "Automated ticket from IOC evaluation pipeline",
    "severity": "medium",
    "source_ref": "shuffle-webhook-demo"
  }'

Sample response:

{
  "success": true,
  "ticket_id": 53,
  "case_id": 53,
  "status": "open"
}

9) Why IOC Was Added to SOC Integrator

• To centralize threat-intelligence logic in one API layer instead of duplicating enrichment/evaluation rules across Shuffle workflows and other services.
• To provide a consistent decision contract (enrich for context, evaluate for action/verdict) that downstream automation can trust.
• To improve traceability by storing IOC checks and decisions in soc-integrator history for audit, tuning, and UAT evidence.
• To simplify integrations with multiple intelligence providers (VirusTotal, AbuseIPDB, and future sources) behind one internal interface.
• To reduce workflow complexity in Shuffle so playbooks focus on orchestration (branching, ticketing, notifications) while IOC decisioning stays in backend logic.

10) Sequence Diagram (MermaidJS)

sequenceDiagram
    autonumber
    participant Sim as Log Simulator
    participant Wz as Wazuh
    participant Sh as Shuffle
    participant SI as soc-integrator
    participant VT as VirusTotal/AbuseIPDB
    participant IR as IRIS

    Sim->>Wz: Send FortiGate/Endpoint simulated logs
    Wz->>Wz: Parse + correlate + trigger alert rule
    Wz->>Sh: Trigger workflow (webhook/API)
    Sh->>SI: POST /ioc/enrich (ioc_type, ioc_value)
    SI->>VT: Query IOC intelligence
    VT-->>SI: Enrichment data
    SI-->>Sh: Enrichment result
    Sh->>SI: POST /ioc/evaluate (ioc + enrichment context)
    SI->>SI: Apply decision logic + write ioc_trace
    SI-->>Sh: matched/severity/reason
    alt matched == true
        Sh->>SI: POST /iris/tickets
        SI->>IR: Create ticket/case
        IR-->>SI: ticket_id/case_id
        SI-->>Sh: Ticket creation success
    else matched == false
        Sh-->>Sh: End workflow without ticket
    end

11) SOC Integrator API Inventory

Group	Method	Endpoint	Notes
Core	GET	/health	Service health and target configuration
Core	POST	/ingest/wazuh-alert	Normalize inbound Wazuh alert payload
Core	POST	/action/create-incident	Create PagerDuty incident
Core	POST	/action/trigger-shuffle	Trigger Shuffle workflow execution
Core	POST	/action/create-iris-case	Create IRIS case (legacy action endpoint)
IRIS	POST	/iris/tickets	Create IRIS ticket/case via soc-integrator
IRIS	GET	/iris/tickets	List/query IRIS tickets/cases
IOC	POST	/ioc/enrich	IOC enrichment from configured intel sources
IOC	POST	/ioc/evaluate	IOC decisioning/verdict
IOC	POST	/ioc/upload-file	Upload file to IOC backend (VirusTotal flow)
IOC	GET	/ioc/analysis/{analysis_id}	Retrieve IOC analysis status/result
IOC	POST	/ioc/evaluate-file	Evaluate file indicator or uploaded sample
IOC	GET	/ioc/history	Retrieve stored IOC trace history
Shuffle	GET	/shuffle/health	Shuffle service reachability check
Shuffle	GET	/shuffle/auth-test	Validate Shuffle API key access
Shuffle	POST	/shuffle/login	Login against Shuffle API
Shuffle	POST	/shuffle/generate-apikey	Generate Shuffle API key from credentials
Shuffle	GET	/shuffle/workflows	List workflows
Shuffle	GET	/shuffle/workflows/{workflow_id}	Get workflow detail
Shuffle	POST	/shuffle/workflows/{workflow_id}/execute	Execute specific workflow
Shuffle	GET	/shuffle/apps	List installed/available Shuffle apps
Shuffle	POST	/shuffle/proxy	Generic proxy request to Shuffle API
Wazuh	GET	/sync/wazuh-version	Fetch Wazuh version information
Wazuh	GET	/wazuh/auth-test	Validate Wazuh API authentication
Wazuh	GET	/wazuh/manager-info	Manager information
Wazuh	GET	/wazuh/agents	List Wazuh agents
Wazuh	GET	/wazuh/alerts	Query recent Wazuh alerts
Wazuh	GET	/wazuh/manager-logs	Read manager logs
Wazuh	POST	/wazuh/sync-to-mvp	Sync Wazuh alerts into MVP pipeline
Wazuh	GET	/wazuh/auto-sync/status	Auto-sync loop status
MVP	POST	/mvp/incidents/ingest	Ingest incident into MVP flow
MVP	POST	/mvp/ioc/evaluate	Evaluate IOC under MVP policy
MVP	POST	/mvp/vpn/evaluate	Evaluate VPN event under MVP policy
MVP	GET	/mvp/config/policies	Read MVP policy configuration
MVP	PUT	/mvp/config/policies	Update MVP policy configuration
MVP	GET	/mvp/health/dependencies	Dependency health snapshot

Additional FastAPI-generated endpoints:

• GET /docs
• GET /openapi.json

---

Date: March 4, 2026 Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)

Appendix C (C1-C3) Production Log Mapping Update

This update documents production log sources and required fields for Appendix C detections implemented in soc-integrator.

C1. Impossible Travel Detection

• Use case:
  • C1-01 Impossible Travel
• Primary production log sources:
  • VPN authentication success logs
  • Active Directory / Windows authentication success logs (event_id=4624)
  • Cloud IdP login success logs (Entra/Okta/Google Workspace)
• Required normalized fields:
  • asset.user
  • network.src_ip
  • timestamp
  • login success indicator (payload.success=true or equivalent)
  • geo context (network.country and network.src_lat/src_lon) or GeoIP enrichment from source IP
• Detection logic summary:
  • Compare consecutive successful logins for same user
  • Calculate distance and travel time
  • Trigger when computed travel speed exceeds threshold (c1_max_travel_speed_kmph)

C2. Advanced Credential Abuse & Privilege Misuse

• Use cases:
  • C2-01 Privileged off-hours login
  • C2-02 Dormant account activation
  • C2-03 Service account interactive logon
  • C2-04 Rapid privilege escalation followed by sensitive access
• Primary production log sources:
  • Windows Security logs (4624, 4672, 4728, 4732, 5145)
  • Linux auth/sudo/PAM logs
  • VPN/IdP authentication logs
• Required normalized fields:
  • asset.user, asset.is_admin, asset.is_service
  • payload.logon_type, payload.event_id, payload.action, payload.success
  • network.src_ip, network.dst_host, network.dst_port
  • timestamp

C3. Lateral Movement & Internal Reconnaissance

• Use cases:
  • C3-01 Multi-host authentication success burst
  • C3-02 SMB/RDP lateral movement burst pattern
  • C3-03 Admin account accessing many servers rapidly
  • C3-04 Internal scanning/enumeration burst
• Primary production log sources:
  • Windows authentication and share access logs
  • East-west firewall telemetry
  • IDS/NDR internal movement/scanning alerts
  • Endpoint network telemetry (e.g., Sysmon network events)
• Required normalized fields:
  • asset.user, asset.is_admin
  • network.src_ip, network.dst_host, network.dst_port
  • login success indicator where applicable
  • timestamp

Minimum Windows Event IDs for Initial Rollout

• 4624 Successful logon
• 4672 Special privileges assigned to new logon
• 4728, 4732 Privileged group membership changes
• 5145 Detailed file share access

Implementation Note

• Simulation scripts exist for Appendix C validation and UAT replay:
  • scripts/send-wazuh-proposal-appendix-c-events.sh
• In production, these simulated events are replaced by actual VPN/AD/cloud/endpoint/network telemetry sources listed above.

Appendix C Production Data Onboarding Checklist

Source	Log Path / Channel	Must-Have Fields	Use Cases	Verification Query (Wazuh/Indexer)
VPN Gateway (FortiGate/SSL-VPN)	Syslog export from firewall/VPN device	timestamp, user, src_ip, action/result, event_id (if mapped), country (optional)	C1, C2	full_log:*vpn* AND full_log:*user=*
Active Directory / Windows DC	Windows Security Event Log (agent/forwarder)	event_id, timestamp, user/account, src_ip (where present), logon_type, success/failure	C1, C2, C3	rule.id:* AND (data.win.system.eventID:4624 OR full_log:*event_id=4624*)
Cloud IdP (Entra/Okta/Google)	API export / SIEM connector -> syslog/json	user, src_ip, event_time, outcome, geo.country (if available), app/service	C1, C2	full_log:*source=*idp* OR full_log:*okta* OR full_log:*entra*
Windows Endpoints/Servers	Wazuh agent + Sysmon/Security logs	event_id, user, src_ip, dst_host, dst_port, process/action	C2, C3	full_log:*source=windows* AND rule.id:*
Linux Servers	auth.log / secure / sudo / sshd	timestamp, user, src_ip, action, success	C2, C3	full_log:*sshd* OR full_log:*sudo*
East-West Firewall	Internal traffic logs (allow/deny/flow)	src_ip, dst_ip/dst_host, dst_port, action, timestamp	C3	full_log:*src_ip=* AND full_log:*dst_port=*
IDS/NDR	IDS alerts / network detection logs	src_ip, dst_ip/dst_host, dst_port, signature/category, timestamp	C3	full_log:*scan* OR full_log:*lateral* OR full_log:*enumeration*

Acceptance Checklist (Per Source)

• Parsing/decoder is stable (no malformed key fields in sampled logs)
• Required fields are present and normalized into event model used by soc-integrator
• Timestamp format is valid ISO-8601 after normalization
• Sample events can be found in wazuh-alerts-* within expected ingestion latency
• At least one C-use-case evaluation run confirms source contributes to detection context

---

Date: March 17, 2026 Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)

Incremental Progress Since March 4, 2026

1) IRIS Stack Stabilization

• Fixed Alembic DB revision mismatch caused by DB being initialized from a different branch:
  • Identified correct head revision (d5a720d1b99b) and stamped the DB directly
  • Fixed missing ioc.case_id column (ALTER TABLE ioc ADD COLUMN IF NOT EXISTS case_id ...) skipped by the migration
• Fixed IRIS static file 404 errors (jQuery, Bootstrap, app JS):
  • Root cause: Svelte/Vite frontend (ui/dist) had never been built
  • Fix: ran npm install && npm run build inside the iris-web/ui container to produce dist/
• All IRIS web UI pages now load correctly including case management and KPI dashboard

2) KPI Dashboard (IRIS)

• Added a custom KPI dashboard page to IRIS web (/kpi):
  • Files: iris-web/source/app/blueprints/pages/kpi_dashboard/
  • Registered in Flask app and added to sidebar navigation
  • Frontend JS/CSS: iris-web/ui/src/pages/kpi_dashboard.js, iris-web/ui/src/css/kpi_dashboard.css
• Added KPI test data seeder: scripts/seed-kpi-test-data.py
• Updated manage_cases.html and case.html templates for KPI-related display fields

3) Shuffle Credential Persistence

• Investigated and resolved the Shuffle admin password being unknown after restarts:
  • Password hash is stored in OpenSearch under Password (capital P) as bcrypt ($2a$ prefix for Go compat)
  • Root cause: SHUFFLE_DEFAULT_PASSWORD was blank; password was auto-generated and not saved
  • Fix: set SHUFFLE_DEFAULT_USERNAME, SHUFFLE_DEFAULT_PASSWORD, and SHUFFLE_DEFAULT_APIKEY in Shuffle/.env
  • API key is now fixed across restarts via SHUFFLE_DEFAULT_APIKEY
• Created pass.txt at project root documenting all system credentials (IRIS, Shuffle, Wazuh, SOC Integrator, third-party API keys)

4) SOC Integrator → Shuffle API Fix

• Fixed /shuffle/apps and /shuffle/workflows returning 401:
  • Root cause: container had an old API key baked from a previous creation — docker restart does NOT re-read env_file
  • Fix: generated fresh Shuffle API key via session cookie, updated soc-integrator/.env, and used --force-recreate to pick up the new key
• Both Shuffle proxy endpoints now return 200 from soc-integrator

5) Wazuh Dashboard — Real-Data Filters

• Fixed "Could not locate index-pattern" error on all custom dashboards:
  • All 4 dashboard ndjson files had their own private index-pattern objects that conflicted with the canonical wazuh-alerts-* pattern
  • Fix: removed private index-pattern objects from each ndjson; updated all references[] and searchSourceJSON.indexRefName entries to point to wazuh-alerts-*
• Replaced simulation-era full_log:*usecase_id* filters across all dashboards with production-data filters:
  • Custom rules dashboard: filter changed to rule.groups: soc_prod*
  • Note: rule.id is a keyword field in OpenSearch — range queries are lexicographic, not numeric; ID-based range filtering is unreliable
  • rule.groups: soc_prod* is the correct discriminator — all SOC custom rules carry this group; no built-in Wazuh rules do
  • Required/appendix dashboards: all 51 usecase_id sub-filters remapped to actual rule.id: 110xxx values and rule.groups: appendix_a/b splits
  • Appendix A/B dashboard: top filter full_log:*usecase_id* → rule.groups: soc_prod*; A/B panel splits use rule.groups: appendix_a and rule.groups: appendix_b
  • 7 use cases with no implemented production rule set to rule.id: __no_rule__ (intentional no-match)
• All 4 dashboards imported and rendering correctly against live wazuh-alerts-* data

6) Firewall Syslog Testing

• Added scripts/test-firewall-syslog.py:
  • Sends FortiGate-style syslog UDP to Wazuh port 514 with 10 test scenarios
  • Supports --via-docker flag to preserve source IP through Docker NAT
• Added 7 firewall source IPs to wazuh_manager.conf allowed list
• Enabled logall and logall_json in Wazuh manager config for improved debug visibility

7) Production Log Samples

• Added production log sample files under samples/:
  • samples/appendix-a-production-samples.log
  • samples/appendix-b-production-samples.log
  • samples/appendix-c-production-samples.log
• Added samples/README.md documenting log format and usage

8) Rule Match Evidence (Live Data, 2026-03-14 → 2026-03-17)

• Queried all SOC custom rules against OpenSearch wazuh-alerts-* and generated summary_rule_match.md
• Total events matched today (2026-03-17): 286,931 across 49 implemented rules

Active rules with events (2026-03-17):

Rule	Description	Events
110341	A4-01 Windows privileged account auth failure	1
110342	A4-02 Windows service account auth failure	46
110354	A4-13 Windows DC DSRM password set (4794)	285,769 ⚠️
110359	A4-19 Windows authentication failure (4625)	55

• ⚠️ Rule 110354 (DSRM / event 4794 via parent 60103) accounts for 99.6% of event volume — under investigation to confirm parent SID scope
• A1, A2, B2, C1 rules that fired on earlier dates (Mar 14–16) had 0 events today — likely simulator runs that have since stopped
• Log sources not yet forwarding: FortiGate syslog (A1/A2), FortiGate VPN (A3), VMware (B1), Sysmon/endpoints (B3), Windows 4624 auth success (C2/C3)

9) Tooling and Documentation Updates

• run-combined-stack.sh: added dedup command (fixed missing elif branch) and recreate command
• README.md: full rewrite covering all stack commands, KPI dashboard, current endpoint list, and macOS bind-mount note
• scripts/README.md: documented test-firewall-syslog.py, seed-kpi-test-data.py, and new dashboard ndjson files
• docs/wazuh-decoders-rules.md: documentation for custom decoder/rule structure

---

Date: March 4, 2026 Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)

Git Diff Progress Summary (Base: 0de071e -> Head: 5e215c0)

Diff Snapshot

• Base commit: 0de071e7c9327c8c9135f0c15cf80c31c9b2e59a
• Head commit: 5e215c0
• Net change: 40 files changed, 6083 insertions(+), 108 deletions(-)

Major Progress Areas

1. SOC Integrator Expansion
2. Added full admin UI stack:
   • soc-integrator/app/ui/index.html
   • soc-integrator/app/ui/assets/app.js
   • soc-integrator/app/ui/assets/styles.css
3. Added Appendix C correlation/detection service:
   • soc-integrator/app/services/c_detection_service.py
4. Extended API/data layers for monitoring, simulation control, IOC, and detection history:
   • soc-integrator/app/main.py
   • soc-integrator/app/models.py
   • soc-integrator/app/repositories/mvp_repo.py
5. Added GeoIP adapter integration:
   • soc-integrator/app/adapters/geoip.py
6. Wazuh Simulation and Dashboard Delivery
7. Added Appendix-specific event generators:
   • scripts/send-wazuh-proposal-appendix-b-events.sh
   • scripts/send-wazuh-proposal-appendix-c-events.sh
8. Added dashboard artifacts/import pipeline:
   • scripts/events/*.ndjson
   • scripts/import-wazuh-dashboard.sh
9. Added Wazuh custom decoder/rules artifacts for proposal scenarios:
   • wazuh-docker/single-node/config/wazuh_cluster/local_decoder.xml
   • wazuh-docker/single-node/config/wazuh_cluster/local_rules.xml
   • wazuh-docker/single-node/config/wazuh_cluster/rules/soc-*.xml

Wazuh Custom Rules Added (Current Active Set)

Active custom rules are currently defined in:

• wazuh-docker/single-node/config/wazuh_cluster/local_rules.xml

Rule groups/ranges implemented:

• Base and appendix classifiers:
  • 100200: base marker for synthetic SOC events (soc_mvp_test=true)
  • 100210: Appendix A classifier
  • 100220: Appendix B classifier
  • 100230: Appendix C classifier
• Appendix A:
  • A1 IOC/DNS: 100301-100302
  • A2 FortiGate firewall/IPS/IDS: 100311-100320
  • A3 VPN anomalies: 100331-100335
  • A4 Windows/AD behaviors: 100341-100364
• Appendix B:
  • B1 VMware/vCenter/ESXi: 100401-100403
  • B2 Log-loss monitor signal: 100411
  • B3 Sysmon-focused detections: 100421-100426
• Appendix C (implemented scope C1-C3):
  • C1 Impossible travel: 100501
  • C2 Credential abuse/privilege misuse: 100511-100514
  • C3 Lateral movement/internal recon: 100521-100524

Operational note:

• Split rule files under wazuh_cluster/rules/soc-*.xml exist as staging artifacts in this workspace; active detection content is loaded from local_rules.xml.
• Operations and Runtime Hardening
• Updated orchestration and runtime configuration:
  • run-combined-stack.sh
  • compose-overrides/soc-integrator.yml
  • soc-integrator/Dockerfile
  • soc-integrator/.env.example

Documentation Progress Included in This Range

• Added/updated proposal revision document:
  • Security Detection & Threat Intelligence Enhancement Proposal-revise.md
• Expanded progress log coverage in this file (progress-update.md) including:
  • Appendix C (C1-C3) production log mapping
  • Production data onboarding checklist
  • Acceptance criteria for source onboarding and validation