# Project Progress Update
Date: February 13, 2026
Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)
## 1) Executive Summary
The MVP platform is operational and running end-to-end in the lab environment.
Core integrations are in place:
- Detection: Wazuh
- Automation: Shuffle
- Case management: IRIS-web (replacing DFIRTrack)
- Escalation (MVP): PagerDuty Stub
- Orchestration/API layer: soc-integrator
All major containers are currently up, and key health checks are passing.
## 2) Completed Work
### Platform orchestration and operations
- Combined stack runner created and improved (`run-combined-stack.sh`)
- Added command support for:
- `up`, `down`, `logs`, `status`, `help`
- per-target control (`wazuh`, `iris`, `shuffle`, `pagerduty`, `integrator`)
- Added consolidated health/status script (`soc-status.sh`)
### Integration architecture
- Connected Wazuh, Shuffle, IRIS-web, PagerDuty Stub, and soc-integrator on shared network
- Resolved startup conflicts and runtime issues (port, compose, routing compatibility)
### SOC Integrator (MVP)
- Added/validated integration APIs for:
- Wazuh
- Shuffle
- IRIS-web
- PagerDuty Stub
- Implemented MVP orchestration endpoints:
- `POST /mvp/incidents/ingest`
- `POST /mvp/ioc/evaluate`
- `POST /mvp/vpn/evaluate`
- `GET /mvp/config/policies`
- `PUT /mvp/config/policies`
- `GET /mvp/health/dependencies`
- Added internal API-key protection for mutation endpoints
### Persistence layer
- Added PostgreSQL service for soc-integrator (`soc-integrator-db`)
- Added incident/policy/audit schema and startup initialization
- Enabled deduplication and audit tracking for incident processing
### Testing utilities and documentation
- Added Wazuh test-event injection script:
- `scripts/send-wazuh-test-events.sh`
- Added root project docs:
- `README.md`
- Added root ignore rules:
- `.gitignore`
## 3) Current Live Status (Lab)
Current stack status: **UP**
Healthy/available components:
- Wazuh manager, indexer, dashboard
- IRIS-web app/nginx/worker/db/rabbitmq
- Shuffle backend/frontend/opensearch/orborus
- PagerDuty Stub
- soc-integrator + soc-integrator-db
Endpoint checks:
- Wazuh Dashboard: OK
- Wazuh API: OK (auth-protected, expected 401 on unauthenticated root)
- IRIS Web: OK
- Shuffle Frontend: OK
- Shuffle Backend: reachable
- Shuffle OpenSearch: reachable (auth-protected)
- PagerDuty Stub: OK
- soc-integrator `/health`: OK
## 4) System Architecture Diagram (PlantUML)
```plantuml
@startuml
title FoodProject SOC Platform - System Architecture (MVP)
skinparam componentStyle rectangle
actor "Analyst" as analyst
cloud "External Log Sources" as logs
rectangle "SOC Shared Docker Network" {
node "Wazuh Stack" as wazuh {
component "Wazuh Manager" as wazuh_mgr
component "Wazuh Indexer" as wazuh_idx
component "Wazuh Dashboard" as wazuh_dash
}
node "Shuffle Stack" as shuffle {
component "Shuffle Frontend" as shuf_fe
component "Shuffle Backend" as shuf_be
component "Shuffle Orborus" as shuf_orb
component "Shuffle OpenSearch" as shuf_os
}
node "IRIS-web Stack" as iris {
component "IRIS Web App" as iris_app
database "IRIS DB" as iris_db
component "IRIS RabbitMQ" as iris_mq
}
node "SOC Integrator Stack" as integ {
component "soc-integrator API" as soc_api
database "soc-integrator-db" as soc_db
}
component "PagerDuty Stub" as pd_stub
}
logs --> wazuh_mgr : Security events
wazuh_mgr --> wazuh_idx : Index alerts
analyst --> wazuh_dash : Investigate alerts
wazuh_dash --> wazuh_idx : Query data
wazuh_mgr --> soc_api : Alert/incident input
soc_api --> soc_db : Persist incidents\npolicies\naudit
soc_api --> iris_app : Create/update cases
soc_api --> pd_stub : Escalation (MVP)
soc_api --> shuf_be : Trigger automation
shuf_fe --> shuf_be : UI/API
shuf_be --> shuf_os : Read/write workflow data
shuf_orb --> shuf_be : Execution queue polling
shuf_orb --> shuf_os : Workflow state interactions
iris_app --> iris_db : Case data
iris_app --> iris_mq : Async jobs
@enduml
```
## 5) In Progress / Remaining for Customer UAT
1. Detection content tuning
- Fine-tune Wazuh rules/decoders for customer log patterns and false-positive reduction
2. Use-case calibration
- Validate risk/severity mapping per approved use cases
- Tune exception list and threshold logic (especially VPN geo anomaly)
3. UAT evidence package
- Capture deterministic UAT scenarios and outputs for:
- IOC flow
- VPN outside-TH flow
- IRIS case creation/update
- PagerDuty Stub escalation path
4. Production hardening items
- Rotate default/local secrets used in lab config
- Lock down internal API keys and access boundaries
## 6) Risks / Notes
- Current escalation target is **PagerDuty Stub** by design for MVP.
Real PagerDuty production integration is the next stage.
- Some Wazuh config certificate directories are root-owned in the local lab clone, which may affect local git add operations if not excluded/fixed.
## 7) Next Milestone (Proposed)
Next milestone: **MVP UAT Completion**
Target outputs:
- Approved UAT checklist execution
- Tuned policy thresholds for customer environment
- Signed-off incident lifecycle flow:
Wazuh event -> soc-integrator decision -> IRIS case -> PagerDuty Stub escalation
---
Date: February 26, 2026
Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)
## Incremental Progress Since February 13, 2026
### 1) IOC Enrichment and Evaluation
- Added IOC APIs in `soc-integrator`:
- `POST /ioc/enrich`
- `POST /ioc/evaluate`
- `GET /ioc/history`
- `POST /ioc/upload-file`
- `POST /ioc/evaluate-file`
- `GET /ioc/analysis/{analysis_id}`
- Integrated VirusTotal adapter for domain/hash/file intelligence and analysis lookups.
- Integrated AbuseIPDB adapter for IP reputation checks.
- Added IOC trace persistence (`ioc_trace`) and repository methods for audit/history.
### 2) IRIS Integration Enhancements
- Added IRIS ticket APIs in `soc-integrator`:
- `POST /iris/tickets`
- `GET /iris/tickets`
- Updated IRIS API key in environment and verified ticket creation path via API.
- Added demo data seeding script:
- `scripts/seed-iris-demo-data.sh`
### 3) Shuffle Workflow Automation
- Created and updated sample Shuffle workflow assets for webhook-driven IRIS ticket creation:
- `shuffle-workflows/sample-webhook-soc-integrator-iris-workflow.json`
- `shuffle-workflows/sample-webhook-soc-integrator-iris-workflow.md`
- Added workflow update helper script:
- `scripts/update-shuffle-workflow-from-template.sh`
- Updated target workflow (`07ecad05-ff68-41cb-888d-96d1a8e8db4b`) with:
- webhook trigger
- HTTP action (`http 1.4.0`) to call `soc-integrator` ticket API
- tested webhook execution path to successful completion
### 4) Networking and Runtime Fixes
- Resolved Shuffle action DNS failure to `soc-integrator` by attaching `soc-integrator` service to Shuffle execution network(s) in:
- `compose-overrides/soc-integrator.yml`
- Verified connectivity from Shuffle execution context to:
- `http://soc-integrator:8080/health`
### 5) Security and Repository Hygiene
- Added `.env` and `.env.*` to root `.gitignore` (kept `.env.example` tracked).
- Removed tracked env files from git cache to prevent secret leakage.
- Updated operational API keys in `soc-integrator/.env` for Shuffle, IRIS, VirusTotal, and AbuseIPDB.
### 6) Current Status (Lab)
- `soc-integrator` health endpoint: reachable.
- IOC enrich/evaluate flows: operational for domain/hash and file submission paths.
- Shuffle webhook-to-IRIS automation: operational after network fix.
- Core stack components remain available for continued UAT and tuning.
### 7) Simulation Logs Workstream
#### Completed
- Added FortiGate simulation coverage for multiple models:
- 40F
- 60F
- 80F
- 501E
- Added endpoint agent simulation coverage for:
- Windows clients
- macOS clients
- Linux clients
- Added continuous run mode (`--forever`) to simulation scripts for long-running lab traffic generation.
- Extended script set to support realistic event streams for Wazuh ingestion and rule validation.
#### Operational scripts
- `scripts/send-wazuh-test-events.sh`
---
Date: March 4, 2026
Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)
## Incremental Progress Since Previous Update (March 4, 2026)
### 1) Production-Profile Simulator Payloads
- Added production payload mode to proposal simulators:
- `scripts/send-wazuh-proposal-required-events.sh`
- `scripts/send-wazuh-proposal-appendix-b-events.sh`
- New argument:
- `--profile=simulation|production` (default remains `simulation`)
- In `production` profile, simulator messages omit `section/usecase_id/usecase` markers and emit production-like key/value fields to support real parser/decoder testing.
### 2) Wazuh Normalize API Improvements
- Enhanced `POST /ingest/wazuh-alert` in `soc-integrator`:
- returns both legacy normalized shape and SOC normalized event shape.
- Added `GET /ingest/wazuh-alert/samples` with practical sample request/response cases for:
- DNS IOC
- VMware auth
- Windows Sysmon
- C1 impossible travel
### 3) C1 Normalization (Production-First)
- Updated C1 normalization logic in:
- `soc-integrator/app/services/mvp_service.py`
- C1 now maps from production characteristics (not only simulator markers):
- identity/vpn source context
- successful login/auth indicator
- user + source IP present
- geo context present (`country` and/or `src_lat/src_lon`)
- Legacy `section/usecase_id` C1 markers are kept as fallback for backward compatibility.
### 4) Current Validation Status
- Production-profile simulator events are being sent to Wazuh successfully.
- Some runs still show only base-rule matching during verification due to current Wazuh manager runtime instability in lab (intermittent restart/init issues), which affects deterministic decoder/rule validation windows.
- Next validation step after stable manager window:
- re-run production-profile A1/B1/B2/B3 batches
- confirm `110xxx` production rules with consistent hit evidence.
### 5) Mermaid Diagram: C1-C3 Normalization Flow (SOC Integrator)
```mermaid
flowchart LR
A[Wazuh Raw Alert
full_log + rule + agent] --> B[POST /wazuh/sync-to-mvp
or /ingest/wazuh-alert]
B --> C[mvp_service.normalize_wazuh_hit]
C --> D[KV parse from full_log
src_ip user country event_type]
D --> E[Normalized Event Schema
source event_type timestamp severity
asset network payload risk_context]
E --> F{C-Detection Evaluate}
F --> C1[C1 Impossible Travel
geo context + success login + user/src_ip]
F --> C2[C2 Credential Abuse
off-hours / dormant / service interactive / rapid privilege]
F --> C3[C3 Lateral Movement
multi-host auth / SMB-RDP burst / internal scan]
C1 --> G[Persist c_detection_events]
C2 --> G
C3 --> G
G --> H[Optional Incident Pipeline
IRIS case + Shuffle + PagerDuty stub]
```
- `scripts/send-wazuh-endpoint-agent-test-events.sh`
- additional simulation scripts under `scripts/` for firewall and endpoint scenarios with continuous mode enabled
#### Detection alignment status
- Simulation work has been aligned to the detection objectives documented in:
- `Security Detection & Threat Intelligence Enhancement Proposal-2.md`
- Proposal use-case mapping explicitly covered in simulation:
- **A1. DNS / Firewall (IOC)**:
- DNS network communication to malicious domain
- DNS/Firewall malicious domain IOC detection events
- **A2. FortiGate IPS/IDS & Firewall**:
- allowed RDP from public IP
- admin password change
- create/add admin account
- disable email notification
- config download
- multiple critical/high IDS alerts
- port scanning (public/private source variants)
- IOC detection and communication to malicious IP
- **A3. FortiGate VPN**:
- authentication success from guest account
- authentication success from multiple countries
- brute-force success pattern
- multiple fail patterns (many accounts from one source)
- authentication success from outside Thailand
- **A4. Windows / Active Directory**:
- privileged/service account authentication failures
- password spray and multi-source fail patterns
- success from public IP / guest account
- pass-the-hash style success indicators
- account/group privilege change and account lifecycle events (create/re-enable)
- AD enumeration behavior indicators
- Endpoint client simulations were added to complement proposal scope for heterogeneous environments:
- Windows agent events
- macOS agent events
- Linux agent events
- Current use is suitable for pipeline and workflow validation (ingest -> detect -> automate -> case creation).
- Remaining work is focused on fine-grained scenario calibration:
- event frequency tuning
- field/value realism per source
- expected alert volume by use case for cleaner UAT evidence
### 8) API Request/Response Samples
#### IOC Enrich
Request:
```bash
curl -sS -X POST http://localhost:8088/ioc/enrich \
-H 'Content-Type: application/json' \
-d '{
"ioc_type": "domain",
"ioc_value": "google.com",
"sources": ["virustotal"]
}'
```
Sample response:
```json
{
"success": true,
"ioc_type": "domain",
"ioc_value": "google.com",
"enrichment": {
"virustotal": {
"reputation": 120,
"last_analysis_stats": {
"malicious": 0,
"suspicious": 0,
"harmless": 90
}
}
}
}
```
#### IOC Evaluate
Request:
```bash
curl -sS -X POST http://localhost:8088/ioc/evaluate \
-H 'Content-Type: application/json' \
-d '{
"ioc_type": "hash",
"ioc_value": "44d88612fea8a8f36de82e1278abb02f",
"sources": ["virustotal"]
}'
```
Sample response:
```json
{
"success": true,
"matched": true,
"severity": "high",
"reason": "VirusTotal marked IOC as malicious",
"ioc_type": "hash",
"ioc_value": "44d88612fea8a8f36de82e1278abb02f"
}
```
#### Create IRIS Ticket (via soc-integrator)
Request:
```bash
curl -sS -X POST http://localhost:8088/iris/tickets \
-H 'Content-Type: application/json' \
-d '{
"title": "Suspicious domain detected",
"description": "Automated ticket from IOC evaluation pipeline",
"severity": "medium",
"source_ref": "shuffle-webhook-demo"
}'
```
Sample response:
```json
{
"success": true,
"ticket_id": 53,
"case_id": 53,
"status": "open"
}
```
### 9) Why IOC Was Added to SOC Integrator
- To centralize threat-intelligence logic in one API layer instead of duplicating enrichment/evaluation rules across Shuffle workflows and other services.
- To provide a consistent decision contract (`enrich` for context, `evaluate` for action/verdict) that downstream automation can trust.
- To improve traceability by storing IOC checks and decisions in `soc-integrator` history for audit, tuning, and UAT evidence.
- To simplify integrations with multiple intelligence providers (VirusTotal, AbuseIPDB, and future sources) behind one internal interface.
- To reduce workflow complexity in Shuffle so playbooks focus on orchestration (branching, ticketing, notifications) while IOC decisioning stays in backend logic.
### 10) Sequence Diagram (MermaidJS)
```mermaid
sequenceDiagram
autonumber
participant Sim as Log Simulator
participant Wz as Wazuh
participant Sh as Shuffle
participant SI as soc-integrator
participant VT as VirusTotal/AbuseIPDB
participant IR as IRIS
Sim->>Wz: Send FortiGate/Endpoint simulated logs
Wz->>Wz: Parse + correlate + trigger alert rule
Wz->>Sh: Trigger workflow (webhook/API)
Sh->>SI: POST /ioc/enrich (ioc_type, ioc_value)
SI->>VT: Query IOC intelligence
VT-->>SI: Enrichment data
SI-->>Sh: Enrichment result
Sh->>SI: POST /ioc/evaluate (ioc + enrichment context)
SI->>SI: Apply decision logic + write ioc_trace
SI-->>Sh: matched/severity/reason
alt matched == true
Sh->>SI: POST /iris/tickets
SI->>IR: Create ticket/case
IR-->>SI: ticket_id/case_id
SI-->>Sh: Ticket creation success
else matched == false
Sh-->>Sh: End workflow without ticket
end
```
### 11) SOC Integrator API Inventory
| Group | Method | Endpoint | Notes |
| ------- | ------ | ------------------------------------------ | -------------------------------------------- |
| Core | GET | `/health` | Service health and target configuration |
| Core | POST | `/ingest/wazuh-alert` | Normalize inbound Wazuh alert payload |
| Core | POST | `/action/create-incident` | Create PagerDuty incident |
| Core | POST | `/action/trigger-shuffle` | Trigger Shuffle workflow execution |
| Core | POST | `/action/create-iris-case` | Create IRIS case (legacy action endpoint) |
| IRIS | POST | `/iris/tickets` | Create IRIS ticket/case via soc-integrator |
| IRIS | GET | `/iris/tickets` | List/query IRIS tickets/cases |
| IOC | POST | `/ioc/enrich` | IOC enrichment from configured intel sources |
| IOC | POST | `/ioc/evaluate` | IOC decisioning/verdict |
| IOC | POST | `/ioc/upload-file` | Upload file to IOC backend (VirusTotal flow) |
| IOC | GET | `/ioc/analysis/{analysis_id}` | Retrieve IOC analysis status/result |
| IOC | POST | `/ioc/evaluate-file` | Evaluate file indicator or uploaded sample |
| IOC | GET | `/ioc/history` | Retrieve stored IOC trace history |
| Shuffle | GET | `/shuffle/health` | Shuffle service reachability check |
| Shuffle | GET | `/shuffle/auth-test` | Validate Shuffle API key access |
| Shuffle | POST | `/shuffle/login` | Login against Shuffle API |
| Shuffle | POST | `/shuffle/generate-apikey` | Generate Shuffle API key from credentials |
| Shuffle | GET | `/shuffle/workflows` | List workflows |
| Shuffle | GET | `/shuffle/workflows/{workflow_id}` | Get workflow detail |
| Shuffle | POST | `/shuffle/workflows/{workflow_id}/execute` | Execute specific workflow |
| Shuffle | GET | `/shuffle/apps` | List installed/available Shuffle apps |
| Shuffle | POST | `/shuffle/proxy` | Generic proxy request to Shuffle API |
| Wazuh | GET | `/sync/wazuh-version` | Fetch Wazuh version information |
| Wazuh | GET | `/wazuh/auth-test` | Validate Wazuh API authentication |
| Wazuh | GET | `/wazuh/manager-info` | Manager information |
| Wazuh | GET | `/wazuh/agents` | List Wazuh agents |
| Wazuh | GET | `/wazuh/alerts` | Query recent Wazuh alerts |
| Wazuh | GET | `/wazuh/manager-logs` | Read manager logs |
| Wazuh | POST | `/wazuh/sync-to-mvp` | Sync Wazuh alerts into MVP pipeline |
| Wazuh | GET | `/wazuh/auto-sync/status` | Auto-sync loop status |
| MVP | POST | `/mvp/incidents/ingest` | Ingest incident into MVP flow |
| MVP | POST | `/mvp/ioc/evaluate` | Evaluate IOC under MVP policy |
| MVP | POST | `/mvp/vpn/evaluate` | Evaluate VPN event under MVP policy |
| MVP | GET | `/mvp/config/policies` | Read MVP policy configuration |
| MVP | PUT | `/mvp/config/policies` | Update MVP policy configuration |
| MVP | GET | `/mvp/health/dependencies` | Dependency health snapshot |
Additional FastAPI-generated endpoints:
- `GET /docs`
- `GET /openapi.json`
---
Date: March 4, 2026
Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)
## Appendix C (C1-C3) Production Log Mapping Update
This update documents production log sources and required fields for Appendix C detections implemented in `soc-integrator`.
### C1. Impossible Travel Detection
- Use case:
- `C1-01` Impossible Travel
- Primary production log sources:
- VPN authentication success logs
- Active Directory / Windows authentication success logs (`event_id=4624`)
- Cloud IdP login success logs (Entra/Okta/Google Workspace)
- Required normalized fields:
- `asset.user`
- `network.src_ip`
- `timestamp`
- login success indicator (`payload.success=true` or equivalent)
- geo context (`network.country` and `network.src_lat/src_lon`) or GeoIP enrichment from source IP
- Detection logic summary:
- Compare consecutive successful logins for same user
- Calculate distance and travel time
- Trigger when computed travel speed exceeds threshold (`c1_max_travel_speed_kmph`)
### C2. Advanced Credential Abuse & Privilege Misuse
- Use cases:
- `C2-01` Privileged off-hours login
- `C2-02` Dormant account activation
- `C2-03` Service account interactive logon
- `C2-04` Rapid privilege escalation followed by sensitive access
- Primary production log sources:
- Windows Security logs (`4624`, `4672`, `4728`, `4732`, `5145`)
- Linux auth/sudo/PAM logs
- VPN/IdP authentication logs
- Required normalized fields:
- `asset.user`, `asset.is_admin`, `asset.is_service`
- `payload.logon_type`, `payload.event_id`, `payload.action`, `payload.success`
- `network.src_ip`, `network.dst_host`, `network.dst_port`
- `timestamp`
### C3. Lateral Movement & Internal Reconnaissance
- Use cases:
- `C3-01` Multi-host authentication success burst
- `C3-02` SMB/RDP lateral movement burst pattern
- `C3-03` Admin account accessing many servers rapidly
- `C3-04` Internal scanning/enumeration burst
- Primary production log sources:
- Windows authentication and share access logs
- East-west firewall telemetry
- IDS/NDR internal movement/scanning alerts
- Endpoint network telemetry (e.g., Sysmon network events)
- Required normalized fields:
- `asset.user`, `asset.is_admin`
- `network.src_ip`, `network.dst_host`, `network.dst_port`
- login success indicator where applicable
- `timestamp`
### Minimum Windows Event IDs for Initial Rollout
- `4624` Successful logon
- `4672` Special privileges assigned to new logon
- `4728`, `4732` Privileged group membership changes
- `5145` Detailed file share access
### Implementation Note
- Simulation scripts exist for Appendix C validation and UAT replay:
- `scripts/send-wazuh-proposal-appendix-c-events.sh`
- In production, these simulated events are replaced by actual VPN/AD/cloud/endpoint/network telemetry sources listed above.
### Appendix C Production Data Onboarding Checklist
| Source | Log Path / Channel | Must-Have Fields | Use Cases | Verification Query (Wazuh/Indexer) |
| ------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------- |
| VPN Gateway (FortiGate/SSL-VPN) | Syslog export from firewall/VPN device | `timestamp`, `user`, `src_ip`, `action/result`, `event_id` (if mapped), `country` (optional) | C1, C2 | `full_log:*vpn* AND full_log:*user=*` |
| Active Directory / Windows DC | Windows Security Event Log (agent/forwarder) | `event_id`, `timestamp`, `user/account`, `src_ip` (where present), `logon_type`, `success/failure` | C1, C2, C3 | `rule.id:* AND (data.win.system.eventID:4624 OR full_log:*event_id=4624*)` |
| Cloud IdP (Entra/Okta/Google) | API export / SIEM connector -> syslog/json | `user`, `src_ip`, `event_time`, `outcome`, `geo.country` (if available), `app/service` | C1, C2 | `full_log:*source=*idp* OR full_log:*okta* OR full_log:*entra*` |
| Windows Endpoints/Servers | Wazuh agent + Sysmon/Security logs | `event_id`, `user`, `src_ip`, `dst_host`, `dst_port`, `process/action` | C2, C3 | `full_log:*source=windows* AND rule.id:*` |
| Linux Servers | auth.log / secure / sudo / sshd | `timestamp`, `user`, `src_ip`, `action`, `success` | C2, C3 | `full_log:*sshd* OR full_log:*sudo*` |
| East-West Firewall | Internal traffic logs (allow/deny/flow) | `src_ip`, `dst_ip/dst_host`, `dst_port`, `action`, `timestamp` | C3 | `full_log:*src_ip=* AND full_log:*dst_port=*` |
| IDS/NDR | IDS alerts / network detection logs | `src_ip`, `dst_ip/dst_host`, `dst_port`, `signature/category`, `timestamp` | C3 | `full_log:*scan* OR full_log:*lateral* OR full_log:*enumeration*` |
#### Acceptance Checklist (Per Source)
- Parsing/decoder is stable (no malformed key fields in sampled logs)
- Required fields are present and normalized into event model used by `soc-integrator`
- Timestamp format is valid ISO-8601 after normalization
- Sample events can be found in `wazuh-alerts-*` within expected ingestion latency
- At least one C-use-case evaluation run confirms source contributes to detection context
---
Date: March 17, 2026
Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)
## Incremental Progress Since March 4, 2026
### 1) IRIS Stack Stabilization
- Fixed Alembic DB revision mismatch caused by DB being initialized from a different branch:
- Identified correct head revision (`d5a720d1b99b`) and stamped the DB directly
- Fixed missing `ioc.case_id` column (`ALTER TABLE ioc ADD COLUMN IF NOT EXISTS case_id ...`) skipped by the migration
- Fixed IRIS static file 404 errors (jQuery, Bootstrap, app JS):
- Root cause: Svelte/Vite frontend (`ui/dist`) had never been built
- Fix: ran `npm install && npm run build` inside the `iris-web/ui` container to produce `dist/`
- All IRIS web UI pages now load correctly including case management and KPI dashboard
### 2) KPI Dashboard (IRIS)
- Added a custom KPI dashboard page to IRIS web (`/kpi`):
- Files: `iris-web/source/app/blueprints/pages/kpi_dashboard/`
- Registered in Flask app and added to sidebar navigation
- Frontend JS/CSS: `iris-web/ui/src/pages/kpi_dashboard.js`, `iris-web/ui/src/css/kpi_dashboard.css`
- Added KPI test data seeder: `scripts/seed-kpi-test-data.py`
- Updated `manage_cases.html` and `case.html` templates for KPI-related display fields
### 3) Shuffle Credential Persistence
- Investigated and resolved the Shuffle admin password being unknown after restarts:
- Password hash is stored in OpenSearch under `Password` (capital P) as bcrypt (`$2a$` prefix for Go compat)
- Root cause: `SHUFFLE_DEFAULT_PASSWORD` was blank; password was auto-generated and not saved
- Fix: set `SHUFFLE_DEFAULT_USERNAME`, `SHUFFLE_DEFAULT_PASSWORD`, and `SHUFFLE_DEFAULT_APIKEY` in `Shuffle/.env`
- API key is now fixed across restarts via `SHUFFLE_DEFAULT_APIKEY`
- Created `pass.txt` at project root documenting all system credentials (IRIS, Shuffle, Wazuh, SOC Integrator, third-party API keys)
### 4) SOC Integrator → Shuffle API Fix
- Fixed `/shuffle/apps` and `/shuffle/workflows` returning 401:
- Root cause: container had an old API key baked from a previous creation — `docker restart` does NOT re-read `env_file`
- Fix: generated fresh Shuffle API key via session cookie, updated `soc-integrator/.env`, and used `--force-recreate` to pick up the new key
- Both Shuffle proxy endpoints now return 200 from `soc-integrator`
### 5) Wazuh Dashboard — Real-Data Filters
- Fixed "Could not locate index-pattern" error on all custom dashboards:
- All 4 dashboard ndjson files had their own private index-pattern objects that conflicted with the canonical `wazuh-alerts-*` pattern
- Fix: removed private index-pattern objects from each ndjson; updated all `references[]` and `searchSourceJSON.indexRefName` entries to point to `wazuh-alerts-*`
- Replaced simulation-era `full_log:*usecase_id*` filters across all dashboards with production-data filters:
- **Custom rules dashboard**: filter changed to `rule.groups: soc_prod*`
- Note: `rule.id` is a `keyword` field in OpenSearch — range queries are lexicographic, not numeric; ID-based range filtering is unreliable
- `rule.groups: soc_prod*` is the correct discriminator — all SOC custom rules carry this group; no built-in Wazuh rules do
- **Required/appendix dashboards**: all 51 usecase_id sub-filters remapped to actual `rule.id: 110xxx` values and `rule.groups: appendix_a/b` splits
- **Appendix A/B dashboard**: top filter `full_log:*usecase_id*` → `rule.groups: soc_prod*`; A/B panel splits use `rule.groups: appendix_a` and `rule.groups: appendix_b`
- 7 use cases with no implemented production rule set to `rule.id: __no_rule__` (intentional no-match)
- All 4 dashboards imported and rendering correctly against live `wazuh-alerts-*` data
### 6) Firewall Syslog Testing
- Added `scripts/test-firewall-syslog.py`:
- Sends FortiGate-style syslog UDP to Wazuh port 514 with 10 test scenarios
- Supports `--via-docker` flag to preserve source IP through Docker NAT
- Added 7 firewall source IPs to `wazuh_manager.conf` allowed list
- Enabled `logall` and `logall_json` in Wazuh manager config for improved debug visibility
### 7) Production Log Samples
- Added production log sample files under `samples/`:
- `samples/appendix-a-production-samples.log`
- `samples/appendix-b-production-samples.log`
- `samples/appendix-c-production-samples.log`
- Added `samples/README.md` documenting log format and usage
### 8) Rule Match Evidence (Live Data, 2026-03-14 → 2026-03-17)
- Queried all SOC custom rules against OpenSearch `wazuh-alerts-*` and generated `summary_rule_match.md`
- Total meaningful events today (2026-03-17, post rule-fix): **199** across 49 implemented rules
**Rule 110354 bug found and fixed:**
- Root cause: parent SID 60103 is "Windows audit success event" — matches ALL `AUDIT_SUCCESS` events, not just 4794
- Rule had no `eventID` constraint → was firing ~313,000 times/day on events like 4624, 4634, 4688, 4793
- Fix: added `^4794$` to rule 110354 in `soc-a4-windows-ad-rules.xml`
- Rule is now silent (0 events post-fix); fix applied directly in running container and in source file
Active rules with events today (2026-03-17, post-fix):
| Rule | Description | Events |
|------|-------------|--------|
| 110359 | A4-19 Windows auth failure (4625) general | 71 |
| 110342 | A4-02 Service account auth failure (4625) | 50 |
| 110348 | A4-08 NTLM logon type 3 — pass-the-hash indicator | 46 |
| 110523 | C3-03 Admin account auth success — lateral movement candidate | 23 |
| 110522 | C3-02 SMB network logon type 3 | 8 |
| 110341 | A4-01 Privileged account auth failure | 1 |
| 110354 | A4-13 DSRM password set (4794) | 0 ✅ fixed |
- Rules 110348, 110522, 110523 were previously masked by the 110354 false-positive flood — now visible with correct counts
- A1, A2, B2, C1 rules had 0 events today (A1/A2/C1 events from Mar 14–16 were likely simulator runs)
- Log sources not yet forwarding: FortiGate syslog (A1/A2), FortiGate VPN (A3), VMware (B1), Sysmon/endpoints (B3)
### 9) Tooling and Documentation Updates
- `run-combined-stack.sh`: added `dedup` command (fixed missing `elif` branch) and `recreate` command
- `README.md`: full rewrite covering all stack commands, KPI dashboard, current endpoint list, and macOS bind-mount note
- `scripts/README.md`: documented `test-firewall-syslog.py`, `seed-kpi-test-data.py`, and new dashboard ndjson files
- `docs/wazuh-decoders-rules.md`: documentation for custom decoder/rule structure
---
Date: March 4, 2026
Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)
## Git Diff Progress Summary (Base: `0de071e` -> Head: `5e215c0`)
### Diff Snapshot
- Base commit: `0de071e7c9327c8c9135f0c15cf80c31c9b2e59a`
- Head commit: `5e215c0`
- Net change: **40 files changed, 6083 insertions(+), 108 deletions(-)**
### Major Progress Areas
1. SOC Integrator Expansion
- Added full admin UI stack:
- `soc-integrator/app/ui/index.html`
- `soc-integrator/app/ui/assets/app.js`
- `soc-integrator/app/ui/assets/styles.css`
- Added Appendix C correlation/detection service:
- `soc-integrator/app/services/c_detection_service.py`
- Extended API/data layers for monitoring, simulation control, IOC, and detection history:
- `soc-integrator/app/main.py`
- `soc-integrator/app/models.py`
- `soc-integrator/app/repositories/mvp_repo.py`
- Added GeoIP adapter integration:
- `soc-integrator/app/adapters/geoip.py`
2. Wazuh Simulation and Dashboard Delivery
- Added Appendix-specific event generators:
- `scripts/send-wazuh-proposal-appendix-b-events.sh`
- `scripts/send-wazuh-proposal-appendix-c-events.sh`
- Added dashboard artifacts/import pipeline:
- `scripts/events/*.ndjson`
- `scripts/import-wazuh-dashboard.sh`
- Added Wazuh custom decoder/rules artifacts for proposal scenarios:
- `wazuh-docker/single-node/config/wazuh_cluster/local_decoder.xml`
- `wazuh-docker/single-node/config/wazuh_cluster/local_rules.xml`
- `wazuh-docker/single-node/config/wazuh_cluster/rules/soc-*.xml`
### Wazuh Custom Rules Added (Current Active Set)
Active custom rules are currently defined in:
- `wazuh-docker/single-node/config/wazuh_cluster/local_rules.xml`
Rule groups/ranges implemented:
- Base and appendix classifiers:
- `100200`: base marker for synthetic SOC events (`soc_mvp_test=true`)
- `100210`: Appendix A classifier
- `100220`: Appendix B classifier
- `100230`: Appendix C classifier
- Appendix A:
- `A1` IOC/DNS: `100301-100302`
- `A2` FortiGate firewall/IPS/IDS: `100311-100320`
- `A3` VPN anomalies: `100331-100335`
- `A4` Windows/AD behaviors: `100341-100364`
- Appendix B:
- `B1` VMware/vCenter/ESXi: `100401-100403`
- `B2` Log-loss monitor signal: `100411`
- `B3` Sysmon-focused detections: `100421-100426`
- Appendix C (implemented scope C1-C3):
- `C1` Impossible travel: `100501`
- `C2` Credential abuse/privilege misuse: `100511-100514`
- `C3` Lateral movement/internal recon: `100521-100524`
Operational note:
- Split rule files under `wazuh_cluster/rules/soc-*.xml` exist as staging artifacts in this workspace; active detection content is loaded from `local_rules.xml`.
3. Operations and Runtime Hardening
- Updated orchestration and runtime configuration:
- `run-combined-stack.sh`
- `compose-overrides/soc-integrator.yml`
- `soc-integrator/Dockerfile`
- `soc-integrator/.env.example`
### Documentation Progress Included in This Range
- Added/updated proposal revision document:
- `Security Detection & Threat Intelligence Enhancement Proposal-revise.md`
- Expanded progress log coverage in this file (`progress-update.md`) including:
- Appendix C (C1-C3) production log mapping
- Production data onboarding checklist
- Acceptance criteria for source onboarding and validation