Merge branch 'main' of http://git2.simplico.net/tum/soc

progress-update.md

@@ -640,6 +640,112 @@ This update documents production log sources and required fields for Appendix C
 
 ---
 
+Date: March 17, 2026
+Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)
+
+## Incremental Progress Since March 4, 2026
+
+### 1) IRIS Stack Stabilization
+
+- Fixed Alembic DB revision mismatch caused by DB being initialized from a different branch:
+  - Identified correct head revision (`d5a720d1b99b`) and stamped the DB directly
+  - Fixed missing `ioc.case_id` column (`ALTER TABLE ioc ADD COLUMN IF NOT EXISTS case_id ...`) skipped by the migration
+- Fixed IRIS static file 404 errors (jQuery, Bootstrap, app JS):
+  - Root cause: Svelte/Vite frontend (`ui/dist`) had never been built
+  - Fix: ran `npm install && npm run build` inside the `iris-web/ui` container to produce `dist/`
+- All IRIS web UI pages now load correctly including case management and KPI dashboard
+
+### 2) KPI Dashboard (IRIS)
+
+- Added a custom KPI dashboard page to IRIS web (`/kpi`):
+  - Files: `iris-web/source/app/blueprints/pages/kpi_dashboard/`
+  - Registered in Flask app and added to sidebar navigation
+  - Frontend JS/CSS: `iris-web/ui/src/pages/kpi_dashboard.js`, `iris-web/ui/src/css/kpi_dashboard.css`
+- Added KPI test data seeder: `scripts/seed-kpi-test-data.py`
+- Updated `manage_cases.html` and `case.html` templates for KPI-related display fields
+
+### 3) Shuffle Credential Persistence
+
+- Investigated and resolved the Shuffle admin password being unknown after restarts:
+  - Password hash is stored in OpenSearch under `Password` (capital P) as bcrypt (`$2a$` prefix for Go compat)
+  - Root cause: `SHUFFLE_DEFAULT_PASSWORD` was blank; password was auto-generated and not saved
+  - Fix: set `SHUFFLE_DEFAULT_USERNAME`, `SHUFFLE_DEFAULT_PASSWORD`, and `SHUFFLE_DEFAULT_APIKEY` in `Shuffle/.env`
+  - API key is now fixed across restarts via `SHUFFLE_DEFAULT_APIKEY`
+- Created `pass.txt` at project root documenting all system credentials (IRIS, Shuffle, Wazuh, SOC Integrator, third-party API keys)
+
+### 4) SOC Integrator → Shuffle API Fix
+
+- Fixed `/shuffle/apps` and `/shuffle/workflows` returning 401:
+  - Root cause: container had an old API key baked from a previous creation — `docker restart` does NOT re-read `env_file`
+  - Fix: generated fresh Shuffle API key via session cookie, updated `soc-integrator/.env`, and used `--force-recreate` to pick up the new key
+- Both Shuffle proxy endpoints now return 200 from `soc-integrator`
+
+### 5) Wazuh Dashboard — Real-Data Filters
+
+- Fixed "Could not locate index-pattern" error on all custom dashboards:
+  - All 4 dashboard ndjson files had their own private index-pattern objects that conflicted with the canonical `wazuh-alerts-*` pattern
+  - Fix: removed private index-pattern objects from each ndjson; updated all `references[]` and `searchSourceJSON.indexRefName` entries to point to `wazuh-alerts-*`
+- Replaced simulation-era `full_log:*usecase_id*` filters across all dashboards with production-data filters:
+  - **Custom rules dashboard**: filter changed to `rule.groups: soc_prod*`
+    - Note: `rule.id` is a `keyword` field in OpenSearch — range queries are lexicographic, not numeric; ID-based range filtering is unreliable
+    - `rule.groups: soc_prod*` is the correct discriminator — all SOC custom rules carry this group; no built-in Wazuh rules do
+  - **Required/appendix dashboards**: all 51 usecase_id sub-filters remapped to actual `rule.id: 110xxx` values and `rule.groups: appendix_a/b` splits
+  - **Appendix A/B dashboard**: top filter `full_log:*usecase_id*` → `rule.groups: soc_prod*`; A/B panel splits use `rule.groups: appendix_a` and `rule.groups: appendix_b`
+  - 7 use cases with no implemented production rule set to `rule.id: __no_rule__` (intentional no-match)
+- All 4 dashboards imported and rendering correctly against live `wazuh-alerts-*` data
+
+### 6) Firewall Syslog Testing
+
+- Added `scripts/test-firewall-syslog.py`:
+  - Sends FortiGate-style syslog UDP to Wazuh port 514 with 10 test scenarios
+  - Supports `--via-docker` flag to preserve source IP through Docker NAT
+- Added 7 firewall source IPs to `wazuh_manager.conf` allowed list
+- Enabled `logall` and `logall_json` in Wazuh manager config for improved debug visibility
+
+### 7) Production Log Samples
+
+- Added production log sample files under `samples/`:
+  - `samples/appendix-a-production-samples.log`
+  - `samples/appendix-b-production-samples.log`
+  - `samples/appendix-c-production-samples.log`
+- Added `samples/README.md` documenting log format and usage
+
+### 8) Rule Match Evidence (Live Data, 2026-03-14 → 2026-03-17)
+
+- Queried all SOC custom rules against OpenSearch `wazuh-alerts-*` and generated `summary_rule_match.md`
+- Total meaningful events today (2026-03-17, post rule-fix): **199** across 49 implemented rules
+
+**Rule 110354 bug found and fixed:**
+- Root cause: parent SID 60103 is "Windows audit success event" — matches ALL `AUDIT_SUCCESS` events, not just 4794
+- Rule had no `eventID` constraint → was firing ~313,000 times/day on events like 4624, 4634, 4688, 4793
+- Fix: added `<field name="win.system.eventID">^4794$</field>` to rule 110354 in `soc-a4-windows-ad-rules.xml`
+- Rule is now silent (0 events post-fix); fix applied directly in running container and in source file
+
+Active rules with events today (2026-03-17, post-fix):
+
+| Rule | Description | Events |
+|------|-------------|--------|
+| 110359 | A4-19 Windows auth failure (4625) general | 71 |
+| 110342 | A4-02 Service account auth failure (4625) | 50 |
+| 110348 | A4-08 NTLM logon type 3 — pass-the-hash indicator | 46 |
+| 110523 | C3-03 Admin account auth success — lateral movement candidate | 23 |
+| 110522 | C3-02 SMB network logon type 3 | 8 |
+| 110341 | A4-01 Privileged account auth failure | 1 |
+| 110354 | A4-13 DSRM password set (4794) | 0 ✅ fixed |
+
+- Rules 110348, 110522, 110523 were previously masked by the 110354 false-positive flood — now visible with correct counts
+- A1, A2, B2, C1 rules had 0 events today (A1/A2/C1 events from Mar 14–16 were likely simulator runs)
+- Log sources not yet forwarding: FortiGate syslog (A1/A2), FortiGate VPN (A3), VMware (B1), Sysmon/endpoints (B3)
+
+### 9) Tooling and Documentation Updates
+
+- `run-combined-stack.sh`: added `dedup` command (fixed missing `elif` branch) and `recreate` command
+- `README.md`: full rewrite covering all stack commands, KPI dashboard, current endpoint list, and macOS bind-mount note
+- `scripts/README.md`: documented `test-firewall-syslog.py`, `seed-kpi-test-data.py`, and new dashboard ndjson files
+- `docs/wazuh-decoders-rules.md`: documentation for custom decoder/rule structure
+
+---
+
 Date: March 4, 2026
 Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)
 

summary_rule_match.md

@@ -0,0 +1,191 @@
+# Wazuh Rule Match Summary — SOC Proposal Appendices A / B / C
+
+**Query window:** 2026-03-17 (today only)
+**Total meaningful events (post rule-fix):** 199
+**Data source:** OpenSearch index `wazuh-alerts-*` (filter: `rule.groups: soc_prod*`)
+
+> **Note — Rule 110354 fix:** Rule 110354 (A4-13 DSRM) was found to be misconfigured — parent SID 60103
+> is "Windows audit success event" (matches ALL `AUDIT_SUCCESS` events), and the rule had no `eventID`
+> constraint. This caused ~313,000 false-positive fires today on events like 4624, 4634, 4688, 4793, etc.
+> Fix applied: added `<field name="win.system.eventID">^4794$</field>`. Rule is now silent (0 events post-fix,
+> confirmed correct). Pre-fix event count is excluded from the summary totals below.
+
+---
+
+## Appendix A — Threat Detection (FortiGate + Windows/AD)
+
+### A1 — DNS / Firewall IOC  *(file: soc-a1-ioc-rules.xml)*
+
+| Rule ID | Use Case | Description | MITRE | Events |
+|---------|----------|-------------|-------|--------|
+| 110301 | A1-01 | DNS query to malicious domain (IOC traffic indicator) | T1071.004 | 0 |
+| 110302 | A1-02 | DNS IOC domain match from threat intelligence feed | T1568 | 0 |
+
+---
+
+### A2 — FortiGate IPS/IDS & Firewall  *(file: soc-a2-fortigate-fw-rules.xml)*
+
+| Rule ID | Use Case | Description | MITRE | Events |
+|---------|----------|-------------|-------|--------|
+| 110311 | A2-01 | FortiGate: RDP (3389) traffic allowed | T1021.001 | 0 |
+| 110312 | A2-02 | FortiGate: admin account password changed | T1098 | 0 |
+| 110313 | A2-03 | FortiGate: new admin account created | T1136 | 0 |
+| 110314 | A2-04 | FortiGate: alerting/notification disabled via config change | T1562 | 0 |
+| 110315 | A2-05 | FortiGate: firewall configuration file downloaded | T1005 | 0 |
+| 110316 | A2-06 | FortiGate IPS: multiple critical signatures triggered | T1595 | 0 |
+| 110317 | A2-07 | FortiGate: TCP port scan from external IP | T1046 | 0 |
+| 110318 | A2-08 | FortiGate IPS: IOC-based IP indicator detected | T1071.001 | 0 |
+| 110319 | A2-09 | FortiGate: internal port scan from private source IP | T1046 | 0 |
+| 110320 | A2-10 | FortiGate: traffic to known C2/malicious IP allowed | T1071.001 | 0 |
+
+> No FortiGate syslog events received today.
+
+---
+
+### A3 — FortiGate VPN  *(file: soc-a3-fortigate-vpn-rules.xml)*
+
+| Rule ID | Use Case | Description | MITRE | Events |
+|---------|----------|-------------|-------|--------|
+| 110331 | A3-01 | VPN authentication success by guest account | T1078.001 | 0 |
+| 110332 | A3-02 | VPN success from different country than last login | T1078 | 0 |
+| 110333 | A3-03 | VPN success after multiple prior failures (brute-force indicator) | T1110.001 | 0 |
+| 110334 | A3-04 | VPN multiple account failures from single source IP | T1110.003 | 0 |
+| 110335 | A3-05 | VPN authentication success from outside Thailand | T1078 | 0 |
+
+> VPN logs not yet forwarded to Wazuh.
+
+---
+
+### A4 — Windows / Active Directory  *(file: soc-a4-windows-ad-rules.xml)*
+
+| Rule ID | Use Case | Description | MITRE | Events |
+|---------|----------|-------------|-------|--------|
+| 110341 | A4-01 | Windows: privileged account name auth failure (4625) | T1110.001 | **1** |
+| 110342 | A4-02 | Windows: service account auth failure (4625) | T1110.001 | **50** |
+| 110343 | A4-03 | Windows AD: adfind enumeration tool executed (4688) | T1087.002 | 0 |
+| 110346 | A4-06 | Windows: remote interactive auth success logon type 10 (4624) | T1021.001, T1078 | 0 |
+| 110348 | A4-08 | Windows: NTLM network logon type 3 — pass-the-hash indicator (4624) | T1550.002 | **46** |
+| 110349 | A4-09 | Windows: guest account auth success (4624) | T1078.001 | 0 |
+| 110350 | A4-10 | Windows: service account interactive logon type 2 (4624) | T1078.003 | 0 |
+| 110352 | A4-12 | Windows: account added to privileged domain group (4728) | T1098.007 | 0 |
+| 110353 | A4-11 | Windows: account added to privileged local group (4732) | T1098.007 | 0 |
+| 110354 | A4-13 | Windows DC: DSRM account password set (4794) | T1098 | 0 ✅ fixed |
+| 110359 | A4-19 | Windows: authentication failure (4625) — general | T1110.003 | **71** |
+| 110361 | A4-21/23 | Windows: new user account created (4720) | T1136 | 0 |
+| 110362 | A4-22/24 | Windows: user account re-enabled (4722) | T1078 | 0 |
+
+> Rule 110354 now correctly requires `eventID=4794` and is silent (no genuine DSRM events today).
+> Rule 110348 (NTLM/pass-the-hash) was previously masked by 110354 noise — now visible with 46 events.
+
+---
+
+## Appendix B — Expanded Monitoring
+
+### B1 — VMware vCenter / ESXi  *(file: soc-b1-vmware-rules.xml)*
+
+| Rule ID | Use Case | Description | MITRE | Events |
+|---------|----------|-------------|-------|--------|
+| 110401 | B1-01 | vCenter: login failure detected (brute-force indicator) | T1110 | 0 |
+| 110402 | B1-02 | ESXi: SSH service enabled on host | T1021.004 | 0 |
+| 110403 | B1-03 | ESXi: SSH authentication event detected | T1021.004 | 0 |
+
+> VMware logs not yet forwarded to Wazuh.
+
+---
+
+### B2 — Log Monitoring  *(file: soc-b2-logmon-rules.xml)*
+
+| Rule ID | Use Case | Description | MITRE | Events |
+|---------|----------|-------------|-------|--------|
+| 110411 | B2-01 | Log Monitor: log ingestion loss detected on monitored stream | T1562.006 | 0 |
+
+---
+
+### B3 — Windows Sysmon  *(file: soc-b3-sysmon-rules.xml)*
+
+| Rule ID | Use Case | Description | MITRE | Events |
+|---------|----------|-------------|-------|--------|
+| 110421 | B3-01 | Sysmon: LSASS process access detected (event 10) | T1003.001 | 0 |
+| 110422 | B3-02 | Sysmon: SQL keyword in process command line (event 1) | T1190 | 0 |
+| 110423 | B3-03 | Sysmon: web script file created (possible webshell, event 11) | T1505.003 | 0 |
+| 110424 | B3-04 | Sysmon: msiexec uninstall detected (event 1) | T1562.001 | 0 |
+| 110425 | B3-05 | Sysmon: LSASS dump via Task Manager (event 10) | T1003.001 | 0 |
+| 110426 | B3-06 | Sysmon: certutil.exe execution detected (event 1) | T1105 | 0 |
+
+> Sysmon not deployed on endpoints.
+
+---
+
+## Appendix C — Advanced Detection (Correlation)
+
+### C1 — Impossible Travel  *(file: soc-c1-c3-rules.xml)*
+
+| Rule ID | Use Case | Description | MITRE | Events |
+|---------|----------|-------------|-------|--------|
+| 110501 | C1-01 | VPN login success with geo context — impossible travel candidate | T1078 | 0 |
+| 110502 | C1-01 | Impossible travel confirmed by soc-integrator correlation | T1078 | 0 |
+
+---
+
+### C2 — Advanced Credential Abuse & Privilege Misuse  *(file: soc-c1-c3-rules.xml)*
+
+| Rule ID | Use Case | Description | MITRE | Events |
+|---------|----------|-------------|-------|--------|
+| 110511 | C2-01 | Privileged account auth success (4624) | T1078.002 | 0 |
+| 110512 | C2-02 | Dormant/legacy account auth success (4624) | T1078 | 0 |
+| 110513 | C2-03 | Service account remote interactive logon type 10 (4624) | T1078.003 | 0 |
+| 110514 | C2-04 | Privilege escalation: group membership change (4732) | T1098.007 | 0 |
+
+---
+
+### C3 — Lateral Movement & Internal Reconnaissance  *(file: soc-c1-c3-rules.xml)*
+
+| Rule ID | Use Case | Description | MITRE | Events |
+|---------|----------|-------------|-------|--------|
+| 110521 | C3-01/02 | RDP auth success logon type 10 (lateral movement indicator) | T1021.001, T1078 | 0 |
+| 110522 | C3-02 | SMB network logon type 3 (lateral movement indicator) | T1021.002, T1078 | **8** |
+| 110523 | C3-03 | Admin account auth success — lateral movement candidate (4624) | T1021.001, T1078.002 | **23** |
+
+> C3 rules (110522, 110523) were previously masked by 110354 noise — now visible with real event data.
+
+---
+
+## Summary
+
+| Appendix | Section | Rules Implemented | Rules with Events | Total Events |
+|----------|---------|:-----------------:|:-----------------:|:------------:|
+| A | A1 — DNS/IOC | 2 | 0 | 0 |
+| A | A2 — FortiGate FW/IPS | 10 | 0 | 0 |
+| A | A3 — FortiGate VPN | 5 | 0 | 0 |
+| A | A4 — Windows/AD | 13 | 4 | 168 |
+| B | B1 — VMware | 3 | 0 | 0 |
+| B | B2 — Log Monitor | 1 | 0 | 0 |
+| B | B3 — Sysmon | 6 | 0 | 0 |
+| C | C1 — Impossible Travel | 2 | 0 | 0 |
+| C | C2 — Credential Abuse | 4 | 0 | 0 |
+| C | C3 — Lateral Movement | 3 | 2 | 31 |
+| **Total** | | **49** | **6** | **199** |
+
+### Active rules today (post rule-fix)
+
+| Rule | Description | Events | Note |
+|------|-------------|--------|------|
+| 110359 | A4-19 Windows auth failure (4625) general | 71 | Normal auth noise |
+| 110342 | A4-02 Service account auth failure (4625) | 50 | Service account brute-force pattern |
+| 110348 | A4-08 NTLM logon type 3 — pass-the-hash indicator | 46 | Previously masked by 110354 bug |
+| 110523 | C3-03 Admin account auth success — lateral movement candidate | 23 | Previously masked by 110354 bug |
+| 110522 | C3-02 SMB network logon type 3 | 8 | Previously masked by 110354 bug |
+| 110341 | A4-01 Privileged account auth failure | 1 | |
+| 110354 | A4-13 DSRM password set (4794) | 0 ✅ | Fixed — was false-positive firing on all AUDIT_SUCCESS |
+
+### Active log sources (today)
+
+| Source | Appendix | Status |
+|--------|----------|--------|
+| Windows Security Event Log (via Wazuh agent) | A4, C3 | ✅ Active — auth events (4624, 4625) ingesting across multiple agents |
+| FortiGate firewall/IPS syslog | A2 | ❌ No events today |
+| FortiGate VPN syslog | A3, C1 | ❌ Not forwarding |
+| DNS / soc-mvp decoder | A1 | ❌ No events today |
+| soc-integrator log-loss events | B2 | ❌ No events today |
+| VMware vCenter/ESXi syslog | B1 | ❌ Not forwarding |
+| Windows Sysmon (via Wazuh agent) | B3 | ❌ Not deployed |

wazuh-docker/single-node/config/wazuh_cluster/rules/soc-a4-windows-ad-rules.xml

@@ -136,9 +136,10 @@
   </rule>
 
   <!-- A4-13: DSRM password set (event 4794)
-       Parent: 60103 -->
+       Parent: 60103 (Windows audit success event — must constrain to eventID 4794) -->
   <rule id="110354" level="12">
     <if_sid>60103</if_sid>
+    <field name="win.system.eventID">^4794$</field>
     <description>A4-13 [PROD] Windows DC: DSRM account password set (4794)</description>
     <group>soc_prod,a4,persistence,</group>
     <mitre><id>T1098</id></mitre>