progress-update

progress-update.md

@@ -640,6 +640,111 @@ This update documents production log sources and required fields for Appendix C
 
 ---
 
+Date: March 17, 2026
+Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)
+
+## Incremental Progress Since March 4, 2026
+
+### 1) IRIS Stack Stabilization
+
+- Fixed Alembic DB revision mismatch caused by DB being initialized from a different branch:
+  - Identified correct head revision (`d5a720d1b99b`) and stamped the DB directly
+  - Fixed missing `ioc.case_id` column (`ALTER TABLE ioc ADD COLUMN IF NOT EXISTS case_id ...`) skipped by the migration
+- Fixed IRIS static file 404 errors (jQuery, Bootstrap, app JS):
+  - Root cause: Svelte/Vite frontend (`ui/dist`) had never been built
+  - Fix: ran `npm install && npm run build` inside the `iris-web/ui` container to produce `dist/`
+- All IRIS web UI pages now load correctly including case management and KPI dashboard
+
+### 2) KPI Dashboard (IRIS)
+
+- Added a custom KPI dashboard page to IRIS web (`/kpi`):
+  - Files: `iris-web/source/app/blueprints/pages/kpi_dashboard/`
+  - Registered in Flask app and added to sidebar navigation
+  - Frontend JS/CSS: `iris-web/ui/src/pages/kpi_dashboard.js`, `iris-web/ui/src/css/kpi_dashboard.css`
+- Added KPI test data seeder: `scripts/seed-kpi-test-data.py`
+- Updated `manage_cases.html` and `case.html` templates for KPI-related display fields
+
+### 3) Shuffle Credential Persistence
+
+- Investigated and resolved the Shuffle admin password being unknown after restarts:
+  - Password hash is stored in OpenSearch under `Password` (capital P) as bcrypt (`$2a$` prefix for Go compat)
+  - Root cause: `SHUFFLE_DEFAULT_PASSWORD` was blank; password was auto-generated and not saved
+  - Fix: set `SHUFFLE_DEFAULT_USERNAME`, `SHUFFLE_DEFAULT_PASSWORD`, and `SHUFFLE_DEFAULT_APIKEY` in `Shuffle/.env`
+  - API key is now fixed across restarts via `SHUFFLE_DEFAULT_APIKEY`
+- Created `pass.txt` at project root documenting all system credentials (IRIS, Shuffle, Wazuh, SOC Integrator, third-party API keys)
+
+### 4) SOC Integrator → Shuffle API Fix
+
+- Fixed `/shuffle/apps` and `/shuffle/workflows` returning 401:
+  - Root cause: container had an old API key baked from a previous creation — `docker restart` does NOT re-read `env_file`
+  - Fix: generated fresh Shuffle API key via session cookie, updated `soc-integrator/.env`, and used `--force-recreate` to pick up the new key
+- Both Shuffle proxy endpoints now return 200 from `soc-integrator`
+
+### 5) Wazuh Dashboard — Real-Data Filters
+
+- Fixed "Could not locate index-pattern" error on all custom dashboards:
+  - All 4 dashboard ndjson files had their own private index-pattern objects that conflicted with the canonical `wazuh-alerts-*` pattern
+  - Fix: removed private index-pattern objects from each ndjson; updated all `references[]` and `searchSourceJSON.indexRefName` entries to point to `wazuh-alerts-*`
+- Replaced simulation-era `full_log:*usecase_id*` filters across all dashboards with production-data filters:
+  - **Custom rules dashboard**: filter changed to `rule.groups: soc_prod*`
+    - Note: `rule.id` is a `keyword` field in OpenSearch — range queries are lexicographic, not numeric; ID-based range filtering is unreliable
+    - `rule.groups: soc_prod*` is the correct discriminator — all SOC custom rules carry this group; no built-in Wazuh rules do
+  - **Required/appendix dashboards**: all 51 usecase_id sub-filters remapped to actual `rule.id: 110xxx` values and `rule.groups: appendix_a/b` splits
+  - **Appendix A/B dashboard**: top filter `full_log:*usecase_id*` → `rule.groups: soc_prod*`; A/B panel splits use `rule.groups: appendix_a` and `rule.groups: appendix_b`
+  - 7 use cases with no implemented production rule set to `rule.id: __no_rule__` (intentional no-match)
+- All 4 dashboards imported and rendering correctly against live `wazuh-alerts-*` data
+
+### 6) Firewall Syslog Testing
+
+- Added `scripts/test-firewall-syslog.py`:
+  - Sends FortiGate-style syslog UDP to Wazuh port 514 with 10 test scenarios
+  - Supports `--via-docker` flag to preserve source IP through Docker NAT
+- Added 7 firewall source IPs to `wazuh_manager.conf` allowed list
+- Enabled `logall` and `logall_json` in Wazuh manager config for improved debug visibility
+
+### 7) Production Log Samples
+
+- Added production log sample files under `samples/`:
+  - `samples/appendix-a-production-samples.log`
+  - `samples/appendix-b-production-samples.log`
+  - `samples/appendix-c-production-samples.log`
+- Added `samples/README.md` documenting log format and usage
+
+### 8) Rule Match Evidence (Live Data, 2026-03-14 → 2026-03-17)
+
+- Queried all SOC custom rules against OpenSearch `wazuh-alerts-*` and generated `summary_rule_match.md`
+- Total events matched: **252,262** across 49 implemented rules
+
+Active rules with events:
+
+| Rule | Description | Events |
+|------|-------------|--------|
+| 110301 | A1-01 DNS query to malicious domain | 32 |
+| 110302 | A1-02 DNS IOC domain match | 32 |
+| 110312 | A2-02 FortiGate admin password changed | 32 |
+| 110313 | A2-03 FortiGate new admin account created | 32 |
+| 110315 | A2-05 FortiGate config file downloaded | 32 |
+| 110320 | A2-10 FortiGate traffic to known C2 | 32 |
+| 110341 | A4-01 Windows privileged account auth failure | 1 |
+| 110342 | A4-02 Windows service account auth failure | 38 |
+| 110354 | A4-13 Windows DC DSRM password set (4794) | 251,833 ⚠️ |
+| 110359 | A4-19 Windows authentication failure (4625) | 54 |
+| 110411 | B2-01 Log Monitor log loss detected | 16 |
+| 110501 | C1-01 VPN login geo context candidate | 97 |
+| 110502 | C1-01 Impossible travel confirmed | 31 |
+
+- ⚠️ Rule 110354 (DSRM / event 4794 via parent 60105) accounts for 99.8% of event volume — under investigation to confirm parent SID scope
+- Log sources not yet forwarding: FortiGate VPN (A3), VMware (B1), Sysmon/endpoints (B3), Windows 4624 auth success (C2/C3)
+
+### 9) Tooling and Documentation Updates
+
+- `run-combined-stack.sh`: added `dedup` command (fixed missing `elif` branch) and `recreate` command
+- `README.md`: full rewrite covering all stack commands, KPI dashboard, current endpoint list, and macOS bind-mount note
+- `scripts/README.md`: documented `test-firewall-syslog.py`, `seed-kpi-test-data.py`, and new dashboard ndjson files
+- `docs/wazuh-decoders-rules.md`: documentation for custom decoder/rule structure
+
+---
+
 Date: March 4, 2026
 Project: FoodProject SOC Platform (Wazuh + Shuffle + IRIS-web + SOC Integrator)