r/LocalLLaMA • u/Character-Apple-8471 • 11h ago
Discussion Qwen 3.8 27B vs Flash Next 125B on the same threat-monitoring project: code review, tests, and where both failed
I gave both models IQ3_S the same prompt: build a real-time cyber threat intelligence platform with Python, FastAPI, async ingestion, persistent storage, WebSockets, and an interactive Three.js globe.
The difficult requirement was data integrity. The platform had to use actual intelligence sources, preserve attribution and timestamps, distinguish observations from inference, and avoid inventing attacks or destinations to make the globe look busy.
After reviewing the two delivered codebases, the 27B submission is the one I would choose to repair and build on. Neither fulfilled the complete prompt. Flash Next attempted more features and included more tests, but several of its core data contracts were broken.
This is one saved submission per model, with the model names taken from the labels used for these runs. The follow-up review and this write-up were AI-assisted. This is an artifact comparison, not a controlled study of overall model capability.
What the task actually asked for
Both were asked to integrate sources including Cloudflare Radar, GreyNoise, DShield, ThreatFox, and distributed honeypot networks. Beyond fetching data, the requirements included:
- Normalization, deduplication, IP geolocation, classification, and configurable risk scoring.
- Correlation, historical storage, anomaly detection, timelines, geographical filters, and investigation views.
- Animated trajectories only where actual source and destination evidence supported them.
- Resilient integrations, rate-limit handling, structured logging, testing, benchmarking, and deployment.
That makes this a useful test of whether the pieces agree with each other. A source adapter, a database, and a globe can all exist while the application still misrepresents the underlying observations.
Results at a glance
| Check | Qwen 3.8 27B | Flash Next 125B |
|---|---|---|
| Existing pytest suite | 16 passed | 34 passed |
| JavaScript syntax checks | Passed | Passed |
| Downloaded DShield sample | Parsed 5/5 top-IP records | Parsed 20 subnet rows, but read the wrong count column |
| ThreatFox | Documented response discarded | Wrong request selector and incorrect IOC normalization |
| Cloudflare Radar | Legacy endpoint | Request/parser mismatch with documented country-top API |
| WebSocket backpressure | Can block ingestion | Can block ingestion |
| Full prompt satisfied | No | No |
The DShield integrations use different feeds, so their record counts are not a head-to-head coverage measurement. One consumes top-IP observations; the other consumes subnet aggregates.
The failures that mattered most
1. Flash Next could report a successful source run after discarding every record.
Its Feodo adapter expected CSV fields such as c2_server_ip, c2_server_port, and bot. The downloaded public CSV used dst_ip, dst_port, and malware.
Replaying that downloaded body through the unmodified parser produced:
text
Input records: 5
Parsed events: 0
Returned error: None
The retrieved sample itself carried an older update date, so it was evidence of a parser mismatch, not proof of current botnet activity. The application still needed to recognize its schema or report an explicit incompatibility.
2. Its DShield adapter treated a subnet prefix as an activity count.
The feed defines column 3 as the subnet prefix and column 4 as the number of targets reporting scans. In the first downloaded row, those values were 24 and 344.
The normalized count was 24.
That error propagates beyond a label: the count feeds statistics and volume-based scoring. The adapter also represents the subnet by its starting address, which is not proof that that individual address attacked anyone. The DShield feed includes the column definitions.
3. Both implementations mishandled ThreatFox.
27B sent the correct recent-IOC request but expected response fields such as ip, threat, and date. A fixture using the documented IOC schema produced zero records while setting a last-success timestamp.
Flash Next sent:
json
{"request": "fetch_last_24h"}
The documented recent-IOC operation uses:
json
{"query": "get_iocs", "days": 1}
Its normalization also treated a domain IOC as a source IP and lost the source timestamp. These were request/parser checks against the ThreatFox contract, not authenticated live integration tests.
4. The analytics could manufacture confidence or inflate activity without new source evidence.
27B awarded a corroboration bonus after seeing the same IP twice from the same source. Two DShield observations in different deduplication buckets scored 9.2, then 21.2, with the second marked corroborated even though only one distinct source existed.
Flash Next added snapshot counts together. Reprocessing the same count=100 observation produced count=200. That measures repeated ingestion as if it were additional source activity.
Its anomaly flush also inserted a timeline count of 1 during an idle flush with no threat event. No production synthetic-threat generator was found in either project, but this scheduler tick still contaminates a chart presented as threat activity.
5. Some promised features existed in code but could not work through the normal execution path.
Flash Next's cross-source correlation examined only the current batch. Normal polling processes one adapter at a time. Sending the same IP through two separate source batches produced no correlation, despite both records being stored.
Its GreyNoise fallback returned a bare list where the pipeline expected (events, error). An empty fallback caused an unpacking exception. More generally, an unexpected adapter exception escaped the source loop; no supervisor restarted that poller, and the health endpoint did not check task liveness.
27B isolated ingestion exceptions better, but it also lost independent anomaly records: two country/category alerts in the same window produced two broadcasts and only one stored alert because their IDs collided.
6. The frontend had correctness and security problems too.
Node probes executing selected actual frontend functions reproduced these Flash Next issues:
- Redrawing a one-item feed twice grew its backing history from 1 to 2 to 4 entries.
- The country filter rejected a US bootstrap record but accepted an equivalent live record because their field names differed.
- An anomaly message threw a
TypeErrorbecause the client readmsg.data.datainstead ofmsg.data. - Feed-controlled markup reached
innerHTMLwithout escaping. That is an injection sink; a browser script-execution exploit was not attempted.
27B generally used textContent for external values, which was a better choice. It still duplicated events when replaying a reconnect snapshot, and its stored confidence distinction was not clearly exposed in the event UI.
Both WebSocket hubs awaited client sends inline without a deadline. A deliberately blocked fake client kept broadcasting pending until an external test timeout cancelled it. Since ingestion awaits broadcasting, a slow viewer can stall the data pipeline in either project.
Why I still prefer the 27B submission
27B provided an app factory, centralized retries and circuit breaking, JSON file logging, event retention, a persistent geolocation cache with expiry, a README, and a live benchmark entrypoint. Those choices made it the more coherent repair baseline.
Flash Next had useful ideas: explicit confidence classes, geographic precision metadata, batched geolocation, optional offline GeoIP, source-run storage, and richer filters. But more interfaces disagreed with each other, from provider responses to frontend message envelopes.
Neither demonstrated a working current Radar integration. 27B used the dimensionless L3 timeseries endpoint that Cloudflare's release notes say stopped receiving data after August 2023. Flash Next's request and parser differed from the documented country-top contract. No token-authenticated Radar check was performed.
Neither demonstrated a deployed, continuously operating observatory. Both also lacked validated global coverage, calibrated anomaly detection, and evidence of reliable real-source attack trajectories.
How the report scored them
| Category | Weight | 27B | Flash Next |
|---|---|---|---|
| Integration correctness and live coverage | 20 | 8 | 5 |
| Scientific accuracy and provenance | 20 | 10 | 6 |
| Reliability and resource control | 15 | 9 | 5 |
| Architecture and maintainability | 10 | 7 | 6 |
| Dashboard behavior and investigation | 10 | 6 | 6 |
| Storage, correlation, and anomaly quality | 10 | 5 | 4 |
| Tests and validation evidence | 10 | 5 | 6 |
| Deployment and reproducibility | 5 | 2 | 1 |
| Total | 100 | 52 | 39 |
These are subjective review scores under a declared rubric. They are not model benchmark scores or percentages of production readiness. Integration and evidence integrity account for 40 points because the prompt explicitly prioritized trustworthy intelligence.
What was tested, and what was not
The review ran both unmodified test suites, JavaScript syntax checks, focused Python/Node probes, and downloaded-feed parser checks. Provider contracts were checked against primary documentation. Test fixtures stayed in isolated contexts; application source was not repaired for the comparison.
A small storage-only benchmark ran three paired trials of 500 unique serial writes into temporary SQLite databases:
| Storage-only measurement | 27B | Flash Next |
|---|---|---|
| Median time for 500 writes | 211.10 ms | 390.77 ms |
27B was about 1.85x faster in that narrow workload. Flash Next's richer upsert performs an extra lookup and lock. The test excluded HTTP, geolocation, the full pipeline, WebSocket consumers, and browser rendering; cache/order effects were not controlled. It says nothing about model inference speed or sustained live ingestion capacity.
There was no browser/WebGL acceptance run, authenticated keyed-provider test, production deployment, or long-duration soak. Both suites used the same installed Python environment, rather than fresh locked installs.
Generation settings, tool budgets, token usage, cost, hardware, and original agent traces were not available for this review. I cannot establish equal generation conditions or generalize from these two outputs to overall model quality.
For this project, I would repair 27B first and selectively reuse Flash Next's better domain concepts. The result that changed my assessment most was that both test suites passed while independent checks still found silent data loss and incorrect observation semantics. Those are the checks I would add to the next comparison before giving either system credit for a working intelligence integration.
