Pre-Merge Architecture Review

Threat Intel Telegram Pipeline

Trigger flow, concurrency model, and an annotated map of breaking points across three chained Airflow DAGs. Compiled from two independent code reviews of PR #112.

pull_request#112
head_commit19dfaa7
platformtelegram
dags3 chained
blockers3 HIGH
REQUEST CHANGES
A

System Overview

A factory build_platform_dags(PlatformConfig) in src/threat_intel/common/dags.py produces three chained DAGs per platform, instantiated at import time by src/threat_intel_telegram.py. Today only platform=telegram exists. Each DAG hands off to the next; completion ripples back up.

DAG 01 / listener

threat-intel-telegram-job-listener

Polls Kafka on a 1-minute schedule, upserts the Mongo job, triggers discover, waits, then commits the offset.

schedule=1mcatchup=Falsemax_active_runs=1
DAG 02 / discover

threat-intel-telegram-discover

Externally triggered. Runs the discover pod on Kubernetes, then triggers screen and waits for it.

schedule=NoneKubernetesPodOperatortelegram acct #0
DAG 03 / screen

threat-intel-telegram-screen

Externally triggered. A single task claims unscreened docs in batches and screens them with an LLM.

schedule=Nonebatch=20OpenAI gpt-5.4-mini
Kafka
threat-intel-jobs-telegram
single partition (p0) · group airflow-threat-intel-telegram
MongoDB
threat_intel db
jobs collection + discovery collection
Kubernetes
discover pod
namespace airflow · ECR image · session PVC
OpenAI
Chat Completions
screening LLM · default gpt-5.4-mini
Telegram
account session
on PVC, mounted at /app/session
B

Trigger Chain

Directional edges are labeled by trigger mechanism. Hover any node to isolate its edges and reveal labels. Red badges mark nodes involved in a HIGH-severity weak link. The green dashed edges are the completion ripple travelling back up the chain.

task chain TriggerDagRun completion ripple (back up) external I/O hover a node to trace its edges
ingressKafka
external
producer
publishes job envelope
Kafka p0
jobs-telegram
single partition · one msg / run
DAG 01listener
WL1
sensor · deferrable
tasks-reader
ThreatIntelKafkaSensor, max_active_tis=1
PythonOperator
prepare-discover
start_job upsert → "discovering"
TriggerDagRun
discover-trigger
wait_for_completion=False
WL2
PythonSensor · reschedule
discover-wait
poke 30s · timeout 24h
WL1
PythonOperator
commit-offset
trigger_rule none_failed_min_one_success
DAG 02discover
KubernetesPodOperator
discover
discover pod · acct #0 · PVC session
PythonOperator
prepare-screen
build child conf
TriggerDagRun
screen-trigger
reset_dag_run=False
WL2
PythonSensor · reschedule
screen-wait
wait_for_child
DAG 03screen
WL3
PythonOperator
screen
run_screen · claim batch of 20 · finish_job
external systems
MongoDB
threat_intel
jobs + discovery
Kubernetes
discover pod
smc-threat-intel-...-discover:latest
OpenAI
chat completions
JSON in/out · retry ×3
C

Parallelization & Concurrency Model

The pipeline is strictly serial per platform but cheap while waiting. It scales horizontally across platforms, never within one. Substantial latent parallelism in the screener is designed for but never used.

Exactly one job in flight, end to end

max_active_runs=1 on all three DAGs, max_active_tis_per_dag=1 on the sensor, one Kafka message per run, and each parent blocking on its child.
listen
discover
screen
1 job
Throughput ceiling = 1 job / (discover time + screen time)

Executor capacity while waiting

Sensors are engineered to hold almost no worker slot — the good news of this design.
tasks-reader (deferrable)frees slot → triggerer
discover-wait (reschedule)slot freed between pokes
screen-wait (reschedule)slot freed between pokes
discover fan-outsingle pod, hardcoded acct #0
screen concurrency (used / designed)1 of N screeners
good

Waiting is nearly free

The Kafka sensor is deferrable, so it hands its worker slot to the triggerer while idle. Both wait sensors use mode="reschedule", releasing their slots between 30s pokes. Idle time costs almost no executor capacity.

bottleneck

No fan-out in discover

A single pod on a single hardcoded telegram account (index 0). Contrast the repo's stable twitter DAG, which fans scrapers out across accounts. There is no horizontal discovery here.

latent

Screener is sequential by choice

A while-loop, one batch of 20 at a time, one HTTP call per batch. The Mongo claim mechanism (1h claim timeout, atomic claim of unscreened docs, stale reclaim) is designed for concurrent screeners, but the DAG never runs more than one. Unused parallelism.

scaling axis

Platforms parallelize, jobs do not

The factory pattern gives each new platform (e.g. discord) its own topic, consumer group, and 3 DAGs. Platforms run independently of each other; jobs within a platform remain strictly serial.

E

The Reference Fix

The repo's stable connector DAG (sm_connector_dag_stable.py + common_ops/kafka_reader.py + common_ops/dispatch_ops.py) already solves blockers 1, 2, and 6 with a Postgres dispatch-claims ledger: skip offsets already in the ledger, claim-before-dispatch via PK insert on (topic, partition, offset), commit only the highest contiguous completed offset ≥ the current committed position (monotonic), verify ownership before commit, and route invalid messages through the same ledger.

DimensionPR #112 listenerStable ledger pattern
DedupeNone — Mongo upsert overwrites blindly, no offset ledgerSkip via kafka_dispatch_offset_exists before dispatch
Commit monotonicityBlind offset+1; can move group offset BACKWARDadvance_completed_kafka_offsets — highest contiguous offset only
Failed-run handlingFAILED child treated as success → message consumed anywayOwnership verified before commit; failures do not advance
Skip pathLow-watermark fallback replays retained backlogClaim-before-dispatch on unique (topic,partition,offset)
Invalid messages_skip_unprocessable commits offset then re-defersInvalid messages routed through the same ledger
Test coverageNonetests/test_deferred_kafka_reader.py

Minimum viable fix if the full ledger is out of scope: drop the get_first_produced fallback and ignore any fallback offset lower than the committed position.

F

Combined Review Verdict

REQUEST CHANGES3 BLOCKERS

Structurally sound, but must not merge as-is

Three HIGH blockers each let the pipeline silently lose or infinitely reprocess work while reporting success. Fix blockers 1, 2, and 3 before merge. Everything else is Medium/Low and can follow.

B1 · offset replay loop B2 · failed child commits B3 · LLM outage masked
+ prompt-injection-hardened screening prompt + claim-based screening design + deferrable / reschedule sensors + consistent with repo factory patterns