Under the hood: how Dakwah-Lens is built.

TL;DR

Open-source under MIT.

1. 01

   Scrape

   services/{apify,rss,youtube}.py

   Five platforms: X / Instagram / TikTok via Apify, YouTube via the Data API, mainstream RSS via feedparser + trafilatura for full-body extraction. All return a unified `ScrapeResult` shape.

2. 02

   Normalize

   services/normalizers.py

   Per-platform normalizer maps raw payloads → SocialPost row dicts. Author handle extraction, URL truncation, posted_at parsing, region tagging for RSS.

3. 03

   Classify

   services/sentiment.py · relevance.py

   Four parallel Gemini Flash-Lite passes per post — (1) event-valence sentiment unified across every platform (since IndoBERT was retired 2026-05-25), (2) 9-category da'wah relevance, (3) focused da'wah-opportunity score, (4) classification into one of 14 da'wah theme groups (Hukum & Keadilan, Aqidah & Ibadah, Konflik & Geopolitik, Toleransi & Lintas-Iman, …) written to the `theme_group` column on `social_posts` — the primary axis for the dashboard and weekly briefings. Combined cost: ~$0.0003 per post.

4. 04

   Upsert

   scripts/ingest.py → social_posts

   Conflict on (platform, external_id) → update text + classifications + region. Re-running a scrape never produces duplicates.

5. 05

   Topic discovery

   scripts/cluster_topics.py → topics

   Daily Gemini Flash-Lite topic discovery per platform, fired at 04:00 WIB by Celery beat. Reads every post from the last 7 days above a da'wah-opportunity floor (typically 300–600 mainstream posts) and asks Gemini to identify 6–10 themes with human-readable Bahasa Indonesia labels framed from a da'wah angle. Three post-LLM gates protect quality: (a) **elastic-word filter** — labels containing 'misterius/tragis/viral/polemik/kontroversi/heboh/geger' are auto-injected into the exclude list, after a 2026-06-02 audit showed a theme labelled 'Peristiwa Misterius & Horor' was vacuuming up 70% road-accident posts that don't belong; (b) **cosine merge at 0.78** — 3072-dim embeddings dedup near-duplicate themes (lowered from 0.85 when we switched embedding model); (c) **orphan-rescue** — 'Lainnya' posts whose `theme_group` matches a surviving theme and whose cosine to its centroid is ≥0.20 get re-labelled to that theme, so 'Lainnya' doesn't swallow posts that actually fit somewhere. Results land in `topics`.

Scheduled via Celery beat (Asia/Jakarta): mainstream RSS every 2 hours · X weekly every Wednesday 22:00 WIB (500-item cap/keyword, 7-day window) · TikTok weekly Wednesday 22:10 WIB and Instagram weekly Wednesday 22:20 WIB (limit 20 items/keyword, evaluation phase since 2026-05-25) · YouTube whitelist channel sweep weekly every Wednesday 21:00 WIB (Data API uploads path — trusted da'i voices) · daily trending overlay at 12:00 WIB: da'wah-relevant keywords scraped on X (apidojo) + unbounded YouTube search (Data API search.list, with a language gate) · Gemini topic discovery at 04:00 WIB daily (mainstream + YouTube) · X + TikTok + Instagram topic discovery every Thursday 04:00 WIB · insights briefings every Thursday 05:00 WIB — one day before the Friday khutbah (currently paused for cost, triggered manually via Claude) · email digest every Thursday 08:00 WIB · Apify billing reconcile at 06:00 WIB daily. Manual runs via `uv run python -m api.scripts.ingest`.

Why two layers: curated alone leaves 7–25 day refresh gaps per keyword — fine for evergreen topics (riba, ibadah) but blind to a viral incident breaking that morning. Pure trending alone over-indexes on entertainment (K-pop fandom, sports). The hybrid covers both.

Mainstream RSS is the backbone (free, dense); paid-Apify social platforms run weekly on keywords/hashtags to keep cost down; YouTube uses the free Data API with two paths at once.

1. 01

   Trigger

   workers/celery_app.py + scripts/manual_briefing.py

   Celery cron is wired for Thursday 05:00 WIB (1h after the 04:00 topic recluster, a day before the Friday khutbah) — but PAUSED since 2026-05-23 to keep LLM cost at zero during the dev phase. Operator currently runs manually via `manual_briefing.py dump <segment>` which runs the data-prep pipeline and emits the prompt — Claude's reply is saved back via `manual_briefing.py save`. Re-enable the cron by uncommenting the block in celery_app.py.

2. 02

   Compute headline stats

   services/insights_summary.py::_compute_stats

   Seven-day aggregations from `social_posts`: sentiment mix, top categories with share-pct, top topics joined with sample headlines per topic. When no baseline week exists yet, delta_pp emits null instead of fabricating change. Per-segment briefings filter on dominant-cat above the 0.1 punt floor.

3. 03

   Retrieve daleel

   services/kitab_retrieval.py + rerank_daleel (top_n=10)

   Flash-Lite builds a retrieval query from stats. OpenAI embeds it. Qdrant searches all 6 kitab collections (~4 per corpus → 15 candidates). Flash-Lite second pass reranks the pool thematically to the 10 strongest matches. Hadith translation_id is filled in where empty — Quran already has both ID + EN (Kemenag + Sahih International). 10-deep pool (was 5) so each Section-4 sub-section can weave 2-3 inline citations without repeating.

4. 04

   Synthesize (Indonesian)

   Gemini 2.5 Pro (paused) → Claude manual

   Auto mode (cron enabled): Gemini 2.5 Pro called once with max_output 32,768 tokens + 12,288-token thinking budget. Current manual mode: paste prompt + context into Claude (chat), save reply via manual_briefing.py. Output is identical in both modes: 5-section markdown 7,300-9,800 words in Indonesian. Daleel-grounded — model can ONLY cite from the 10-daleel retrieved pool. English version paused since 2026-05-23 (fallback banner shown to EN readers).

5. 05

   Persist + render

   briefings → /briefings/[id] + Puppeteer flyers

   Writes a row to `insights_summaries` (summary_md, headline_stats, daleel_refs, segment, model, tokens, cost). Public page /briefings/brief/{slug} renders with sticky TOC + share/download toolbar. Section 4 is split into 6 interactive deliverable cards (click to open modal). Flyers are rendered on-demand by Puppeteer + HTML/Tailwind — 2 variants (general + Gen-Z) per briefing plus 6 per-deliverable variants. CDN-cached 1h, immutable.

1. 01

   Capture

   app/[locale]/briefs/new

   User submits topic + segment + tone + locale + optional extra context. Onboarding profile attached server-side.

2. 02

   Embed the topic

   lib/kitab-retrieval.ts

   OpenAI text-embedding-3-large turns the topic into a 3072-dim vector. Enriched with the audience segment label to bias retrieval toward audience-relevant verses, hadith, and tafsir passages.

3. 03

   Retrieve daleel

   Qdrant: all kitab collections

   Cosine search across all kitab collections in parallel (~4 candidates per corpus, ~15 total). A second Gemini Flash-Lite pass thematically reranks the pool down to the 5 most relevant — cosine alone surfaces verses that share tokens with the query but miss the actual da'wah angle. If retrieval fails, the brief generator surfaces a typed error to the UI.

4. 04

   Synthesize

   lib/llm.ts → generateJson()

   Gemini Pro (primary) gets the topic, profile, extra context, and retrieved daleel. Falls back to Claude Sonnet 4.5 if Gemini fails. Output is constrained to a JSON schema.

5. 05

   Validate & store

   Zod validation → briefs table

   Zod-validates the response shape, attaches the retrieved daleel array, persists to `briefs`. Failures surface as a typed error to the form, never a half-baked row.

Every entry in Qdrant carries (a) the original Arabic, (b) the translation in the brief's locale, and (c) the canonical citation (book + chapter + ayah/hadith number). Briefs always link back to the named source so the da'i can verify before delivery.

Browse the kitab library at /kitab

Why a monorepo with Python + TypeScript instead of a single language?

Python wins for the data/ML side (Gemini / OpenAI / Anthropic client integrations, Qdrant client, embedding pipeline — all most mature in Python). Next.js + Auth.js + Drizzle wins for the user-facing app. Trying to do ML in TS or Auth.js-equivalents in Python would have meant fighting the ecosystem at every step. The cost is two dependency managers, mitigated by uv + npm both being fast.

Why Qdrant instead of pgvector?

We already had Postgres for everything else, so pgvector was tempting. Qdrant won on filtering ergonomics (named payload fields, hybrid search), and on the fact that it's the dominant choice in production RAG stacks today — easier to find debugging help and tooling.

Why was IndoBERT retired and all sentiment moved to Gemini Flash-Lite?

It used to be a hybrid: Gemini event-valence for mainstream news, IndoBERT for social posts (X/IG/TikTok/YouTube). But a manual eval on 37 'korupsi' tweets on 2026-05-25 showed IndoBERT mislabelled 6 of 7 positives (sarcasm read as positive, supportive opinion as negative, rhetorical critique as neutral) — positive-class accuracy ~14%. After switching every platform to Gemini Flash-Lite, positive-class accuracy on X jumped to ~92% on the same sample. Extra cost: ~$1/mo for X sentiment, immaterial against the budget. Trade-off paid: torch + transformers + the HuggingFace cache (~1.5 GB worker image) leave the dependency tree, plus the worker cold-start gets ~10-15s faster.

Why Gemini Pro primary and Claude Sonnet fallback, not the reverse?

Gemini 2.5 Pro is ~35% cheaper than Sonnet 4.5 at comparable quality bracket for this kind of long-form structured-output task. Putting it first lowers the running cost meaningfully; Sonnet's the backstop when Google has an outage.

Why not use a framework like LangChain or LlamaIndex?

Our retrieval + synthesis flow is ~80 lines of TypeScript. Adding a framework would have added 200 transitive dependencies for value we don't use. We're not chaining many models or supporting many vector DBs — when those needs appear, the trade flips.

Why no Facebook scraper on the schedule?

A `scrape_facebook()` function exists in `apify.py`, but no beat entry calls it. Three reasons. (1) Facebook has been declining as a content surface for younger Indonesian Muslims — most of the genuinely-active da'wah conversation has migrated to TikTok and X. (2) Apify's Facebook actor is among the priciest per item, and the marginal yield per IDR didn't justify it. (3) Instagram already covers the Meta ecosystem on our active schedule. We keep the scraper code ready, so the day data tells us we're missing something it can be enabled with a one-line beat entry.