The single index of every "Bad X" metric Lighthouse reports (the KPIs → Bad
Botim board and the module pages behind it): what each one means, the exact
formula, its thresholds, which system the data comes from, and how fresh /
settled it is. Deeper per-metric specs stay in their own docs (linked); when
this page and a linked spec disagree, the spec wins. Threshold/band changes
happen in server/queries/kpis/bad-botim.ts (BANDS) and per-metric ETLs —
update this page in the same PR.
Last audited: 2026-09-10 (main @ 91ccf94).
| Module | Bad definition | Bands (good ≤ / watch ≤) | Origin source | Lag / maturity |
|---|---|---|---|---|
| Bad P2P call | cnt_any_bad_call / total_calls — any of {video freeze>5%, black-screen, low-res, audio freeze, no-audio, connecting>5%, init-connect-fail, RTT>500} |
11% / 14% | StarRocks ads_botim_voip_result_van_bad_call_1d_di |
T+1, lag-free |
| Bad ring | cnt_total_call_setup_bad_call / total_paired_calls ≈ delay-ring (~3.4%) + silence-never-ring (~13%) |
18% / 22% | StarRocks ads_botim_voip_result_call_setup_1d_di |
T+7 lagged — trailing ~6 zero days trimmed; partially-ingested tail days still render |
| Bad meet call | bad_call_total / total_calls, participant grain — union of 10 flags {join_failed, high_rtt, slow_join, socket_failed/slow, audio/video freeze/loss} |
34% / 40% | StarRocks ads_botim_meet_call_quality_stats_di |
T+1 |
| Bad P2P rating | 1 − (4★+5★)/(all 1–5★) — the 1–3★ share of end-of-call ratings, long+short, Overall | 15% / 18% | StarRocks ex_algento.botim_pd.voip_rating_stat_v2 |
ratings settle ~2 days late |
| Bad MP | sessions with fatal_count>0 ÷ total sessions (session 口径) |
1.0% / 1.5% | Fabric gold.fact_mp_session |
T+1 gold partition |
| Bad app | Bad App Rate (COR-497): users with ≥1 threshold-crossing signal ÷ DAU; composite = EN (footprint errors ∪ network) + CA (fg crash ∪ ANR), android+ios folded DAU-weighted on the KPIs board. Full spec: app_bad_app_definition_v1.md | 2.5% / 3.5% | Lighthouse warehouse fct_bad_app_daily + fct_crash_daily (sources in §2) |
trailing 3 days provisional (ETL re-refresh; ES→Fabric exact swap; next-launch crash reports) — KPIs board renders them dashed |
| Bad KYC | bad applies ÷ all applies; bad = load>200ms OR "didn't work" dropoff OR SDK error OR pending>1d. Days with reason-flag coverage <90% dropped | 2% / 4% | Fabric gold.gold.fact_kyc_attempt (bad_kyc_model) |
T+1; today (partial) dropped |
| Bad server | 1 − traffic-weighted daily success% (Σ success%·qps / Σ qps), all backend services, per-service return-code allowlist | 0.1% / 0.3% | Prometheus zus_server_requests_seconds → warehouse server_overview_hourly |
snapshot ETL every ~10 min, 90d retention |
Related-but-separate definitions (not on the Bad Botim board): - Bad call (V4/in-call detail) — v4_bad_call_definition_v2.md, part1-badcall-definition-en.md. - New-issue crash reach gate (rollout reports) — 0.3% same-window device-space reach + growth guard; in app_bad_app_definition_v1.md.
| Table | Feeds | Notes |
|---|---|---|
ads_botim_voip_result_van_bad_call_1d_di |
Bad P2P call; VoIP quality report | daily, per-dim columns (region/version/operator/line/platform/pp/vpn) |
ads_botim_voip_result_call_setup_1d_di |
Bad ring; never-ring/delay-ring analyses | T+7 late-arriving; target_tag empty on never-ring calls (they die before line assignment) |
ads_botim_meet_call_quality_stats_di |
Bad meet call | participant grain |
ex_algento.botim_pd.voip_rating_stat_v2 |
Bad P2P rating; CSAT module | ~2d settle |
ex_algento.botim_pd.dwd_botim_logs_k8s_statlog_server_call_trace_di |
server signaling/push funnel diagnostics (never-ring RCA) | per-event rows keyed by trace_id; heavy scans — bound windows |
| Table | Feeds | Notes |
|---|---|---|
silver.botim_footprint_extra |
Bad app EN exact (fabric ETL, lookback 4d); ad-hoc stream/code drilldowns | full population; typed columns incl. event_code/event_details/event_context, device_key; endpoint can degrade for hours (metadata COUNTs stay fast while column reads time out — seen 2026-09-02) |
silver.botim_footprint_appactive |
Bad app DAU/denominators exact; version-mix / coverage | device_key = device grain; no launch counts (ES-only) |
gold.fact_mp_session |
Bad MP | session grain, fatal_count |
gold.gold.fact_kyc_attempt |
Bad KYC | bad_kyc_model flags |
| Index / transform | Feeds | Notes |
|---|---|---|
logs-footprint-extra-* |
Bad app EN same-day (1/101 uidHashMod101 sample ×101); event-stream pages |
~2-3 days full retention, then rolling deletion — platform/dim-FILTERED event views fall back to live ES and undercount hard on older days (measured: D-3 ≈ 25%, D-7 ≈ 1%); unfiltered views read the persisted warehouse rollups instead |
logs-footprint-appactive-* |
Bad app DAU (HLL) + launches (SUM(count), the ONLY launch source) |
baseInfo.deviceType: 0=iOS, 1=Android |
tra_messenger_error_daily (ES transform) |
For Dev pages for kMessengerError (All grid / by-dim / event detail / scenario) since 2026-09-09 | daily transform rollup; history starts at transform creation, no backfill; other events stay on the warehouse rollups |
logs-footprint-voipstat-*, logs-footprint-call-analytics-* |
V4/legacy VoIP modules | never sort/aggregate on _id (fielddata incident 2026-08/09) |
| Source | Feeds | Notes |
|---|---|---|
Crashlytics BigQuery export (bot-messenger-8ce54.firebase_crashlytics.*, Android+iOS, REALTIME+batch) |
Bad app CA (fg-only FATAL/ANR devices, installation_uuid); new-issue detection |
crash events only — NO denominator of its own (no GA4 export on this project); day-partitioned on event_timestamp — always bound both ends; reports arrive on next app launch (late accrual) |
| Play Developer Reporting API | Play benchmark panel (fct_play_vitals_daily) |
PLAY days = America/Los_Angeles, 2–3d lag; distinct_users is order-of-magnitude rounded — corroboration only, never a denominator |
| Metric | Feeds | Notes |
|---|---|---|
zus_server_requests_seconds |
Bad server (via server_overview_hourly snapshots) |
every backend service; QPS-weighted daily aggregation |
| Table | Written by | Read by |
|---|---|---|
fct_bad_app_daily |
bad-app ES ETL (sampled, trailing 3d) + fabric ETL (exact, overwrites; src column) |
/app quality pages, KPIs Bad app, app-daily-bar report |
fct_crash_daily |
crashlytics ETL (fg-only, per-dim) | /app quality (CA), rollout reports |
fct_play_vitals_daily, dim_app_version_map |
play-vitals ETL | Play benchmark panel |
fct_app_error_hourly |
app-errors hourly rollup ETL (365d retention; dims: total/version/region/netType/brand/w/platform marginals — no platform×dim composites, hence the live-ES fallback above) | For Dev event pages (except kMessengerError → ES transform) |
server_overview_hourly |
server-overview snapshot ETL (90d) | /botim-server, Bad server |
Value key — ★★★ core (drives decisions directly) · ★★ supporting (context, corroboration, drilldown) · ★ situational (real caveats; fit-for-purpose only).
| Source | Modules | Value | Why / caveats |
|---|---|---|---|
ads_botim_voip_result_van_bad_call_1d_di (+ _summary_ variant) |
VoIP quality, KPIs p2p, VoIP 60d report | ★★★ | THE P2P bad-call ledger; T+1, rich dims |
ads_botim_voip_result_call_setup_1d_di (+ summary) |
Bad ring, never-ring analyses | ★★★ | only setup-funnel source; T+7 lag is the one trap |
ads_botim_meet_call_quality_stats_di, ads_botim_meet_quality_stats_di |
Meet | ★★★ | participant-grain quality |
voip_rating_stat_v2 |
CSAT, Bad rating | ★★★ | the user-voice metric; ~2d settle |
dwd_botim_logs_k8s_statlog_server_call_trace_di |
server-trace funnel, incident RCA | ★★ | per-trace push/signaling funnel; heavy — bound windows; proved out in the Aug-26 never-ring RCA |
dwd_botim_voip_call_roomid_match_7d_di |
call pairing/legacy detail | ★★ | join helper |
dwd_botim_logs_server_otp_session_di |
OTP/onboarding | ★★ | vendor split by ptype; count ROWS not sessions |
ads_botim_voip_user_behavior_event_agg |
Behavior module | ★ | device_type 0=iOS (non-standard), non-additive counts, UNDER-counts Android — never use for platform splits |
| Source | Modules | Value | Why / caveats |
|---|---|---|---|
logs-footprint-voipstat-* |
V4 warehouse chain, legacy V3, compare/eBPF | ★★★ | in-call telemetry backbone; never sort/agg on _id |
logs-footprint-call-analytics-* |
V4 chain, VQC, trace | ★★★ | pairs with voipstat |
logs-footprint-extra-* |
Bad app EN (1/101 sample), event streams, kuitrack/kusage, msg-send-fail, search, homepage funnel | ★★★ same-day / ★ history | irreplaceable for TODAY; ~2-3d full retention — history questions belong to Fabric or the warehouse rollups |
logs-footprint-appactive-* |
DAU + launches | ★★★ | the ONLY launch-count source |
logs-footprint-mp-*, logs-footprint-advertise-* |
MP live drilldowns, Ads ETL | ★★ | same retention rules |
tra_messenger_error_daily (+ _hourly, tra_role) |
For Dev kMessengerError pages | ★★ | daily transform, dims baked in; history from creation only, no backfill |
| Cases per-user call legs (ES backfill) | Cases module | ★★ | on-demand per-user, retention-bound |
| Source | Modules | Value | Why / caveats |
|---|---|---|---|
silver.botim_footprint_extra / _appactive |
Bad app exact (EN/DAU/device_key), version mix, rollout reports | ★★★ | the EXACT full-population source; endpoint can stall for hours (metadata fast, column reads dead — 2026-09-02) |
silver.botim_footprint_mp + gold.fact_mp_session + gold.agg_mp_dau_daily |
MP module | ★★★ | session/fatal 口径; agg_mp_dau is MP-scoped, NOT app DAU |
gold.gold.fact_kyc_attempt |
Money/KYC | ★★★ | bad_kyc_model flags |
gold.gold.fact_loan_book, gold.fact_trans_merchant_payments, silver.merchant_t_merchant_creation_order |
Lending, Merchant | ★★ | finance-side facts |
| Source | Modules | Value | Why / caveats |
|---|---|---|---|
| Crashlytics BigQuery (android+ios, REALTIME/batch) | Bad app CA, new-crash detection, rollout gates | ★★★ | issue-level detail; fg-only filter mandatory; no denominator of its own; next-launch late accrual |
| Play Developer Reporting API | Play benchmark | ★★ | official yardstick; LA days, 2-3d lag, rounded counts — reconcile, never join |
| Source | Modules | Value | Why / caveats |
|---|---|---|---|
Prometheus zus_server_requests_seconds* |
Bad server, /botim-server fleet | ★★★ | every backend service; snapshot ETL is the freshness ceiling |
| OpenObserve traces | service/method dependency maps | ★★ | topology + method edges |
Archery phoneservicelog |
SMS module | ★★ | 5-min aggregates mirror |
CS Connector (aesp / csconnector API) |
Cases feedback ingest | ★★ | ticket→telemetry bridge |
Local warehouse (MySQL) — ods_summary_raw, fct_*, server_overview_hourly, alert_*, appdata |
every page reads HERE first | ★★★ | the serving layer: pages read the warehouse, not upstreams; freshness = each ETL's cadence |
The same inventory pivoted the way the org plans work: one row set per business line — its Lighthouse surfaces, headline metrics, and every upstream it depends on. (A source appearing under two lines is real shared infrastructure, not duplication.)
| Surfaces | /voip quality + legacy V3, V4 model, Meet, CSAT, Behavior, Compare/eBPF, server-trace, VoIP 60d report, voip-daily/weekly Teams cards |
| Headline metrics | Bad P2P call · Bad ring · Bad meet call · Bad P2P rating · V4 bad call (spec v2) |
| Sources | ES logs-footprint-voipstat-* + call-analytics-*; StarRocks van_bad / call_setup / meet ×2 / rating_v2 / behavior_agg (★ caveats) / roomid_match; server_call_trace for signaling/push RCA |
| Watch out | ring T+7; rating ~2d; behavior_agg platform 口径; _id sort ban on ES |
| Surfaces | /app quality (BAR, compare + DAU companion), For Dev event streams + Network Errors, custom dashboards (network success/usage, pipeline health), app-daily-bar Teams report, rollout go/no-go reports |
| Headline metrics | Bad App Rate (EN+CA) · per-stream error users · new-issue crash reach (0.3% gate) |
| Sources | ES footprint extra/appactive (same-day, 1/101 sample; launches ONLY here); Fabric silver footprint (exact history); Crashlytics BQ (fg crash/ANR, issue detail); Play Reporting API (benchmark); warehouse fct_bad_app_daily/fct_crash_daily/fct_play_vitals_daily/fct_app_error_hourly |
| Watch out | trailing 3d provisional; filtered event views fall to live-ES (2-3d) — history via Fabric/warehouse |
| Surfaces | Chat Lighthouse, Message tracking (msg-stats scenarios: GIF, unknown_msg, celebration…), voice-note playback, msg-send-fail |
| Headline metrics | send-fail rate · scenario adoption/error rates · kMessengerError streams |
| Sources | ES footprint extra (kUITrack / kMessengerError / kNetwork); ES transform tra_messenger_error_daily (since 2026-09-09, no backfill); warehouse kuitrack/kusage rollups |
| Watch out | kUITrack warehouse refreshes daily — day-grain by design (bucket pinned) |
| Surfaces | /mp module (fatal rate, sessions, ttfr, reason/source drilldowns), dev-platform stats |
| Headline metrics | Bad MP (session fatal rate) · kMPFatal 三口径 · ttfr_ms |
| Sources | Fabric gold.fact_mp_session + agg_mp_dau_daily (MP-scoped DAU!) + silver footprint_mp; ES logs-footprint-mp-* for on-demand reason/source |
| Watch out | first_render 埋点 under-fires (ttfr caveat); agg_mp_dau ≠ app DAU |
| Surfaces | /money KYC funnel, /lending (EasyCash…), /merchant bad-merchant monitor |
| Headline metrics | Bad KYC · loan-book funnel · bad-merchant families |
| Sources | Fabric gold.gold.fact_kyc_attempt / fact_loan_book / fact_trans_merchant_payments / silver.merchant_t_merchant_creation_order; TiDB-DW via Grafana org-1 (merchant); ES kMPPageView for EasyCash H5 funnel |
| Watch out | KYC completion must use server-side (front-end under-fires); TiDB ds needs org-1 Basic-Auth |
| Surfaces | /ads module (funnel, no-show reasons, revenue), ads-daily-vitals alert |
| Headline metrics | show rate · CTR · skip rate · revenue |
| Sources | ES logs-footprint-advertise-* → warehouse fct_ads_daily; ads-revenue ETL (response_id join) |
| Watch out | join on response_id, not top-level rid |
| Surfaces | /botim-server (fleet, per-service detail, daily AI summaries), dependency maps, Bad server on KPIs |
| Headline metrics | fleet failure rate · per-service success%/latency · uptime |
| Sources | Prometheus zus_server_requests_seconds* → server_overview_hourly (90d); OpenObserve traces (service/method edges) |
| Watch out | pages read snapshots — freshness = snapshot cadence (~10 min) |
| Surfaces | /onboarding OTP page, Bad-OTP dashboard, homepage funnel, /search |
| Headline metrics | OTP success/cost by vendor · funnel conversion · search traffic/CTR |
| Sources | StarRocks dwd_botim_logs_server_otp_session_di; ES footprint (OTP trace, kSearch, homepage events); homepage-funnel + otp-footprint ETLs |
| Watch out | OTP volume by ROWS not deduped sessions; purpose 8 = non-OTP (Etisalat) |
| Surfaces | /cases (feedback → phone → VoIP telemetry root-cause, call archive, cohorts) |
| Headline metrics | case volume · root-cause classes |
| Sources | CS Connector API (tickets); ES per-user call-leg backfill |
| Watch out | ES retention bounds the backfill window |
| Surfaces | /sms module |
| Headline metrics | delivery/latency aggregates |
| Sources | Archery phoneservicelog 5-min aggregates → warehouse |
| Watch out | mirror cadence 15 min |
web/src/routes/kpis/BadBotimSection.tsx (DASH / ANNOT).