tags: cofacts, meeting note GA: UA-98468513-3 —

20250324 會議記錄

:::info

所有會議記錄
線上出席：T
NPO Hub: nonumpa, orz, bil
https://gather.town/app/z3x18KQFgZCX8MeZ/cofacts :::

:potable_water: Release pipeline

:star: Released to production

:electric_plug: API

Langfuse experiments and openapi.json admin access https://github.com/cofacts/rumors-api/releases/tag/release%2F20250318

:robot_face: rumors-line-bot

GraphQL throw batch error https://github.com/cofacts/rumors-line-bot/releases/tag/release/20250318

Langfuse update

Clickhouse 又牙起來了，staging server load 飆到 4
相同的 error 每秒噴好幾次

MergeTreeBackgroundExecutor: Exception while executing background task {a15ed9cd-6969-4d01-a8f5-612a98bcd959::202503_217315_228114_196}: Code: 241. DB::Exception: (total) memory limit exceeded: would use 1.65 GiB (attempt to allocate chunk of 4361808 bytes), current RSS 1.93 GiB, maximum: 1.93 GiB. (MEMORY_LIMIT_EXCEEDED), Stack trace (when copying this message, always include the lines below):

惡性循環： trace_log 變大 –> 觸發 merge tree –> memory 不夠 –> error 又要進入 trace_log –> 觸發 merge tree.

  SELECT
      database,
      `table`,
      formatReadableSize(sum(bytes_on_disk)) AS total_size
  FROM system.parts
  WHERE active
  GROUP BY
      database,
      `table`
  ORDER BY sum(bytes_on_disk) DESC

  Query id: 4e750b17-3069-4029-a706-57c93868820f

      ┌─database─┬─table───────────────────┬─total_size─┐
   1. │ system   │ trace_log               │ 11.07 GiB  │
   2. │ system   │ text_log                │ 598.68 MiB │
   3. │ system   │ metric_log              │ 177.76 MiB │
   4. │ system   │ asynchronous_metric_log │ 94.78 MiB  │
   5. │ system   │ opentelemetry_span_log  │ 16.79 MiB  │
   6. │ default  │ observations            │ 8.89 MiB   │
   7. │ system   │ query_log_0             │ 7.67 MiB   │
   8. │ system   │ part_log                │ 6.85 MiB   │
   9. │ system   │ processors_profile_log  │ 6.40 MiB   │
  10. │ default  │ traces                  │ 5.63 MiB   │
  11. │ default  │ event_log               │ 2.23 MiB   │
  12. │ system   │ query_log               │ 1.83 MiB   │
  13. │ system   │ query_views_log         │ 684.09 KiB │
  14. │ system   │ error_log               │ 619.39 KiB │
  15. │ default  │ scores                  │ 273.98 KiB │
  16. │ system   │ query_metric_log        │ 243.14 KiB │
  17. │ system   │ asynchronous_insert_log │ 207.28 KiB │
  18. │ default  │ schema_migrations       │ 618.00 B   │
  19. │ default  │ project_environments    │ 367.00 B   │
      └──────────┴─────────────────────────┴────────────┘

暫解：truncate 掉 system.*_log 然後重啟 clickhouse server
- 觀察 load 下降
完全關掉 system log tables
- 就算刪掉還是會被在 flush period 後重建 https://kb.altinity.com/altinity-kb-setup-and-maintenance/altinity-kb-system-tables-eat-my-disk/
- Config 裡能關的都關了
- 2025/3/24 20:10 是 13.64MB

CCPRIP

[Op] Automatic takedown

修復 bug: https://github.com/cofacts/takedowns/pull/188
讓 error 能 fail workflow: https://github.com/cofacts/takedowns/pull/189
需要 detect articles & reply request comments QQ
- comment: https://cofacts.github.io/community-builder/#/editorworks?type=2&day=365&userId=gOhKQZUBYrjt7MSMEiku&showAll=1
- articles: https://cofacts.github.io/community-builder/#/editorworks?type=2&day=365&userId=j4S8C_VvVO5-QRczPhyOyfaL-VQU-D1BnonMpo7Ro2HWjjAIg&showAll=1
- articles: https://cofacts.github.io/community-builder/#/editorworks?type=2&day=365&userId=j4S8C_fLmWyFJ6eEbOW0W8yviuJrekyEavCrB1yb9DDeYs34Y&showAll=1
Article & replyrequest –> 開票 https://github.com/cofacts/takedowns/issues/192
Web 送出的時候用 turnstile 偵測 bot –> 開票 https://github.com/cofacts/rumors-site/issues/598
開票就能用 MCP 讀 XD

[Comm] LLM transcript

Total cost forecast this month: 284 USD
Try out audio only gpt-4o-transcribe
- 如果沒人聲，還會 hallucinate 嗎
- 如果很快且在沒人聲時會閉嘴，那或許可以墊在 Gemini 前面？
  - Trade-off: 會無法處理「人聲與字幕不符」or 影像有更多資訊的影片

大松準備

3/29 大松: orz, nonumpa, bil (已新增提案列表)
需要拖鞋，有貓咪
徵求查核協作者、開發者

小聚籌備

4/20 (日) 2pm - 5 pm

Analytics issue

只有 LINE bot 的是好的，其他壞光惹

MAFINDO

搜尋到一個印尼的團隊似乎是與錯假訊息因應有關

MAFINDO (Indonesia) https://mafindo.or.id/