Cofacts multimedia support research

Retrieving file data

https://developers.line.biz/en/reference/messaging-api/#get-content

Search & indexing

Research: all image retrieval methods https://g0v.hackmd.io/bhL6csQ8T1e81E2De7ZS5Q#Search-with-similarity
Image hash bit length https://g0v.hackmd.io/LHhF_VQ1RdS12C0k0ESQ_Q
Indexing in hamming space https://g0v.hackmd.io/xsDcMPySQM69vA0xHO8_dA#Indexing
Large-Scale Video Retrieval Using Image Queries - 介紹 https://www.youtube.com/watch?v=tLqbdQR7kjM
- 見下 vectors / documents 章節

Hashed file names

The OpenDream’s approach.

Convert each reported image / video to perceptual hash (fingerprint) stores the file using the hash in its name.
- hash: https://www.npmjs.com/package/image-hash
- image-hash research
During query, convert the query image / video in the same way and look up for the file with the same name in Google Drive directly.
- Query is performed on client side (chatbot), did not take 3rd party applications into consideration.

Hash for other file formats

Videos

ffmpeg extract

  # skip_frame: https://superuser.com/questions/669716/how-to-extract-all-key-frames-from-a-video-clip
  # remove dup: https://stackoverflow.com/questions/37088517/remove-sequentially-duplicate-frames-when-using-ffmpeg
  ffmpeg -threads 1 -skip_frame nokey -lowres 3 -i [video-file] -vf mpdecimate  -vsync 0 -f image2 output/file-%04d.jpg

https://www.npmtrends.com/ffmpeg-vs-ffmpeg-static-vs-ffmpeg.js-vs-fluent-ffmpeg-vs-@ffmpeg/ffmpeg

Audios

chromaprint (fpcalc)
- 越長的檔案 fingerprint 就越長

Vector / embedding based similarity

Systems supporting vector search

OpenSearch KNN https://opensearch.org/docs/latest/search-plugins/knn/index/
ElasticKNNs https://github.com/alexklibisz/elastiknn
Elasticsearch 8.6
- dense_vector fields
  - Max dimension: 1024 for indexed vectors
  - Indexing are for Approximate kNN
- Exact, brute-force knn w/ scripted score and cosineSimilarity() function
- Approximate kNN w/ HNSW algorithm
  - can combine with ordinary query and calculate score together
- Official guides
  - https://www.elastic.co/guide/en/elasticsearch/reference/8.6/knn-search.html
  - Blog post: https://www.elastic.co/blog/how-to-deploy-nlp-text-embeddings-and-vector-search#
  - Upload Python model on Huggingface to Elasticsearch https://www.elastic.co/guide/en/machine-learning/current/ml-nlp-text-emb-vector-search-example.html
    - Does not support image-related models (reference)

Systems / software for near duplicate video search

Near Duplicate Video Retrieval
- Handles these: side morrored, color-filtered, and waterwashed. Middle row: horizontal screen changed to vertical screen with large black margins. Botton row: rotated
videohash
- Python
- Near Duplicate Video Detection (Perceptual Video Hashing) - Get a 64-bit comparable hash-value for any video.
- Have collisions on unrelated video https://www.reddit.com/r/DataHoarder/comments/q74hkz/videohash_python_package_for_near_duplicate_video/

Methodologies

Image / video vectors

Large-Scale Image Retrieval with Elasticsearch http://nmis.isti.cnr.it/falchi/Draft/2018-SIGIR.pdf
- 2018 SIGIR
- 先把圖變成 R-MAC feature (multi-resolution, dimension=2048)
- 然後想辦法變成可以 encode 成 word 的東西塞進 Elasticsearch
The VISIONE Video Search System: Exploiting Off-the-Shelf Text Search Engines for Large-Scale Video Retrieval https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8321359/
- 影片轉 scene, object, color, CNN descriptors 轉成可以 encode 成 word 的東西
- “Scalar Quantization-based Surrogate Text representation”
Towards Practical Visual Search Engine Within Elasticsearch https://arxiv.org/pdf/1806.08896.pdf
- 也是轉成 word
Fisher vectors
- compact local descriptor as fixed-length vector, 用在 retrieval 上可做 vector comparison
- Asymmetric comparison of FV - 圖只佔一部分可找（think of image in LINE chat screenshots), 一種 bag of features
- Introduced by Perronnin and Dance, CVPR ‘07
- Analyzing Classifiers: Fisher Vectors and Deep Neural Networks
IVWBAE2019 - Video indexing and retrieval based on content: a systematic literature review
- Spreadsheet

Text embeddings

OpenAI text embedding
- Dimension = 1536
- normalized to 1 (dot product for cosine similarity)
- pgvector cosine similarity example https://github.com/mckaywrigley/wait-but-why-gpt
tf.js Multilingual Universal Sentense Encoder
- Input: Variable length text in Chinese, Chinese (Taiwan), English, etc
- Output: 512 dimension vector
- Can compare similarity across languages

Multimodal embeddings (image / text embedding)

Microsoft unilm https://twitter.com/alphasignalai/status/1630651280019292161
https://arXiv.org/abs/2302.14045
https://palm-e.github.io/
Visual foundation models in https://github.com/microsoft/visual-chatgpt
CLIP
- https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1
- Deploy -> Inference API -> JavaScript
- Input layer - max_seq_length: 128
- Output layer - out_features: 512
- https://www.sbert.net/examples/applications/image-search/README.html (seems to support both image and text inputs)
Facebook multimodal model ImageBind
- https://github.com/facebookresearch/ImageBind
- https://imagebind.metademolab.com/
- Also includes audio & video!
- HF: https://github.com/huggingface/transformers/issues/23240
- Input layer: TBD

Analysis

Benefits
- 無論是 FV 還是 embedding 應都有 locality similarity，裁切或截圖後仍有機會找得到
- 中文簡體繁體、英文原始訊息，有機會被 multilingual text embedding 放在很近的地方而檢索得到
- ==AI 繪圖、無聲音無字幕的影片，透過 multimodal embedding 可能可以達成，輸入文字敘述來搜尋相符圖片的事情==
Efforts to get these benefits
- Upgrade to ES 8.6 to get access to vector search
- Integrate tf.js or other vector generation methods
用在 Cofacts？
- 絕大多數有影響力的圖文都是文字，OCR 會比 embedding 更有用
- 文字搜尋上，embedding 是否有比 bm25 好或好多少，要看 Cofacts dataset 決定
- 即使圖片影片中沒有文字可 OCR，有錯的通常是在圖說。這樣一來，surrounding text 又比 vector search 重要。

File storage & hosting

Google drive

The OpenDream’s approach.

Google cloud storage

Database model

Considerations:

Store surrounding text?
Store text content of the image / video / voice message for full-text search

[!TIP] Prefer “In separate articles” - discussion

Use `articles`

The OpenDream’s approach.

Article text does not contain anything but $image__xxx

No surrounding text & text content for full-text search.

In `articles.attachments`

One article stores:

attachments: attachment data and its trasncription text for full-text search
- Therefore search queries should also look up this field
- highlights should also consider this field in addition
text: the surrounding text

✅ In separate `articles`

One article stores:

attachment: the multimedia data
text: its transcription for full-text search

Another article stores:

text: the surrounding text

Connect related articles together in other ways

Easier search (no extra field) but associations after search hit may be complex
- Should seek replies in related article and display?

Separate MediaEntry index

Article stores only

text, which is the transcription for full-text search

New index mediaentries stores:

_id: the hash
articleId: maps to the article storing the transcription and replies
url: the URL to the file on GCS
Other feature vectors for similar image retrieval
Other meta data like media type

Pros

Each hash can only map to 1 media entry
- Cannot have 2 articles having same hash
Faster query time by hash
- Super fast retrieval by hash (by ID, no index time)
- faster time to retrieve by feature vector due to fewer documents to scan / index
Can be separated from Cofacts article logic
- Media manager can just feed an Elastic client to media manager
- Media manager can manage the index by itself

Cons

Still possible to have 2 articles created
- Process: Search mediaentries index –> If no, create article –> write mediaentries index
- It is still a non-atomic read-then-write, so article creation is still possible
- If this occurs, there will be a empty article with no media entries pointing to it.
Cannot mix image & text query together
Increased complexity of managing a separate index

Surrounding text (cooccurrence) model

Separate ArticleGroup index

Create another index, articlegroups, to place co-occurrences of article groups.
ID: sorted article IDs –> unique ID for the same composition
Does not have dedicated reply-request and replies
Just records a link
Q: If a reply is used in multiple articles in the same group, when user gives feedback, whose article-reply-feedback is this?

Processing multiple messages on LINE bot

Article group as an `article`

No text nor attachment, but has an articleIds field pointing to ID of individual articles
Has its own reply requests & replies

Common problems

How to count occurrences / popularity of a specific article group

Processor model

Handle in rumors-line-bot

OpenDream’s approach: search logic, hashing is done on line bot

✅ Dedicated multimedia processor

[!TIP] Design doc: Cofacts media manager implementation

Add-on of current API server
- API server can serve multiple clients
- can limit access of related API using API kjeys
Can be implemented stand-alone
- Replace current google drive first
- Article integration can come later

input:

URL to fetch data
should insert or just query

output: list of deduped

public file url
identifier
similarity (1 for exact match)

processing:

The processor handles download, storage, resolving and search function.
It stream uploads the file onto google cloud storage.
[OPTIONAL] It caches file content of an URL for a short while, so that query –> create via same URL works smoothly

Processor implementation

See Cofacts media processor implementation

Cofacts multimedia support research

Retrieving file data

Search & indexing

Hashed file names

Vector / embedding based similarity

Systems supporting vector search

Systems / software for near duplicate video search

Methodologies

Analysis

File storage & hosting

Google drive

Google cloud storage

Database model

Use articles

In articles.attachments

✅ In separate articles

Separate MediaEntry index

Surrounding text (cooccurrence) model

Separate ArticleGroup index

Article group as an article

Common problems

Processor model

Handle in rumors-line-bot

✅ Dedicated multimedia processor

Processor implementation

Other topics

stream file convert

Transcriptions

OCR

When to apply OCR

Metadata extraction

What to extract

When to extract

How to extract

Video summarization for thumbnails

Videos on Youtube / links

Youtube data API

oembed

schema.org microdata

Crowdsourced metadata

Visualizing audio

Use `articles`

In `articles.attachments`

✅ In separate `articles`

Article group as an `article`