EMBEDDINGS.md

CLIP Embedding Similarity Examples

No longer used due to cost. Original code is in feature-experimental-clip branch. The code here was used to add the embeddings via a Colab notebook.

Below are some screenshots of CLIP Embedding similarity results.

Embeddings were added to Elasticsearch as dense vectors:

image: {
  ...
  embedding: {
    type: 'dense_vector',
    dims: 512,
    index: true,
    similarity: 'cosine',
  },
  ...
}

Example cosine similarity query:

const esQuery: T.SearchRequest = {
    index: 'art',
    query: {
      bool: {
        must: [
          { exists: { field: 'image.embedding' } },
          {
            script_score: {
              query: { match_all: {} },
              script: {
                source:
                  "cosineSimilarity(params.query_vector, 'image.embedding') + 1.0",
                params: { query_vector: input_vector },
              },
            },
          },
        ],
        must_not: {
          term: {
            id: document.id,
          },
        },
      },
    },
    from: 0,
    size: SIMILAR_PAGE_SIZE,
};

Examples: