Skip to content

Latest commit

 

History

History
113 lines (100 loc) · 3.54 KB

ReadMe.md

File metadata and controls

113 lines (100 loc) · 3.54 KB
Raw

Similar Manga Recommendations

This repo has both the scraping and matching utilities to find mangas which are close in content to others. The idea is to create a recommendation system outside MangaDex since there isn't one and thus allow for users to discover other content. Right now it is pretty rudimentary, but seems to give ok results most of the time. If you want to use it, please download Neko and enable recommendations inside the app or use the Chrome Extension.

Setup / Dependencies

Client was generated using swagger. You will need to setup a golang workspace, and then run the following commands. Only manga need to be downloaded / scraped from mangadex to be able to perform similar manga identification.

go get golang.org/x/oauth2
go get github.com/antihax/optional
go get github.com/james-bowman/nlp
go get github.com/caneroj1/stemmer
go get github.com/PuerkitoBio/goquery

Runtime Instructions

  1. Run ./1_cache_mangas to get local list of all mangas on the site (28 hours for 73k manga)
  2. Run ./2_calc_similar to generate similarity index between manga (6 hours)
  3. Run ./3_calc_mappings to export external ID to mangadex UUID mapping files (2 minutes)
  4. Run ./4_calc_mu_ids convert MangaUpdate ids into uniform "new" ids for their new api (27 hours)
  5. Run python utils/create_sqlite.py to generate mapping file for Neko

Manga Links Data

Key Related site URL URL details
al anilist https://anilist.co/manga/`{id}` Stored as id
ap animeplanet https://www.anime-planet.com/manga/`{slug}` Stored as slug
bw bookwalker.jp https://bookwalker.jp/`{slug}` Stored has "series/{id}"
mu mangaupdates https://www.mangaupdates.com/series.html?id=`{id}` Stored has id
nu novelupdates https://www.novelupdates.com/series/`{slug}` Stored has slug
kt kitsu.io https://kitsu.io/api/edge/manga/`{id}` or https://kitsu.io/api/edge/manga?filter[slug]={slug} If integer, use id version of the URL, otherwise use slug one
amz amazon N/A Stored as full URL
ebj ebookjapan N/A Stored as full URL
mal myanimelist https://myanimelist.net/manga/{id} Store as id
raw N/A N/A Stored as full URL, untranslated stuff URL (original language)
engtl N/A N/A Stored as full URL, official english licenced URL