Add Diagrams

[acelinkio]

Adds several diagrams:

• Repo structure
• C4 Context
• C4 Container
• C4 Component: Autosync
• C4 Component: Back
• C4 Component: Front
• C4 Component: Scanner
• C4 Component: Transcoder

[acelinkio]

Do have a couple of questions:

• Why does the frontend need to connect to backend? afaik frontend just serves static content
• What does autosync do?
• What publishes/subscribes to each queue? Some of those queues might be internal to a system and others used to share between.
• Backend has a metadata volume. What exactly is inside of that?
• Transcoder has a metadata volume. What exactly is inside of that?

[zoriya]

Why does the frontend need to connect to backend? afaik frontend just serves static content

The frontend does SSR, if you request /movie/alien, the frontend will first fetch the backend and then give back an html page with all the content (you can see this if you disable the JS on your browser, most pages will work).

What does autosync do?

This is the service that will listen to watch status changes and update it on external website (for now only SIMKL)

What publishes/subscribes to each queue? Some of those queues might be internal to a system and others used to share between.

The scanner pushes to the scanner queue when new files are found in the library. This is read by the matcher.
The scanner.rescan queue is read by the scanner (pushed via an API call to the backend) and triggers a whole library rescan.

Backend has a metadata volume. What exactly is inside of that?

This volume contains images (poster, thumbnails...). This gets downloaded from images URL saved in the database.

Transcoder has a metadata volume. What exactly is inside of that?

This contains:

• Subtitle & Fonts extracted from the video
• Thumbnail preview images used when seeking (images like the one below).
• And metadata information/keyframes extracted from the video and stored in a json file as a cache. I do want to move this to postgres (would make cache invalidation when we update the code easier).

[acelinkio]

Not really happy with the way Mermaid is rendering these diagrams. May look to moving away from their c4 https://mermaid.js.org/syntax/c4.html. Trying to avoid drawing this by hand.

Last commit included updates for:

• RabbitMQ Exchanges/Queue mapping
• List which applications produce/consume from exchanges/queues
• Add autosync and activitytracker info

[zoriya]

Just wrote my thought, keep in mind I had never seen a c4 diagram. Overall, I think those diagrams would make Kyoo more understandable for potential contributors. Thanks for taking the time to make those!

I feel arrows directions sometimes are wrong, mixing both consume/produce makes is hard to know how data flows.

On the scanner diagram:

• scanner.resan is consumed by the scanner, not the matcher.
• why are the scanner queue and the scanner.rescan queue not defined on the same ContainerBoundary? Not sure if this is an oversight or for another reason.

For the transcoder, the API's only interaction with it is to proxy requests from the front (for example, when calling /api/movie/alien/info or /api/episode/got-s1e1/master.m3u8). The backend does not use anything else from the transcoder.
The backend currently proxies requests to check JWT and permissions. I'd ideally want to move this logic elsewhere, but not sure how to proceed.

For the back diagram, there is no mention of postgres or meilisearch. Also, I don't know if mentioning all the others components (like scanner/front/autosync) makes sense. The backend is technically independent of those systems.

[acelinkio]

I feel arrows directions sometimes are wrong, mixing both consume/produce makes is hard to know how data flows.

fixed

scanner.resan is consumed by the scanner, not the matcher.

fixed

why are the scanner queue and the scanner.rescan queue not defined on the same ContainerBoundary? Not sure if this is an oversight or for another reason.

Middleware like messaging is hard to represent. Typically I would always assign ownership to the object producing the exchange/queue. kyoo_back creates the scanner.rescan ContainerQueue and is not a member of scanner. emb (EnterpriseMessageBus) CountainerBoundary is an entity used to represent those inter-container communication and also help with diagram formatting.

Anything intra-container communication is represented inside of the ContainerBoundary. scanner queue is internal to scanner ContainerBoundary.

For the back diagram, there is no mention of postgres or meilisearch. Also, I don't know if mentioning all the others components (like scanner/front/autosync) makes sense. The backend is technically independent of those systems.

postgres and meilisearch are the database icons in the diagram. Screencap to point out where. I'll try to see if I can customize that to make them pop more.

The reason why scanner/front/autosync are mentioned is because their relationships. kyoo_front directly connects to kyoo_back. kyoo_back also produces messages that are consumed by kyoo_scanner and kyoo_autosync.

For the transcoder, the API's only interaction with it is to proxy requests from the front (for example, when calling /api/movie/alien/info or /api/episode/got-s1e1/master.m3u8). The backend does not use anything else from the transcoder.
The backend currently proxies requests to check JWT and permissions. I'd ideally want to move this logic elsewhere, but not sure how to proceed.

Definitely down to talk that out. Diagram is meant to talk about the current state. There are a couple of things we can brainstorm about in the future: TranscoderMetadata, creating a kyoo_matcher image, and etc. Happy to do that through Discord, posts can be a little wordy.

[acelinkio]

Removing draft. Think the diagrams are in a good enough state to check in. I will definitely help maintain them in the future too.

[zoriya]

why is this an half block?

[acelinkio]

Definitely have a desire for each component to be the same size. I do not see a way to do it within the block diagram atm.

[zoriya]

we should specify somehow that this is the matcher, don't you think?

[acelinkio]

I definitely agree. Lets create a matcher image and then update this diagram. These diagrams should be as close to what is actually deployed.

Ideally we should not be required specify docker args at deployment time.

[zoriya]

I don't think having a copy of an image with only the flag changing is a good way.

Even grapahana does that with a flag for reader/writter (I copied their way when doing that).

[zoriya]

I think defining those terms would be helpful

[acelinkio]

We can definitely add a glossary. These are software architecture diagrams, so there is some expectation of knowledge.

Mermaid is still working on adding a legends feature for c4 diagrams. If those were added I would describe the protocol and ports each relationship made.

[zoriya]

Maybe we should add a few lines to explain what each services does. Transcoder seems obvious but at first glance we might not know that the front does SSR or what autosync does.

[acelinkio]

Really happy this merge request is opening this dialog

Diagrams should be supplemental. More than happy to start working on Github pages docs with you. That would be a better place to describe each component in detail.

[zoriya]

Yes I agree with you but I don't want something too boilerplate-y. Having a doc too far from the code would either become outdated or not read.

What do you think about having a short document containing a one line description of each service and their relations to each other (diagrams) and a readme per service which will be code focused (with mermaid if necessary).

I started writing blogs for deep technical stuff about a service, I'll try to keep this going and linking them in the doc here!