Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ feat: Add streaming support #125

Open
wants to merge 39 commits into
base: main
Choose a base branch
from
Open

✨ feat: Add streaming support #125

wants to merge 39 commits into from

Conversation

wanjohiryan
Copy link
Member

@wanjohiryan wanjohiryan commented Nov 3, 2024

This adds:

  • Keyboard and mouse handling on the frontend
  • Video and audio streaming from the backend to the frontend
  • Input server that works with Websockets

Update - 17/11

  • Master docker container to run this
  • Steam runtime
  • Entrypoint.sh

Copy link

cloudflare-workers-and-pages bot commented Nov 3, 2024

Deploying nestri with  Cloudflare Pages  Cloudflare Pages

Latest commit: ed9b48d
Status: ✅  Deploy successful!
Preview URL: https://e515058f.nestri.pages.dev
Branch Preview URL: https://feat-stream.nestri.pages.dev

View logs

wanjohiryan and others added 10 commits November 11, 2024 16:31
I've finished cooking 🍝

---------

Co-authored-by: DatCaptainHorse <[email protected]>
…ging (#129)

Adds a functional NESTRI_PARAMS env variable to container image,
allowing usage like:
```docker run -e NESTRI_PARAMS='--no-audio=true --verbose=true'```

Also adds a no-audio option to help debug stream issues.
Also also, changes the container builder portion to use the actual main nestri repo.

---------

Co-authored-by: DatCaptainHorse <[email protected]>
@wanjohiryan
Copy link
Member Author

wanjohiryan commented Nov 14, 2024

I just added the first part of handling websockets thru a relay, partykit

It works locally, i have yet to deploy the server side end of things

FIXME: I think I overwrote the changes done by DatHorse in the prior branch :(
TODO:

  • Make sure it works with the remote Cloudflare partykit server(s)
  • Test the latency, whether it has improved or dropped

UPDATE: This was scraped, too much overhead :/

wanjohiryan and others added 4 commits November 15, 2024 13:23
… WHIP output (#130)

I feel things can be improved still, however rather than making perfect
CLI parameter handling, gonna PR to help move on.

WHIP output option added for starters, we may need to write our own
WebRTC element based off examples and API from:
https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/tree/main/net/webrtc

---------

Co-authored-by: DatCaptainHorse <[email protected]>
Co-authored-by: Wanjohi <[email protected]>
DatCaptainHorse and others added 12 commits November 17, 2024 10:23
Adds WebRTC WHIP/WHEP relay, frontend code changes and container file
changes.

---------

Co-authored-by: DatCaptainHorse <[email protected]>
Co-authored-by: Wanjohi <[email protected]>
- Adds heartbeat to keep WS alive on nestri-server side
- Fixes inputs getting stuck on repeated keys, while also allowing
modifiers to be sent after.
- For now treat Home keys as Esc to make games escapable.
- Add button to start stream, to fix browser preventing play issue.

Co-authored-by: DatCaptainHorse <[email protected]>
This adds functionality to:
1. Broadcast input to the server using WebRTC Data Channels
2. Change some terms for easy understanding `stream -> room`, `viewer =>
participant`
3. Remove input functionality currently relying on Cloudflare's D3 and
move it to our relay

---------

Co-authored-by: Kristian Ollikainen <[email protected]>
Co-authored-by: DatCaptainHorse <[email protected]>
@wanjohiryan
Copy link
Member Author

Update

  • We are using WebRTC for streaming video and audio from the server-side to the client-side. (Yeap, we dropped QUIC)
  • We are using WebRTC data channels to share everything else - input, docker commands etc etc

NOTE: No ports were opened during this operation B)

@wanjohiryan
Copy link
Member Author

TODO

It has been reported that input has a lot of latency, there might be many causes of this but here's my two cents:

  • JSON might be "too heavy" to send, so maybe use ArrayBuffers or protobuf
  • We could optimise data-channels to use unordered streams
  • Handle input much better in the Golang relay, need to investigate on how many milliseconds we lose there
  • Check nestri-server and adjust buffers accordingly - need to investigate the latency here as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[🐛 Bug] Mouse Pointer is not locked [🐛 Bug] Cannot find .X11-unix directory
2 participants