Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ feat: Add streaming support #125

Open
wants to merge 46 commits into
base: main
Choose a base branch
from
Open

✨ feat: Add streaming support #125

wants to merge 46 commits into from
+12,481 −5,228

Conversation

wanjohiryan
Copy link
Member

@wanjohiryan wanjohiryan commented Nov 3, 2024 •
edited
Loading

This adds:

  • Keyboard and mouse handling on the frontend
  • Video and audio streaming from the backend to the frontend
  • Input server that works with Websockets

Update - 17/11

  • Master docker container to run this
  • Steam runtime
  • Entrypoint.sh

@cloudflare-workers-and-pages Cloudflare Workers and Pages
Copy link

cloudflare-workers-and-pages bot commented Nov 3, 2024 •
edited
Loading

Deploying nestri with  Cloudflare Pages  Cloudflare Pages

Latest commit: 8d4b39e
Status: ✅  Deploy successful!
Preview URL: https://399123eb.nestri.pages.dev
Branch Preview URL: https://feat-stream.nestri.pages.dev

View logs

This was linked to issues Nov 11, 2024
[🐛 Bug] Mouse Pointer is not locked #77
Open
[🐛 Bug] Cannot find .X11-unix directory #74
Open
wanjohiryan and others added 10 commits November 11, 2024 16:31
@wanjohiryan
feat: Update lib-moq
b3720cc
@DatCaptainHorse
✨ feat: GPU information and encoder picking (#126)
cc436bc
@DatCaptainHorse
✨ feat: Pipewire (#128)
87cfbd1 
I've finished cooking 🍝

---------

Co-authored-by: DatCaptainHorse <[email protected]>
@wanjohiryan
🧹 chore: Add visual clarity to the Containerfile
12a1e4f
@wanjohiryan
Merge branch 'main' of ssh://github.com/nestriness/nestri into feat/s…
022c455 
…tream
@wanjohiryan
🐛 fix: Fix the canvas dimensions
2239c3b
@wanjohiryan
TODO: Add docker-compose.yml and Dockerfile.relay
ab19e0d
@DatCaptainHorse
🐛 fix: Container parameter passing, option to disable audio for debug…
12c6444 
…ging (#129)

Adds a functional NESTRI_PARAMS env variable to container image,
allowing usage like:
```docker run -e NESTRI_PARAMS='--no-audio=true --verbose=true'```

Also adds a no-audio option to help debug stream issues.
Also also, changes the container builder portion to use the actual main nestri repo.

---------

Co-authored-by: DatCaptainHorse <[email protected]>
@wanjohiryan
✨ feat: Migrate to Websocket relay
cf064ae
@wanjohiryan
Merge branch 'feat/stream' of ssh://github.com/nestriness/nestri into…
ad7322d 
… feat/stream
@wanjohiryan
Copy link
Member Author

wanjohiryan commented Nov 14, 2024 •
edited
Loading

I just added the first part of handling websockets thru a relay, partykit

It works locally, i have yet to deploy the server side end of things

FIXME: I think I overwrote the changes done by DatHorse in the prior branch :(
TODO:

  • Make sure it works with the remote Cloudflare partykit server(s)
  • Test the latency, whether it has improved or dropped

UPDATE: This was scraped, too much overhead :/

wanjohiryan and others added 4 commits November 15, 2024 13:23
@wanjohiryan
fix: Add partysocket to apps/www
25b941c
@DatCaptainHorse @wanjohiryan
feat: Separate CLI parameters to files, move few things to functions,…
69b35ae 
… WHIP output (#130)

I feel things can be improved still, however rather than making perfect
CLI parameter handling, gonna PR to help move on.

WHIP output option added for starters, we may need to write our own
WebRTC element based off examples and API from:
https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/tree/main/net/webrtc

---------

Co-authored-by: DatCaptainHorse <[email protected]>
Co-authored-by: Wanjohi <[email protected]>
@wanjohiryan
chore: Some modular changes
a975b89
@wanjohiryan
Revert "chore: Some modular changes"
e21ac83 
This reverts commit a975b89.
DatCaptainHorse and others added 12 commits November 17, 2024 10:23
@DatCaptainHorse @wanjohiryan
✨feat: WebRTC streaming support and improvements (#131)
874e47d 
Adds WebRTC WHIP/WHEP relay, frontend code changes and container file
changes.

---------

Co-authored-by: DatCaptainHorse <[email protected]>
Co-authored-by: Wanjohi <[email protected]>
@wanjohiryan
fix: Move the Containerfile
b2e4b1f
@DatCaptainHorse
🐛fix: some input fixes, missing heartbeat (#132)
06a952d 
- Adds heartbeat to keep WS alive on nestri-server side
- Fixes inputs getting stuck on repeated keys, while also allowing
modifiers to be sent after.
- For now treat Home keys as Esc to make games escapable.
- Add button to start stream, to fix browser preventing play issue.

Co-authored-by: DatCaptainHorse <[email protected]>
@wanjohiryan
fix: Add stuff
8d9a221
@wanjohiryan
Merge branch 'feat/stream' of ssh://github.com/nestriness/nestri into…
6831a6f 
… feat/stream
@wanjohiryan
fix: Fix canvas overflowing because of aspect-ratio
36cac1d
@wanjohiryan
feat: Rename webrtcrelay to relay
9a8eddb
@wanjohiryan
feat: Add websocket server v1
42fbeb7
@wanjohiryan
feat: Add participant's names
2ef1f0b
@wanjohiryan
chore: Add styling to logs
581f466
@wanjohiryan
feat: Add msg types
4d9b941
@wanjohiryan @DatCaptainHorse
✨ feat: Add rooms and use WebRTC DataChannel for input (#133)
8bc770f 
This adds functionality to:
1. Broadcast input to the server using WebRTC Data Channels
2. Change some terms for easy understanding `stream -> room`, `viewer =>
participant`
3. Remove input functionality currently relying on Cloudflare's D3 and
move it to our relay

---------

Co-authored-by: Kristian Ollikainen <[email protected]>
Co-authored-by: DatCaptainHorse <[email protected]>
@wanjohiryan
Copy link
Member Author

Update

  • We are using WebRTC for streaming video and audio from the server-side to the client-side. (Yeap, we dropped QUIC)
  • We are using WebRTC data channels to share everything else - input, docker commands etc etc

NOTE: No ports were opened during this operation B)

@wanjohiryan
Copy link
Member Author

TODO

It has been reported that input has a lot of latency, there might be many causes of this but here's my two cents:

  • JSON might be "too heavy" to send, so maybe use ArrayBuffers or protobuf
  • We could optimise data-channels to use unordered streams
  • Handle input much better in the Golang relay, need to investigate on how many milliseconds we lose there
  • Check nestri-server and adjust buffers accordingly - need to investigate the latency here as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[🐛 Bug] Mouse Pointer is not locked [🐛 Bug] Cannot find .X11-unix directory
2 participants
@wanjohiryan @DatCaptainHorse