Remote Display proof-of-concept

[ewired]

A Remote Display application for Plato, tested on Clara 2E and Libra 2. An MQTT server relays messages between a Kobo device and Firefox extension. The extension controls tabs according to the device's commands, and converts screenshots of tabs into JPEG XL images to display on the device. Using only JPEG XL, the user can easily choose between monochrome, lossless or lossy greyscale depending on their needs for bandwidth consumption or speed. Lossless JPEG XL gets slightly better image compression than basic PBM/PGM with zlib, which was used in a previous version.

Currently the gestures are:

• Swipe north/south to scroll element under start position or viewport by swipe amount
• Back/forward device buttons to scroll up/down at center of viewport by 50%
• Swipe east/west >50% of width to switch to previous/next tab in window
• Swipe east/west <50% of width to focus the previous/next window
• Arrow east/west to navigate tab forward/back
• Arrow north to close current tab
• Arrow south is handled by device, closes remote display app
• Pinch or spread to zoom by 10%
• Corner northeast to scale up image output & scale down browser window by 10% (northwest for opposite)
• Corner southeast/southwest to increase/decrease contrast filter by 25% (removes filter at lowest bound 100%)
• Corner southeast with contrast at 100% to toggle invert filter
• Short hold finger to manually request updated image, resize the window to desired scale, and full update the display
• Long hold finger to open link under finger in new background tab
• Tap to send click event to element under finger
• Draw circle clockwise around held pivot point to refresh page
• Draw circle counterclockwise around pivot to reopen closed tab