vt-input-mode.md

status: draft

VT Input Mode Protocol

The goal of the vt-input-mode protocol is to make command line interactivity cross-platform.

• No TTY required.
• No OS-level signal tracking required.

Audience

Anyone who wants to:

• Operate without TTY.
• Share applications on LAN (using inetd, netcat, etc).
• Track every key press and key release.
• Track position dependent keys such as WASD.
• Distinguish between Left and Right physical keys.
• Get consistent output regardless of terminal window resize.
• Track mouse on a pixel-wise level.
• Track mouse outside the terminal window (getting negative coordinates).
• Take advantage of high-resolution wheel scrolling.
• Track scrollback text manipulation.
• Track application closing and system shutdown.
• Be independent of operating system and third party libraries.

Existing approaches have the following drawbacks:

• There is no uniform way to receive keyboard events.
• Window size tracking requires platform-specific calls with no way to synchronize the output.
• Mouse tracking modes lack support for negative coordinates, high-resolution wheel scrolling, and have a limited set of buttons.
• Bracketed paste mode does not support the transfer of binary data and data containing sequences of bracketed paste mode itself.

Conventions

• All numeric values used in this protocol are decimal and zero-based. todo: Should we use HEX-form of the uint32 (IEEE-754 32-bit binary float, Little-Endian) for the floating point value representation?
• Space characters are not used in sequence payloads and are only used for readability of the description.
• All unescaped symbols outside of this protocol should be treated as clipboard pasted data.

Format

Signaling uses APC ESC _ <payload> ESC \ with an event-specific payload syntax.

The payload consists of a list of attributes in the following format:

<attr>=<val>,...,<val>; ...; <attr>=<val>,...,<val>

todo: representation of the floating point values: float32 -> uint32 IEEE-754 form -> hex? or just round it up to 0.00001?

Field	Descriprtion
<attr>	Attribute name.
<val>,...,<val>	Comma-separated value list.

Initialization

Set:   ESC _ events=<Source0>,...,<SourceN> ESC \
Reset: ESC _ events ESC \

Source	Events to track
keyboard	Keyboard.
mouse	Mouse.
focus	Focus.
format	Line format.
clipoard	Clipboard.
window	Window size and selection.
system	System signals.

This sequence enables vt-input-mode and event tracking for the specified event Sources. The vt-input-mode is deactivated if none of the Sources is specified.

Note: By enabling vt-input-mode, all current terminal modes are automatically saved (to be restored on exit) and switched to something like "raw" mode, in which input is available character by character, echoing is disabled, and all special processing of terminal input and output characters is disabled (except for LF to CR+LF conversion).

Events

• Keyboard

  ESC _ event=keyboard ; kbmods=<KeyMods> ; keyid=<KeyId> ; pressed=<KeyDown> ; scancode=<ScanCode> ; chord=<KeyId0>,...,<KeyIdN> ; print=<C0>,...,<Cn> ESC \
  

• Mouse

  ESC _ event=mouse ; kbmods=<KeyMods> ; coord=<X>,<Y> ; buttons=<ButtonState> ; wheel=<DeltaY>,<DeltaX> ESC \
  

• Focus

  ESC _ event=focus ; state=<FocusState> ESC \
  

• Format

  ESC _ event=format ; wrapping=<State> ; alignment=<State> ; rtl=<State> ESC \
  

• Clipboard

  ESC _ event=clipoard ; format=<ClipFormat> ; security=<SecLevel> ; data=<Data> ESC \
  

• Window

  ESC _ event=window ; size=<Width>,<Height> ; cursor=<X>,<Y> ; region=<Left>,<Top>,<Right>,<Bottom> ; selection=<StartX>,<StartY>,<EndX>,<EndY>,<Mode> ESC \
  

• System

  ESC _ event=system ; signal=<Signal> ESC \
  

Keyboard

ESC _ event=keyboard ; kbmods=<KeyMods> ; keyid=<KeyId> ; pressed=<KeyDown> ; scancode=<ScanCode> ; chord=<KeyId0>,...,<KeyIdN> ; print=<C0>,...,<Cn> ESC \

Q: Do we need to track scancode chord? scanchord=<Code0>,...,<CodeN>?

Attribute	Description
kbmods=<KeyMods>	Keyboard modifiers.
keyid=<KeyId>	Physical key ID.
pressed=<KeyDown>	Key state: <KeyDown>=1 - Pressed. <KeyDown>=0 - Released.
scancode=<ScanCode>	Scan code.
chord=<KeyId0>,...,<KeyIdN>	Simultaneously pressed key id's in ascending order.
print=<C0>,...,<Cn>	Codepoints of the generated string.

In response to the activation of keyboard tracking, the application receives a vt-sequence containing keyboard modifiers state:

ESC _ event=keyboard ; kbmods=<KeyMods> ESC \

The full sequence is fired after every key press and key release. The sequence can contain a string generated by a keystroke as a set of codepoints: C0 + ... + Cn. The string can be fragmented and delivered by multiple consecutive events.

The chord=<KeyId0>,...,<KeyIdN> attribute contains the set of simultaneously pressed key id's in ascending order and is used to track key combinations. It is possible to track both chord presses + (e.g. Ctrl+F1) and chord releases - (e.g. Ctrl-F1 or Ctrl-Alt):

• Pressed chord: The pressed attribute has the value 1.
• Release chord: The pressed attribute has the value 0.

Keyboard modifiers

The state kbmods=<KeyMods> of keyboard modifiers is the binary OR of all currently pressed modifiers and enabled modes.

Bit	Side	Modifier Key	Value
0	Left	⌃ Ctrl	0x0001
1	Right	⌃ Ctrl	0x0002
2	Left	⎇ Alt ⌥ Option	0x0004
3	Right	⎇ Alt ⌥ Option ⇮ AltGr	0x0008
4	Left	⇧ Shift	0x0010
5	Right	⇧ Shift	0x0020
6	Left	⊞ Win ⌘ Command ◆ Meta ❖ Super	0x0040
7	Right	⊞ Win ⌘ Command ◆ Meta ❖ Super	0x0080
8		reserved	0x0100
9		reserved	0x0200
10		reserved	0x0400
11		reserved	0x0800
12		⇭ NumLock Mode	0x1000
13		⇪ CapsLock Mode	0x2000
14		⇳ ScrollLock Mode	0x4000
15		reserved Mode	0x8000

Scan codes

Scan codes are usually useful for applications that need to know which key is pressed, regardless of the current keyboard layout. For example, the WASD (Up, Left, Down, Right) keys for games, which ensure a consistent key formation across QWERTY, AZERTY or Dvorak keyboard layouts.

QWERTY: "W" = 11  "A" = 1E  "S" = 1F  "D" = 20
AZERTY: "Z" = 11  "Q" = 1E  "S" = 1F  "D" = 20
Dvorak: "," = 11  "A" = 1E  "O" = 1F  "E" = 20

Scan codes for the keys on a standard 104-key keyboard:

┌────┐  ┌────╥────╥────╥────┐  ┌────╥────╥────╥────┐  ┌────╥────╥────╥────┐  ┌────╥────╥────┐
| 01 │  | 3B ║ 3C ║ 3D ║ 3E |  | 3F ║ 40 ║ 41 ║ 42 |  | 43 ║ 44 ║ 57 ║ 58 |  |E037║ 46 ║E045|
└────┘  └────╨────╨────╨────┘  └────╨────╨────╨────┘  └────╨────╨────╨────┘  └────╨────╨────┘
┌────╥────╥────╥────╥────╥────╥────╥────╥────╥────╥────╥────╥────╥────────┐  ┌────╥────╥────┐  ┌────╥────╥────╥────┐
| 29 ║ 02 ║ 03 ║ 04 ║ 05 ║ 06 ║ 07 ║ 08 ║ 09 ║ 0A ║ 0B ║ 0C ║ 0D ║     0E |  |E052║E047║E049|  | 45 ║E035║ 37 ║ 4A |
╞════╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══════╡  ╞════╬════╬════╡  ╞════╬════╬════╬════╡
| 0F   ║ 10 ║ 11 ║ 12 ║ 13 ║ 14 ║ 15 ║ 16 ║ 17 ║ 18 ║ 19 ║ 1A ║ 1B ║   2B |  |E053║E04F║E051|  | 47 ║ 48 ║ 49 ║    |
╞══════╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩╦═══╩══════╡  └────╨────╨────┘  ╞════╬════╬════╣ 4E |
| 3A    ║ 1E ║ 1F ║ 20 ║ 21 ║ 22 ║ 23 ║ 24 ║ 25 ║ 26 ║ 27 ║ 28 ║       1C |                    | 4B ║ 4C ║ 4D ║    |
╞═══════╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩═╦══╩══════════╡       ┌────┐       ╞════╬════╬════╬════╡
| 2A      ║ 2C ║ 2D ║ 2E ║ 2F ║ 30 ║ 31 ║ 32 ║ 33 ║ 34 ║ 35 ║          36 |       |E048|       | 4F ║ 50 ║ 51 ║    |
╞══════╦══╩═╦══╩═══╦╩════╩════╩════╩════╩════╩════╩╦═══╩╦═══╩╦═════╦══════╡  ┌────┼────┼────┐  ╞════╩════╬════╣E01C|
| 1D   ║ 5B ║ 38   ║               39              ║E038║ 5C ║  5D ║ E01D |  |E04B|E050|E04D|  |      52 ║ 53 ║    |
└──────╨────╨──────╨───────────────────────────────╨────╨────╨─────╨──────┘  └────┴────┴────┘  └─────────╨────╨────┘

Physical keys

The <KeyId> is incremented by 2 for each generic key, providing two <KeyId> placeholders for each physical key to distinguish between Left and Right (or Numpad) in the last bit. Ignore the last bit of <KeyId> for tracking generic keys.

Key ID	Name	Physical Key	Scan Code	Notes
0	undef
2	LeftCtrl	Left Ctrl	29 0x1D
3	RightCtrl	Right Ctrl	57373 0xE01D
4	LeftAlt	Left Alt	56 0x38
5	RightAlt	Right Alt	57400 0xE038
6	LeftShift	Left Shift	42 0x2A
7	RightShift	Right Shift	54 0x36
8	LeftWin	Left Win	91 0x5B
9	RightWin	Right Win	92 0x5C
10	Apps	Apps	93 0x5D
12	NumLock	Num Lock	69 0x45
14	CapsLock	Caps Lock	58 0x3A
16	ScrollLock	Scroll Lock	69 0x45
18	Esc	Esc	1 0x01
20	Space	Space	57 0x39
22	Backspace	Backspace	14 0x0E
24	Tab	Tab	15 0x0F
26	Break	Break	57414 0xE046	Ctrl + Pause
28	Pause	Pause	57412 0xE045
30	Select	Select
32	SysRq	SysRq	84 0x54	Alt + PrintScreen
34	PrintScreen	Print Screen	57399 0xE037
36	Enter	Enter	28 0x1C
37	NumpadEnter	Numpad Enter	57372 0xE01C
38	PageUp	PageUp	57417 0xE049
39	NumpadPageUp	Numpad PageUp	73 0x49
40	PageDown	PageDown	57425 0xE051
41	NumpadPageDown	Numpad PageDown	81 0x51
42	End	End	57423 0xE04F
43	NumpadEnd	Numpad End	79 0x4F
44	Home	Home	57415 0xE047
45	NumpadHome	Numpad Home	71 0x47
46	LeftArrow	Left Arrow	57419 0xE04B
47	NumpadLeftArrow	Numpad Left Arrow	75 0x4B
48	UpArrow	Up Arrow	57416 0xE048
49	NumpadUpArrow	Numpad Up Arrow	72 0x48
50	RightArrow	Right Arrow	57421 0xE04D
51	NumpadRightArrow	Numpad Right Arrow	77 0x4D
52	DownArrow	Down Arrow	57424 0xE050
53	NumpadDownArrow	Numpad Down Arrow	80 0x50
54	Key0	0	11 0x0B
55	Numpad0	Numpad 0	82 0x52
56	Key1	1	2 0x02
57	Numpad1	Numpad 1	79 0x4F
58	Key2	2	3 0x03
59	Numpad2	Numpad 2	80 0x50
60	Key3	3	4 0x04
61	Numpad3	Numpad 3	81 0x51
62	Key4	4	5 0x05
63	Numpad4	Numpad 4	75 0x4B
64	Key5	5	6 0x06
65	Numpad5	Numpad 5	76 0x4C
66	Key6	6	7 0x07
67	Numpad6	Numpad 6	77 0x4D
68	Key7	7	8 0x08
69	Numpad7	Numpad 7	71 0x47
70	Key8	8	9 0x09
71	Numpad8	Numpad 8	72 0x48
72	Key9	9	10 0x0A
73	Numpad9	Numpad 9	73 0x49
74	Insert	Insert	57426 0xE052
75	NumpadInsert	Numpad Insert	82 0x52
76	Delete	Delete	57427 0xE053
77	NumpadDelete	Numpad Delete	83 0x55
78	Clear	Clear	76 0x4C
79	NumpadClear	Numpad Clear	76 0x4C	Numpad 5
80	Multiply	*
81	NumpadMultiply	Numpad *	55 0x37
82	Plus	+
83	NumpadPlus	Numpad +	78 0x4E
84	Separator	Separator
85	NumpadSeparator	Numpad Separator
86	Minus	-	12 0x0C
87	NumpadMinus	Numpad -	74 0x4A
88	Period	.	52 0x34
89	NumpadDecimal	Numpad .	83 0x53
90	Slash	/	53 0x35
91	NumpadSlash	Numpad /	57397 0xE035
92	BackSlash	\	43 0x2B
94	OpenBracket	[	26 0x1A
96	ClosedBracket	]	27 0x1B
98	Equal	=	13 0x0D
100	BackQuote	`	41 0x29
102	SingleQuote	'	40 0x28
104	Comma	,	51 0x33
106	Semicolon	;	39 0x27
108	F1	F1	59 0x3B
110	F2	F2	61 0x3C
112	F3	F3	62 0x3D
114	F4	F4	63 0x3E
116	F5	F5	64 0x3F
118	F6	F6	65 0x40
120	F7	F7	66 0x41
122	F8	F8	67 0x42
124	F9	F9	68 0x43
126	F10	F10	69 0x44
128	F11	F11	87 0x57
130	F12	F12	88 0x5B
132	F13	F13
134	F14	F14
136	F15	F15
138	F16	F16
140	F17	F17
142	F18	F18
144	F19	F19
146	F20	F20
148	F21	F21
150	F22	F22
152	F23	F23
154	F24	F24
156	KeyA	A
158	KeyB	B
160	KeyC	C
162	KeyD	D
164	KeyE	E
166	KeyF	F
168	KeyG	G
170	KeyH	H
172	KeyI	I
174	KeyJ	J
176	KeyK	K
178	KeyL	L
180	KeyM	M
182	KeyN	N
184	KeyO	O
186	KeyP	P
188	KeyQ	Q
190	KeyR	R
192	KeyS	S
194	KeyT	T
196	KeyU	U
198	KeyV	V
200	KeyW	W
202	KeyX	X
204	KeyY	Y
206	KeyZ	Z
208	Sleep	Sleep
210	Calculator	Calculator
212	Mail	Mail
214	MediaVolMute	Media Vol Mute
216	MediaVolDown	Media Vol Down
218	MediaVolUp	Media Vol Up
220	MediaNext	Media Next
222	MediaPrev	Media Prev
224	MediaStop	Media Stop
226	MediaPlayPause	Media Play/Pause
228	MediaSelect	Media Select
230	BrowserBack	Browser Back
232	BrowserForward	Browser Forward
234	BrowserRefresh	Browser Refresh
236	BrowserStop	Browser Stop
238	BrowserSearch	Browser Search
240	BrowserFavorites	Browser Favorites
242	BrowserHome	Browser Home

Mouse

ESC _ event=mouse ; kbmods=<KeyMods> ; coord=<X>,<Y> ; buttons=<ButtonState> ; wheel=<DeltaY>,<DeltaX> ESC \

Attribute	Description
kbmods=<KeyMods>	Keyboard modifiers (see Keyboard event).
coord=<X>,<Y>	Pixel-wise coordinates of the mouse pointer. Each coordinate is represented in the form of a floating point value of the sum of the integer coordinate of the cell in the terminal window grid and the relative offset within the cell in the range [0.0f, 1.0f).
buttons=<ButtonState>	Mouse button state.
wheel=<DeltaY>,<DeltaX>	Vertical and horizontal wheel high-resolution delta represented as floating point value.

In response to the activation of mouse tracking, the application receives a vt-sequence containing current mouse state:

ESC _ event=mouse ; kbmods=<KeyMods> ; coord=<X>,<Y> ; buttons=<ButtonState> ESC \

The mouse tracking event fires on any mouse activity, as well as on keyboard modifier changes.

Mouse button state

Bit	Active button
0	Left
1	Right
2	Middle
3	4th
4	5th

Note: Mouse tracking will continue outside the terminal window as long as the mouse button pressed inside the window is active. In this case, coordinates with negative values are possible.

Focus

ESC _ event=focus ; state=<FocusState> ESC \

Attribute	Description
state=<FocusState>	Terminal window focus: <FocusState>=1 - Focused. <FocusState>=0 - Unfocused.

In response to the activation of focus tracking, the application receives a vt-sequence containing current focus state.

Format

ESC _ event=format ; wrapping=<State> ; alignment=<State> ; rtl=<State> ESC \

Attribute	Description
wrapping=<State>	Line wrapping: <State>=0 - Unwrapped. <State>=1 - Wrapped.
alignment=<State>	Line alignment: <State>=0 - Left. <State>=1 - Right. <State>=2 - Center.
rtl=<State>	Text flow direction: <State>=0 - Left to Right. <State>=1 - Right to Left.

In response to the activation of format tracking, the application receives a vt-sequence containing current line format.

Clipboard

ESC _ event=clipoard ; format=<ClipFormat> ; security=<SecLevel> ; data=<Data> ESC \

Attribute	Description
format=<ClipFormat>	Clipboard data format.
security=<SecLevel>	Security level.
data=<Data>	Base64 encoded data.

Clipboard data format

Format	Description
text	Plain text.
rich	Rich text format.
html	HTML format.
ansi	ANSI/VT format.

Security level

Bit	Description
0x01	Exclude clipboard content from monitor processing.
0x02	Can include in clipboard history.
0x04	Can upload to cloud clipboard.

Window

ESC _ event=window ; size=<Width>,<Height> ; cursor=<X>,<Y> ; region=<Left>,<Top>,<Right>,<Bottom> ; selection=<StartX>,<StartY>,<EndX>,<EndY>,<Mode> ESC \

Attribute	Description
size=<Width>,<Height>	Terminal window size in cells.
cursor=<X>,<Y>	Current text cursor position.
region=<Left>,<Top>,<Right>,<Bottom>	Scrolling region margins.
selection=<StartX>,<StartY>,<EndX>,<EndY>,<Mode>	Coordinates of the text selection start/end (half-open interval) and text selection mode: <Mode>=0 - Line-based. <Mode>=1 - Rect-based.

In response to the activation of window tracking, the application receives a vt-sequence containing current window state.

Mandatory synchronization

When the terminal window is resized, the changes are only applied after a handshake between the terminal and the application. Successive multiple resizing of the window initiates the same number of handshakes.

Handshake steps:

1. Terminal requests a new size, omitting all other parameters.
   • Application receives the request and prepares for resizing, e.g. by updating, cropping or deleting visible lines.
2. Application replies with the same message as the request.
3. Terminal applies the new size and sends the changes.
   • Application updates its UI.

Terminal:    ESC _ event=window ; size=<Width>,<Height> ESC \
Application: ESC _ event=window ; size=<Width>,<Height> ESC \
Terminal:    ESC _ event=window ; size=<Width>,<Height> ; cursor=<X>,<Y> ; region=<Left>,<Top>,<Right>,<Bottom> ; selection=<StartX>,<StartY>,<EndX>,<EndY>,<Mode> ESC \

Note that the terminal window resizing always reflows the scrollback, so the window size, cursor position, scrolling regions, and selection coordinates are subject to change during step 3. Upon receiving the resize request (step 1), a fullscreen application can prepare a scrollback by cropping visible lines to avoid unwanted line wrapping or line extrusion, then send a resize confirmation (step 2). In case the aplication's output is anchored to the current cursor position or uses scrolling regions, the application should wait after step 2 for the updated values before continuing to output.

Hypothetical case with Far Manager (FM):

• FM saves visible original scrollback.
• FM draws its UI on top of the visible original scrollback.
• Terminal sends a resize request.
• FM receives request.
• FM restores the original scrollback.
• FM replies on resize request.
• Terminal receives reply.
• Terminal resizes the window and reflows the scrollback.
• Terminal sends modified window parameters.
• FM receives modified window parameters.
• FM saves the modified scrollback.
• FM draws its UI on top of the scrollback.

Selection tracking

The window size tracking sequence is fired after every scrollback text selection changed. In the case of using the mouse for selection, a single left click is treated as a special case of selection when the start and end are the same (empty selection).

Note that selected text in the scrollback above the top index row will produce negative Y-coordinate values.

System

ESC _ event=system ; signal=<Signal> ESC \

Signal	Description
0	Window closing.
1	System shutdown.

The application must respond to the terminal within 5 seconds with the same message confirming that it will close itself without being forced. After a response to Signal=0, the application can continue running and closing the terminal window will be silently aborted. In the absence of confirmation, and also in the case of Signal=1, the application will be forced to close.

Usage Examples

C++20

#include <iostream>

int main()
{
    std::cout << "test" << '\n';
}

Python

#!/bin/python

while True:
    print('test\n')

PowerShell

while ($True)
{
    "test\n";
}