ESP32-S3 USB device emulation

[cicciocb]

Hi all,
Hello, following the request to implement USB emulation for a keyboard, I have developed support for keyboard and mouse emulation on the ESP32-S3. It is now possible to type entire texts, press buttons, move the mouse on the screen, and click the mouse buttons.
To use it, connect the module to the USB port labeled USB (not the COM port), and then execute the command USB.START. This will initialize the USB emulation for both keyboard and mouse functionalities, allowing you to interact with your device seamlessly.
You should hear the classic sound in Windows when USB devices are recognized.
This sound indicates that the system has successfully detected the connected devices.

Then you can use any command like :
USB.TYPETEXT "OK"

or

USB.MOUSEMOVE 100,100

Take care that you can easily lose control of the PC with the command USB.KEYPRESS, as the key will remain pressed until the command USB.KEYRELEASE is executed. This can result in your screen being filled with repeated characters from the key that is pressed.

If the text "typed" is not correct, this probably means that you need to set up the keymap for your specific keyboard language.
I've implemented support for specific keyboards using a hex string that can be used with the command

Code: [Local Link Removed for Guests]

USB.START keymap$ 

You will find the values for several languages further down in this message.

as a little example, you can test this code :

Code: [Local Link Removed for Guests]

USB.START
pause 1000
px = 200 : py = 0
for z = 0 to 2*pi+0.001 step pi/360
  x = cos(z) * 200
  y = sin(z) * 200
  usb.mousemove x-px, y-py
  px = x : py = y
  pause 1
next z

for z = 0 to 15
  usb.typetext chr$(&hB3) ' TAB
  pause 500
next z

usb.mousemove 10, 127
usb.mousemove 10, 127
usb.mousemove 10, 127
usb.mousemove 10, 127
usb.mouseclick 1

usb.typetext "USB OK " + str$(rnd())

The version I'm releasing is the one without USB serial port support and is QIO - OPI; this means that it works with any 16MB module and, optionally, can be connected with a 2 USB cables (COM + USB) to have simultaneously the "classic" serial port plus the USB emulation.

I'm releasing this under the reference 1.70.51 to identify it as a beta version and contains "OTG" inside the version name.

It can be installed using the flasher website.

I'll release a version for the ESP32-S3 GEEK very soon (need some extra testing)

This is a preliminary version, so there may be some bugs present. I look forward to your feedback.

USB HID Command Documentation
USB.START [keymap$]

• Description: Initializes the USB system and keyboard emulation. If the additional parameter is provided, it sets up a custom key mapping.
• It is Mandatory to use this command before using the others
• Input Arguments:
• keymap$: A string containing hex values representing key mappings (if not at the end of line). If not specified the default keyboard mapping is used (EN_US).

• Note: this command can be used again for changing the keyboard mapping

USB.TYPETEXT text$

• Description: Simulates typing a string of text.
• Input Arguments:
• text$: A string representing the text to be typed.

USB.KEYPRESS keycode

• Description: Simulates pressing and holding down a specific key.
• Input Arguments:
• keycode: A numerical value representing the key code of the key to be pressed.

USB.KEYRELEASE keycode

• Description: Simulates releasing a specific key.
• Input Arguments:
• keycode: A numerical value representing the key code of the key to be released.

USB.KEYRELEASEALL

• Description: Simulates releasing all currently pressed keys.
• Input Arguments: None.

USB.MOUSEMOVE width, height, [wheel [, pan]]

• Description: Simulates moving the mouse cursor by specified distances and optionally adjusts scrolling and panning.
• Input Arguments:

• width: A numerical value indicating horizontal movement.
• height: A numerical value indicating vertical movement.
• wheel (optional): A numerical value for wheel movement (scrolling). Defaults to zero if not provided.
• pan (optional): A numerical value for panning. Defaults to zero if not provided.

• Note: the value of these arguments is limited between -127 and +127

USB.MOUSECLICK button

• Description: Simulates clicking a mouse button.
• Input Arguments:

• button: A numerical value representing the mouse button to click (e.g., left or right).

USB.MOUSEPRESS button

• Description: Simulates pressing and holding down a mouse button.
• Input Arguments:

• button: A numerical value representing the mouse button to press.

USB.MOUSERELEASE button

• Description: Simulates releasing a mouse button.
• Input Arguments:

• button: A numerical value representing the mouse button to release.

Description of Buttons:

• 1: Represents the left button of the mouse, commonly used for selection and clicking.
• 2: Represents the right button of the mouse, often used for context menus.
• 4: Represents the middle button (or scroll wheel) of the mouse, typically used for scrolling or additional functions.
• 8: Represents a button that typically navigates back in web browsers or file explorers.
• 16: Represents a button that typically navigates forward in web browsers or file explorers.
• 31: Represents all mouse buttons being pressed simultaneously.

These are the Keymaps for regional keyboards

Code: [Local Link Removed for Guests]

EN_US$ = "00000000000000002A2B280000000000000000000000000000000000000000002C9EB4A0A1A2A434A6A7A5AE362D3738271E1F20212223242526B333B62EB7B89F8485868788898A8B8C8D8E8F909192939495969798999A9B9C9D2F3130A3AD350405060708090A0B0C0D0E0F101112131415161718191A1B1C1DAFB1B0B500"
FR_FR$ = "00000000000000002A2B280000000000000000000000000000000000000000002C38206030B41E21222D31AE1023B6B7A79E9FA0A1A2A3A4A5A63736322EB290679485868788898A8B8C8D8E8FB39192938495969798999D9B9C9A62656D6625641405060708090A0B0C0D0E0F331112130415161718191D1B1C1A61636E5F00"
IT_IT$ = "00000000000000002A2B280000000000000000000000000000000000000000002C9E9F74A1A2A32DA5A6B030363837A4271E1F20212223242526B7B632A7B2AD738485868788898A8B8C8D8E8F909192939495969798999A9B9C9D6F3570AEB8000405060708090A0B0C0D0E0F101112131415161718191A1B1C1D00B5000000"
DE_DE$ = "00000000000000002A2B280000000000000000000000000000000000000000002C9E9F31A1A2A3B1A5A6B030363837A4271E1F20212223242526B7B632A7B2AD548485868788898A8B8C8D8E8F909192939495969798999A9B9D9C656D6600B8000405060708090A0B0C0D0E0F101112131415161718191A1B1D1C6472677000"
ES_ES$ = "00000000000000002A2B280000000000000000000000000000000000000000002C9E9F74A1A2A32DA5A6B030363837A4271E1F20212223242526B7B632A7B2AD738485868788898A8B8C8D8E8F909192939495969798999A9B9C9D6F3570AEB8000405060708090A0B0C0D0E0F101112131415161718191A1B1C1D00B5000000"
PT_BR$ = "00000000000000002A2B280000000000000000000000000000000000000000002C9E9F20A1A2A3B1A5A6B030363837A4271E1F20212223242526B7B632A7B2AD548485868788898A8B8C8D8E8F909192939495969798999A9B9D9C656D6600B8000405060708090A0B0C0D0E0F101112131415161718191A1B1D1C6472677000"
PT_PT$ = "00000000000000002A2B280000000000000000000000000000000000000000002C9E9FA0A1A2A32DA5A6AF2F363837A4271E1F20212223242526B7B632A7B2AD5F8485868788898A8B8C8D8E8F909192939495969798999A9B9C9D65356600B8000405060708090A0B0C0D0E0F101112131415161718191A1B1C1D64B5670000"
SV_SE$ = "00000000000000002A2B280000000000000000000000000000000000000000002C9E9FA061A2A331A5A6B12D363837A4271E1F20212223242526B7B632A7B2AD5F8485868788898A8B8C8D8E8F909192939495969798999A9B9C9D656D6600B8000405060708090A0B0C0D0E0F101112131415161718191A1B1C1D6472670000"
DA_DK$ = "00000000000000002A2B280000000000000000000000000000000000000000002C9E9FA061A2A331A5A6B12D363837A4271E1F20212223242526B7B632A7B2AD5F8485868788898A8B8C8D8E8F909192939495969798999A9B9C9D65726600B8000405060708090A0B0C0D0E0F101112131415161718191A1B1C1D646E670000"
HU_HU$ = "00000000000000002A2B280000000000000000000000000000000000000000002CA19F5B73A2469EA5A678A0363837A3351E1F20212223242526B77672A45DB6598485868788898A8B8C8D8E8F909192939495969798999A9B9D9C49544A60B8640405060708090A0B0C0D0E0F101112131415161718191A1B1D1C455A515E00"

For example, for the italian language you can do :

Code: [Local Link Removed for Guests]

IT_IT$ = "00000000000000002A2B280000000000000000000000000000000000000000002C9E9F74A1A2A32DA5A6B030363837A4271E1F20212223242526B7B632A7B2AD738485868788898A8B8C8D8E8F909192939495969798999A9B9C9D6F3570AEB8000405060708090A0B0C0D0E0F101112131415161718191A1B1C1D00B5000000"
USB.START IT_IT$

This table provides the key definitions along with their hexadecimal values:

KEY_LEFT_CTRL &h80
KEY_LEFT_SHIFT &h81
KEY_LEFT_ALT &h82
KEY_LEFT_GUI &h83
KEY_RIGHT_CTRL &h84
KEY_RIGHT_SHIFT &h85
KEY_RIGHT_ALT &h86
KEY_RIGHT_GUI &h87
KEY_UP_ARROW &hDA
KEY_DOWN_ARROW &hD9
KEY_LEFT_ARROW &hD8
KEY_RIGHT_ARROW &hD7
KEY_BACKSPACE &hB2
KEY_TAB &hB3
KEY_RETURN &hB0
KEY_ESC &hB1
KEY_INSERT &hD1
KEY_DELETE &hD4
KEY_PAGE_UP &hD3
KEY_PAGE_DOWN &hD6
KEY_HOME &hD2
KEY_END &hD5
KEY_CAPS_LOCK &hC1
KEY_F1 &hC2
KEY_F2 &hC3
KEY_F3 &hC4
KEY_F4 &hC5
KEY_F5 &hC6
KEY_F6 &hC7
KEY_F7 &hC8
KEY_F8 &hC9
KEY_F9 &hCA
KEY_F10 &hCB
KEY_F11 &hCC
KEY_F12 &hCD
KEY_F13 &hF0
KEY_F14 &hF1
KEY_F15 &hF2
KEY_F16 &hF3
KEY_F17 &hF4
KEY_F18 &hF5
KEY_F19 &hF6
KEY_F20 &hF7
KEY_F21 &hF8
KEY_F22 &hF9
KEY_F23 &hFA
KEY_F24 &hFB

[cicciocb]

This example is even funnier :

Code: [Local Link Removed for Guests]

USB.START
pause 1000
px = 200 : py = 0
for z = 0 to 2*pi+0.001 step pi/360
  x = cos(z) * 200
  y = sin(z) * 200
  usb.mousemove x-px, y-py
  px = x : py = y
  pause 1
next z

for z = 1 to 16
  usb.typetext chr$(&hB3) ' TAB
  pause 100
next z
pause 500

usb.typetext chr$(&hB0) ' ENTER

usb.typetext "This is a Text typed directly by Annex into another TAB " + str$(rnd())

[cicciocb]

The version of the ESP32-S3 GEEK is online :

Annex32-S3_BLE_CAN_USB_DMT_VGA_HID_OTG_1.70.51_qio_qspi_LFS.bin

p.s.
I fixed the bug on the TFT

[bugs]

This example is even funnier :

Yes - it's alive!

[Electroguard]

Here's a little mouse screen coordinate finder called Pixie which some may find useful: http://www.nattyware.com/pixie.php
It's available as a windows portable exe which needs no installation, and it also runs fine on my MX Linux just by double-clicking it which runs it in Wine.
The little popup window includes a useful pixel colour grabber.
It stays on top, but can be moved around out of the way, or minimised to the system tray using the Minus button, then popped up again with a double click.
When done, the window is closed by the tiny X button at center left.

Pixie.jpg

[bugs]

One useful key on the Windows keyboard is the LEFT_GUI (or "windows" key).
In conjunction with the "Run" command a program may be easily started remotely.
Example:-

Code: [Local Link Removed for Guests]

' USB HID test of the "Windows" Key
' Uses the "Run" command to open a specified program
' e.g.  Notepad

USB.START
pause 1000
  
USB.KEYPRESS  &h83  'KEY_LEFT_GUI '
usb.typetext "r"  'the run command
USB.KEYRELEASE  &h83  'KEY_LEFT_GUI '
usb.typetext "Notepad"
usb.keypress  &hB0  'KEY_RETURN
USB.KEYRELEASE &hB0  'KEY_RETURN

Alternatively any program pinned to the taskbar can be opened (or closed) by using the windows key with a number.
e.g. winkey+2 if the second item is required.

Code: [Local Link Removed for Guests]

' USB HID test of "Windows" Key
' Opens (OR CLOSES) the second item pinned to the taskbar
' the first item may be CoPilot (worm can....) so wise not to use

USB.START
pause 1000
  
USB.KEYPRESS  &h83  'KEY_LEFT_GUI '
usb.typetext "2"  'select second item pinned to taskbar
USB.KEYRELEASE  &h83  'KEY_LEFT_GUI '

[karlkloss]

The version of the ESP32-S3 GEEK is online :

Annex32-S3_BLE_CAN_USB_DMT_VGA_HID_OTG_1.70.51_qio_qspi_LFS.bin

p.s.
I fixed the bug on the TFT

I can confirm that it works.
Settings are:

image.png