Read the pad, detect unknown ones, layout editor

added a feature that now can read the inputs off the board, meaning if you already have saved settings and you uninstall / reinstall the app or use it on a new device it can auto detect what each key has saved.

using the above feature i now also do believe it is possible to auto detect the macros pads rough layout / how many keys and dials it has!. hopfully this means that more people can use the app easily without needing to hard code a bunch of new layouts. its not truly universal but it should open the door to much more pads working out of the box right now!

and lastly in case the extra layouts do work i made a little layout editor so you can the app to look like your actual macropad!

README.md

@@ -23,7 +23,7 @@
 </p>
 
 <p align="center">
-  <a href="#-install-it-the-whole-world-is-counting-on-you">Install</a> &nbsp;|&nbsp;
+  <a href="#-install">Install</a> &nbsp;|&nbsp;
   <a href="#-your-first-binding">First binding</a> &nbsp;|&nbsp;
   <a href="#-when-something-goes-wrong">Help</a> &nbsp;|&nbsp;
   <a href="#-other-pads-in-this-family">Other pads</a> &nbsp;|&nbsp;
@@ -37,7 +37,7 @@
 > ```
 > curl -fsSL https://raw.githubusercontent.com/Probler-Yt/MacroPad/main/install.sh | bash
 > ```
-> Never used a terminal? Perfect. [The step by step guide](#-install-it-the-whole-world-is-counting-on-you) was written for you.
+> Never used a terminal? Perfect. [The step by step guide](#-install) was written for you.
 
 ---
 
@@ -73,13 +73,13 @@ Probably, if yours looks like this:
 - Came with software called **`MINI_KEYBOARD.exe`**, or a download link to it.
 - Plugs in with USB and works as a keyboard straight away.
 
-To be sure, open a terminal ([how?](#step-1-open-a-terminal)), type `lsusb` and press Enter. Look for this line:
+To check, run `lsusb` in a terminal and look for a line like:
 
 ```
 Bus 001 Device 011: ID 1189:8840 Acer Communications & Multimedia USB Composite Device
 ```
 
-The important bit is **`1189:8840`**. Don't worry about the "Acer" part: it isn't made by Acer, these pads just borrow that ID.
+The part that matters is **`1189:8840`**. Ignore "Acer": these pads borrow that ID and have nothing to do with Acer.
 
 > [!NOTE]
 > **Different number, like `1189:8890` or `1189:8842`?** That's a sibling of this pad with a different layout. MacroPad will spot it and tell you it isn't supported yet, rather than risk sending it the wrong thing. [ch57x-keyboard-tool](https://github.com/kriomant/ch57x-keyboard-tool) supports several of them today, and [you can help add yours here](#-other-pads-in-this-family).
@@ -95,7 +95,10 @@ The important bit is **`1189:8840`**. Don't worry about the "Acer" part: it isn'
 | 🎵 **Media keys** | Play/pause, next, previous, stop, volume, mute, **screen brightness**, calculator, email, browser, my computer. |
 | 🚫 **Nothing** | Switch a key off completely, so a stray press does nothing. |
 | 🔄 **Rotate view** | Stand your pad up or lay it flat. The drawing turns to match. |
-| 🧠 **Remembers what it wrote** | The pad can't be read back, so the app keeps its own notes and marks anything it doesn't know. |
+| 📖 **Reads your pad** | Opens and asks the pad what it's actually holding, so you start from the truth. |
+| 🔎 **Identifies unknown pads** | Asks an unrecognised pad about itself, read only, and refuses to write if it doesn't answer properly. |
+| 📐 **Layout editor** | Drag the keys around until the drawing matches your pad. |
+| 🧠 **Remembers what it wrote** | Keeps its own notes too, and marks anything it isn't sure about rather than guessing. |
 | 🩺 **Tells you what's wrong** | Unplugged? No permission? Wrong pad? You get a plain explanation and the exact fix, with a copy button. |
 | 📥 **Imports from the vendor app** | Set your pad up on Windows before? Bring that setup across. |
 
@@ -125,6 +128,7 @@ Both write straight to the pad's own memory, so you can even use both. Pick whic
 | Key sequences with delays | Not yet | Yes |
 | Mouse actions, LED modes, extra layers | Not yet | Yes |
 | Record a shortcut by pressing it | Yes | No |
+| Read what's on the pad | Yes | No |
 | Install and permissions | One line does both | AUR, a download or `cargo install`; rule by hand |
 
 **Short version:** new to Linux, or just want to click things? MacroPad. Want every feature, a different pad, or a config file you can keep in git? ch57x-keyboard-tool.
@@ -133,26 +137,20 @@ Other people have built GUIs for these pads too, mostly for the smaller 3 and 6
 
 ---
 
-## 🚀 Install it (the whole world is counting on you)
+## 🚀 Install
 
-Welcome. Maybe this is your first week on Linux. Maybe it's your first *hour*. Either way, you are about to do something brave and important, and I'm going to walk you through every single click. Nothing here can break your computer. Deep breath. Let's save the world.
+One command does everything: the app, the Qt libraries if you need them, the menu entry, and the USB permission rule. It asks before each step that needs your password, and it's safe to run again to update.
 
-### Step 0: Plug in your pad
+If you're comfortable in a terminal, [skip to the command](#step-2-copy-this-line). The steps below spell it out for anyone newer to Linux, since these pads are a lot of people's first reason to open one.
-
-Plug the macro pad into a USB port. That's it. Step zero complete. You're doing amazing.
 
 ### Step 1: Open a terminal
 
-A **terminal** is a window where you type instructions instead of clicking. It looks scary. It isn't. Think of it as a very literal assistant who does exactly what you type.
+A terminal is a window where you type commands instead of clicking. Nothing below can damage your system.
-
-How to open one:
 
 - **KDE Plasma** (CachyOS, Bazzite, Kubuntu, Fedora KDE, KDE neon): press the **Meta key** (the one with the Windows logo), type **`Konsole`**, press **Enter**.
 - **GNOME** (Ubuntu, Fedora, Pop!_OS): press the **Meta key**, type **`Terminal`**, press **Enter**.
 - **Almost anywhere**: try pressing **`Ctrl` + `Alt` + `T`** together.
 
-A window with a blinking cursor appears. Hello, terminal. 👋
-
 ### Step 2: Copy this line
 
 Click the copy button on the right side of this box (or select the text and press `Ctrl+C`):
@@ -168,46 +166,40 @@ In plain words: "download the MacroPad installer from GitHub (`curl`), and run i
 
 </details>
 
-### Step 3: Paste it into the terminal
+### Step 3: Paste it and press Enter
 
 > [!IMPORTANT]
-> **In a terminal, pasting is `Ctrl` + `Shift` + `V`**, not `Ctrl+V`. This catches out everyone coming from Windows. (Right click, then **Paste** works too.)
+> Pasting into a terminal is **`Ctrl` + `Shift` + `V`**, not `Ctrl+V`. Right click, then Paste, also works.
-
-Paste it, then press **Enter**.
 
 ### Step 4: Answer the questions
 
-The installer explains what it's doing as it goes. It will ask a couple of questions like **"Add it now? [Y/n]"**. Just press **Enter** to say yes.
+The installer says what it's doing as it goes, and asks before anything that needs your password. Pressing Enter accepts the default.
 
-When it asks for your **password**, type the password you use to log in to your computer and press **Enter**.
+It needs the password for two things: installing Qt on systems that don't have it, and writing one permission file so the app can talk to your pad. Everything else stays in your home folder.
 
-> [!WARNING]
+> [!NOTE]
-> **Nothing appears while you type your password.** No dots, no stars, nothing. That's normal! Linux hides it completely so nobody can even count the letters. Just type it and press Enter. If you get it wrong it'll simply ask again.
+> Nothing appears on screen while you type a password in a terminal. No dots, no stars. That's deliberate, not a frozen prompt.
 
-It needs your password for two small things: installing Qt (the toolkit that draws the window) on some systems, and adding one file that lets the app talk to your pad. Everything else goes in your own home folder.
+First run downloads around 100 MB on some systems. You'll see **All done.** when it's finished.
 
-The first install downloads about 100 MB on some systems, so give it a minute. You'll see **All done.** in green when it's finished.
+### Step 5: Replug the pad
 
-### Step 5: Unplug the pad and plug it back in
+Unplug it and plug it back in, so Linux applies the new permission.
-
-This makes Linux notice the new permission. Unplug. Count to two. Plug back in.
 
 ### Step 6: Open MacroPad
 
-Open your app menu (press the **Meta key**), type **`MacroPad`**, and click it. 🎉
+Open your app menu, type **MacroPad**, and click it.
 
 <p align="center">
   <img src="docs/images/first-launch.png" width="820" alt="MacroPad on its first launch: every key is striped because the app hasn't written anything yet">
 </p>
 
-**Everything is striped the first time. That's correct.** The app hasn't written anything to your pad yet, and the pad has no way of telling it what's already there. You'll fix that in about thirty seconds.
+A **green dot** and **Pad ready** at the bottom left means it's working. Anything else, and the panel on the right explains it, or see [When something goes wrong](#-when-something-goes-wrong).
 
-Look at the bottom left. A **green dot** and **Pad ready** means everything worked. If you see something else, the panel on the right will tell you exactly what to do, or jump to [When something goes wrong](#-when-something-goes-wrong).
+The app reads your pad when it opens, so the drawing should fill in with whatever is already on it. Keys still showing as hatched are ones it couldn't read; press **Read pad** to ask again.
 
 > [!TIP]
-> Want it on your taskbar? Right click **MacroPad** in your app menu and choose **Pin to Task Manager** (Plasma) or **Pin to Dash** (GNOME).
+> To keep it handy: right click **MacroPad** in your app menu and choose **Pin to Task Manager** (Plasma) or **Pin to Dash** (GNOME).
-
-**You did it.** The world is saved. Probably. Let's make sure by setting up a key.
 
 ---
 
@@ -215,7 +207,7 @@ Look at the bottom left. A **green dot** and **Pad ready** means everything work
 
 ### 1. Click a key on the drawing
 
-Pick any key. It gets an orange outline, and the panel on the right shows what it does now.
+It gets an orange outline, and the panel on the right shows what it currently does.
 
 ### 2. Type what you want it to do
 
@@ -246,7 +238,7 @@ Click **Media key** and pick from the list. Volume and brightness on the knobs i
 
 ### Or make it do nothing
 
-Bound something you don't want any more? Click **Nothing**, then write it. The key goes quiet: pressing it does nothing at all. Keys set this way show **Nothing** in grey on the drawing.
+To clear a binding, click **Nothing** and write it. Keys set this way show **Nothing** in grey.
 
 <p align="center">
   <img src="docs/images/step-nothing.png" width="820" alt="Key 5 set to Nothing, waiting to be written, with key 1 already showing Nothing in grey">
@@ -266,9 +258,9 @@ The bottom bar tells you exactly what happened. Green means it worked.
   <img src="docs/images/step-written.png" width="820" alt="After writing: the keys are solid white and the bottom bar says Wrote 3 changes to the pad">
 </p>
 
-### 4. Press the key on your pad
+### 4. Press the key
 
-It does the thing. You can close the app now. **Your bindings live on the pad itself**, so they keep working after a reboot, on another computer, even on Windows. Nothing needs to stay running.
+**Your bindings live on the pad itself**, so they keep working after a reboot, on another computer, even on Windows. Nothing stays running in the background, and you can close the app.
 
 ---
 
@@ -289,6 +281,18 @@ You can also click the three lines of text under each knob. Every click of a tur
 
 ---
 
+## 📐 If the drawing doesn't match your pad
+
+Press **Edit layout** at the bottom. The keys start wobbling, and you can drag them around until the picture matches the pad in front of you. Two keys swap when you drop one onto the other, so nothing is ever lost.
+
+The boxes along the bottom set the grid size and how many keys and knobs there are. **Done** saves it, **Cancel** puts it back.
+
+<p align="center">
+  <img src="docs/images/edit-layout.png" width="820" alt="Edit layout mode: keys outlined and being dragged into a new position">
+</p>
+
+This only changes the picture. It never writes anything to the pad. It's there because the pad reports how many keys it has but not how they're arranged, so on a pad we haven't seen before the first guess can be wrong.
+
 ## 🔄 Rotate view
 
 Some people stand the pad up with the knobs on top. Some lay it flat with the knobs on the right. Click **Rotate view** at the bottom and the drawing turns to match. It remembers your choice.
@@ -307,17 +311,24 @@ Only do this if nothing has changed on the pad since that capture.
 
 ---
 
-## ❓ Why does it say "Unknown"?
+## 📖 Reading your pad
 
-Because **these pads are write only.** You can send them new bindings, but there's no known way to ask one what it's currently set to. Even the official Windows app opens with every key blank.
+MacroPad asks the pad what it's holding when it opens, so the drawing shows what's really on the hardware rather than what some file thinks. There's a **Read pad** button at the bottom to ask again at any time.
 
-So the app keeps its own notes of everything it has successfully written, and it's careful about it:
+Anything you've changed but not written yet is left alone when it reads, since that's your intent, not the pad's state.
 
-- If a write finishes, the key is marked with what was written.
+### So what does "Unknown" mean now?
-- If a write fails before anything reaches the pad, nothing changes.
-- If a write fails **halfway**, that key goes back to **Unknown**, because the honest answer is "no idea". Just write it again.
 
-Striped keys aren't broken. They're just keys this app hasn't touched yet.
+Hatched keys are ones the app hasn't been told about: either the pad didn't answer, or you haven't pressed Read pad. Press it and they fill in.
+
+<p align="center">
+  <img src="docs/images/read-pad.png" width="820" alt="The app after reading the pad: every key and dial filled in from the hardware">
+</p>
+
+The app still keeps its own notes, because a write can fail halfway. If that happens, that one control goes back to Unknown rather than claiming either the old or the new binding, and reading the pad clears it up.
+
+> [!NOTE]
+> This took a while to work out. The project was built assuming these pads were write only, because nothing documented a way to read them and the first captures only recorded traffic in one direction. They can be read. The story is in [For the nerds](#-for-the-nerds).
 
 ---
 
@@ -429,7 +440,24 @@ Siblings come in other sizes, and several share vendor ID `1189`. **They don't a
   <img src="docs/images/unsupported.png" width="820" alt="The app explaining that it found a 1189:8890 pad that it doesn't support yet">
 </p>
 
-### You can add yours
+### Detecting a pad automatically
+
+If MacroPad sees a pad it doesn't recognise, it offers to **ask the pad what it is**. That's read only, and it needs the pad to pass two tests before the app will write anything to it:
+
+1. It answers the `0xFB` query with a sensible key and knob count.
+2. It then reports a full layer, describing exactly the control slots that count implies.
+
+A pad from a different family fails one of those, and the app leaves it alone rather than guessing. On the command line:
+
+```
+macropad detect
+```
+
+The pad reports how many keys and knobs it has, but not how they're arranged, so the app's first guess at rows and columns may be wrong. Check the drawing against the real thing.
+
+If your pad passes, please [open an issue](https://github.com/Probler-Yt/MacroPad/issues) with the output so it can be added to the known list and everyone else gets it working first time.
+
+### Adding one by hand
 
 If you'd like your pad in MacroPad too, you're exactly who this project needs. It's the same process used for this one. You'll need the vendor software (`MINI_KEYBOARD.exe`) and Wine.
 
@@ -544,6 +572,41 @@ rOzzy1987's format is for different product IDs, which is why porting it to this
 
 </details>
 
+<details>
+<summary><b>Reading the pad, and how we nearly missed it</b></summary>
+
+The project ran for weeks on the assumption that these pads were write only. Two things caused that:
+
+- Nothing documented a read. Neither rOzzy1987's project nor ch57x-keyboard-tool implements one.
+- The capture tool only recorded **outgoing** transfers. So even though the pad had been answering all along, nothing was ever written down.
+
+What settled it was noticing that the vendor software has a "view settings" button, and that it showed bindings written by MacroPad. It couldn't have known those from a file, so it was asking the pad. Capturing both directions gave the answer.
+
+Two query commands, both answered on the interrupt IN endpoint of the same report id:
+
+```
+03 fb ...                       what are you?
+03 fb 0c 02                     reply: 12 keys, 2 knobs
+
+03 fa 0f 03 01 02 ...           send me layer 1
+                                (15 key slots, 3 layers, layer 1, 2 knobs)
+03 fa 02 01 01 00 ... 01 05 17  key 2 is ctrl+alt+t
+03 fa 10 01 02 00 ... 01 ea 00  dial 1 left is volume down
+                                ... one report per control, then silence
+```
+
+Replies use the same layout as a config write, with `0xFA` in place of the `0xFD` magic. Media replies carry a count of 1 where writes send 2; both are accepted.
+
+Three things fell out of it:
+
+- **The pad reports its own layout**, so key and knob counts don't have to be guessed from the USB ID.
+- **There are three layers**, and the firmware keeps 15 key slots and 3 knobs whatever the hardware has. Ours answers for 6 controls that don't physically exist.
+- **The dial mapping is confirmed from the other direction.** Reading back what we had written returns exactly the mapping we assumed, which is stronger evidence than a write that appeared to work.
+
+The vendor's request has trailing bytes that change between runs and look like uninitialised memory from the Windows program. Zeros work fine.
+
+</details>
+
 <details>
 <summary><b>usbmon only shows you 32 bytes</b></summary>
 
@@ -572,7 +635,7 @@ The app's permission check reads `udevadm info` to see whether the device got th
 <details>
 <summary><b>Where the app keeps its notes</b></summary>
 
-`~/.config/macropad/state-1189-8840.json` holds everything the app has successfully written. Every entry is exactly a command the CLI understands:
+`~/.config/macropad/state-1189-8840.json` holds what the app last read from or wrote to the pad. Since the pad can be read, this is a cache rather than the source of truth. Every entry is exactly a command the CLI understands:
 
 ```json
 {
@@ -646,14 +709,18 @@ tests/                   real captures and the tests that replay them
 - [x] Keys, shortcuts and media keys on `1189:8840`
 - [x] Both knobs, all three actions each
 - [x] Upright and flat views
+- [x] Reading the pad, so the app shows what's really on it
 - [x] One line installer for any distro
-- [ ] **Key sequences with delays** (ch57x-keyboard-tool shows the likely format; one capture to confirm)
+- [ ] **Key sequences with delays**. Reading the pad shows two-key sequences stored in the layout we assumed, so this is close
-- [ ] Layers 2 and 3, and LED modes
+- [ ] **Layers 2 and 3**, which the pad turns out to have
+- [ ] LED modes
 - [ ] AUR package, so Arch users can install it like anything else
 - [ ] Knob press to toggle brightness between 0% and 100%
 - [ ] Save and switch between named profiles (gaming, editing, streaming)
 - [ ] **Windows and macOS** via a cross platform USB layer
-- [ ] **More pads**: every capture sent in gets us closer
+- [x] Detecting a pad's key and knob counts by asking it
+- [x] A layout editor, since the pad doesn't report how its keys are arranged
+- [ ] **More pads**: every report sent in gets us closer
 
 ---
 

docs/make_screenshots.py

@@ -139,6 +139,13 @@ def scene_first_launch():
     save(w, "first-launch")
 
 
+def scene_read():
+    w = window(bindings=DEMO)
+    w._say("Read 18 controls from the pad.", A.GOOD)
+    select(w, "dial1-left")
+    save(w, "read-pad")
+
+
 def scene_select_and_type():
     w = window(bindings={k: v for k, v in DEMO.items() if k.startswith("dial")})
     select(w, "key2")
@@ -191,6 +198,19 @@ def scene_written():
     save(w, "step-written")
 
 
+def scene_edit_layout():
+    w = window(bindings=DEMO)
+    w._edit_layout(True)
+    spots = {k: (r, c) for k, r, c
+             in __import__("macropad_gui.padview", fromlist=["padview"])
+             ._key_grid(w.pad.layout, w.pad.orientation)}
+    w.pad.drag = "key2"
+    w.pad.drag_at = w.pad.keys["key11"].center()
+    w.pad.drop = spots["key11"]
+    save(w, "edit-layout")
+    w._edit_layout(False, keep=False)
+
+
 def scene_flat():
     w = window(bindings=DEMO)
     w.pad.set_orientation("flat")

macropad-capture.py

@@ -1,9 +1,10 @@
 #!/usr/bin/env python3
 """
-macropad-capture - watch what the vendor's software actually sends.
+macropad-capture - watch what the vendor's software sends, and what the pad
+says back.
 
-Uses the kernel's usbmon facility to record USB traffic going to your macro
+Uses the kernel's usbmon facility to record USB traffic to and from your
-pad, then decodes the outgoing reports so we can see the real wire format.
+macro pad, then decodes the reports so we can see the real wire format.
 
 This is passive. It only observes - it never sends anything itself.
 
@@ -16,15 +17,16 @@ How to use it:
      under Wine:
          wine MINI_KEYBOARD.exe
 
-  3. In that software, change ONE key to something distinctive - key 1 to
+  3. Do ONE thing in that software, then come back and press Ctrl+C.
-     the letter 'a' is ideal, because we know 'a' is HID code 0x04 and it
-     will be easy to spot in the bytes.
 
-  4. Click whatever it uses to apply or save.
+     To learn how a binding is written: change ONE key to the letter 'a'
+     and apply it. ('a' is HID code 0x04, easy to spot in the bytes.)
 
-  5. Come back here and press Ctrl+C.
+     To learn whether the pad can be read back: press whatever the
+     software calls "view settings" or "read". Anything the pad sends is
+     recorded and marked with a <- arrow.
 
-It writes macropad-capture.txt, which has everything we need.
+  4. It writes macropad-capture.txt, which has everything we need.
 """
 
 import os
@@ -57,63 +59,135 @@ def find_pad():
     return candidates
 
 
+def kernel_has_usbmon():
+    """
+    'y' built into the kernel, 'm' a loadable module, None not built at all.
+    Read from the running kernel's own config where it's available.
+    """
+    try:
+        import gzip
+        with gzip.open("/proc/config.gz", "rt") as f:
+            text = f.read()
+    except OSError:
+        try:
+            text = open(f"/boot/config-{os.uname().release}").read()
+        except OSError:
+            return "?"
+    for line in text.splitlines():
+        if line.startswith("CONFIG_USB_MON="):
+            return line.split("=", 1)[1].strip()
+    return None
+
+
+def debugfs_mounted():
+    try:
+        return " /sys/kernel/debug " in open("/proc/mounts").read()
+    except OSError:
+        return False
+
+
 def ensure_usbmon():
+    """
+    Get /sys/kernel/debug/usb/usbmon to exist, and explain it if we can't.
+
+    Two separate things have to be true: the usbmon facility has to be in
+    the kernel, and debugfs has to be mounted for it to appear. A missing
+    module and an unmounted debugfs look identical until you check both.
+    """
     if os.path.isdir(DEBUGFS):
         return True
+
+    built = kernel_has_usbmon()
+
+    if built == "m":
         print(f"{DIM}Loading usbmon module...{O}")
         subprocess.run(["modprobe", "usbmon"], capture_output=True)
-    if os.path.isdir(DEBUGFS):
+
-        return True
+    if not debugfs_mounted():
-    # debugfs may simply not be mounted yet
+        print(f"{DIM}Mounting debugfs...{O}")
         subprocess.run(["mount", "-t", "debugfs", "none", "/sys/kernel/debug"],
                        capture_output=True)
-    return os.path.isdir(DEBUGFS)
+
+    if os.path.isdir(DEBUGFS):
+        return True
+
+    # Still nothing. Say which of the two is actually missing.
+    print(f"\n{R}Can't get at usbmon.{O}  {DIM}kernel {os.uname().release}{O}")
+    if built is None:
+        print("Your kernel was built without usbmon (CONFIG_USB_MON is not set),")
+        print("so USB capture isn't available on it at all.")
+        print(f"\n{B}What to do{O}")
+        print("  Boot a kernel that has it. On Arch and CachyOS the LTS kernel does:")
+        print("    sudo pacman -S linux-lts linux-lts-headers")
+        print("  Then pick it in the boot menu and run this again.")
+    elif built == "m":
+        mod = f"/lib/modules/{os.uname().release}/kernel/drivers/usb/mon"
+        print("Your kernel lists usbmon as a module, but it wouldn't load.")
+        if not os.path.isdir(mod):
+            print(f"{mod} is missing, which usually means the kernel was")
+            print("updated and not rebooted into yet.")
+            print(f"\n{B}What to do{O}\n  Reboot, then run this again.")
+        else:
+            print(f"\n{B}What to do{O}")
+            print("  sudo depmod -a && sudo modprobe usbmon")
+    else:
+        print("usbmon should be built in, but the folder still isn't there.")
+        print(f"\n{B}What to do{O}")
+        print("  sudo mount -t debugfs none /sys/kernel/debug")
+        print("  sudo ls /sys/kernel/debug/usb")
+    return False
 
 
 def decode(line, want_dev):
     """
     usbmon text format, roughly:
-      <tag> <ts> <S|C> <type>:<bus>:<dev>:<ep> <flags> <len> = <hex data>
+      <tag> <ts> <S|C|E> <type><dir>:<bus>:<dev>:<ep> <flags> <len> = <hex>
 
-    We want submissions (S) carrying data toward our device: control
+    Direction is the second letter: 'o' toward the device, 'i' from it.
-    transfers (Co) and interrupt out (Io).
+    Data rides on the submission (S) of an OUT transfer and on the
+    completion (C) of an IN transfer, so we want both.
     """
     parts = line.split()
     if len(parts) < 5:
         return None
 
-    event = parts[2]
+    event, addr = parts[2], parts[3]
-    addr = parts[3]
-    if event != "S":
-        return None
-
     bits = addr.split(":")
     if len(bits) < 4:
         return None
-    xfer, bus, dev, ep = bits[0], bits[1], bits[2], bits[3]
+    xfer, ep = bits[0], bits[3]
 
-    if int(dev) != want_dev:
+    try:
+        if int(bits[2]) != want_dev:
             return None
-    if not xfer.endswith("o"):        # 'o' = out, toward the device
+    except ValueError:
         return None
 
-    if "=" not in line:
+    outgoing = xfer.endswith("o")
+    if outgoing and event != "S":        # OUT data rides on the submission
+        return None
+    if not outgoing and event != "C":    # IN data rides on the completion
+        return None
+
+    if "=" not in line:                  # no payload on this line
         return None
     data = line.split("=", 1)[1].replace(" ", "").strip()
     if not data:
         return None
 
-    kind = {"Co": "control out", "Io": "interrupt out",
+    kind = {"Co": "control out", "Ci": "control in",
-            "Bo": "bulk out"}.get(xfer, xfer)
+            "Io": "interrupt out", "Ii": "interrupt in",
+            "Bo": "bulk out", "Bi": "bulk in"}.get(xfer, xfer)
 
     setup = ""
-    if xfer == "Co":
+    if xfer.startswith("C"):
         # Control transfers carry an 8-byte setup packet in the flags field.
         for p in parts[4:9]:
             if re.fullmatch(r"[0-9a-f]{2,4}", p):
                 setup += p + " "
 
-    return {"kind": kind, "ep": ep, "setup": setup.strip(), "data": data}
+    return {"kind": kind, "ep": ep, "setup": setup.strip(), "data": data,
+            "outgoing": outgoing}
 
 
 def main():
@@ -154,8 +228,9 @@ def main():
 
     print(f"{G}Capturing.{O} Now, in another terminal:")
     print(f"  {B}wine MINI_KEYBOARD.exe{O}")
-    print("\nChange ONE key to the letter 'a', then apply it.")
+    print("\nDo ONE thing in it, then come back here.")
-    print(f"{DIM}('a' is HID code 0x04 - easy to spot in the bytes.){O}")
+    print(f"{DIM}  ->  the software talking to the pad")
+    print(f"  <-  the pad talking back{O}")
     print(f"\nPress {B}Ctrl+C{O} when you're done.\n")
     print("-" * 62)
 
@@ -171,7 +246,8 @@ def main():
                 n = len(packets)
                 hexstr = rec["data"]
                 pretty = " ".join(hexstr[i:i + 2] for i in range(0, min(len(hexstr), 40), 2))
-                print(f"{n:3}  {rec['kind']:<14} {pretty}"
+                arrow = f"{DIM}->{O}" if rec["outgoing"] else f"{G}<-{O}"
+                print(f"{n:3} {arrow} {rec['kind']:<14} {pretty}"
                       + (" ..." if len(hexstr) > 40 else ""))
                 out.write(f"[{n}] {rec['kind']} ep={rec['ep']}"
                           + (f" setup={rec['setup']}" if rec["setup"] else "")
@@ -186,15 +262,15 @@ def main():
     print("\n" + "-" * 62)
     if not packets:
         print(f"{Y}Nothing captured.{O}")
-        print("The software may not have written, or it may be talking to a")
+        print("The software may not have talked to the pad, or it may be on a")
         print("different device number. Check lsusb while Wine is running.")
         return 1
 
-    print(f"{G}Captured {len(packets)} outgoing transfer(s){O} -> {OUTFILE}")
+    sent = sum(1 for p in packets if p["outgoing"])
-
+    got = len(packets) - sent
-    hits = [p for p in packets if "04" in p["data"]]
+    print(f"{G}Captured {sent} sent, {got} received{O} -> {OUTFILE}")
-    if hits:
+    if got:
-        print(f"{DIM}{len(hits)} contain byte 0x04, which may be your 'a' keycode.{O}")
+        print(f"{G}The pad answered back.{O} It can be read, not only written.")
 
     print(f"\nSend me {B}{OUTFILE}{O} and I'll decode the format.")
     return 0

macropad-read.py

@@ -0,0 +1,184 @@
+#!/usr/bin/env python3
+"""
+macropad-read - find out what the pad will tell us about itself.
+
+Two ways of listening, neither of which writes a configuration:
+
+  1. Feature reports. A HID "get feature" is a read. We ask for each
+     report id in turn and print anything that comes back.
+
+  2. Passive listening. Several programs can have a hidraw device open at
+     once, and input reports go to all of them. So we can sit and watch
+     while the vendor software talks to the pad, and see the pad's
+     replies, with no kernel module and no root.
+
+Neither of these changes your keybinds.
+
+    python3 macropad-read.py                 # find the pad, do both
+    python3 macropad-read.py /dev/hidraw10   # a specific node
+    python3 macropad-read.py --listen 60     # listen for 60 seconds
+
+To catch the pad answering the vendor software:
+
+    1. Start:   python3 macropad-read.py --listen 60
+    2. In another window:   wine MINI_KEYBOARD.exe
+    3. Press whatever it calls "view settings" or "read".
+    4. Come back here. Anything the pad said is printed and saved.
+
+It writes macropad-read.txt in the same format as macropad-capture.py,
+so the same tools can read it.
+"""
+
+import fcntl
+import os
+import select
+import sys
+import time
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+import macropad as proto                                   # noqa: E402
+
+OUTFILE = "macropad-read.txt"
+REPORT_IDS = (0, 1, 2, 3, 4)
+LEN = 65
+
+G = "\033[32m"; R = "\033[31m"; Y = "\033[33m"; B = "\033[1m"
+DIM = "\033[2m"; O = "\033[0m"
+
+
+def hidiocgfeature(size):
+    """_IOC(READ|WRITE, 'H', 0x07, size) - hidraw's GET_FEATURE ioctl."""
+    return (3 << 30) | (size << 16) | (ord("H") << 8) | 0x07
+
+
+def pretty(data, width=16):
+    lines = []
+    for off in range(0, len(data), width):
+        chunk = data[off:off + width]
+        hexs = " ".join(f"{b:02x}" for b in chunk)
+        text = "".join(chr(b) if 32 <= b < 127 else "." for b in chunk)
+        lines.append(f"    {off:04x}  {hexs:<{width * 3}} {text}")
+    return "\n".join(lines)
+
+
+def try_features(path, out):
+    """Ask the pad for each report id. Reading can't change anything."""
+    print(f"{B}Feature reports{O}  {DIM}asking the pad for each report id{O}")
+    found = 0
+    for rid in REPORT_IDS:
+        buf = bytearray(LEN)
+        buf[0] = rid
+        try:
+            fd = os.open(path, os.O_RDWR)
+        except OSError as e:
+            print(f"  {R}can't open {path}: {e.strerror}{O}")
+            return 0
+        try:
+            fcntl.ioctl(fd, hidiocgfeature(len(buf)), buf, True)
+        except OSError as e:
+            print(f"  id {rid}:  {DIM}nothing ({e.strerror}){O}")
+            continue
+        finally:
+            os.close(fd)
+
+        if not any(buf[1:]):
+            print(f"  id {rid}:  {DIM}all zeros{O}")
+            continue
+        found += 1
+        print(f"  id {rid}:  {G}answered{O}")
+        print(pretty(bytes(buf)))
+        out.write(f"[feature] report id {rid}\n    {bytes(buf).hex()}\n")
+    return found
+
+
+def listen(path, seconds, out):
+    print(f"\n{B}Listening for {seconds}s{O}  "
+          f"{DIM}anything the pad sends shows up here{O}")
+    print(f"{DIM}Now use the vendor software and press its read or view "
+          f"settings button.{O}\n")
+    try:
+        fd = os.open(path, os.O_RDONLY | os.O_NONBLOCK)
+    except OSError as e:
+        print(f"{R}Can't open {path}: {e.strerror}{O}")
+        return 0
+
+    seen = 0
+    end = time.time() + seconds
+    try:
+        while time.time() < end:
+            ready, _, _ = select.select([fd], [], [], 0.5)
+            if not ready:
+                continue
+            try:
+                data = os.read(fd, 128)
+            except BlockingIOError:
+                continue
+            except OSError as e:
+                print(f"{R}read failed: {e.strerror}{O}")
+                break
+            if not data:
+                continue
+            seen += 1
+            left = int(end - time.time())
+            print(f"{G}<-{O} {len(data)} bytes  {DIM}({left}s left){O}")
+            print(pretty(data))
+            out.write(f"[{seen}] interrupt in\n    {data.hex()}\n")
+            out.flush()
+    except KeyboardInterrupt:
+        print(f"\n{DIM}stopped early{O}")
+    finally:
+        os.close(fd)
+    return seen
+
+
+def main():
+    args = [a for a in sys.argv[1:] if not a.startswith("--")]
+    seconds = 30
+    if "--listen" in sys.argv:
+        i = sys.argv.index("--listen")
+        if i + 1 < len(sys.argv) and sys.argv[i + 1].isdigit():
+            seconds = int(sys.argv[i + 1])
+
+    if args:
+        path = args[0]
+    else:
+        pads = proto.find_devices()
+        if not pads:
+            print(f"{R}No pad found.{O} Run macropad-probe.py to see what's there,")
+            print("or pass the node yourself:  python3 macropad-read.py /dev/hidrawN")
+            return 1
+        if len(pads) > 1:
+            print(f"{B}Several candidates:{O}")
+            for p in pads:
+                print(f"  {p['path']}  {p['vid']}:{p['pid']}")
+            print("\nPick one and pass it as an argument.")
+            return 1
+        path = pads[0]["path"]
+
+    if not os.access(path, os.R_OK):
+        print(f"{R}{path} isn't readable.{O} Add the udev rule, or use sudo.")
+        return 1
+
+    print(f"{B}Reading from {path}{O}")
+    print(f"{DIM}Nothing here writes a configuration.{O}\n")
+
+    with open(OUTFILE, "w") as out:
+        out.write(f"# read attempt on {path}\n\n")
+        found = try_features(path, out)
+        heard = listen(path, seconds, out)
+
+    print("\n" + "-" * 62)
+    if found or heard:
+        print(f"{G}Got something.{O}  "
+              f"{found} feature report(s), {heard} message(s) while listening.")
+        print(f"Saved to {B}{OUTFILE}{O} - send that over.")
+    else:
+        print(f"{Y}The pad said nothing.{O}")
+        print("That doesn't prove it can't be read, only that it wasn't")
+        print("answering us. The next thing to try is capturing the vendor")
+        print("software with macropad-capture.py.")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

macropad.py

@@ -223,6 +223,47 @@ def native_media(report_id, action, usage, layer=1):
     return [bytes([report_id]) + bytes(data), native_commit(report_id)]
 
 
+# ----------------------------------------------------------- reading back
+#
+# The pad can be read, which we only found by capturing the vendor software's
+# "view settings" button. Two commands, both answered on the interrupt IN
+# endpoint of the same report id:
+#
+#   0xFB  what are you?    reply: 03 fb <keys> <knobs>
+#   0xFA  send me a layer  reply: one report per control, magic 0xFA, then
+#                                 nothing more
+#
+# Observed, reading layer 1:
+#   03 fa 0f 03 01 02 ...          request (trailing bytes were uninitialised
+#                                  memory in the vendor program; zeros work)
+#   03 fa 02 01 01 00 ... 01 05 17 key2 is ctrl+alt+t
+#   03 fa 10 01 02 00 ... 01 ea 00 dial 1 left is volume down
+
+MAGIC_INFO = 0xFB
+MAGIC_READ = 0xFA
+KEY_SLOTS = 0x0F      # the firmware keeps 15 key slots whatever the pad has
+KNOB_SLOTS = 3
+LAYERS = 3
+
+
+def native_info(report_id):
+    """Ask the pad what it is. Reply carries the key and knob counts."""
+    data = bytearray(REPORT_LEN - 1)
+    data[0] = MAGIC_INFO
+    return bytes([report_id]) + bytes(data)
+
+
+def native_read(report_id, layer=1):
+    """Ask the pad for every control in one layer."""
+    data = bytearray(REPORT_LEN - 1)
+    data[0] = MAGIC_READ
+    data[1] = KEY_SLOTS
+    data[2] = LAYERS
+    data[3] = layer
+    data[4] = 2
+    return bytes([report_id]) + bytes(data)
+
+
 # ------------------------------------------------------- legacy protocol
 #
 # The older format. Unlike Extended, a single binding is a *sequence* of

macropad_gui/__main__.py

@@ -35,7 +35,8 @@ def cmd_doctor(_):
 def cmd_state(_):
     s = core.State()
     print(f"{DIM}{s.path}{'' if s.path.exists() else '  (not created yet)'}{O}\n")
-    for c in core.CONTROL_IDS:
+    print(f"{DIM}layout: {s.layout.describe()}{O}\n")
+    for c in s.layout.control_ids:
         e = s.get(c)
         byte = f"0x{core.action_byte(c):02x}"
         if e.unknown:
@@ -46,6 +47,26 @@ def cmd_state(_):
     return 0
 
 
+def cmd_detect(_):
+    d = core.diagnose(keyd=False)
+    if not d.device:
+        print(f"{R}No pad found at all.{O}")
+        return 1
+    print(f"{B}Asking {d.device.vid}:{d.device.pid} on {d.device.path}{O}")
+    print(f"{DIM}This only reads. Nothing is configured.{O}\n")
+    found = core.detect(d.device)
+    print(core.detection_report(d.device, found))
+    if found.speaks:
+        lay = found.layout
+        print(f"\n{G}It speaks this protocol.{O}")
+        print(f"Best guess at the shape: {lay.describe()}. The pad reports how "
+              f"many\nkeys it has, not how they're arranged, so check that "
+              f"against yours.")
+        return 0
+    print(f"\n{Y}It didn't answer properly, so the app won't write to it.{O}")
+    return 1
+
+
 def cmd_write(a):
     if a.control not in core.CONTROL_IDS:
         print(f"Unknown control. One of: {', '.join(core.CONTROL_IDS)}")
@@ -78,6 +99,7 @@ def main():
     sub = ap.add_subparsers(dest="cmd")
     sub.add_parser("doctor", help="check the pad is connected and writable")
     sub.add_parser("state", help="show what this tool believes is on the pad")
+    sub.add_parser("detect", help="ask an unrecognised pad what it is (read only)")
     w = sub.add_parser("write", help="write one binding and record it")
     w.add_argument("control")
     w.add_argument("keys", nargs="?", help="e.g. 'ctrl+c' or 'a,b,c'")
@@ -92,7 +114,8 @@ def main():
             print("Install Qt for Python:  sudo pacman -S pyside6")
             return 1
         return run()
-    return {"doctor": cmd_doctor, "state": cmd_state, "write": cmd_write}[a.cmd](a)
+    return {"doctor": cmd_doctor, "state": cmd_state, "write": cmd_write,
+            "detect": cmd_detect}[a.cmd](a)
 
 
 if __name__ == "__main__":

macropad_gui/app.py

@@ -13,7 +13,7 @@ from PySide6.QtGui import (QColor, QFont, QFontDatabase, QGuiApplication, QIcon,
 from PySide6.QtWidgets import (QApplication, QButtonGroup, QFileDialog, QFrame,
                                QGridLayout, QHBoxLayout, QLabel, QLineEdit,
                                QMainWindow, QMessageBox, QPlainTextEdit,
-                               QPushButton, QVBoxLayout, QWidget)
+                               QPushButton, QSpinBox, QVBoxLayout, QWidget)
 
 from . import core, keymap
 from .padview import INK, INK_DIM, TAPE, WINDOW, Look, PadView
@@ -57,6 +57,10 @@ QFrame[role="banner"] {{ border: 1px solid {LINE}; border-left: 3px solid {BAD};
 QFrame[role="banner"][tone="quiet"] {{ border-left: 3px solid {INK_DIM.name()}; }}
 QFrame[role="banner"] QLabel, QFrame[role="banner"] QPushButton {{ background: transparent; }}
 QFrame[role="footer"] {{ border-top: 1px solid {LINE}; }}
+QFrame[role="editbar"] {{ border-top: 1px solid {TAPE.name()}; background: #1c1a17; }}
+QFrame[role="editbar"] QLabel {{ background: transparent; }}
+QSpinBox {{ background: {FIELD}; border: 1px solid {LINE}; border-radius: 4px;
+            padding: 4px 6px; }}
 """
 
 
@@ -144,12 +148,14 @@ class Inspector(QWidget):
     changed = Signal(str, object)       # control, Binding or None (revert)
     write = Signal(str)
     revert = Signal(str)
+    action = Signal()
 
     def __init__(self):
         super().__init__()
         self.setFixedWidth(360)
         self.control = None
         self._loading = False
+        self.editing_layout = False
 
         v = QVBoxLayout(self)
         v.setContentsMargins(24, 24, 24, 16)
@@ -170,7 +176,10 @@ class Inspector(QWidget):
         self.banner_fix.setLineWrapMode(QPlainTextEdit.NoWrap)
         self.banner_copy = _button("Copy commands")
         self.banner_copy.clicked.connect(self._copy)
-        for w in (self.banner_head, self.banner_text, self.banner_fix, self.banner_copy):
+        self.banner_action = _button("")
+        self.banner_action.clicked.connect(lambda: self.action.emit())
+        for w in (self.banner_head, self.banner_text, self.banner_fix,
+                  self.banner_copy, self.banner_action):
             bl.addWidget(w)
         self.banner_copy.setSizePolicy(self.banner_copy.sizePolicy().horizontalPolicy(),
                                        self.banner_copy.sizePolicy().verticalPolicy())
@@ -290,6 +299,8 @@ class Inspector(QWidget):
         self.banner_fix.setFixedHeight(lh * max(1, len(cmds)) + 30)
         self.banner_fix.setVisible(bool(cmds))
         self.banner_copy.setVisible(bool(cmds))
+        self.banner_action.setVisible(diag.status == "unsupported")
+        self.banner_action.setText("Ask the pad what it is")
         if diag.fix and diag.fix[-1].startswith("#"):
             self.banner_text.setText(self.banner_text.text() + "\n\nThen unplug the pad and plug it back in.")
         self.banner.show()
@@ -307,11 +318,15 @@ class Inspector(QWidget):
         if control is None:
             self.editor.hide()
             self.empty.setText(
-                "Click a key or dial to change what it does.\n\n"
+                ("Drag the keys around until the drawing matches your pad, "
-                + ("The pad can't report what's on it, so this app only knows "
+                 "then press Done.\n\nUse the boxes below to change the "
-                   "what it has written itself. Hatched controls are unknown. "
+                 "grid or the number of keys and knobs. This only changes "
-                   "If you set the pad up with the vendor software, you can "
+                 "the picture, never the pad."
-                   "import that capture instead." if all_unknown else ""))
+                 if self.editing_layout else
+                 "Click a key or dial to change what it does.\n\n")
+                + ("Hatched controls are ones this app hasn't been told about. "
+                   "Press Read pad to ask the pad itself what it's holding."
+                   if all_unknown and not self.editing_layout else ""))
             self.empty.show()
             self._loading = False
             return
@@ -422,6 +437,9 @@ class Window(QMainWindow):
         self.state = core.State()
         self.desired = {}                    # control -> Binding, unwritten edits
         self.diag = None
+        # Pads the user has identified by asking them. Remembered so the app
+        # doesn't go back to calling them unsupported on the next check.
+        self.adopted = set(self.settings.value("adopted", []) or [])
 
         root = QWidget()
         outer = QVBoxLayout(root)
@@ -431,12 +449,41 @@ class Window(QMainWindow):
         body = QHBoxLayout()
         body.setSpacing(0)
         self.pad = PadView()
+        self.pad.set_layout(self.state.layout)
         self.pad.set_orientation(self.settings.value("orientation", "upright"))
         self.inspector = Inspector()
         body.addWidget(self.pad, 1)
         body.addWidget(self.inspector)
         outer.addLayout(body, 1)
 
+        # editor bar, hidden unless you're rearranging the pad
+        self.editbar = QFrame()
+        self.editbar.setProperty("role", "editbar")
+        eb = QHBoxLayout(self.editbar)
+        eb.setContentsMargins(16, 8, 16, 8)
+        eb.setSpacing(8)
+        eb.addWidget(_label("Drag the keys to match your pad.", "dim"))
+        eb.addSpacing(12)
+        self.spin = {}
+        for name, label, lo, hi in (("rows", "Rows", 1, 15),
+                                    ("cols", "Columns", 1, 15),
+                                    ("keys", "Keys", 1, 15),
+                                    ("knobs", "Knobs", 0, 3)):
+            eb.addWidget(_label(label, "dim"))
+            box = QSpinBox()
+            box.setRange(lo, hi)
+            box.setFixedWidth(56)
+            box.valueChanged.connect(self._resize_layout)
+            self.spin[name] = box
+            eb.addWidget(box)
+        eb.addStretch()
+        self.edit_done = _button("Done", "primary")
+        self.edit_cancel = _button("Cancel", "flat")
+        eb.addWidget(self.edit_cancel)
+        eb.addWidget(self.edit_done)
+        self.editbar.hide()
+        outer.addWidget(self.editbar)
+
         footer = QFrame()
         footer.setProperty("role", "footer")
         fl = QHBoxLayout(footer)
@@ -445,6 +492,10 @@ class Window(QMainWindow):
         self.dot.setFixedSize(9, 9)
         self.status = _label("", "dim")
         self.message = _label("")
+        self.edit_btn = _button("Edit layout", "flat")
+        self.edit_btn.setToolTip("Rearrange the drawing to match your pad")
+        self.read_btn = _button("Read pad", "flat")
+        self.read_btn.setToolTip("Ask the pad what's on it right now")
         self.rotate = _button("Rotate view", "flat")
         self.import_btn = _button("Import capture…", "flat")
         self.write_all = _button("", "primary")
@@ -452,6 +503,8 @@ class Window(QMainWindow):
         fl.addWidget(self.status)
         fl.addSpacing(16)
         fl.addWidget(self.message, 1)
+        fl.addWidget(self.edit_btn)
+        fl.addWidget(self.read_btn)
         fl.addWidget(self.rotate)
         fl.addWidget(self.import_btn)
         fl.addWidget(self.write_all)
@@ -461,8 +514,14 @@ class Window(QMainWindow):
         self.pad.selected.connect(self._select)
         self.inspector.changed.connect(self._changed)
         self.inspector.revert.connect(self._revert)
+        self.inspector.action.connect(self._identify)
         self.inspector.write.connect(lambda c: self._write([c]))
         self.write_all.clicked.connect(lambda: self._write(self._pending()))
+        self.edit_btn.clicked.connect(lambda: self._edit_layout(True))
+        self.edit_done.clicked.connect(lambda: self._edit_layout(False, keep=True))
+        self.edit_cancel.clicked.connect(lambda: self._edit_layout(False, keep=False))
+        self.pad.rearranged.connect(self._rearranged)
+        self.read_btn.clicked.connect(lambda: self._read(quiet=False))
         self.rotate.clicked.connect(self._rotate)
         self.import_btn.clicked.connect(self._import)
 
@@ -473,12 +532,15 @@ class Window(QMainWindow):
         self.timer.start(2000)
         self._poll()
         self._refresh()
+        # The pad can tell us what it holds, so ask rather than assume.
+        if self.diag and self.diag.ready:
+            QTimer.singleShot(0, lambda: self._read(quiet=True))
 
     # ------------------------------------------------------------- model
 
     def _pending(self):
         out = []
-        for c in core.CONTROL_IDS:
+        for c in self.state.layout.control_ids:
             b = self.desired.get(c)
             if b is None:
                 continue
@@ -490,7 +552,7 @@ class Window(QMainWindow):
     def _refresh(self, editor=True):
         pending = set(self._pending())
         looks = {}
-        for c in core.CONTROL_IDS:
+        for c in self.state.layout.control_ids:
             e = self.state.get(c)
             if c in pending:
                 looks[c] = Look(self.desired[c].label(), known=not e.unknown, pending=True)
@@ -510,7 +572,8 @@ class Window(QMainWindow):
         if not editor:
             self.inspector.update_buttons(c in pending, can_write)
             return
-        all_unknown = all(self.state.get(x).unknown for x in core.CONTROL_IDS)
+        all_unknown = all(self.state.get(x).unknown
+                          for x in self.state.layout.control_ids)
         self.inspector.show_control(
             c, self.state.get(c) if c else None,
             self.desired.get(c) if c in pending else None,
@@ -539,8 +602,15 @@ class Window(QMainWindow):
         except Exception as e:            # never let polling kill the window
             self._say(f"Couldn't check the pad: {e}", BAD)
             return
+        d = self.diag
+        if d.status == "unsupported" and d.device and \
+                f"{d.device.vid}:{d.device.pid}" in self.adopted:
+            self.diag = core.Diagnosis("ready", f"Ready on {d.device.path}.",
+                                       device=d.device, warnings=d.warnings)
         colour = {"ready": GOOD, "unplugged": INK_DIM.name()}.get(self.diag.status, BAD)
         self.dot.setStyleSheet(f"background: {colour}; border-radius: 4px;")
+        if self.diag.device:
+            self.pad.device_id = f"{self.diag.device.vid}:{self.diag.device.pid}"
         self.status.setText({
             "ready": f"Pad ready on {self.diag.device.path}" if self.diag.device else "Pad ready",
             "unplugged": "No pad connected",
@@ -551,6 +621,8 @@ class Window(QMainWindow):
         self.inspector.show_problem(self.diag, self.note)
         if self.diag.status != old:
             self._refresh(editor=False)
+            if self.diag.ready and old in ("unplugged", "no-interface", None):
+                self._read(quiet=True)
 
     # ------------------------------------------------------------- write
 
@@ -590,12 +662,122 @@ class Window(QMainWindow):
             self._say(f"Wrote {len(done)} changes to the pad.", GOOD)
         self._refresh()
 
+    def _read(self, quiet=False):
+        """
+        Ask the pad what it holds and believe the answer. Anything you have
+        changed but not written is left alone, since that is your intent
+        rather than the pad's state.
+        """
+        if not (self.diag and self.diag.ready):
+            if not quiet:
+                self._say(self.diag.headline if self.diag else "No pad.", BAD)
+            return
+        self.read_btn.setEnabled(False)
+        self.read_btn.setText("Reading...")
+        QApplication.processEvents()
+        try:
+            got = core.read_into_state(self.diag.device, self.state)
+        except core.ReadError as e:
+            if not quiet:
+                self._say(f"Couldn't read the pad: {e}", BAD)
+        else:
+            n = len(got)
+            self._say(f"Read {n} control{'s' if n != 1 else ''} from the pad.", GOOD)
+        finally:
+            self.read_btn.setEnabled(True)
+            self.read_btn.setText("Read pad")
+        self._refresh()
+
+    def _identify(self):
+        """
+        Interrogate a pad we don't recognise. Reads only. If it answers
+        properly we adopt its shape and carry on; if not we say so and
+        leave it alone.
+        """
+        dev = self.diag.device if self.diag else None
+        if not dev:
+            return
+        self.inspector.banner_action.setEnabled(False)
+        self.inspector.banner_action.setText("Asking...")
+        QApplication.processEvents()
+        try:
+            found = core.detect(dev)
+        finally:
+            self.inspector.banner_action.setEnabled(True)
+            self.inspector.banner_action.setText("Ask the pad what it is")
+
+        if not found.speaks:
+            self._say(f"That pad doesn't speak this protocol: {found.why}", BAD)
+            return
+
+        layout = found.layout
+        self.state = core.State(layout=layout)
+        self.state.layout = layout
+        for control, binding in found.bindings.items():
+            self.state.record(control, binding)
+        self.state.save()
+        self.pad.set_layout(layout)
+        self.desired.clear()
+        self.adopted.add(f"{dev.vid}:{dev.pid}")
+        self.settings.setValue("adopted", sorted(self.adopted))
+        self._poll()
+        self._say(f"It says {layout.keys} keys and {layout.knobs} "
+                  f"knob{'s' if layout.knobs != 1 else ''}. Check the drawing "
+                  f"matches your pad.", GOOD)
+        self._refresh()
+
     def _say(self, text, colour=None):
         self.message.setText(text)
         self.message.setStyleSheet(f"color: {colour};" if colour else "")
 
     # ------------------------------------------------------------- misc
 
+    def _edit_layout(self, on, keep=False):
+        if on:
+            self.editing_from = self.state.layout
+            for name, value in (("rows", self.state.layout.rows),
+                                ("cols", self.state.layout.cols),
+                                ("keys", self.state.layout.keys),
+                                ("knobs", self.state.layout.knobs)):
+                self.spin[name].blockSignals(True)
+                self.spin[name].setValue(value)
+                self.spin[name].blockSignals(False)
+            self._say("")
+        elif not keep:
+            self._set_layout(self.editing_from)
+        else:
+            self.state.save()
+            lay = self.state.layout
+            self._say(f"Layout saved: {lay.describe()}.", GOOD)
+        self.editbar.setVisible(on)
+        self.edit_btn.setEnabled(not on)
+        self.read_btn.setEnabled(not on)
+        self.inspector.editing_layout = on
+        self.pad.select(None if on else self.pad.current)
+        self.pad.set_editing(on)
+        self._refresh()
+
+    def _set_layout(self, layout):
+        self.state.layout = layout
+        self.pad.set_layout(layout)
+        if self.pad.current not in layout.control_ids:
+            self.pad.current = None
+        self._refresh()
+
+    def _rearranged(self, layout):
+        self.state.layout = layout
+
+    def _resize_layout(self):
+        lay = self.state.layout.resized(
+            rows=self.spin["rows"].value(), cols=self.spin["cols"].value(),
+            keys=self.spin["keys"].value(), knobs=self.spin["knobs"].value())
+        if lay.rows != self.spin["rows"].value():     # it had to grow to fit
+            self.spin["rows"].blockSignals(True)
+            self.spin["rows"].setValue(lay.rows)
+            self.spin["rows"].blockSignals(False)
+        self._set_layout(lay)
+        self.pad.set_editing(True)
+
     def _rotate(self):
         o = "flat" if self.pad.orientation == "upright" else "upright"
         self.pad.set_orientation(o)

macropad_gui/core.py

@@ -26,7 +26,7 @@ import subprocess
 import sys
 import tempfile
 import time
-from dataclasses import dataclass, field
+from dataclasses import dataclass, field, replace
 from datetime import datetime
 from pathlib import Path
 
@@ -42,23 +42,155 @@ except ImportError:                      # running from inside the package dir
 VID, PID = "1189", "8840"
 DEVICE_ID = f"{VID}:{PID}"
 
-KEYS = tuple(f"key{i}" for i in range(1, 13))
-DIALS = ("dial1", "dial2")
 DIAL_ACTIONS = ("left", "push", "right")
-CONTROL_IDS = KEYS + tuple(f"{d}-{a}" for d in DIALS for a in DIAL_ACTIONS)
-
-# Action bytes come from macropad.CONTROLS, never from here.
-for _cid in CONTROL_IDS:
-    assert _cid in proto.CONTROLS, f"macropad.py has no control {_cid}"
-
 WRITE_GAP = 0.02      # same pause the CLI leaves between reports
 
 
 def action_byte(control):
-    return proto.CONTROLS[control]
+    """
+    The byte the pad uses for one control. Keys count from 1; knobs start
+    at 0x10, three apiece. Confirmed on 1189:8840 both by writing and by
+    reading back, and it is what the pad reports for its own slots.
+    """
+    control = control.lower()
+    if control.startswith("key"):
+        return int(control[3:])
+    dial, act = control.split("-")
+    return proto.KNOB_BASE + (int(dial[4:]) - 1) * 3 + DIAL_ACTIONS.index(act)
 
 
-CONTROL_FOR_ACTION = {action_byte(c): c for c in CONTROL_IDS}
+@dataclass(frozen=True)
+class Layout:
+    """
+    The shape of one pad: how many keys and knobs, and how the keys are
+    arranged when you look at it.
+
+    keys and knobs come from the pad itself (the 0xFB query). Rows and
+    columns do not: the pad knows how many keys it has, not how they are
+    laid out, so that part is either known from the model or chosen by
+    whoever owns it.
+
+    Rows and columns describe the pad lying flat, the way its keys are
+    printed. Standing it upright just turns the drawing.
+    """
+    keys: int = 12
+    knobs: int = 2
+    rows: int = 3
+    cols: int = 4
+    name: str = ""
+    places: tuple = ()      # ((row, col), ...) per key; empty means left to right
+
+    def __post_init__(self):
+        if not 1 <= self.keys <= proto.KEY_SLOTS:
+            raise ValueError(f"a pad can have 1 to {proto.KEY_SLOTS} keys")
+        if not 0 <= self.knobs <= proto.KNOB_SLOTS:
+            raise ValueError(f"a pad can have up to {proto.KNOB_SLOTS} knobs")
+        if self.rows * self.cols < self.keys:
+            raise ValueError(f"{self.rows}x{self.cols} can't hold "
+                             f"{self.keys} keys")
+        if self.places:
+            if len(self.places) != self.keys:
+                raise ValueError("places must give a spot for every key")
+            if len(set(self.places)) != len(self.places):
+                raise ValueError("two keys can't share a spot")
+            for r, c in self.places:
+                if not (0 <= r < self.rows and 0 <= c < self.cols):
+                    raise ValueError(f"({r}, {c}) is outside {self.rows}x{self.cols}")
+
+    def spots(self):
+        """[(key_id, row, col)] with the pad lying flat."""
+        if self.places:
+            return [(f"key{i + 1}", r, c) for i, (r, c) in enumerate(self.places)]
+        return [(f"key{i + 1}", i // self.cols, i % self.cols)
+                for i in range(self.keys)]
+
+    def rearranged(self, places):
+        """The same pad with its keys in different spots."""
+        return replace(self, places=tuple((int(r), int(c)) for r, c in places))
+
+    def resized(self, rows=None, cols=None, keys=None, knobs=None):
+        """
+        Change the grid, keeping any arrangement that still fits and
+        falling back to left to right for keys that no longer do.
+        """
+        rows = self.rows if rows is None else rows
+        cols = self.cols if cols is None else cols
+        keys = self.keys if keys is None else keys
+        knobs = self.knobs if knobs is None else knobs
+        if rows * cols < keys:                 # grow to fit rather than refuse
+            rows = -(-keys // cols)
+
+        taken, places = set(), []
+        old = {i: p for i, p in enumerate(self.places)}
+        free = [(r, c) for r in range(rows) for c in range(cols)]
+        for i in range(keys):
+            spot = old.get(i)
+            if spot and spot[0] < rows and spot[1] < cols and spot not in taken:
+                places.append(spot)
+                taken.add(spot)
+            else:
+                places.append(None)
+        for i, spot in enumerate(places):
+            if spot is None:
+                nxt = next(p for p in free if p not in taken)
+                places[i] = nxt
+                taken.add(nxt)
+        return replace(self, rows=rows, cols=cols, keys=keys, knobs=knobs,
+                       places=tuple(places) if self.places else ())
+
+    @property
+    def key_ids(self):
+        return tuple(f"key{i}" for i in range(1, self.keys + 1))
+
+    @property
+    def dial_ids(self):
+        return tuple(f"dial{i}" for i in range(1, self.knobs + 1))
+
+    @property
+    def control_ids(self):
+        return self.key_ids + tuple(f"{d}-{a}" for d in self.dial_ids
+                                    for a in DIAL_ACTIONS)
+
+    @property
+    def control_for_action(self):
+        return {action_byte(c): c for c in self.control_ids}
+
+    def describe(self):
+        knobs = (f" and {self.knobs} knob{'s' if self.knobs != 1 else ''}"
+                 if self.knobs else "")
+        return f"{self.keys} keys in {self.rows}x{self.cols}{knobs}"
+
+    def to_json(self):
+        d = {"keys": self.keys, "knobs": self.knobs,
+             "rows": self.rows, "cols": self.cols, "name": self.name}
+        if self.places:
+            d["places"] = [list(p) for p in self.places]
+        return d
+
+    @classmethod
+    def from_json(cls, d):
+        return cls(keys=int(d["keys"]), knobs=int(d["knobs"]),
+                   rows=int(d["rows"]), cols=int(d["cols"]),
+                   name=d.get("name", ""),
+                   places=tuple(tuple(p) for p in d.get("places", ())))
+
+
+# Pads we have laid eyes on. Anything else has to be detected or described
+# by its owner, because a USB id alone does not pin down a layout: 1189:8840
+# covers several different pads.
+KNOWN_LAYOUTS = {
+    ("1189", "8840"): Layout(12, 2, 3, 4, "12 keys, 2 knobs"),
+}
+DEFAULT_LAYOUT = KNOWN_LAYOUTS[(VID, PID)]
+
+# Kept so older code and the CLI keep working; the app uses a Layout.
+KEYS = DEFAULT_LAYOUT.key_ids
+DIALS = DEFAULT_LAYOUT.dial_ids
+CONTROL_IDS = DEFAULT_LAYOUT.control_ids
+CONTROL_FOR_ACTION = DEFAULT_LAYOUT.control_for_action
+
+for _cid in CONTROL_IDS:
+    assert action_byte(_cid) == proto.CONTROLS[_cid], _cid
 
 _DIAL_WORDS = {"left": "turn left", "push": "press", "right": "turn right"}
 
@@ -246,7 +378,7 @@ def _atomic_write_json(path, data):
         raise
 
 
-FORMAT = 1
+FORMAT = 2      # 1 had no layout, and one build saved rows and cols swapped
 
 
 @dataclass
@@ -264,16 +396,23 @@ class State:
     should call record()/mark_unknown(); the GUI just reads it.
     """
 
-    def __init__(self, path=None):
+    def __init__(self, path=None, layout=None):
         self.path = Path(path) if path else config_dir() / f"state-{VID}-{PID}.json"
         self.entries = {}                    # (layer, control) -> Entry
+        self.layout = layout or DEFAULT_LAYOUT
         if self.path.exists():
             self._load()
 
     def _load(self):
         data = json.loads(self.path.read_text())
-        if data.get("format") != FORMAT:
+        version = data.get("format")
-            raise ValueError(f"{self.path}: unsupported format {data.get('format')}")
+        if version not in (1, FORMAT):
+            raise ValueError(f"{self.path}: unsupported format {version}")
+        # Format 1 files either have no layout or have one from the build that
+        # stored rows and columns the wrong way round. Ignore it either way;
+        # the bindings are still good.
+        if version == FORMAT and data.get("layout"):
+            self.layout = Layout.from_json(data["layout"])
         for layer, controls in data.get("layers", {}).items():
             for control, e in controls.items():
                 if e.get("unknown"):
@@ -293,6 +432,7 @@ class State:
                 d = {"binding": e.binding.to_json(), "written": e.written}
             layers.setdefault(str(layer), {})[control] = d
         _atomic_write_json(self.path, {"format": FORMAT, "device": DEVICE_ID,
+                                       "layout": self.layout.to_json(),
                                        "layers": layers})
 
     def get(self, control, layer=1):
@@ -315,6 +455,8 @@ class Device:
     report_id: int
     writable: bool
     name: str = ""
+    vid: str = VID
+    pid: str = PID
 
 
 @dataclass
@@ -435,17 +577,19 @@ def diagnose(keyd=True):
 
     if not found and everything:
         other = everything[0]
+        dev = Device(path=other["path"], report_id=other["report_id"],
+                     writable=other["writable"], name=other.get("name", ""),
+                     vid=other["vid"], pid=other["pid"])
         return Diagnosis(
             "unsupported",
-            f"Found a {other['vid']}:{other['pid']} pad, which this app doesn't know yet.",
+            f"Found a {other['vid']}:{other['pid']} pad, which isn't one this "
-            detail=[f"So far only {DEVICE_ID} (12 keys, 2 dials) has been "
+            "app knows.",
-                    "worked out. Pads in this family differ in layout and "
+            device=dev,
-                    "protocol, so writing to yours with the wrong one could "
+            detail=["Pads in this family differ in both layout and protocol, "
-                    "fail or scramble it. The app won't try.",
+                    "so the app won't write to one it hasn't identified.",
-                    "ch57x-keyboard-tool, a command line tool, supports several "
+                    "It can ask the pad about itself first. That only reads, "
-                    "of these pads today. To get yours into this app, the "
+                    "and if the pad doesn't answer properly nothing else "
-                    "README's \"Other pads\" section shows how to capture what "
+                    "happens."],
-                    "the vendor software sends."],
             warnings=warnings)
 
     if not found:
@@ -464,7 +608,8 @@ def diagnose(keyd=True):
 
     d = found[0]
     dev = Device(path=d["path"], report_id=d["report_id"],
-                 writable=d["writable"], name=d.get("name", ""))
+                 writable=d["writable"], name=d.get("name", ""),
+                 vid=d["vid"], pid=d["pid"])
 
     if dev.writable:
         return Diagnosis("ready", f"Ready on {dev.path}.", device=dev,
@@ -569,13 +714,20 @@ def write_binding(device, control, binding, state, layer=1, _write=None):
 
 # --------------------------------------------------------------- decoding
 
+def decode_info(report):
+    """(keys, knobs) from a 0xFB reply, or None."""
+    if len(report) < 4 or report[1] != proto.MAGIC_INFO:
+        return None
+    return report[2], report[3]
+
+
 def decode_config(report):
     """
     The inverse of Binding.reports()[0]: (control, layer, Binding) from a
     native config report, or None if it isn't one we understand. Used to
     import captures and to prove round trips in the tests.
     """
-    if len(report) < 13 or report[1] != proto.MAGIC_NATIVE:
+    if len(report) < 13 or report[1] not in (proto.MAGIC_NATIVE, proto.MAGIC_READ):
         return None
     action, layer, kind = report[2], report[3], report[4]
     control = CONTROL_FOR_ACTION.get(action)
@@ -584,17 +736,17 @@ def decode_config(report):
     delay = report[5] | (report[6] << 8)
     count = report[10]
 
-    if kind == proto.KeyType.MULTIMEDIA:
+    if kind == proto.KeyType.MULTIMEDIA and count:
         media = _MEDIA_BY_USAGE.get(report[11] | (report[12] << 8))
         return (control, layer, Binding(media=media)) if media else None
 
-    if kind == proto.KeyType.BASIC and count == 0:
+    if count == 0 and kind in (proto.KeyType.NONE, proto.KeyType.BASIC):
         mods, code = report[11], report[12]
         if not mods and not code:
             return control, layer, Binding(none=True)
         return None
 
-    if kind == proto.KeyType.BASIC and count:
+    if kind in (proto.KeyType.NONE, proto.KeyType.BASIC) and count:
         steps = []
         for i in range(count):
             off = 11 + 2 * i
@@ -616,11 +768,21 @@ def is_commit(report):
         (report[2], report[3]) == proto.TERMINATOR
 
 
-def read_capture(path):
+def read_capture(path, incoming=False):
-    """Raw report bytes from a macropad-capture.py output file, in order."""
+    """
+    Reports from a macropad-capture.py output file, in order.
+
+    Captures can now hold both directions. By default we return only what
+    was sent TO the pad, since that is what a binding looks like; pass
+    incoming=True for what the pad sent back.
+    """
     out = []
+    want = " in " if incoming else " out "
+    keep = False
     for line in Path(path).read_text().splitlines():
-        if line.startswith("    "):
+        if line.startswith("["):
+            keep = want in f" {line.split(']', 1)[-1].strip()} "
+        elif line.startswith("    ") and keep:
             try:
                 out.append(bytes.fromhex(line.strip()))
             except ValueError:
@@ -648,3 +810,192 @@ def import_capture(path, state):
     if done:
         state.save()
     return done
+
+
+# ---------------------------------------------------------------- reading
+
+READ_TIMEOUT = 1.5       # total, per request
+READ_QUIET = 0.25        # replies have stopped once this much silence passes
+
+
+class ReadError(Exception):
+    pass
+
+
+def _exchange(path, request, want=None):
+    """
+    Send one request and collect replies until the pad goes quiet.
+
+    Reading is the one place we write to the pad without changing it: both
+    commands here are queries, byte for byte what the vendor software sends.
+    """
+    import select
+
+    try:
+        fd = os.open(path, os.O_RDWR | os.O_NONBLOCK)
+    except OSError as e:
+        raise ReadError(_explain(e)) from e
+
+    replies = []
+    try:
+        while select.select([fd], [], [], 0)[0]:      # drain anything stale
+            try:
+                os.read(fd, 128)
+            except OSError:
+                break
+        try:
+            os.write(fd, request)
+        except OSError as e:
+            raise ReadError(_explain(e)) from e
+
+        deadline = time.monotonic() + READ_TIMEOUT
+        last = time.monotonic()
+        while time.monotonic() < deadline:
+            if not select.select([fd], [], [], 0.05)[0]:
+                if replies and time.monotonic() - last > READ_QUIET:
+                    break
+                continue
+            try:
+                data = os.read(fd, 128)
+            except BlockingIOError:
+                continue
+            except OSError as e:
+                raise ReadError(_explain(e)) from e
+            if not data:
+                continue
+            last = time.monotonic()
+            replies.append(data)
+            if want is not None and len(replies) >= want:
+                break
+    finally:
+        os.close(fd)
+    return replies
+
+
+def read_info(device):
+    """(keys, knobs) as the pad reports them, or None if it doesn't answer."""
+    for reply in _exchange(device.path, proto.native_info(device.report_id), want=1):
+        got = decode_info(reply)
+        if got:
+            return got
+    return None
+
+
+def read_layer(device, layer=1):
+    """{control: Binding} for one layer, as it is on the pad right now."""
+    out = {}
+    for reply in _exchange(device.path, proto.native_read(device.report_id, layer)):
+        decoded = decode_config(reply)
+        if not decoded:
+            continue
+        control, got_layer, binding = decoded
+        if got_layer == layer:
+            out[control] = binding
+    return out
+
+
+def read_into_state(device, state, layer=1):
+    """Read the pad and make the shadow state match it. Returns what was read."""
+    bindings = read_layer(device, layer)
+    if not bindings:
+        raise ReadError("the pad didn't answer")
+    for control, binding in bindings.items():
+        state.record(control, binding, layer)
+    state.save()
+    return bindings
+
+
+# -------------------------------------------------------------- detection
+
+@dataclass
+class Detection:
+    """What a read-only interrogation of an unrecognised pad turned up."""
+    speaks: bool = False          # it answered our protocol properly
+    keys: int = 0
+    knobs: int = 0
+    layers: int = 0
+    bindings: dict = field(default_factory=dict)
+    why: str = ""                 # when speaks is False, what went wrong
+
+    @property
+    def layout(self):
+        """
+        A first guess at the shape. The pad reports counts, not geometry,
+        so rows and columns are a guess the owner should confirm.
+        """
+        if not self.speaks:
+            return None
+        cols = 3 if self.keys % 3 == 0 else (4 if self.keys % 4 == 0 else self.keys)
+        rows = -(-self.keys // cols)
+        return Layout(self.keys, self.knobs, rows, cols)
+
+
+def detect(device):
+    """
+    Ask an unknown pad what it is, without configuring anything.
+
+    Both commands are queries. We only conclude the pad speaks this
+    protocol if it answers both in the right shape: a well formed info
+    reply, and a layer whose controls are exactly the slots that reply
+    implies. A pad from a different family will fail one of those and we
+    leave it alone.
+    """
+    try:
+        info = read_info(device)
+    except ReadError as e:
+        return Detection(why=f"it didn't answer: {e}")
+    if not info:
+        return Detection(why="it didn't answer the 'what are you' query")
+
+    keys, knobs = info
+    if not 1 <= keys <= proto.KEY_SLOTS or not 0 <= knobs <= proto.KNOB_SLOTS:
+        return Detection(why=f"it reported {keys} keys and {knobs} knobs, "
+                             "which isn't a pad this protocol can describe")
+
+    probe = Layout(keys, knobs, rows=keys, cols=1)      # geometry irrelevant here
+    expected = set(probe.control_for_action)
+
+    try:
+        replies = _exchange(device.path, proto.native_read(device.report_id, 1))
+    except ReadError as e:
+        return Detection(why=f"it answered the first query but not the second: {e}")
+
+    seen, bindings = set(), {}
+    for reply in replies:
+        if len(reply) < 13 or reply[1] != proto.MAGIC_READ:
+            continue
+        action, layer = reply[2], reply[3]
+        seen.add(action)
+        if action not in expected or layer != 1:
+            continue
+        control = probe.control_for_action[action]
+        decoded = decode_config(reply)
+        if decoded:
+            bindings[control] = decoded[2]
+
+    missing = expected - seen
+    if missing:
+        return Detection(why=f"it described {keys} keys and {knobs} knobs but "
+                             f"didn't report {len(missing)} of them, so this "
+                             "isn't the protocol it speaks")
+
+    return Detection(speaks=True, keys=keys, knobs=knobs,
+                     layers=proto.LAYERS, bindings=bindings)
+
+
+def detection_report(device, detected):
+    """A block of text to paste into an issue when adding a new pad."""
+    lines = [f"Device: {device.vid}:{device.pid}",
+             f"Name:   {device.name or 'not reported'}",
+             f"Node:   {device.path}, report id {device.report_id}", ""]
+    if not detected.speaks:
+        lines += ["This pad did NOT answer the 1189:8840 protocol.",
+                  f"Reason: {detected.why}"]
+        return "\n".join(lines)
+    lines += [f"The pad reports {detected.keys} keys and {detected.knobs} knobs.",
+              f"It answered a layer read for every slot that implies.", "",
+              "Layer 1 as read from the pad:"]
+    for control, binding in sorted(detected.bindings.items(),
+                                   key=lambda kv: action_byte(kv[0])):
+        lines.append(f"  0x{action_byte(control):02x}  {control:<12} {binding.label()}")
+    return "\n".join(lines)

macropad_gui/padview.py

@@ -13,7 +13,7 @@ Visual states:
 import math
 from dataclasses import dataclass
 
-from PySide6.QtCore import QPointF, QRectF, Qt, Signal
+from PySide6.QtCore import QPointF, QRectF, Qt, QTimer, Signal
 from PySide6.QtGui import (QColor, QFont, QFontDatabase, QFontMetricsF,
                            QPainter, QPainterPath, QPen, QTransform)
 from PySide6.QtWidgets import QSizePolicy, QWidget
@@ -33,11 +33,6 @@ R_OUT, R_IN, R_KNOB = 54.0, 40.0, 30.0
 ROW_H, ROW_GAP = 22.0, 12.0
 DIAL_W = 160.0
 
-# Which dial is dial 1 (action bytes 0x10-0x12). Upright, dials on top:
-# assumed left. Flat (the vendor app's view): assumed top.
-DIALS_IN_ORDER = ("dial1", "dial2")
-
-
 @dataclass
 class Look:
     text: str = ""           # what's on the pad, or will be once written
@@ -46,22 +41,28 @@ class Look:
     quiet: bool = False      # known, but does nothing: legend drawn dimmed
 
 
-def _key_grid(orientation):
+def _key_grid(layout, orientation):
-    """[(key_id, row, col)] with row 0 at the top."""
+    """
-    out = []
+    [(key_id, row, col)] as drawn, with row 0 at the top.
-    if orientation == "upright":              # 4 rows x 3 cols, dials above
+
-        for r in range(4):
+    The layout describes the pad lying flat. Upright is the same pad turned
-            for c in range(3):
+    a quarter turn anticlockwise, so a key at (r, c) in an R by C grid ends
-                out.append((f"key{(3 - r) + 1 + c * 4}", r, c))
+    up at (C - 1 - c, r) in a C by R one.
-    else:                                     # 3 rows x 4 cols, dials right
+    """
-        for r in range(3):
+    flat = layout.spots()
-            for c in range(4):
+    if orientation != "upright":
-                out.append((f"key{r * 4 + c + 1}", r, c))
+        return flat
-    return out
+    return [(k, layout.cols - 1 - c, r) for k, r, c in flat]
+
+
+def _grid_size(layout, orientation):
+    return ((layout.cols, layout.rows) if orientation == "upright"
+            else (layout.rows, layout.cols))
 
 
 class PadView(QWidget):
     selected = Signal(str)
+    rearranged = Signal(object)
 
     def __init__(self, parent=None):
         super().__init__(parent)
@@ -69,9 +70,19 @@ class PadView(QWidget):
         self.setSizePolicy(QSizePolicy.Expanding, QSizePolicy.Expanding)
         self.setMinimumSize(360, 420)
         self.orientation = "upright"
-        self.looks = {c: Look() for c in core.CONTROL_IDS}
+        self.layout = core.DEFAULT_LAYOUT
+        self.device_id = core.DEVICE_ID
+        self.looks = {c: Look() for c in self.layout.control_ids}
         self.current = None
         self.hover = None
+        self.editing = False
+        self.drag = None          # key being moved
+        self.drag_at = None       # where the pointer is, in board units
+        self.drop = None          # (row, col) it would land in
+        self._wobble = 0
+        self._wobbler = QTimer(self)
+        self._wobbler.setInterval(130)
+        self._wobbler.timeout.connect(self._tick)
         self.mono = QFontDatabase.systemFont(QFontDatabase.FixedFont)
         self._layout()
 
@@ -82,6 +93,14 @@ class PadView(QWidget):
         self._layout()
         self.update()
 
+    def set_layout(self, layout):
+        """Redraw for a pad of a different shape."""
+        self.layout = layout
+        if self.current not in layout.control_ids:
+            self.current = None
+        self._layout()
+        self.update()
+
     def set_looks(self, looks):
         self.looks = looks
         self.update()
@@ -90,6 +109,21 @@ class PadView(QWidget):
         self.current = control
         self.update()
 
+    def set_editing(self, on):
+        """
+        Edit mode: keys can be dragged between cells. They wobble, which is
+        the clearest way anyone has found to say "these can be moved".
+        """
+        self.editing = on
+        self.drag = self.drop = self.drag_at = None
+        self._wobbler.start() if on else self._wobbler.stop()
+        self.setCursor(Qt.ArrowCursor)
+        self.update()
+
+    def _tick(self):
+        self._wobble = 1 - self._wobble
+        self.update()
+
     # ------------------------------------------------------------ layout
 
     def _layout(self):
@@ -97,23 +131,35 @@ class PadView(QWidget):
         self.dials = {}             # dial -> centre QPointF
         self.rows = {}              # dial action -> legend row QRectF
 
+        lay = self.layout
+        grid_rows, grid_cols = _grid_size(lay, self.orientation)
+        # from a knob's centre down to the bottom of its legend rows
+        stack = R_OUT + ROW_GAP + len(core.DIAL_ACTIONS) * ROW_H
+
         if self.orientation == "upright":
-            dial_centres = [QPointF(M + DIAL_W / 2, M + R_OUT),
+            # knobs in a row above the keys
-                            QPointF(M + DIAL_W * 1.5, M + R_OUT)]
+            span = max(grid_cols * (K + G) - G, lay.knobs * (DIAL_W + G))
-            grid_top = M + 2 * R_OUT + ROW_GAP + 3 * ROW_H + 28
+            step = span / max(lay.knobs, 1)
+            dial_centres = [QPointF(M + step * (i + 0.5), M + R_OUT)
+                            for i in range(lay.knobs)]
+            grid_top = M + (R_OUT + stack + 28 if lay.knobs else 0)
             grid_left = M
         else:
+            # knobs in a column down the right
             grid_top, grid_left = M, M
-            x = M + 4 * K + 3 * G + 30 + DIAL_W / 2
+            x = M + grid_cols * (K + G) + 30 + DIAL_W / 2
-            first = M + R_OUT
+            dial_centres = [QPointF(x, M + R_OUT + i * (stack + 20 + R_OUT))
-            second = first + R_OUT + ROW_GAP + 3 * ROW_H + 20 + R_OUT
+                            for i in range(lay.knobs)]
-            dial_centres = [QPointF(x, first), QPointF(x, second)]
 
-        for key, r, c in _key_grid(self.orientation):
+        self.cells = {}
-            self.keys[key] = QRectF(grid_left + c * (K + G),
+        for r in range(grid_rows):
+            for c in range(grid_cols):
+                self.cells[(r, c)] = QRectF(grid_left + c * (K + G),
                                             grid_top + r * (K + G), K, K)
+        for key, r, c in _key_grid(lay, self.orientation):
+            self.keys[key] = self.cells[(r, c)]
 
-        for dial, centre in zip(DIALS_IN_ORDER, dial_centres):
+        for dial, centre in zip(lay.dial_ids, dial_centres):
             self.dials[dial] = centre
             top = centre.y() + R_OUT + ROW_GAP
             for i, act in enumerate(core.DIAL_ACTIONS):
@@ -121,10 +167,9 @@ class PadView(QWidget):
                     centre.x() - DIAL_W / 2 + 6, top + i * ROW_H,
                     DIAL_W - 12, ROW_H)
 
-        right = max([r.right() for r in self.keys.values()] +
+        boxes = list(self.cells.values()) + list(self.rows.values())
-                    [r.right() for r in self.rows.values()])
+        right = max(r.right() for r in boxes)
-        bottom = max([r.bottom() for r in self.keys.values()] +
+        bottom = max(r.bottom() for r in boxes)
-                     [r.bottom() for r in self.rows.values()])
         self.board = QRectF(0, 0, right + M, bottom + 46)
 
     def _transform(self):
@@ -172,7 +217,27 @@ class PadView(QWidget):
                     return f"{dial}-{act}"
         return None
 
+    def _board_pos(self, pos):
+        inv, ok = self._transform().inverted()
+        return inv.map(QPointF(pos)) if ok else None
+
+    def _cell_at(self, point):
+        for spot, rect in self.cells.items():
+            if rect.adjusted(-G / 2, -G / 2, G / 2, G / 2).contains(point):
+                return spot
+        return None
+
     def mouseMoveEvent(self, e):
+        if self.editing:
+            if self.drag:
+                self.drag_at = self._board_pos(e.position())
+                self.drop = self._cell_at(self.drag_at) if self.drag_at else None
+                self.update()
+            else:
+                over = self._hit(e.position())
+                key = over if over and over.startswith("key") else None
+                self.setCursor(Qt.OpenHandCursor if key else Qt.ArrowCursor)
+            return
         h = self._hit(e.position())
         if h != self.hover:
             self.hover = h
@@ -184,12 +249,52 @@ class PadView(QWidget):
         self.update()
 
     def mousePressEvent(self, e):
+        if e.button() != Qt.LeftButton:
+            return
         h = self._hit(e.position())
-        if h and e.button() == Qt.LeftButton:
+        if self.editing:
+            if h and h.startswith("key"):
+                self.drag = h
+                self.drag_at = self._board_pos(e.position())
+                self.setCursor(Qt.ClosedHandCursor)
+                self.update()
+            return
+        if h:
             self.current = h
             self.selected.emit(h)
             self.update()
 
+    def mouseReleaseEvent(self, e):
+        if not (self.editing and self.drag):
+            return
+        moved, target = self.drag, self.drop
+        self.drag = self.drag_at = self.drop = None
+        self.setCursor(Qt.OpenHandCursor)
+        if target:
+            self._place(moved, target)
+        self.update()
+
+    def _place(self, key, target):
+        """
+        Drop a key into a cell. If another key is already there the two
+        swap, so a drag never loses one.
+        """
+        spots = {k: (r, c) for k, r, c in _key_grid(self.layout, self.orientation)}
+        if spots.get(key) == target:
+            return
+        for other, spot in spots.items():
+            if spot == target:
+                spots[other] = spots[key]
+                break
+        spots[key] = target
+
+        if self.orientation == "upright":       # back to the flat arrangement
+            spots = {k: (c, self.layout.cols - 1 - r) for k, (r, c) in spots.items()}
+        places = [spots[f"key{i + 1}"] for i in range(self.layout.keys)]
+        self.layout = self.layout.rearranged(places)
+        self._layout()
+        self.rearranged.emit(self.layout)
+
     # ------------------------------------------------------------ paint
 
     def paintEvent(self, _):
@@ -204,17 +309,54 @@ class PadView(QWidget):
         p.setPen(QPen(INK_DIM, 1.2))
         p.drawPath(board)
 
+        if self.editing:
+            for spot, rect in self.cells.items():
+                if spot == self.drop or not any(r is rect for r in self.keys.values()):
+                    path = QPainterPath()
+                    path.addRoundedRect(rect, 5, 5)
+                    pen = QPen(TAPE if spot == self.drop else HATCH, 1.4)
+                    pen.setStyle(Qt.DashLine)
+                    p.setPen(pen)
+                    p.setBrush(Qt.NoBrush)
+                    p.drawPath(path)
+
         for key, rect in self.keys.items():
+            if key == self.drag:
+                continue
+            p.save()
+            if self.editing:
+                # a small alternating tilt, the way icons shake when movable
+                lean = 0.55 if (hash(key) + self._wobble) % 2 else -0.55
+                p.translate(rect.center())
+                p.rotate(lean)
+                p.translate(-rect.center())
             self._paint_key(p, key, rect)
+            p.restore()
+
         for dial, centre in self.dials.items():
             self._paint_dial(p, dial, centre)
 
+        if self.drag and self.drag_at:
+            rect = QRectF(0, 0, K, K)
+            rect.moveCenter(self.drag_at)
+            p.save()
+            p.setOpacity(0.92)
+            shadow = QPainterPath()
+            shadow.addRoundedRect(rect.adjusted(2, 4, 2, 4), 5, 5)
+            p.fillPath(shadow, QColor(0, 0, 0, 90))
+            p.fillPath(shadow, QColor(0, 0, 0, 60))
+            fill = QPainterPath()
+            fill.addRoundedRect(rect, 5, 5)
+            p.fillPath(fill, BOARD)
+            self._paint_key(p, self.drag, rect)
+            p.restore()
+
         f = QFont(self.mono)
         f.setPixelSize(10)
         p.setFont(f)
         p.setPen(INK_DIM)
         p.drawText(QRectF(M, self.board.bottom() - 34, 200, 16),
-                   Qt.AlignLeft | Qt.AlignVCenter, core.DEVICE_ID)
+                   Qt.AlignLeft | Qt.AlignVCenter, self.device_id)
 
     def _outline_pen(self, control, known):
         if control == self.current:

tests/captures/1189-8840-read.txt

@@ -0,0 +1,160 @@
+# capture of Bus 001 Device 005: ID 1189:8840 Acer Communications & Multimedia USB Composite Device
+# the vendor software's "view settings" button, reading all three layers.
+# Trailing bytes of the two outgoing requests are uninitialised stack memory
+# from the Windows program, not part of the protocol.
+# bus 1 device 5
+
+[1] interrupt out ep=4
+    03fbfbfb3200488f2f0218c82102b25676003cd03200b2834c005002f201488f
+[2] interrupt in ep=4
+    03fb0c0200000000000000000000000000000000000000000000000000000000
+[3] interrupt out ep=4
+    03fa0f03010230cc3200f0cc32007cf20269000000004d004402f0cc32005d01
+[4] interrupt in ep=4
+    03fa010101000000000000000000000000000000000000000000000000000000
+[5] interrupt in ep=4
+    03fa020101000000000001051700000000000000000000000000000000000000
+[6] interrupt in ep=4
+    03fa030102000000000001b60000000000000000000000000000000000000000
+[7] interrupt in ep=4
+    03fa040101000000000000000000000000000000000000000000000000000000
+[8] interrupt in ep=4
+    03fa050101000000000000000000000000000000000000000000000000000000
+[9] interrupt in ep=4
+    03fa0601010000000000010f4500000000000000000000000000000000000000
+[10] interrupt in ep=4
+    03fa070102000000000001cd0000000000000000000000000000000000000000
+[11] interrupt in ep=4
+    03fa080101000000000000000000000000000000000000000000000000000000
+[12] interrupt in ep=4
+    03fa090101000000000000000000000000000000000000000000000000000000
+[13] interrupt in ep=4
+    03fa0a0101000000000000000000000000000000000000000000000000000000
+[14] interrupt in ep=4
+    03fa0b0102000000000001b50000000000000000000000000000000000000000
+[15] interrupt in ep=4
+    03fa0c0101000000000000000000000000000000000000000000000000000000
+[16] interrupt in ep=4
+    03fa0d01000000000000020059005b0000000000000000000000000000000000
+[17] interrupt in ep=4
+    03fa0e01000000000000020059005c0000000000000000000000000000000000
+[18] interrupt in ep=4
+    03fa0f01010000000000020059005d0000000000000000000000000000000000
+[19] interrupt in ep=4
+    03fa100102000000000001ea0000000000000000000000000000000000000000
+[20] interrupt in ep=4
+    03fa110102000000000001e20000000000000000000000000000000000000000
+[21] interrupt in ep=4
+    03fa120102000000000001e90000000000000000000000000000000000000000
+[22] interrupt in ep=4
+    03fa130102000000000001700000000000000000000000000000000000000000
+[23] interrupt in ep=4
+    03fa140102000000000001920100000000000000000000000000000000000000
+[24] interrupt in ep=4
+    03fa1501020000000000016f0000000000000000000000000000000000000000
+[25] interrupt in ep=4
+    03fa180101000000000001002400000000000000000000000000000000000000
+[26] interrupt in ep=4
+    03fa170101000000000001005200000000000000000000000000000000000000
+[27] interrupt in ep=4
+    03fa160101000000000001002600000000000000000000000000000000000000
+[28] interrupt out ep=4
+    03fa0f03020230cc3200f0cc32007cf20269000000004d004402f0cc32005d01
+[29] interrupt in ep=4
+    03fa010201000000000001000400000000000000000000000000000000000000
+[30] interrupt in ep=4
+    03fa020201000000000001000500000000000000000000000000000000000000
+[31] interrupt in ep=4
+    03fa030201000000000001000600000000000000000000000000000000000000
+[32] interrupt in ep=4
+    03fa040201000000000001000700000000000000000000000000000000000000
+[33] interrupt in ep=4
+    03fa050201000000000001000800000000000000000000000000000000000000
+[34] interrupt in ep=4
+    03fa060201000000000001000900000000000000000000000000000000000000
+[35] interrupt in ep=4
+    03fa070201000000000001000a00000000000000000000000000000000000000
+[36] interrupt in ep=4
+    03fa080201000000000001000b00000000000000000000000000000000000000
+[37] interrupt in ep=4
+    03fa090201000000000001000c00000000000000000000000000000000000000
+[38] interrupt in ep=4
+    03fa0a0201000000000001000d00000000000000000000000000000000000000
+[39] interrupt in ep=4
+    03fa0b0201000000000001000e00000000000000000000000000000000000000
+[40] interrupt in ep=4
+    03fa0c0201000000000001000f00000000000000000000000000000000000000
+[41] interrupt in ep=4
+    03fa0d0201000000000001001000000000000000000000000000000000000000
+[42] interrupt in ep=4
+    03fa0e0201000000000001001100000000000000000000000000000000000000
+[43] interrupt in ep=4
+    03fa0f0201000000000001001200000000000000000000000000000000000000
+[44] interrupt in ep=4
+    03fa120201000000000001001e00000000000000000000000000000000000000
+[45] interrupt in ep=4
+    03fa110201000000000001001f00000000000000000000000000000000000000
+[46] interrupt in ep=4
+    03fa100201000000000001002000000000000000000000000000000000000000
+[47] interrupt in ep=4
+    03fa150201000000000001002100000000000000000000000000000000000000
+[48] interrupt in ep=4
+    03fa140201000000000001002200000000000000000000000000000000000000
+[49] interrupt in ep=4
+    03fa130201000000000001002300000000000000000000000000000000000000
+[50] interrupt in ep=4
+    03fa180201000000000001002400000000000000000000000000000000000000
+[51] interrupt in ep=4
+    03fa170201000000000001002500000000000000000000000000000000000000
+[52] interrupt in ep=4
+    03fa160201000000000001002600000000000000000000000000000000000000
+[53] interrupt out ep=4
+    03fa0f03030230cc3200f0cc32007cf20269000000004d004402f0cc32005d01
+[54] interrupt in ep=4
+    03fa010301000000000001000400000000000000000000000000000000000000
+[55] interrupt in ep=4
+    03fa020301000000000001000500000000000000000000000000000000000000
+[56] interrupt in ep=4
+    03fa030301000000000001000600000000000000000000000000000000000000
+[57] interrupt in ep=4
+    03fa040301000000000001000700000000000000000000000000000000000000
+[58] interrupt in ep=4
+    03fa050301000000000001000800000000000000000000000000000000000000
+[59] interrupt in ep=4
+    03fa060301000000000001000900000000000000000000000000000000000000
+[60] interrupt in ep=4
+    03fa070301000000000001000a00000000000000000000000000000000000000
+[61] interrupt in ep=4
+    03fa080301000000000001000b00000000000000000000000000000000000000
+[62] interrupt in ep=4
+    03fa090301000000000001000c00000000000000000000000000000000000000
+[63] interrupt in ep=4
+    03fa0a0301000000000001000d00000000000000000000000000000000000000
+[64] interrupt in ep=4
+    03fa0b0301000000000001000e00000000000000000000000000000000000000
+[65] interrupt in ep=4
+    03fa0c0301000000000001000f00000000000000000000000000000000000000
+[66] interrupt in ep=4
+    03fa0d0301000000000001001000000000000000000000000000000000000000
+[67] interrupt in ep=4
+    03fa0e0301000000000001001100000000000000000000000000000000000000
+[68] interrupt in ep=4
+    03fa0f0301000000000001001200000000000000000000000000000000000000
+[69] interrupt in ep=4
+    03fa100302000000000001700000000000000000000000000000000000000000
+[70] interrupt in ep=4
+    03fa110301000000000001001f00000000000000000000000000000000000000
+[71] interrupt in ep=4
+    03fa1203020000000000016f0000000000000000000000000000000000000000
+[72] interrupt in ep=4
+    03fa150301000000000001002100000000000000000000000000000000000000
+[73] interrupt in ep=4
+    03fa140301000000000001002200000000000000000000000000000000000000
+[74] interrupt in ep=4
+    03fa130301000000000001002300000000000000000000000000000000000000
+[75] interrupt in ep=4
+    03fa180301000000000001002400000000000000000000000000000000000000
+[76] interrupt in ep=4
+    03fa170301000000000001002500000000000000000000000000000000000000
+[77] interrupt in ep=4
+    03fa160301000000000001002600000000000000000000000000000000000000

tests/test_core.py

@@ -77,6 +77,8 @@ class EveryCapture(unittest.TestCase):
         seen = 0
         for path in sorted(CAPTURES.glob("*.txt")):
             for report in core.read_capture(path):
+                if report[1] in (proto.MAGIC_INFO, proto.MAGIC_READ):
+                    continue                      # a query, not a binding
                 if core.is_commit(report):
                     self.assertEqual(proto.native_commit(3)[:len(report)], report)
                     continue
@@ -105,6 +107,22 @@ class EveryCapture(unittest.TestCase):
             self.assertEqual(state.get("dial1-right").binding, core.Binding(keys="3"))
             self.assertTrue(state.get("dial2-left").unknown)
 
+    def test_capture_directions_are_separate(self):
+        """A reply from the pad must never be mistaken for a binding."""
+        with tempfile.TemporaryDirectory() as d:
+            cap = Path(d) / "both.txt"
+            cap.write_text(
+                "[1] interrupt out ep=4\n    03fd010101000000000001000400\n"
+                "[2] interrupt in ep=3\n    03fd020101000000000001001500\n"
+                "[3] interrupt out ep=4\n    03fdfeff00000000\n")
+            self.assertEqual(len(core.read_capture(cap)), 2)
+            self.assertEqual(len(core.read_capture(cap, incoming=True)), 1)
+
+            state = core.State(Path(d) / "s.json")
+            got = core.import_capture(cap, state)
+            self.assertEqual([(c, b.keys) for c, _, b in got], [("key1", "a")])
+            self.assertTrue(state.get("key2").unknown)
+
     def test_uncommitted_configs_are_not_imported(self):
         with tempfile.TemporaryDirectory() as d:
             cap = Path(d) / "c.txt"
@@ -115,6 +133,281 @@ class EveryCapture(unittest.TestCase):
             self.assertTrue(state.get("key1").unknown)
 
 
+class ReadingThePad(unittest.TestCase):
+    """The vendor software's "view settings", captured and decoded."""
+
+    def setUp(self):
+        self.cap = CAPTURES / "1189-8840-read.txt"
+        self.sent = core.read_capture(self.cap)
+        self.got = core.read_capture(self.cap, incoming=True)
+
+    def test_our_requests_match_the_vendors(self):
+        info, read1, read2, read3 = self.sent
+        self.assertEqual(proto.native_info(3)[:2], info[:2])
+        for layer, sent in ((1, read1), (2, read2), (3, read3)):
+            # the vendor's trailing bytes are uninitialised memory; ours are zero
+            self.assertEqual(proto.native_read(3, layer)[:6], sent[:6])
+
+    def test_info_reply(self):
+        self.assertEqual(core.decode_info(self.got[0]), (12, 2))
+        self.assertEqual(len(core.KEYS), 12)
+        self.assertEqual(len(core.DIALS), 2)
+
+    def test_layer_1_is_what_the_pad_holds(self):
+        layer1 = {}
+        for r in self.got:
+            d = core.decode_config(r)
+            if d and d[1] == 1:
+                layer1[d[0]] = d[2]
+        # dials, which is the mapping we could never confirm by writing alone
+        self.assertEqual(layer1["dial1-left"], core.Binding(media="volumedown"))
+        self.assertEqual(layer1["dial1-push"], core.Binding(media="mute"))
+        self.assertEqual(layer1["dial1-right"], core.Binding(media="volumeup"))
+        self.assertEqual(layer1["dial2-left"], core.Binding(media="brightnessdown"))
+        self.assertEqual(layer1["dial2-push"], core.Binding(media="calculator"))
+        self.assertEqual(layer1["dial2-right"], core.Binding(media="brightnessup"))
+        # keys, including modifiers and an empty one
+        self.assertEqual(layer1["key2"], core.Binding(keys="ctrl+alt+t"))
+        self.assertEqual(layer1["key6"], core.Binding(keys="ctrl+shift+alt+super+f12"))
+        self.assertEqual(layer1["key3"], core.Binding(media="prev"))
+        self.assertTrue(layer1["key1"].none)
+
+    def test_a_read_binding_re_encodes_to_the_same_write(self):
+        """Whatever we read, writing it back must produce the same meaning."""
+        for r in self.got:
+            d = core.decode_config(r)
+            if not d:
+                continue
+            control, layer, binding = d
+            again = core.decode_config(
+                binding.reports(3, core.action_byte(control), layer)[0])
+            self.assertIsNotNone(again, binding)
+            self.assertEqual(again[0], control)
+            self.assertEqual(again[2], binding)
+
+    def test_the_pad_stores_two_key_sequences(self):
+        """
+        Slots 0x0d to 0x0f hold two keystrokes each, in the layout we had
+        assumed for sequences: count at byte 10, then (modifier, keycode)
+        pairs. This is factory data for a 15 key sibling rather than
+        something we watched being written, so it supports the encoding
+        without proving the pad accepts it from us.
+        """
+        multi = [r for r in self.got
+                 if r[1] == proto.MAGIC_READ and r[10] == 2 and r[4] != 2]
+        self.assertTrue(multi)
+        r = multi[0]
+        self.assertEqual((r[11], r[13]), (0, 0))         # no modifiers
+        self.assertTrue(r[12] and r[14])                  # two real keycodes
+
+    def test_replies_cover_the_firmware_not_the_hardware(self):
+        """
+        The pad answers for 24 slots per layer: 15 keys and 3 knobs, which
+        is the most this firmware supports. Ours has 12 keys and 2 knobs,
+        so 6 of those slots are for hardware that isn't there. Those decode
+        to no control, which is what we want: the app must not offer them.
+        """
+        configs = [r for r in self.got if r[1] == proto.MAGIC_READ]
+        self.assertEqual(len(configs), 24 * 3)
+
+        real = [r for r in configs if core.decode_config(r)]
+        self.assertEqual(len(real), len(core.CONTROL_IDS) * 3)
+
+        phantom = {r[2] for r in configs if not core.decode_config(r)}
+        self.assertEqual(phantom, {0x0d, 0x0e, 0x0f,       # keys 13 to 15
+                                   0x16, 0x17, 0x18})      # a third knob
+
+    def test_reading_fills_in_the_state(self):
+        from unittest import mock
+        replies = [r for r in self.got if r[1] == proto.MAGIC_READ and r[3] == 1]
+        dev = core.Device(path="/dev/null-pad", report_id=3, writable=True)
+        with tempfile.TemporaryDirectory() as d:
+            state = core.State(Path(d) / "s.json")
+            with mock.patch.object(core, "_exchange", return_value=replies):
+                got = core.read_into_state(dev, state)
+            self.assertEqual(len(got), len(core.CONTROL_IDS))
+            fresh = core.State(Path(d) / "s.json")
+            self.assertFalse(any(fresh.get(c).unknown for c in core.CONTROL_IDS))
+            self.assertEqual(fresh.get("dial2-push").binding,
+                             core.Binding(media="calculator"))
+
+
+class Detection(unittest.TestCase):
+    """
+    Detecting an unknown pad, using the real capture as the stand-in for a
+    pad that speaks this protocol, and mangled versions for ones that don't.
+    """
+
+    def setUp(self):
+        self.dev = core.Device(path="/dev/null-pad", report_id=3, writable=True,
+                               name="Some Other Pad", vid="1189", pid="8890")
+        got = core.read_capture(CAPTURES / "1189-8840-read.txt", incoming=True)
+        self.info = [r for r in got if r[1] == proto.MAGIC_INFO]
+        self.layer1 = [r for r in got if r[1] == proto.MAGIC_READ and r[3] == 1]
+
+    def run_detect(self, info=None, layer=None):
+        from unittest import mock
+        calls = [info if info is not None else self.info,
+                 layer if layer is not None else self.layer1]
+        with mock.patch.object(core, "_exchange", side_effect=calls):
+            return core.detect(self.dev)
+
+    def test_a_pad_that_speaks_is_recognised(self):
+        d = self.run_detect()
+        self.assertTrue(d.speaks, d.why)
+        self.assertEqual((d.keys, d.knobs), (12, 2))
+        self.assertEqual(d.layout, core.Layout(12, 2, 4, 3))
+        self.assertEqual(d.bindings["dial1-left"], core.Binding(media="volumedown"))
+
+    def test_silence_is_not_taken_as_yes(self):
+        d = self.run_detect(info=[])
+        self.assertFalse(d.speaks)
+        self.assertIn("didn't answer", d.why)
+
+    def test_a_reply_that_is_not_ours_is_rejected(self):
+        d = self.run_detect(info=[bytes.fromhex("03aa5501") + bytes(60)])
+        self.assertFalse(d.speaks)
+
+    def test_nonsense_counts_are_rejected(self):
+        bad = bytearray(self.info[0])
+        bad[2] = 99                                  # 99 keys
+        d = self.run_detect(info=[bytes(bad)])
+        self.assertFalse(d.speaks)
+        self.assertIn("99 keys", d.why)
+
+    def test_a_pad_that_answers_one_query_but_not_the_other(self):
+        """The dangerous case: sounds right, then doesn't describe itself."""
+        d = self.run_detect(layer=self.layer1[:4])
+        self.assertFalse(d.speaks)
+        self.assertIn("didn't report", d.why)
+
+    def test_a_six_key_pad(self):
+        info = bytearray(self.info[0])
+        info[2], info[3] = 6, 1
+        keep = {1, 2, 3, 4, 5, 6, 0x10, 0x11, 0x12}
+        layer = [r for r in self.layer1 if r[2] in keep]
+        d = self.run_detect(info=[bytes(info)], layer=layer)
+        self.assertTrue(d.speaks, d.why)
+        self.assertEqual(d.layout, core.Layout(6, 1, 2, 3))
+        self.assertEqual(len(d.layout.control_ids), 9)
+
+    def test_report_is_pasteable(self):
+        text = core.detection_report(self.dev, self.run_detect())
+        self.assertIn("1189:8890", text)
+        self.assertIn("0x10  dial1-left", text)
+        refused = core.detection_report(self.dev, self.run_detect(info=[]))
+        self.assertIn("did NOT answer", refused)
+
+
+class LayoutIsRemembered(unittest.TestCase):
+
+    def test_state_keeps_the_pad_shape(self):
+        with tempfile.TemporaryDirectory() as d:
+            path = Path(d) / "s.json"
+            lay = core.Layout(9, 2, 3, 3, "nine")
+            st = core.State(path, layout=lay)
+            st.record("key9", core.Binding(keys="f5"))
+            st.save()
+            again = core.State(path)
+            self.assertEqual(again.layout, lay)
+            self.assertEqual(len(again.layout.control_ids), 15)
+            self.assertEqual(again.get("key9").binding, core.Binding(keys="f5"))
+
+    def test_an_old_state_file_still_loads(self):
+        with tempfile.TemporaryDirectory() as d:
+            path = Path(d) / "s.json"
+            path.write_text(json.dumps({
+                "format": core.FORMAT, "device": core.DEVICE_ID,
+                "layers": {"1": {"key1": {"binding": {"keys": "a"},
+                                          "written": "2026-01-01T00:00:00"}}}}))
+            st = core.State(path)
+            self.assertEqual(st.layout, core.DEFAULT_LAYOUT)
+            self.assertEqual(st.get("key1").binding, core.Binding(keys="a"))
+
+
+class Arranging(unittest.TestCase):
+    """Moving keys around the grid, and keeping it consistent when rotated."""
+
+    def setUp(self):
+        self.lay = core.DEFAULT_LAYOUT
+
+    def test_the_default_matches_the_printed_pad(self):
+        flat = {k: (r, c) for k, r, c in self.lay.spots()}
+        self.assertEqual(flat["key1"], (0, 0))       # top left, reading across
+        self.assertEqual(flat["key4"], (0, 3))
+        self.assertEqual(flat["key12"], (2, 3))
+
+    def test_upright_is_the_same_pad_turned(self):
+        from macropad_gui import padview
+        up = {k: (r, c) for k, r, c in padview._key_grid(self.lay, "upright")}
+        self.assertEqual(up["key1"], (3, 0))         # bottom left when stood up
+        self.assertEqual(up["key4"], (0, 0))
+        self.assertEqual(up["key12"], (0, 2))
+        self.assertEqual(len(set(up.values())), self.lay.keys)
+
+    def test_swapping_two_keys(self):
+        places = list(self.lay.spots())
+        moved = self.lay.rearranged(
+            [(2, 3) if k == "key1" else (0, 0) if k == "key12" else (r, c)
+             for k, r, c in places])
+        flat = {k: (r, c) for k, r, c in moved.spots()}
+        self.assertEqual(flat["key1"], (2, 3))
+        self.assertEqual(flat["key12"], (0, 0))
+        self.assertEqual(len(set(flat.values())), 12)
+
+    def test_resizing_keeps_what_still_fits(self):
+        moved = self.lay.rearranged(
+            [(2, 3) if k == "key1" else (0, 0) if k == "key12" else (r, c)
+             for k, r, c in self.lay.spots()])
+        self.assertEqual(moved.resized(cols=6).cols, 6)
+        smaller = moved.resized(rows=2, cols=2, keys=4)
+        self.assertEqual(len(smaller.spots()), 4)
+        for _, r, c in smaller.spots():
+            self.assertLess(r, 2)
+            self.assertLess(c, 2)
+
+    def test_a_grid_too_small_grows_instead_of_failing(self):
+        grown = self.lay.resized(rows=1, cols=3)
+        self.assertEqual((grown.rows, grown.cols), (4, 3))
+        self.assertEqual(len(grown.spots()), 12)
+
+    def test_layout_survives_a_save(self):
+        with tempfile.TemporaryDirectory() as d:
+            path = Path(d) / "s.json"
+            moved = self.lay.rearranged(
+                [(2, 3) if k == "key1" else (0, 0) if k == "key12" else (r, c)
+                 for k, r, c in self.lay.spots()])
+            st = core.State(path, layout=moved)
+            st.save()
+            self.assertEqual(core.State(path).layout, moved)
+
+
+class Layouts(unittest.TestCase):
+
+    def test_action_bytes_agree_with_the_protocol_module(self):
+        for c in core.DEFAULT_LAYOUT.control_ids:
+            self.assertEqual(core.action_byte(c), proto.CONTROLS[c], c)
+
+    def test_layouts_of_other_shapes(self):
+        three = core.Layout(3, 1, 1, 3)
+        self.assertEqual(three.control_ids,
+                         ("key1", "key2", "key3",
+                          "dial1-left", "dial1-push", "dial1-right"))
+        self.assertEqual(core.action_byte("dial1-left"), 0x10)
+        fifteen = core.Layout(15, 3, 5, 3)
+        self.assertEqual(core.action_byte("dial3-right"), 0x18)
+        self.assertEqual(len(fifteen.control_ids), 24)
+
+    def test_impossible_layouts_are_refused(self):
+        for bad in ((0, 2, 4, 3), (99, 2, 40, 3), (12, 9, 4, 3), (12, 2, 2, 3)):
+            with self.assertRaises(ValueError, msg=bad):
+                core.Layout(*bad)
+
+    def test_round_trip(self):
+        l = core.Layout(9, 2, 3, 3, "nine")
+        self.assertEqual(core.Layout.from_json(l.to_json()), l)
+
+
 class BindingModel(unittest.TestCase):
 
     def test_round_trip(self):
@@ -244,8 +537,9 @@ class PermissionDiagnosis(unittest.TestCase):
 
     def diag(self, rules, tags):
         from unittest import mock
-        dev = [{"path": "/dev/hidraw10", "pid": "8840", "report_id": 3,
+        dev = [{"path": "/dev/hidraw10", "vid": "1189", "pid": "8840",
-                "writable": False, "name": "USB Composite Device"}]
+                "report_id": 3, "writable": False,
+                "name": "USB Composite Device"}]
         with mock.patch.object(proto, "find_devices", return_value=dev), \
              mock.patch.object(core, "_matching_rules", return_value=rules), \
              mock.patch.object(core, "_udev_tags", return_value=tags), \
@@ -278,7 +572,9 @@ class PermissionDiagnosis(unittest.TestCase):
             d = core.diagnose()
         self.assertEqual(d.status, "unsupported")
         self.assertIn("1189:8890", d.headline)
-        self.assertIsNone(d.device)
+        # the device comes with it now, so the app can offer to interrogate it
+        self.assertEqual(d.device.path, "/dev/hidraw4")
+        self.assertEqual((d.device.vid, d.device.pid), ("1189", "8890"))
 
     def test_rule_fine_but_not_local_session(self):
         d = self.diag(["/etc/udev/rules.d/60-macropad.rules"], {"uaccess"})
@@ -306,6 +602,30 @@ class GuiSmoke(unittest.TestCase):
                 w.pad.select(c)
                 w._refresh()
                 self.assertEqual(w.inspector.title.text(), core.control_name(c))
+            # the drawing must cope with any shape the protocol allows
+            for lay in (core.Layout(3, 1, 1, 3), core.Layout(15, 3, 5, 3),
+                        core.Layout(4, 0, 1, 4)):
+                w.pad.set_layout(lay)
+                for orientation in ("upright", "flat"):
+                    w.pad.set_orientation(orientation)
+                    w.pad.grab()
+                    for c in lay.control_ids:
+                        self.assertIsNotNone(core.action_byte(c))
+            # dragging a key in edit mode swaps it with whatever is there
+            w.pad.set_layout(core.DEFAULT_LAYOUT)
+            w._edit_layout(True)
+            from macropad_gui import padview
+            before = {k: (r, c) for k, r, c
+                      in padview._key_grid(w.pad.layout, w.pad.orientation)}
+            w.pad.drag = "key1"
+            w.pad._place("key1", before["key12"])
+            after = {k: (r, c) for k, r, c
+                     in padview._key_grid(w.pad.layout, w.pad.orientation)}
+            self.assertEqual(after["key1"], before["key12"])
+            self.assertEqual(after["key12"], before["key1"])
+            w._edit_layout(False, keep=False)
+            self.assertEqual(w.state.layout, core.DEFAULT_LAYOUT)
+            self.assertFalse(w.pad.editing)
             w._changed("key1", core.Binding(media="mute"))
             self.assertEqual(w._pending(), ["key1"])
             # choosing Nothing marks the control as changed straight away