Cursorless

A spoken language for rapid, structurally aware voice coding

[Cursorless @GoCursorless]

Check out the new Cursorless tutorial series! https://t.co/w9Rm0R2xqD

[Changelog @changelog]

If you've never seen/heard @PokeyRule's @GoCursorless project in action... 🤯👇

[Cursorless @GoCursorless]

@eyewarebeam @PokeyRule @lunixbochs https://t.co/xSXyDaGshR

[Cursorless @GoCursorless]

Re https://t.co/xAeIoqsNqa: I can provide you a bit more info on what Cursorless / eye tracking integration looks like, though. There are two kinds of integration: passive and active.

[Cursorless @GoCursorless]

Cursorless itself is primarily 1) a language, and 2) a text manipulation engine implementing the semantics of that language. Tasks like communicating with an eye tracker, preforming speech recognition, etc, are things we leave to Talon.

[Cursorless @GoCursorless]

Re https://t.co/xAeIoqsNqa: I can provide you a bit more info on what Cursorless / eye tracking integration looks like, though. There are two kinds of integration: passive and active.

[Beam Eye Tracker @beameyetracker]

@PokeyRule @GoCursorless @lunixbochs I see, thanks. Cursorless users figure out their multimodal approach individually. @lunixbochs curious to get your feedback on Beam's performance compared to your existing eye tracking solution.

[Cursorless @GoCursorless]

Note that in both of the active integrations, we're relying on Talon to give us gaze information at the point that you say a word. We are happy to rely on Talon to provide that abstraction rather than trying to build it ourselves.

[Cursorless @GoCursorless]

@eyewarebeam @PokeyRule @lunixbochs Cursorless itself is primarily 1) a language, and 2) a text manipulation engine implementing the semantics of that language. Tasks like communicating with an eye tracker, preforming speech recognition, etc, are things we leave to Talon.

[Cursorless @GoCursorless]

@eyewarebeam @PokeyRule @lunixbochs Note that in both of the active integrations, we're relying on Talon to give us gaze information at the point that you say a word. We are happy to rely on Talon to provide that abstraction rather than trying to build it ourselves.

[Cursorless @GoCursorless]

@eyewarebeam @PokeyRule @lunixbochs Another example of an active integration would be relying on gaze to allow us to reuse good hats. If two hats are not too close to each other, then it's ok if they're the same hat, because we can disambiguate based on which you're looking at.

[Cursorless @GoCursorless]

@eyewarebeam @PokeyRule @lunixbochs For example, we could map a mark called "here", which refers to the place you're looking when you say that word. So then you could say "swap funk here with here" to swap two functions, making sure you look at the right function at the moment you say each "here"

[Cursorless @GoCursorless]

@eyewarebeam @PokeyRule @lunixbochs An active integration is one that requires effort on our part to build. In general this would involve getting word-level timing from Talon so that we know where the user is looking at the moment they say a particular word.

[Cursorless @GoCursorless]

@eyewarebeam @PokeyRule @lunixbochs So when he looks and pops, then all the hats he needs in that area are the good, one-syllable ones. In addition, he can refer to scopes like "funk", "class", etc without needing to mention a hat, because they are implicitly referring to the ones containing his cursor.

[Cursorless @GoCursorless]

@eyewarebeam @PokeyRule @lunixbochs He will commonly look somewhere, then pop to move his cursor. He has Cursorless configured with the "greedy" hat allocation setting, which means that it tries really hard to give the best hats to those near the cursor.

[Cursorless @GoCursorless]

@eyewarebeam @PokeyRule @lunixbochs Passive integrations are ones where a user benefits from using an eye tracker alongside Cursorless without any formal integration on our part. For example, one of our core maintainers uses an eye tracker, and makes a popping sound to click.

[Cursorless @GoCursorless]

@eyewarebeam @PokeyRule @lunixbochs @eyewarebeam I’m not sure I’d say they figure out their multimodal approach individually. Cursorless is built on top of talon; anything multimodal will be handled by that layer of the stack