I believe touch screens are fundamentally a bad interface for productivity. Consider the range of actions provided by a mouse: You can hover without clicking, you can left click, or you can right click, all with nearly pixel-level precision. Add in a keyboard and your options expand even further.
A smartphone is like a computer with a one-button mouse and an abnormally large, irregularly shaped cursor, where can never be sure which part of the cursor indicates your actual click target. Software on this computer is not aware of the cursor's location until after the mouse has been clicked, and portions of the screen are blacked out when you move the mouse to certain positions. Your keyboard only works when you bring up an on-screen overlay which takes up ~35% of your screen real-estate, on a monitor which is abnormally small to begin with.
Could any amount of well-designed software make text entry efficient on this machine?
This is a hardware problem, not a software problem.
It does seem like a missed opportunity to have taken the keyboard/mouse approach and then transferred it to touch devices. Even the keyboard layout has no real advantage for two thumb typing on a screen.
Approaches that adapt the interface whilst leaning heavily on letter based inference could be interesting for one handed / single digit entry of letters. Something like dasher: https://www.inference.org.uk/dasher/dashersummary.html
It's interesting to me that Google had Tablet Tuesdays. One of the things I've said many times is that it's obvious when a company actually does the thing they're pressing users to do. Google obviously uses gmail, and just as clearly never used any of the social products they released. But I think Tablet Tuesdays doesn't accomplish the thing they hoped it would: if you can use a regular computer 4 days out of 5 -- or maybe 5 out of 6 :-( then you can limp along on that one day and not have enough incentive to actually solve the problems. "Tablet Tuesdays" should have been "Tablet Teams" -- whole groups of people forced to use nothing but a tablet, with no way to hide from the problems that caused.
Eloquent seems like an excellent existence proof that better is possible. Personally, I would try multi-touch gestures to solve some of the problems. It might be (okay, likely is) too complex a solution, but Apple (at least) can detect up to ten(?) separate touchpoints. That would be absurd, but I'd be curious to try copy/paste with multi-touch shortcuts. And it seems that selection might benefit from multi-touch as opposed to the (admittedly clever) pressure hack Eloquent is using.
Once I have to scroll (vertically or, even worse, horizontally) in addition to moving the selection handles the UX is just terrible.
One small thing: I don't know how it works on Android, but I used to have an N9 and then the Jolla phone and you could tap anywhere in text to place the cursor there.
iOS doesn't let you, except confusingly on the very first tap that activates the cursor. For subsequent taps, you can tap exactly where you want in the middle of a word and it always snaps the cursor to the start or end of that word.
I'm pretty good at aiming at the right character to edit even with big fingers on a small screen. Let me do it!
On blackberry, a mistake was one wrong character. On screen keyboards with swipe and autocorrect, a mistake can be inserting 1 or 2 random words
Screen keyboard doesn't work in the rain
while we're griping:
- on an older android device the built-in keyboard is such a pig that it sometimes requires you to slow down to like 1 character per second. Note that this worked fine on a nexus 5 with aosp a million years ago, so it's not like it's not a solved problem
- Swipe keyboard is in theory good, but the keyboard can't switch from swipe <-> tap smoothly enough and usually causes an error
- droid has the ability to drag inside the spacebar to move the cursor, but the first time you do this, it inserts a word instead because it's confused about what mode it's in
It is amusing that it's hard to convince people this is a problem, but I sort-of understand. Over time people have learned to just, not edit text on mobile. There's relatively powerful versions of office suites on modern mobile OSes, certainly more powerful than Windows CE devices that had full keyboards would ever ship with, and yet most people don't even really consider doing much on mobile other than sending messages and taking notes, two things that rarely require dragging the cursor. When editing things you quickly type out, gestures like dragging the spacebar to move the cursor around is usually "good enough" for making small edits to fix typos or change the wording, which makes it feel like a non-issue.
On Pinephone with Squeekboard, I greatly miss the ability to drag on the spacebar to move text, and even slightly miss the ability to swipe across keys to type. And yet, the weird thing is, even though text editing on Phosh is significantly less refined than either Android or iOS... I ultimately don't have much harder of a time doing it. And I think that speaks volumes on its own.
- Wait for someone to point out that text editing is no longer practical
- "I am not anti-mobile. My goal is not to return back to the desktop, but to move mobile forward."
Why? Why should we privilege an intentionally nerfed computing experience as the inevitable future? Almost every trend "mobile" is pioneering is bad.
Cursor keys to edit, a scroll wheel to navigate one's inbox, it was a dream that "they" stole from us with RIM's demise.
Fix that before literally anything else on Android, please.
> too many people mistakenly see text editing as “done”
I can't imagine anyone that uses both desktop and mobile to think mobile text editing doesn't need improvements.
In other news, I typed this on my phone, having to edit parts of it, and failing to simply drag my cursor to the end!
It doesn’t support desktop modifiers like Ctrl or Shift, but at least you can properly place the cursor where you want without going mad. For some reason, only LineageOS has that feature, and not even every other ROM. I once tried a different ROM and switched back because the feature was missing, horrible.
[0] https://play.google.com/store/apps/details?id=org.pocketwork...
The other benefit is that you can quickly copy/cut/paste/select all with a single swipe action (from the swipe button to the c, x, v, a keys respectively).
There is just an single downside: no swipe on the space bar to move the cursor. But with another gesture you can turn the keyboard into an arrows one.
It is possible to connect a keyboard to a phone though obviously this is a bit silly. But it does seem obvious that people don’t care that much about the text-editing experience. And iPhones tried being better at text editing and apple dropped those changes, so it does seem they at least didn’t feel the change was worth it:
- you used to be able to select a word and pressing shift would toggle the suggested correction between regular, capitalised, and uppercase. That feature was dropped
- with the pressure-sensitivity feature, you could press on the keyboard to turn it into a kind of touch-pad for moving the cursor, this meant your finger didn’t get in the way of the text so you could see where the cursor was going. You could press harder to begin a word-by-word selection then move to select more or press harder to upgrade to sentence/paragraph selection. Apple got rid of the pressure-sensitive screens in newer phones and give them a long-press space bar gesture, but this doesn’t allow for selection (as far as I know) and makes it hard to move the cursor down.
In place of touch, I'd prefer a new keyboard screen containing a joystick to move the text cursor with. On the opposite side of the keyboard, you could have all the context buttons, together with a 'select' button which can be held while moving the joystick to make a selection. Add a toggle button to the existing keyboard to switch from and to these new input options and you're all set.
Whether this solution is intuitive enough for the average mobile user is up for discussion.
It made intuitive sense coming from a desktop environment and didn't suffer from having one's fingers obscuring the caret or handles.
I still miss this method of highlighting text in Android.
This is before we get into the silliness of autocorrect repeatedly autocorrecting the same thing when you delete it and retype it.
Dread writing on mobile.
I’m also not sure that the inline magnifier is such a good idea, he never gives an example with a fat finger overlaid, I think that would immediately show it’s just not feasible.
I’m also sure that Apple has dozens of user tests with all type of text editing strategies, Ken Kocienda even wrote a book about his process developing text input and auto-correct.
https://www.amazon.com/Creative-Selection-Inside-Apples-Proc...
In practice, I only use the thing to receive SMS messages containing verification codes (which I can't receive on my laptop). The phone doesn't travel with me (i.e. it's not in fact mobile). It stays near the laptop, which also doesn't travel.
My eyesight isn't that great these days. And I have a distinct sense that my fingers and thumbs have got fatter with age; one thumb covers three "buttons" on the virtual keyboard.
You may say that I should get a bigger, more-modern fondleslab. But the buttons on the virtual keyboard are the same size on big fondleslabs as they are on my old one; they keep the keyboard as small as possible, so that app designers have more screen to play with.
But that meant I needed to make updates from the hospital with my phone. I mean yes there were probably better options but I was a bit busy at the time to think of them.
Let me just say that writing raw HTML files using textedit for Android was not a great experience. It's just not the right interface for making complex text.
Maybe LLMs will help with this, allowing us to describe what we want at a higher level, through voice or text. But God help me I do not want to try to write valid HTML on a phone (after being awake for 35 of the previous 36 hours).
I believe that the relation should be reversed: Make the best system possible, usable by well-trained users, and only then add easy modes, tutorials, child safe modes, etc. with paths for improvement at every level. But when using modern systems, I often feel like I’m reduced to using Fisher-Price toys with one (inoperative) button.
It appeared as if writing and text editing (writing a novel is a lot of editing) for mobile phones was solved for Japanese, and it was a matter of time until it would arrive in Latin scripts.
Still waiting for that future. I wonder if things in Japane regressed too?
These people must have never used a phone. This article hits 100%, there's quite a lot of these micro-issues on things we use constantly that could be major differentiators and extremely easy sells, but no one implements.
My very personal pet peeve is text reflow on mobile browsers. Just zooming into a long paragraph and having it wrap so you only have to scroll vertically and never horizontally to read it. It's a killer feature and almost no browsers have it!
1. The autocorrect is the worst part and the only real struggle as it often change the word not logically with the previous words (softkey) or it doesn't know the correct conjugation and changes it. Finally, the menu shortcut is way too close to the letter 'a' and I can't stop hitting it.
2. On the ergonomy side, only 'select text' is problematic for me, but not the long press + cursors, that's okay, it's more about cursors themselves when they reach the edges of the screen when I select whole paragraphs. I often find that I can't select the first letters. Therefore, I always need to select some letters from the paragraph above and remove, or select less and rewrite.. Another problem with it is selecting very big chunks of text, I would love a click somewhere + 'shift' click elsewhere to select all the text. Finally, the 'Semantic' web doesn't help sometimes when I can't select several paragraphs because they are not in the same Semantic unit.
I the past I was carrying with me a small Bluetooth keyboard that could switch with a button between my phone and my tablet, but finally, writing on the phone was good enough.
I have way less good experiences with writing code on the phone or interacting with a ssh session.
Thumb-key takes a radical approach, and I've watched someone who has become comptetent at it (which i haven't) type elegantly on it.
https://github.com/dessalines/thumb-key
But I always wonder why noone has tried building a hardware chord keyboard for a phone. It seems perfectly suited.
But in addition the keyboards just don’t seem to be very good. I can’t tell you for how long now I’ve been infuriated by entering a search in iOS Safari only to have every word separated by a ‘.’ because of how it overloads the right edge of the to have a period right where my thumb likes to go.
I’ve tried numerous third party keyboards and they are all some sort of combination of “bad”, so I always end up going back to the built-in one after a while.
These are all far from solved problems, it’s just that it seems like a lot of people are so accustomed to the friction that it just seems like one of those inevitable things.
* Immediately start dragging. When you pause it immediately enters text selection. If you move slowly you get the loupe otherwise it starts jumping word by word in the text
* When you release you get the popup options to do something with your text
* Holding the end of the selection and moving slowly gives you fine control over the cursor.
* When editing this comment, it doesn’t immediately jump to selection but gives you cursor control right away after long press with the magnified loupe. Move fast enough and the magnifier disappears. This is different behavior from long pressing the space bar.
* Double tap and drag gives you the selection behavior.
* Double tap and no drag selects the word.
Actual coding on a touchscreen is still impossible even with the articles improvements. I’d like to see more work on that. I wonder if speaking to our devices inaudibly could be the long term way forward.
Mobile? Since it seemed like highly inferior computing, I delayed getting one. Finally I did. Tried to do some little thing, and for four hours the fake keyboard came and went for no good reason, several times had to power off the device and log in again.
Conclusion: I hate that mobile device.
Result: I put the thing in a Faraday cage envelope and put that in a box in my car. I hope never to use it. My only intended use is to contact 911 in an emergency, and I hope never to have one of those.
To me, mobile is a big step backwards and very unwelcome.
I'm putting up with Windows 10 Home Edition but ASAP returning to Windows 7 Professional. Most important application -- KEdit.
Theme in computing: Mostly just text, in files, in the Windows hierarchical file system. Next, macros for KEdit. Next, command line scripts in Rexx. Next, software in the Microsoft .NET version of Visual Basic.
Main use of computing: My startup and its Web site. I have the Web site code, 100,000 lines of typing, all from KEdit, all in .NET Visual Basic, running, apparently as intended.
Latest Irritation from Mobile: Wanted to order some Chinese food for pick up. Used their Web site. The site assumed I had a mobile phone. So, couldn't order via the site and had to call the Chinese restaurant and give me order to a person, item by item.
The assumption of mobile for the Web site ruined the on-line ordering and much of the site.
The site was complicated and in the end all they needed was one page of HTML. That one page could easily be plenty good, and even a static page would be better.
Good example of computing and a Web site: Hacker News!!!!!
Hint: all misspelled or weirdly capitalized comments in my HN history were made on mobile.
While the bigger screen is indeed a step in the right direction, most of the shortcomings from smaller screens are still here. Text input, as described by this post is just as silly on a phablet like it is on an iphone. I even went as far, as purchasing a logitech mobile keyboard (with a tablet/phone slot) so that I can reply to something longer than a short sentence.
For me its obvious, that without a physical keyboard, the phone will remain a device to scroll up and down and will not replace a pc, no matter how big the screen is.
This means I cannot curl my thumb without my pointer finger also curling. This is not something that can be stretched or relaxed or otherwise fixed without surgery- my tendons are cross-connected. The result is that typing on a cell phone keyboard ends up cramping my hand very quickly.
https://en.wikipedia.org/wiki/Linburg%E2%80%93Comstock_varia...
The original IPhone "un-solved" the problem and then every phone company copied Apple because that's what they all do. And here we all are complaining about this like it's 2010 all over again.
It largely solves this problem.
Long press on spacebar, keep it down and move around to move the cursor. While dragging tap with a second finger to start selection mode.
The weirder part to me was that the iPad didn't bring much improvement on that front. Even with a bigger screen and "desktop class" application, text editing only barely works with a keyboard/touchpad attached. The only improvement has been on moving the text cursor, yet it's still a real pain to select random text in the middle of a page.
On the solution though .... I think we should really do something simpler than introduce new paradigms and further dig the "that's how we do it on mobile" well.
Just give people damn arrow keys and a crontrol key: let them put a cursor on the middle of the text they want, move the cursor with ultimate precision exactly where they want it, and shift select the text they need.
Android does the arrow part right, Windows' on screen keyboard does both right. It's incredible how liberating it is to just select text with the arrow keys, hit ctrl+C, crl+V it elsewhere and be done. That's really not that much to ask, Apple not doing anything in this front feels like sheer laziness at this point.
Allegedly this is standard in her age group. Makes me wonder about the next few waves of people entering the workforce and whether they're going to hate using keyboards for input.
> For highly proficient users, this gets even worse as their is no command key equivalents for cut, copy, or paste. […] Part of the unspoken reason desktop clipboard use is so high is the speed in which it can be used. Mobile has none of this.
I recently discovered [1] that iOS has several three-finger pinch/swipe gestures for copy, paste, cut, undo, and redo. Apparently these gestures were introduced in 2019/iOS 13 [2]. While they are certainly not the easiest gestures to reliably perform, they at least do not suffer from the issues of ambiguity described by the author.
[1]: accidentally, of course, because the discoverability of many features on mobile is quite poor
[2]: https://9to5mac.com/2019/06/12/gestures-undo-ios-iphone-ipad...
They weren’t and then they were. Palm, Nokia, and Blackberry were all devoting around half their user-facing surface area to content creation with physical keyboards.
Near then end Nokia did make some really nice Linux-powered slide out keyboard devices. Best of both worlds if not for their thickness.
Browsers could honestly use a modifier (additional mouse button or key held down) that put them in a "text selection mode," in which all other meanings of click and click-drag are switched off. That would give the user full control to excerpt text from the page regardless of its context.
copy and paste should be dedicated soft buttons, at least as long presses.
On the topic of mobile text editing: I use an iPhone SE (2nd gen), which has a 4.7" screen. The other day, I needed to translate some text. In iOS's Notes, I couldn't type and see my source at the same time. So I did what any iOS user would do and opened vim (3), made a v-split for the source, then wrote and edited my translation.
Vim-users, do you also find vim good on a small touchscreen?
1: https://youtu.be/j15MLgnIKfI?t=66
2: https://youtu.be/_9YPm0EghvU?t=224
3: in a-shell.
For writing prose, gboard is bad, but it's the least bad solution, its sliding gestures and predictions are good enough.
I use unexpected keyboard for programming on the go. Free and open source, originally developed for termux. It has arrow keys and modifiers, and undo also works (ctrl z). TBH it's not perfect for me as it doesn't have predictive text and its arrow keys are fiddly, so I use it sparingly.
This used to work perfectly on iOS until iOS 17. You could hold and drag text handle or you could double tap to select text around it and then drag selection handles. Now select-dragging a handle in place was made impossible, I don't know why.
The points mentioned in the article are painful, but something I could get adjusted to and get better at with time. There are a couple of things for me though, that I just can't work around, and which are subsequently the biggest pain points for me. These are: lack of vertical space in landscape mode and broken text selection feature. Let me elaborate. Note that I own a google pixel phone, so can't really speak for iphones or androids from other companies.
Pixels gravitate towards long and narrow displays. This can be great for portrait mode, but sucks for landscape most of the time. The issue is made so much worse that neither OS or most apps seem to try to optimize for this mode, further wasting the space available. For example, I'm typing this post in Firefox on my phone right now. About 1/3rd of the screen is taken up by the keyboard. This is fine and is actually the reason I use landscape mode - bigger keyboard allows for faster typing. Another 1/3rd however is taken up by a combination of notification bar at the top, gesture hint bar at the bottom and firefox's address bar. With some manipulation you can make the adress bar go away, but the other two bars can not be hidden on pixels period. Perhaps the biggest frustration is the fact that the "fullscreen" mode I want is already implemented on Android for games and such, there's just no way to turn it on for firefox in either the app itself or the OS.
Text selection tends to be broken when selecting several paragraphs of text, especially when selecting existing text from a web page - the teardrop just jumps around violently not letting you to put it where you want it. Sometimes you can wrestle it with multiple attempts, but other times it just won't let you leaving you with the only workaround of selecting in smaller blocks.
This reminds me there actually was one somewhat recent improvement to text editing on androids from few years back: ability to move cursor by sliding your finger over the space bar. That eliminated one big pain point I used to have.
I also want multiple undo everywhere and consistently like any decent desktop.
https://youtu.be/Y36rUWeVnMM?si=yqwrOAanvSJ0fOEt
It lets you combine speech to text and markdown editing.
Partly for fun and practice and partly because it helps me to quickly commence new posts and either iterate on the go or return to the open PR when I'm back at my laptop.
The article did skip over one important QOL feature that every modern keyboard has: Swiping the spacebar to move the cursor. It solves a lot of the things the author complains about.
The thing that causes me physical pain when writing something on mobile is the web. Browsers/websites misinterpret inputs into text fields all the time. Trying to scroll the text field while it has an active selection? Great, now your keyboard is closed and the selection is gone.
Why no customizable menu buttons? Good note editing apps like Bear on iOS have them, and predictable position makes choice faster/more precise vs a near-cursor popup
A selection undo stack (and action undo tree stack) would also help in fixing mistapping and other mistakes
And maybe modal editing paradigm could also be a good source of learning?
Also fully customizable keyboards would help in text editing
(don't like their proposed animation , seems like distracting noise, though maybe in actual use it's not bad)
Of all the possible ways to input text with your thumbs, this is undoubtedly one of the worst.
I'm sure plenty of people have tried to come up with alternate keyboards, maybe a radial layout or something, but when I look, I don't see a whole lot of activity in this space.
There's got to be a better way, surely.
I have used a number of non-qwerty/typewriter style mobile keyboards. I have landed on MessagEase, but really loved a few others. Thankfully MessagEase still works, most died somewhere in the mid 10s.
I still wish text editing was better, I like taking notes on things usually on Obsidian and the entire mobile text writing and more importantly editing is just not too great, but as the demo shows it can be improved a lot... :/
There. No need to reinvent the wheel, just do the same stuff desktop does but with the actual keys desktop has.
The spacebar-drag trick is cool, but it's not enough.
What a strange attitude to take! What’s the point of the research if it’s not aimed at shipping anything?
Hopefully this is just a miscommunication and isn’t meant to be as passive-aggressive as it sounds.
In the meantime if anyone knows a good "Graffiti" app keyboard for me to try out feel free to recommend.
1) I use the arrows in my keyboard to move the cursor. (Or letters in vi) Consider four new buttons to be able to move around once the cursor is active.
2) Love this concept. Editing this very post is too hard. This is a big change. Put together the progressive UI that adopts these changes over time. Do the research on user adoption and least impact. Move this from idealist end state, to practical sequence of steps to get there.
https://www.youtube.com/watch?v=76b3c_ssyPQ&t=181s
It's a long video but the segment I mean is only a few minutes, and it's really fascinating if you're interested in people having to work out surprisingly hard, invisible problems.
https://support.microsoft.com/en-us/topic/how-do-i-enable-di...
Edit: fixed link
Vim is usable, with Hacker's Keyboard (Android keyboard app that gives a full on-screen keyboard) or a Bluetooth keyboard.
The Vim paradigm solves the problem of not having a mouse or menu bars, since it doesn't rely on them.
> (...) there is a completely new gesture, double tap. Wait, there is actually a third gesture long press, which also works.
TIL about the double tap -- always used the long press to select text. It is interesting that the author refers to long press as an alternative. Do most people use double tap for that?
https://developer.apple.com/documentation/swiftui/texteditor
I can move the cursor by pressing and holding the spacebar or 3D touching anywhere on the keyboard. I can then 3D touch harder in the same movement to select a word and drag to select more. One tap on that selecting brings up the menu. Double tapping selects a word, double tap and drag lets you select more. Here the menu appears instantly.
And the proficient users can even three finger pinch to copy and three finger zoom to paste. Or three finger tap to get a special menu.
——
This one might be specific to me but I can type on mobile nearly as fast as I can on my keyboard (I might just be a slow typer). I also sometimes feel like I’m the only one who highly appreciates autocorrect. It rarely makes mistakes (great, now I get roasted for every typo).
Still, editing on the computer is much better. I use a lot of shortcuts and vi modes where I can. But I don’t think this can be emulated on mobile. Please prove me wrong.
Maybe with a device that read your mind and allowed you to type by thinking, but with that aside, I don't think there is a solution. I think this is a fundamental limitation of mobile.
One way could be to implement some or all of this along with changes to Android's "spacebar drag" cursor movement mode. Currently it can only move left and right unlike Apple's touchpad approach. Maybe I'm missing something but it feels like you could implement a lot of the features through a keyboard touchpad while keeping regular editing clunky but familiar. By expanding on an otherwise very limited feature it might be welcomed by power users without depriving anyone of their existing workflow and muscle memory.
One problem that comes to mind is that people are used to selecting from beginning to end and triggering the interaction from the bottom of the screen gives very little area go down while keeping 1:1 movement of the cursor. It could be solved by having the cursor hit the "roof" until the users finger matches the translated position then switch to a scrolling mode at the edges.
Even those React rich text editors don't support React Native.
It's just way too hard to solve this problem for medium-sized teams.
vi, cat, or emacs to generate the original text.
Too bad that ship has sailed. I still miss my Bold every time I try to edit text on IOS
Pasting is particularly hellish because of what essentially amounts to a zero-width tap target. Then add in poorly designed third party apps changing the interface (looking at you microsoft) and it's a nightmare.
> Text editing on mobile was considered “good enough.” Since people weren’t complaining, there was little motivation to improve it.
Put it another way, "make things better" just because they can be made better.
Also, undo.
To a lesser extend, using a trackpad vice mouse. Or a controller in FPS games.
We don't need more low quality content.
I use these two constantly
Text editing involves a pop-up keyboard. Why not add a menu with arrow keys, selection toggles, and copy/cut/paste there?
You can just hold down the spacebar to move the cursor. I assume Android does this as well?
It's less WYSIWYG and more "what you're getting is what we interpreted you wanting, good luck"
Time when I really appreciated Vim was when I realized it is such a great way to edit text file on a small touch keyboard.
I call text editing on mobile solved issue ;)
The post echoed many of my frustrations and inspired me to consider adopting the fisheye-zoom to my editor.
I'm looking for people interested in doing shader programming on the go to try this early version, hit me up if you are interested [email protected]
(pop your phone into the case if you anticipate work session(s) )
Since I found me proven both wrong and right in some sense. Looks we rather sacrificed quality for immediate reaction and enthusiastically do bad text editing and sloppy writing on the go.
This sounds like arguing that a hammer makes a terrible screwdriver.
Like in the old days, with voice-recorders.
My phone doesn't have such a sensor, so this wouldn't work.
No, it's completely broken. And mobile is an absolute mess. The touch system is so utterly overloaded with functionality that it effectively makes all these wonderful things completely opaque and, in a lot of cases, not discoverable at all.
Ask any iOS user what single, double, triple clicks mean. Ask them if they know about long touch on the spacebar. Ask them to show you everything they know about how to operate the device.
What you'll learn is that the vast majority of people pretty know nothing about the function overloading of touch and gestures. They know simple touch interaction and that's about it.
The touch screen, whether it is on a phone or in a car, exists for one simple reason: It is a lot easier to make UI decisions when you don't have to make them. It's cheaper. It's faster to design. Manufacturing costs are lower. Testing is simpler.
Having design many physical control panels during my career, for products ranging from commercial through aerospace, I can tell you it is a difficult grueling process. In short, pain in the ass. And, if you make a mistake or change your mind, you can't just compile your way out of it. Hardware is hard (and expensive) for a reason.
Today's smart phone designers flat-out refuse to add physical UI elements to their phones, instead, choosing contorted ways to overload the touch interface.
Prior to iOS devices I owned some very interesting phones and PDA's by companies like Motorola, Blackberry and Palm. In the aggregate, these companies had interesting physical UI ideas that worked very well.
A stylus, for example, is brilliantly simple and better than your finger, particularly if more sophisticated, as seen on Wacom tablets.
Blackberry had a small trackball you could also click right on the keyboard (another very useful UI element). It was great for editing text and other use cases. Heck, you could use a browser with a proper desktop-style cursor.
Motorola had a click-wheel on a PDA-like add-on to their flip phones. Fantastic for navigation and super easy to use. The clip-on also had additional buttons, that made such things as single-handed almost-blind address book navigation very simple.
Also, touch screens are incredibly fragile. Example, during a call, any accidental touch will cause problems. I have resorted to switching away from the phone application during my call and, if I have to move around (while wearing a headset) placing the phone facing outward in my pant pocket. I've had too many cases of the phone doing stupid things during a call when in my pocket facing inward.
In short, touch screens are terrible for the consumer in a lot of use cases. They are fantastic for the manufacturer because they don't have to make any decisions, manufacture a single product and all is well so long as the torture they inflict on their customers is just below a pain threshold.
There are so many potentially interesting ideas that could be explored. Adding a set of user-programmable physical macro buttons somewhere (sides or back). Adding a small cursor-only touch pad to the back for, well, pointing control. Click and right click could be actual buttons. Blackberry's mini trackball was brilliant. And a real keyboard, as a first-class citizen add-on, not something left to third parties with no real support past the basics.
If someone wants to make heavy use of a device they would love a real full-width keyboard that clips-on and folds open with a Blackberry style trackball and a few extra keys and functions. Lots of people buy cases for their phones. I would absolutely buy an official fold-out keyboard/case if it was truly integrated so I could actually use the thing without having to remember what triple-click-press-slide-right-and-face-north f-ing means.
For me, the phone is unusable for anything beyond the trivial. I hate browsing the internet with it. I don't buy anything other than Uber rides with it. I don't use it to buy anything on Amazon. Text messaging for work is useless other than the very basic, because it is horrible to edit anything and there is no real desktop integration to track and search as you can with an email client.
Which brings me to the other problem with mobile:
It's 2023.
Why isn't my phone seamlessly integrated with my computer?
When I plug-in or otherwise connect my phone to my desktop or laptop, the phone's screen should appear on my desktop as a first-class citizen for full interaction without having to touch my phone. I should be able to edit text on my phone from my desktop in a phone window. I should be able to cut, copy and paste anything between desktop and mobile. I should be able to run applications on my phone and interact with them on one of my three 27 inch screens. I should be able to click on a phone number on the desktop and dial it on the phone. I should be able to use the same headphones I use on my desktop to run a call on the phone through the desktop interface. I should be able to move, copy and manage files on the phone from my laptop. Etc.
I don't really understand what mobile designers are doing. It seems that they stopped being creative and holding their user's best interests as their primary guidance. They slap a touch screen on an aluminum frame, use as few buttons as absolutely possible. Limit desktop integration to the absolute minimum (iTunes is garbage) and leave third parties to hack stuff together rather than develop exquisitely integrated solution.
I could be wrong on this. I firmly believe Microsoft still has a real shot at a next generation phone. Their Windows Phone attempt was a failure for many reasons. What is lacking today are the things I mentioned above, and more. As ridiculous as it may sound in 2023 --because it should have been done by now-- having a device that fully integrates with desktop in an exquisitely usable symbiotic relationship would be nothing less than amazing. That, I believe, would gain traction.
Buttons.
Modern phones are horrifying examples of UX, largely because they don't have buttons. You get two buttons for volume, even though volume accounts for probably 1/10,000th of the UX of the user. You get one button for power, because how else do you turn it on? And if you're lucky you get a button where the fingerprint reader is. You also increasingly get an "assistant" button, which can often only be used for activating a shitty voice assistant specific to the make of phone you have.
Navigating text editing requires buttons. You can try to use touch tricks, but they're complicated, clunky and inaccurate. For ease of use you need buttons specific to editing. A mouse and keyboard have buttons and shortcuts for editing because a person needs specific functionality for specific use cases. So the phone needs more buttons to make this work.
For physical buttons used for editing, nothing I've seen has ever surpassed the clickwheel. It worked for iPod, it worked for Blackberry. You can open menus, scroll with precision, and select items. And a flat layout allows use as arrow keys, compared to the side-mounted version that only has two dimensions. You can emulate this on-screen, though it'll be inferior as haptic feedback is one of the important aspects of the precision of a clickwheel.
Arrow keys are another important button layout. They allow precise navigation within a text field so you can get to the place you want to make a specific change in, without the clumsy finger-tapping that never works right. On-screen these need to be dedicated buttons, as a virtual clickwheel can't easily differentiate between just clicking on the top of the clickwheel versus using the arrow functionality, and the UX suffers.
Another important button is the delete key, as opposed to backspace. You should be able to delete things in front of the cursor as well as behind it.
And then there's the "select characters" cursor modifier. You should be able to start selecting text where the cursor is, navigate using your arrow keys, and then stop the selection, and then click one of the modifiers for copy/cut/paste or just start writing to replace it.
And of course we need an undo button, in addition to buttons for cut/copy/paste so we don't need to long-finger-press to show an editing menu.
Basically all the buttons I've mentioned could easily fit within an editing box at the bottom of the screen, with the text in the top screen. Having the buttons right there for the entirety of the editing session is the right UX. It's actually faster than keyboard shortcuts, it's more intuitive, and more accurate, with less precision required for the user.
Maybe one day these stupid hardware vendors will get off their obsession with chasing Apple and finally introduce some useful buttons on the device again. Until then, fake buttons in an editing modal is the best choice.
The problem is the target audience for this is business customers, and particularly those in management, so it’s a small market. ICs aren’t up and down as much between meetings and can largely stick to their desktop setup. But the manager’s life is one of constant movement, reviewing and making comments on things, drafting, editing, tweaking, etc… and usually in blocks of time where it doesn’t quite make sense to sit all the way down (unless you’re really good at calendar management, which.. also sucks on mobile!)
> fixing text editing isn’t seen as important enough in the war between Android and iOS.
I think there’s a group of people for whom this would make all the difference. It’s just not your typical consumer.
EDIT: and before you say “sounds like you just want a tablet” - I don’t! A tablet is the difference between lugging around a briefcase with me everywhere, or not.
Doing anything on a mobile device is far more painful than people assume. A generation of people are now unfamiliar with using a PC and although it has its problems, the use cases are far easier.
Are these really solved cases?
- multiple tabs in a browser. Why not put tabs/shortcuts on the desktop (errr mobile top?)
- apps in general, why do they exist at all? Most are just slightly better web pages--ok I know it's just to give Apple&Google gatekeeper status and create a way to get paid for the app. but at a cost to users that is more and more annoying. Why have a mobile aware NYTimes.com AND an app? Ditch the apps where not really needed and do a decent one-page web app instead. No need to update, no special gatekeeper for install. If the first issue was solved, it would make mobile devices easier. I have no sympathy for the Mush burdened x-twitter but if they have a gripe with Apple do a decent web page and no gatekeeper is involved.
- &^&%$#$$% passwords. Partly because of apps in general, all your passwords are hidden from your password manager, which is of dubious value on a mobile to begin with. Typing in that auto-generated password created by a password manager is HELL on a mobile device, text being hell as this article points out.
- Use a hardware keyboard.
- Install Termux.
- Install and use vim (or emacs, or nano, or whatever) to edit. That is, use a goddamned fucking text editor designed for the task.
The fact that devices similar to the 1990s-era Psion Series 5 are rare as hen's teeth doesn't help. This was pocketable (given sufficiently large pockets), had basic office productivity apps preinstalled (word processor, spreadsheet, database, email, contact and diary manager), as well as its own scripting language. It was optimised for power consumption and battery life. I know at least one journalist who used the device to file copy on the road --- this was no toy.
<https://en.wikipedia.org/wiki/Psion_Series_5>
I'd love to see a modern update based on e-ink rather than LCD, as I wrote a few months ago:
<https://news.ycombinator.com/item?id=36171805>
(I did receive a highly intriguing email suggesting such a device might be in development but have had no further follow-ups.)
There's a delicate balance between size, usefulness, and portability which is challenging to get right. In virtually all cases even the worst netbook-type compact laptop is superior for any significant composing task than the best smartphone or tablet. Touch-only interfaces are ultimately a dead-end.
The section of Jenson's article on how taps can be misinterpreted should be tattooed to the forehead of every UI/UX designer, PM, and marketer on the planet, as well as stenciled to all office walls, windows, doors, and whiteboards. I've railed about the ambiguity of selection vs. movement before (see: <https://news.ycombinator.com/item?id=31771672>), but Scott highlights additional nuances. Does a screen touch: place the cursor? bring up the menu? start a drag? start a double tap? start a long press?
And what of when the user is disabled (Parkinsons, other motor/neuron disorder), tired, ill, drunk, in a moving vehicle (preferable not as the driver/operator), juggling children (don't drop), or in any other way at less than 100% focus, attention, ability, and dexterity?
I've had some variant of a smartphone or tablet since the mid-aughts. The two I've the fondest recollections of were the first (a Palm Centro, with a hardware keyboard) and my most recent (Onyx BOOX Max Lumi). Each has different strengths. The Centro was good for basic text entry, though its display was minimal. The BOOX is a large e-ink device which is excellent for reading and has Termux (a Linux userland for Android) installed. With an external keyboard it's passable local Linux system and can be used to SSH elsewhere, though the combined set is not something that can be crammed into a pocket. Slides easily into my messenger bag though.
As I'd listed out in table-form three years ago, there is no use-case for an emissive-display tablet for which it is superior to a different device, and the distinction is little changed for smartphones. Display + keyboard remains the ultimate productivity environment. For capability, privacy, and flexibility, I'd prefer separate audio, image/video, and comms devices, each of which individually can be quite small. Technological advances for each is slow enough that these need not be replaced frequently (once a decade or more should be sufficient). The main driver of late has been storage capacity, though with 1 TB being now widely available, that's no longer a driver for text, audio, or even to a large extent images, though video can of course consume prodigious storage.
The privacy advantages of airgapped generative / capture tools, with a separate comms-capable device also seem increasingly advisable.
I find it telling that one of the more popular addons for ipads are covers with a builtin keyboard. It's a way bigger device than an iphone. But yet the keyboard sucks enough that Apple sells covers with a keyboard. Of course, all the touchscreen keyboard problems that the ipad has are magnified on their iphone. Yet, they don't have a solution for that. And they also sell a stylus for the ipad. Because fingers lack precision. It's the same OS but there seem to be no such options for the iphone. Does the stylus even work with an iphone? Is that deliberate? It's not like people are going to be magically more precise on an iphone relative to a huge ipad. Conclusion, Apple just accepts that that's the way things are. And besides, Steve Jobs would turn in his grave if they dared to ship an iphone with a stylus.
Hardware keyboards on phones used to be a thing. I worked at Nokia back in the day. Really nice keyboards. Blackberries were popular too. People wrote lots of stuff on those things. I wouldn't mind a little pocket laptop. It's not like my pixel 6 is small or subtle in my pocket. It would be more useful with a slide out keyboard.