Sunday, July 5, 2009

Convergence (or, the device I want ...)

It's been, OK, more years than any of us want to think about since I first wrote about Trent walking around with a handheld hooked up to the internet through radio packets. Over the years bits and pieces of that device have become real -- I had a nice letter many years back from a Compaq engineer who wrote that his design for the iPaq had been based in large part on my description of a handheld.

Currently, Apple is out front in the handheld market. The iPhone is a superb piece of technology, and while it doesn't do everything I want, it does a lot of the things I want, and does them somewhere between acceptably and well. But there's an opportunity here for device convergance -- if some company like Microsoft or Dell or even Sony or Nintendo were serious about absolutely owning the convergent device, here's what they need to build.

The device I want has roughly the same physical form factor as an iPhone, though there's no reason the screen can't be a little larger -- the iPhone masks off both the top and bottom of the phone with black bars. I want the entire surface of the phone to be a touchscreen. It needs a higher resolution screen than the iPhone, something in 16:9 format -- 640 by 360, say, or 800x450, as compared to the 480x320 screen the iPhone currently sports.

Next, of course, it needs to work as a phone, with high speed internet access. It needs a good enough microphone and speakers that it can be used as a speakerphone for conference calls. It needs bluetooth and high speed WiFi.

It needs to work as a computing device. This means multi-tasking built in, as in the Palm Pre, and probably some flavor of Linux-like OS. It means the ability to manage other devices -- the ability to work as a USB master, not just as a slave. I want to plug my USB hub into it and have my keyboard and mouse and hard drives and MIDI keyboard and Wacom tablet available.

I want a fast CPU, lots of storage, and lots of RAM and I want it all to go to sleep when I'm not using it. I shouldn't have to power up the 60 Gigs of storage I'm not using to get at the 2 Gigs I am using at the moment. I want a multi-core processor, with the cores turning themselves off when not in use.

I want to use standard peripherals -- cheap, standardized memory cards, power chargers, and spare batteries.

I want a wireless router built into it, so that I can use the connection to provide internet service to people or devices around me if I choose.

I want a video processor to offload 3D processing, for movies and games. I want a standard headphone jack. I want AM, FM, and HD radio.

The handheld needs a variety of cradles that it can be slid into. One cradle would be a games controller -- something like the Nintendo DS or PlayStation Portable -- with the various buttons that the kids know how to use to play those games they like. (Take a look at a PSP and you can see what I'm talking about; a PSP is a set of controls wrapped around a device about the same size as an iPhone screen.)

It should work as a still and video camera -- a good one. The controls can be entirely software based, but the camera should be able to take pictures pointed either outward, or inward. Possibly the camera itself can be mounted on a swivel (within the body of the device) to permit this. This way, when used as a camera, you can see on the screen the picture you're about to take; when used for video conferencing, you can see the image you're sending out. (A potential alternative to a rotating lense is to put lcd panels on both sides of the handheld and use a smaller and cheaper panel as the viewfinder for photographs and video.)

A camera cradle would offer improvements to the camera functions and would give you the ability to mount a flash and improved 35mm optics on the device (hello, depth of field) ... and perhaps more importantly, to mount the device itself on a tripod. The camera cradle would also have a port for microphone-in so that quality audio can be recorded on the device.

The device should be able to record 720 or 1080P video to the memory card, in 24 or 30 frames per second.

It should offer a simple binocular function that permits me to zoom in on things easily to look at them. It should offer functional night vision -- not just light enhancement, but infrared stepup.

The next cradle I want is a simple waterproof enclosure. I want to read e-books in the tub. I've dropped a half dozen books in the tub over the years, or had them get wet on the bathroom floor -- that's unfortunate for a book, but a disaster for my handheld. The enclosure should also permit people to use the camera/video functions, for people to take photos when surfing, scuba diving, etc.

I want a projector. A projector cradle is the likely way to do this in Version 1.0, but by version 2.0 I want a little projector built in. Maybe the projector can double as the flash for the still camera. (There's a little gadget floating around out there that projects a keyboard on a flat surface, and then watches your fingers when you try to type on it. I'm skeptical, but they should include this anyway, just in case.) Also I want a flashlight -- not the "light up the screen" thing the current iPhone does, but a real little light (the projector, again) that will illuminate a room clearly.

I want a scanner built in -- the camera doing double duty, but I want to reliably be able to point the camera at a page of text and have it OCR the text and store it.

I want it to replace all my remote controls. I want it to open the door to my car and start the engine.

I want it tied into my bank and I want it to replace my credit cards. When I'm paying a bill, I put my thumb on the optical sensor (the camera, again) and wave my handheld in the general direction of the store's payment device. And we're done without having to wait for our stoned or stupid waiter to try to figure out the bill and bring it to us and wander away with my credit card and steal the numbers off it while he's out of sight.

I want it to recognize the faces of the people who are allowed to pick it up. If someone not on the authorized list picks it up I want it to yell for help and/or call 911. (Maybe 811 ... the "lost phone" registry: "Help! Someone I don't know has picked me up! I'm at these coordinates!")

I want a large-form tablet as well. I don't need multiple form factor versions of my handheld; I just need a cradle that has a big pressure-sensitive screen on it for when I want to lie in bed and paint.

I can't be missing phone calls. When someone calls me, the device stops whatever it's doing, and at my voice command either answers on speakerphone, or puts the caller on hold with the message that I'll be with them in a moment while I snap the device out of the cradle it's in and take the call privately.

I want the built-in GPS to give me real-time ground traffic control information. I also want it to talk to my radar detector and to share that data with everyone else using my brand handheld, so that when people's radar detectors start going off right before the 3rd offramp on the 405 after the 101, I hear about it ten miles back rather than when my radar detector goes off.

I want the GPS in my car to tell everyone where I am and how fast I'm going, and to tell me where everyone else is and how fast they're going, so that the same service that alerts me about the speed trap on the 405 can tell me, "Take Sepulveda. No, seriously, trust me on this one. Turn right at Mulholland and take Stone Canyon Road over to Kester and you'll be home 20 minutes faster. About twenty cars ahead of you using this service failed to do that, and they're stuck now."

None of this is impossible, though much of it is at the edge of the possible. But so were modern phones, just a few years ago. (What? A phone and an MP3 player and a camera and a little tv all in one device?!) At some point, someone is going to build a close approximation of the device I'm describing ... even money on Apple. Done correctly, it'll be a complete ecosystem and will simply own the handheld form factor.


Ran across an old file with business ideas in it recently -- two of them struck me as interesting, looking back. One was a television where the screen was built with fiber optics -- 20+ years ago, it wasn't a bad idea. Picture tubes monitors displaying 800x600 were state of the art, back then; that's a mere 480,000 pixels. Pixels were also, except in trinitron screens, circular and didn't actually cover the entire surface area; and in all televisions the pixels were created by grouping 3 separate light sources, RGB, with varying degrees of brightness. I sketched out a design for a television using a single very bright white light, along with red, blue, and green filters applied in succession at the base of each length of optic fiber to apply the correct color to each pixel; the pixels themselves would have been actual squares, covering the entire surface of the screen much as LCD screens do today.

It's not practical today -- and might not have been then -- a 1080P screen has over two million pixels. That's a lot of optic fiber. But twenty years ago it's not clear to me it might not have been workable, if probably a niche product.

The other one I like, looking back at it, was the use of film as a data storage medium. This one I'm a little more confident about -- I see no real reason it couldn't have been used as a data archival tool, anyway. The technology has certainly existed long enough. Film is an analog medium but a single frame of 35 mm film still has somewhere between 4 and 20 million measurable pixels, depending on a variety of factors. (Better film, better optics, more pixels, short form.) And each pixel has a realistic color depth of something in the range of 36 bits, again depending on a variety of factors. Taking conservative numbers, though, 4 million pixels at 24 bit color depth, you get 3 bytes per pixel x 4 million, or 12 million bytes of storage, or 1.5MB per frame of 35 millimeter film. Spool that through a film printer, and you could certainly get hundreds of megabytes of usable storage even using 1980s technology. I'm a little surprised no one ever did.

Today, with holographic storage around the corner, the idea is quaint. If you could actually print an 8-1/2x11 sheet of acetate as 1200 dpi, with 32 bit color, and reliably read it back, you'd have a write-once medium of 93.5 square inches, with each square inch having 1.4 million pixels, for a total raw pixel count per page of 134 million pixels. At four bytes per pixel (optimistic, I suspect) you'd have a storage medium capable of about 67 megabytes per page. You can fiddle with these numbers to suit yourself -- if you can only reliably get 8 bit color, that's about 17 megabytes per page. If you can get 4000 dpi, you get a page with one and a half billion pixels, and a storage capacity of six billion bytes --- roughly the storage capacity of a DVD.

One possible benefit might be in speed -- I can imagine a scanner scanning 6 GB of data much faster than a single read head can read a DVD. But that's the only real real benefit that comes to mind with modern technology. The next generation of holographic storage is coming in at around 500GB on a single disc -- I don't see any traditional film-based technology likely to challenge that. But someone (me, maybe) probably missed out on an interesting startup, back in 1987.


ditmars1929 said...

Wow, Dan, do you want it to mow the lawn and paint the house too?

David Pitkin said...

Many of the items you ask for were there in a Nokia N800 or even a handset in Japan about 4 years ago, minus the real CPU vs. battery issues. I think another problem that gets even less airtime is a large culprit, this is largely a software problem.

Our desktop computers windows/mac and linux are so damn complicated and the development tools to enable the types of incredible scenarios laid out are just not there even if a lot of the hardware is.

My question is how could our society build open technology to enable this next generation of devices.

A long way to go:

Sean Fagan said...

That was a lot to digest. At the moment, one thing springs to mind, related to displays.

If we exclude VR glasses or implants, then what I think you'd like is something with a hell of a lot of computing power, a hell of a lot of storage (RAM and non-volatile -- or combined if some technology is fast enough)... and then some sort of wireless display.

I mean, you're asking for a lot from a hand-held device. But if we can use the same device on the desktop and as a mobile device, it gets a lot closer, I think. Being able to just sit in front of a display and input device (keyboard/mouse/touchpad/whatever) and have it use that, and whip it out and use a touch-screen on the device when you're not... I think that'd integrate into your life more.

Anonymous said...

I like the film storage idea, that's kinda neat. But cellulose acetate film chemically degrades over roughly a 50 year timescale. How long would it take for a single pixel to become unreliable under chemical alteration?

And imagine if you currently had your ideal handheld, with all the bells and whistles save the size. I wonder what size a system like that would be now.

A good laptop can offer most of those functions (cept the camera) right now (if the software and networking allowed). So shrinking a 5lb laptop down to the size of an 5oz iPhone basically, and you get to remove the large screen, keyboard, and a good chunk of battery. 5 years? 10?

Casey said...

I'd bet that a second (fixed) camera would be cheaper (and easier to protect from damage) than a second display. That's the choice Samsung made with the Q1Ultra.

Casey said...

Anonymous - see Samsung's latest Ultra Mobile PC, the Q1EX-71G Tablet. It's most of the way there.

Omegix said...

one of the technical challenges in making the entire screen a touch device will be lighting. Underneath the framing around the phone is typically some kind of lighting source to shine through the sides of the LCD/touch panel.

Unknown said...

The first thing that popped into my head while reading the section about the traffic alerts, etc. was that that would be just another step towards a Traffic Control system. Why worry about knowing where the traffic delays are when you don't even get to drive?

Daniel Keys Moran said...

>Wow, Dan, do you want it to mow the lawn and paint the house too?

No, I'm waiting on the Three Laws before I start handing dangerous painting and lawnmowing equipment out to the robots.


Yeah, a lot of this stuff is out there in one form or another. None of it is really bleeding edge except that I want it all in something about the size and weight of an iphone. We'll get it, too, though I wouldn't hazard a guess as to when.

Software development is a bitch no matter what language you use. I have no opinion about java versus objective c -- I've coded a little java, and no objective c at all, and I'm not competent to have an opinion. But any genuinely convergent device is going to have the same classes of problems Windows has now -- not the same severity, one hopes -- too much functionality to do any one thing with real elegance. That's OK -- I'll live with one device that does 100 things at an average B+, versus a hundred devices that do everything at an average A-. Which I think is the real tradeoff, too -- how many remotes have you owned that absolutely sucked?


Yeah, if wireless connectivity is good enough, I don't care that much where the CPU and RAM and storage all actually live. Cloud is fine, as long as I can't tell the difference when I go to access it.

>I like the film storage idea, that's kinda neat. But cellulose
>acetate film chemically degrades over roughly a 50 year
>timescale. How long would it take for a single pixel to.
>become unreliable under chemical alteration?

Certainly my theoretical film storage would have needed correction -- hefty, big, mean, serious error correction such as modern SDDs have.

>A good laptop can offer most of those functions (cept
>the camera) right now (if the software and networking
>allowed). So shrinking a 5lb laptop down to the size of
>an 5oz iPhone basically, and you get to remove the
>large screen, keyboard, and a good chunk of battery.
>5 years? 10?

Right timeframe, I think. The two pieces I'd really like to see (the one piece I'd really like to see, really) is the cradle functionality. Or we can just cut to the chase and get a waterproof ebook reader -- until there's a waterbook ebook reader, there will always be paper books at my house.

>just another step towards a Traffic Control system.

I shopped a startup around about 5 years ago called "Ground Traffic Control." It was the idea I just included above -- a communications network built around wireless communications, GPS, and laser detectors, to give people real real-time road alerts. I got some meetings on it, but no startup cash, and as it was predicated on a piece of hardware -- GPS + Radar + some kind of wireless connectivity -- without the hardware, there was no company. So GTC went to the shelf.

Sean Fagan said...

Have you read Charlie Stross' Halting State? It's set in the near future; one of the things he does in it is use people's phones for distributed processing. (With their -- implicit, in a EULA for games or other services -- permission.)

Your comment about the cloud being fine reminded me of that.

Daniel Keys Moran said...

I've only read one thing by Charlie Strauss -- lobsters, I think -- and I liked it.

I'm skeptical about handheld devices being a meaningful cloud computing resource, though, any time soon. Clients to the cloud, sure.

Unknown said...

Well, the recently leaked "Light Peak" technology demo from Intel certainly looked interesting. That gives you the optical connection for a very high bandwidth data pipe. Excellent for a handheld with the specs you've outlined.

Frank J Nagy said...

Film storage... Well there once existed such a device. I used it whan I was a graduate student at CalTech in the 70's and using the CDC-7600 system at LBL. They had the one and only IBM PhotoDigital Storage System or it was called something like that. It was a Rube-Goldberg apparatus. Used film "chips" about 1"x2". I don't remember the capacity and I sort of remember that it wrote the chips with an electron beam in a vacuum chamber, move the chip to an online developer, stored the chips in little boxes with slide-open caps in drawers that slide in and out (gears) for the picker to pick a box, open it and pick a chip to be moved to the optical reader. Yeah, it had a set of transport tubes to move the chips around (don't quite remember how they worked).

The system was call the MSS - Mass Store System and provide online rapid access (faster than tape) to data. I used it to store my final reduced data for my thesis work for ruther analysis.

I believe that the installation was the only one ever built due to some legal issues. It was a kludge but it did work.

I also remember that after a chip was written and developed, it was read to check it and if that failed it was ejected out a little slot and fell on the floor (or into a box the data center placed there for discards). I used to have several such chips picked up when I got a tour but they have long since vanished.

So, yes we did have film storage once upon at time. Also, I did bubble chamber physics where the pzrticle events were recorded on film (3 views on three separate spools) for later scanning and measuring so in its own way that was film storage as well.