If you take the guts of a Blu-ray or DVD player, blow it up, and spread it across a work bench, it looks like this. So you might be surprised to know that you’re looking at the future of storage.

A laser beam whose wavelength is being monitored by this Soviet-looking machine is being bounced from mirror to mirror to mirror before it lands on a spinning disc the size of CD, but orange, and transparent. It’s reading the holograms that are embedded buried inside the disc, gigabytes of random test data.

This work table is deep inside the labyrinthine complex that is GE’s Global Research Lab, 550 acres of big machines and big brains, in the hinterlands of Niskayuna, New York. It’s where the company that brought us 30 Rock invents the future of energy, aviation, healthcare, and dozens of other mega-industries, including, as it turns out, data storage.

Hard drives, DVDs, USB sticks: This is where we store our digital lives. But while our data is timeless, our storage devices aren’t. So, what’s next? And then what?

Data storage is something most people don’t spend much time thinking about, and if we do, it’s in abstract terms. Laptops have a fixed amount of space; we pay for more, but accept less. DVDs hold a certain length of video, or a healthy chunk of a music collection; these are disposable. Flash drives move stuff from one place to another; we sense that they’re different than hard drives; but we’re not sure how.

What we know is that we need to store stuff, somewhere. And by we, I mean we: our network infrastructure won’t be ready for widespread cloud computing, or that fantasy of downloading everything you’ll ever watch in full HD, for a very, very long time, and until then—or for people with unease about that concept, even then—storage is something we need to think about.

In 2010, storage tech is in flux. Here’s how we—and the people and companies we’re slowly (but surely) handing our data over to, store stuff now, and more importantly, later.

Hard Drives Aren’t Dead

Hard drives! You almost certainly own at least one of these, in you laptop, desktop, or even portable music player. The basic principle revolves (ha!) around the reading and writing of data onto a magnetized, metallic platter, which is assembled inside a hard drive’s case alongside a head, which is roughly analogous to the needle on a record player, except instead reading variations in a physical groove, this head floats above the platter, reading little tiny magnetic variations from a short distance.

If the immediate evocation of a record player didn’t tip you off, this technology has a long legacy (read: It’s old as hell): The first machine to utilize the concept was built in 1956; the first modern-looking, reasonably small hard drive (at 5MB, no less!) shipped in 1980, from Seagate.

The story since then has been surprisingly uncomplicated, with steady advances in data storage density, decreases in size and a drastic drops in price. The first 1GB hard drive, built in 1980, weighed over 500 pounds. Today, a 2 terabyte—that’s 2,000 times more capacious—hard drive is small enough to tuck into a loose jeans pocket, and can be had for under $140.

But surely this technology is reaching a breaking point, right? Not quite. With storage density approaching practical maximum’s, hard drive manufacturers resurrected an old theory somewhere around 2005: Perpendicular storage. Seagate senior vice president, Recording Media R & D and Operations Mark E. Re:

We use to use a recording method called longitudinal recording, which is called that because the magnetization and the storage layer on the disk or platter is a plane. It’s parallel to the surface. And when we moved to perpendicular [storage], we change the magnetization layer on the disk so now it aligns perpendicular to the surface

Why?

When you’re trying to get your bits closer and closer together with longitudinal storage, the magnetization didn’t want to say there. It wanted to spring apart, like if you’re putting two bar magnets together. But if align them perpendicular…they want to be closer together.

Translation: More data, less surface space.

Seagate saw longitudinal recording limiting their hard drives to somewhere around 100 gigabits (12.5 gigabytes) per square inch, and at the rate things were going, without perpendicular storage, hard drive makers would be up against a wall.

With perpendicular recording, though, they think they can eventually hit somewhere around 1 terabit (about 128 gigabytes) per square inch. Today, in 2010, they’re maxing out at about 400 gigabits per square inch in stuff you can buy off the shelf. There are quite a few years left of regular hard drives getting larger, faster and cheaper before the technology runs its course, and that’s not even counting the wilder hard drive research that’s going on. Heat assisted magnetic recording uses localized heating of disc surfaces, for ultra-dense data writing. Bit pattern media could reduce the space needed for a bit on a hard drive’s surface from 50 to 1 magnetic grains, by encoding the platter’s substrate with molecular patterns.

Seagate’s hazy prediction for what this actually means for hard drives: Upwards of 50 terabits (6.25 terabytes) per square inch, which companies be working towards, and making money from, for years. Hard drives aren’t going anywhere—at least, not for now.

The Inevitable Rise of SSDs

So what about SSDs, or solid-state drives? They’re by far the buzziest of the storage options, and we’re constantly told that solid-state drives will replace hard drives, like, now. That’s not quite right. Solid-state drives, which have no moving parts and store data with electrical charge rather than magnetism, are taking over—just, not everything.

The basics, from our last Giz Explains on the subject:

What’s inside is a bunch of flash memory chips and a controller running the show. There are no moving parts, so an SSD doesn’t need to start spinning, doesn’t need to physically hunt data scattered across the drive and doesn’t make a whirrrrr. The result is that it’s crazy faster than a regular hard drive in nearly every way, so you have insanely quick boot times (an old video, but it stands), application launches, random writes and almost every other measure of drive performance (writing large files excepted).

So, they’re fast. They don’t catastrophically fail (though they do slowly degrade). They’re perfect for laptops! And you probably want one.

But the future of SSDs is a fairly narrow one, at least for now: Consumer applications range from notebooks to desktops to NAS storage, but they’re all just that: consumer solutions. While we’re going to have to wait a few more years for Flash storage to reach a truly reasonable price point for our new gaming PCs and notebooks, the enterprise world—where data needs are rapidly outpacing ours, and the scale of storage is so much larger—will have to wait much longer.

The fastest area of growth for solid-state storage isn’t even in HDD-like SSDs anyway—it’s in portable devices, like smartphones (and soon, tablets). This storage is of a different nature, though: speed isn’t terribly important in a mobile device, nor is capacity. People are going to be fine with their iPad’s low-mid-range chips of flash storage, because they’ll run apps, play movies and store magazines just fine. Meanwhile, Google will continue to buy hundreds of thousands of massive hard drives to keep up with demand, and the rest of us will gleefully shell out for the rapidly cheapening solid-state drives that will power our laptops. This will continue in parallel, for as far as the eye can see.

But what will the SSDs of the future be like? Research now is focused on eliminating their comparative weaknesses more than anything else. They’ll become more buyable, I guess? Cheaper? Longer-lived? (Current flash storage of the more affordable multi-level cell variety can only be written to about 10,000 before failure.) Yes, all of that. General Manager of SanDisk’s SSD group, Doron Myersdorf, from our SSD Giz Explains: “More granular algorithms with caching and prediction means there’s less unnecessary erasing and writing.” In simpler terms, companies are getting smarter about writing data to SSDs, with their limited lifespan in mind. And on the storage capacity/price issue:

There have been several walls in history of the [flash] industry—there was transition to MLC, then three bits per cell, then four—every time there is some physical wall, that physics doesn’t allow you to pass, there is always a new shift of paradigm as to how we make the next step on the performance curve.

SSDs as we know them today are still a young, and they’ve got a long way to go. And before the technology can completely take over the consumer space, we’re going to see more and more awkward hybrid products, like Samsung’s MH80 drive, which uses a small bank of flash memory for some tasks, and spins up the hard drive only when necessary. Progress!

Your next computer probably won’t have one. But the one after that? Sure. Meanwhile, cheap flash storage, like the stuff inside your crappy USB key, will only get cheaper. And when 64GB thumb drives are commonplace and cheap, you’ll probably stop caring about optical media, like Blu-ray discs, for file storage and sharing. Or not.

Our Holographic Future

Optical media isn’t going anywhere, either. Put another way, Blu-ray isn’t going to be the last disc you buy—it’s just the last one where data will be stored only on the surface. Holographic storage, like GE is working on, and which we got to see up close at their Global Research labs, stores data down inside in many, many layers (GE’s demoed up to 75), encoding the data using thousands and thousands of tiny holograms throughout the entire disc. The secret sauce is the material the disc is made out of, and how it reacts to light. On a broader level, where GE’s holographic storage differs from the other major approach to holographic storage (called page-based), and what allows it to reach densities of 1TB per disc, is that it uses even tinier micro holograms that store less data per individual hologram, but more in aggregate.

While GE is mostly pitching the tech to archivists for now—like our friends at the Library of Congress, who wanna hold onto stuff for a real long time—since the discs, GE says, last for 30 years, what makes it viable as a storage tech you might get your hands on soon after it launches in 2012 is that it’s designed to fit in with the current optical media infrastructure, meaning it’ll be cheaper and easier to roll out than some radically different tech. That is, the discs are the same physical size and shape as CDs and DVDs, and they use a laser that’s very similar to Blu-ray’s, even using the same wavelength. On a hardware level, it just uses a slightly different optical element, but the rest basically comes down to software/firmware, meaning you might still be able to play your Blu-ray discs in a holographic storage drive. (This exploded view of a disc being read, that orange spinning thing, is what all readers look like in a laboratory, even Blu-ray drives—because it’s easier to tweak settings than in their actual product form.)

Sci-Fi

After SSDs and hard drives are reduced to hilarious relics, mentioned only to shock classrooms full of children to attention with a jolt of pure absurdity (“so you’re saying the spun? In circles?), how will we store data? A few of the nuttier possibilities:

• Carbon Nanoballs:

Interest is growing in the use of metallofullerenes – carbon “cages” with embedded metallic compounds – as materials for miniature data storage devices. Researchers at Empa have discovered that metallofullerenes are capable of forming ordered supramolecular structures with different orientations. By specifically manipulating these orientations it might be possible to store and subsequently read out information.

Two of pop-science’s favorite buzz words, united.

• Molecular memory:

What if, instead of carving transistors and other microelectronic devices out of chunks of silicon, you used organic molecules? Even large molecules are only a few nanometers in size; an integrated circuit using molecules could contain trillions of electronic devices-making possible tiny supercomputers or memories with a million times the storage density of today’s semiconductor chips.

A thumb drive larger than your entire NAS would actually have to be made arbitrarily larger, just so you wouldn’t lose it.

• Bacteria:

Trust your data with tiny bugs: Artificial DNA with encoded information can be added to the genome of common bacteria, thus preserving the data….

According to researchers, up to 100 bits of data can be attached to each organism. Scientists successfully encoded and attached the phrase “e=mc2 1905″ to the DNA of bacillus subtilis, a common soil bacteria.

Your storage drive could literally be alive, one day.

• Quantum mechanics: Data encoded on an unfathomable scale:

In a quantum computer, a single bit of information is encoded into a property of a quantum mechanical system-the spin of an electron, for example. In most arrangements that rely on Nitrogen atoms in diamond to store data, reading the information also resets the qubit, which means there is only one opportunity to measure the state of the qubit.

Granted, research into this now is focused on storing tiny amounts of data for a matter of seconds, which is just long enough to allow a quantum computer to barely function, but still: potential!

Data: It’s everywhere. And one day, we’ll be able to take advantage of that.

[Bacteria pic via]

Still something you wanna know? Send questions about platters, disks, bits, bops, beeps or boops here, with “Giz Explains” in the subject line.

Memory [Forever] is our week-long consideration of what it really means when our memories, encoded in bits, flow in a million directions, and might truly live forever.

Bits don’t have expiration dates. But memories will only live forever if the media and file formats holding them remain intact and coherent. Time can be as deadly to data storage as it is to carbon-based life forms.

There are lots of ways data can die: YouTube can pull a video offline before anybody snags it, your hard drive can crash, taking ultra-rare Grateful Dead bootlegs that you never got a chance to upload to Usenet with it, or maybe you designed a brilliant piece of visual art a decade ago in some kooky file format that simply doesn’t exist anymore, and there’s no possible way to view the file without traveling to some creepy dude’s basement a thousand miles away.

What we’re talking about is digital rot—or data rot or bit decay or whatever you’d like to call it—systemic processes which can mean death to data. Kind of a problem when you’d like to keep it around forever. Let’s paint this in broad strokes: You can roughly break the major kinds of rot into hardware, software and network. That is, the hardware that breaks down, the formats that go extinct, and the online stuff that vanishes one way or another.

The Hard Life of Hardware

Everything’s gotta be stored on something. And guess what? All media age. (Except diamonds—bling bling, biatch.) Brain cells die, film degrades and hard drives break.

A sampling of common digital media and their life expectancies (assuming you take care of them):
• Floppy disk – This can theoretically survive between 3 and 10 million passes
• CD and DVDs – It depends heavily on the materials used in their construction (PDF), but you’re looking at anywhere between 2 and 10 and 25 years, in the best of circumstances
• Flash storage – Also depends on the type, letting you write between 10,000 cycles with multi-level flash memory, or 100,000 with single-cell flash
• Hard disk drives – Kind of a crapshoot—anecdotally, five years is a good average, though they can last shorter or longer, depending, again, on how they’re built

Google, with its millions of servers, is in the best position to test hard drives from every manufacturer, and conducted a massive study of HDD failure. Basically, if a drive makes it past the first six months, it’s pretty likely to make it through Year 4, but it is going to die at some point (and makes/models die in batches). As you probably don’t need to be told, hard drives can fail in any number of ways.

In other words, whatever you’re storing your precious data on, back it up, preferably with a mix of drives or media from different manufacturers/time periods.

But what if you’re, say, the Library of Congress, the largest library in the world, charged with a mission “to sustain and preserve a universal collection of knowledge and creativity for future generations,” and suddenly confronted—after 200 years of relatively tranquil existence—by an unending, ever-expanding digital deluge that must be archived and cataloged? On top of a copy of every piece of material that’s registered through the United States Copyright Office, and the two centuries of (oftentimes badly damaged) cultural history you’re already trying to preserve? How do you store stuff?

“DVDs and CDs aren’t even considered storage,” say Martha Anderson and Beth Dulaban, from the LoC’s Office of Strategic Initiatives. They need to transfer shiny-silver-disc content to something sturdier to meet their mission requirements. For digital content, the Library uses a mix of hard disks and tape, like Oracle’s StorageTek T10000B 1TB tape drives, rated for 30 years of archive life. At the Packard Campus, the main battle station for the LoC’s audio-visual preservation, they have 10,000 tapes providing 10 petabytes of capacity, Gregory Lukow, from the LoC’s Motion Picture, Broadcasting & Recorded Sound Division told me. In the video above, you can see a SAMMA robot hard at work. These do analog-to-digital conversion en masse, and the LoC has four of ‘em.

The key, though, is that even though the LoC works with drive manufacturers on boosting reliability and meeting the Library’s technical specifications, is that they have a policy of redundancy and diversity—two to three copies, maybe spread across different states, and stored in different kinds of hardware running different kinds of software. The Packard Campus, which is where music and video are archived and preserved in crazy labs with robots, mirrors everything to a secret location via fiber optic cable. While you probably don’t have secret bunkers to stash your porn, it’s a good general guideline: More copies on more disks is more better.

A Format Can Be a Tomb

It’s obvious, though, that storage media age and die. The more insidious problem, particularly with “born digital” content—stuff that started life as bits—is format obsolescence. That is, just ’cause a video wrapped up in MKV, or an Ogg Vorbis music file, or a DOCX file is readable on computers today doesn’t mean they will be 20 years from now. And if nothing can read what’s inside the file, the data inside is basically lost.

The way you might’ve already experienced this, in a way, is via DRM that’s been deactivated (like a bunch of digital music stores did after being crushed by iTunes), rendering your songs wrapped up in it completely useless. I suspect people who bought into ebooks early, before the emergence of EPUB, are going to be effed in the ay in a similar manner. And don’t even get us started on HD DVD and other failed video and audio physical formats—that’s potentially a double whammy of format death.

It’s important, then, to store your memories using formats that are legit standards that’ll be around for a longass time, if not quite forever. Growing recognition of the problem, particularly as it pertains to ephemeral web content, is part of what’s behind the push for open standards—proprietary standards, from a long-term survival standpoint, are not the best idea, ’cause once whoever makes them dies, the format may die too.

The Library of Congress has picked out seven points that’ll give you an idea of how sustainable a format is—that is, likely to outlast your current Lady Gaga obsession:
• Disclosure – how open the specs are
• Adoption – “an open format that nobody’s adopted isn’t too useful to us”
• Transparency – how readable it is on a technical level
• Self-documentation – decent metadata, which is in some ways the secret challenge, given that it becomes more valuable as the amount of data you have grows exponentially
• External dependencies – how much you need particular hardware to read it, for example
• Impact of patents
• Technical protection mechanisms – is DRM in the way?

Quality is also an issue. So, for instance, for master digital archives of video, the Library uses mtion JPEG-2000 in an MXF wrapper, because it’s mathematically lossless. It uses MPEG2 for sub-masters, which are the source material for MPEG-4 copies that patrons can access. Or, as another example, for a long time, “PDF was considered persona non-grata” because it was proprietary, but since Adobe’s opened it up, they’re now working with Adobe on an archivable form of PDF.

The advantage the Library has with analog-to-digital conversions is that they get to dictate the format and specs—that’s not so with most of the content out there. For instance, there’s not really an agreed upon web video standard—witness the H.264 vs. Ogg Theora codec war, though that’s lookin’ more and more like it’s going toward H.264—so web video is considered “highly at risk.” Despite the large amount of web video the Library has captured—after a year working out the process for doing so, Martha and Beth “don’t have real high hopes for them surviving.” YouTube provides one form of hope, though, in that there’s so many YouTube videos, and so many copies, “there’s bound to be some community interest in keeping them alive over time.”

Pulling the Plug

There might be community interest in keeping the copies of Trolololo alive and playable for the next generation from a format standpoint, but what if Google suddenly pulls the plug on YouTube? How much of it what’s there would be lost forever? Or photos uploaded to Flickr and Facebook that have been wiped from hard drives, since they’re in the cloud. Consider, for instance, everything that would be lost if Wikipedia really did run out of money, and was shut down. Or Twitter.

This isn’t a patently “what if” scenario. Last year, Yahoo, who has a habit of closing services, killed GeoCities—you had a GeoCities page, right?—nuking not just people’s personal pages on an individual level, but really deleting a massive archive of web history. Yahoo paid more than $3.5 billion for GeoCities just over 10 years ago. So it could happen, even to popular services—especially ones that operate under the radar, legal or otherwise, like say, Oink.CD.

They’re fragile, yeah, but bits, unlike ink on paper or brain cells, can live forever, if they’re taken care of. As we’re awash in an ever-cresting tsunami of data, sometimes it’s easy to forget that can be a pretty big if.

Thanks to Beth, Martha and Greg at the Library of Congress, the friendliest government employees I’ve ever talked to! Still something you wanna know? Send questions about data, Data or Reading Rainbow here with “Giz Explains” in the subject line.

Original photo from RAMAC Restoration site

Memory [Forever] is our week-long consideration of what it really means when our memories, encoded in bits, flow in a million directions, and might truly live forever.

“We use the epub format: It is the most popular open book format in the world.” That’s how Steve Jobs announced the iPad. And wow, that sounds like all the ebooks you own will just work on anything. Um, no.

The idea of an open ebook format that works on any reader sounds nice. Buy it from any source, read it on any device. In a few cases, it’s true, and that open format thing can work for you. But, in reality, right now? You’re pretty much going to be stuck reading books you buy for one device or ecosystem in that same little puddle, thanks to DRM. And well, Amazon.

The Hardware

Okay, so the easiest way to put this in perspective is to quickly list what formats the major ebook readers support. (Why these four? Well, they’re the ones due to sell over 2 million units this year, except for Barnes & Noble‘s, which we’re including as a direct contrast to Kindle just because.)

• Amazon Kindle: Kindle (AZW, TPZ), TXT, MOBI, PRC and PDF natively; HTML and DOC through conversion
• Apple iPad: EPUB, PDF, HTML, DOC (plus iPad Apps, which could include Kindle and Barnes & Noble readers)
• Barnes & Noble Nook: EPUB, PDB, PDF
• Sony Reader: EPUB, PDF, TXT, RTF; DOC through conversion

You’ll notice a pattern there: Everybody (except for Amazon) supports EPUB as their primary ebook format. Turns out, there’s a good reason for that.

EPUB, the MP3 of Book Publishing

The reason just about every ebook uses EPUB is because the vast majority of the publishing industry has decided that EPUB is the industry standard file format for ebooks. It’s a free and open standard, based on open specifications. The successor to Open eBook, it’s maintained by the International Digital Publishing Forum, which has a pretty lengthy list of members, both of the dead-tree persuasion (HarperCollins and McGraw Hill) and of the technological kind (Adobe and HP). Google’s million-book library is all in EPUB too.

It’s based on XML—extensible markup language—which you see all over the place, from RSS to Microsoft Office, ’cause it lays out rules for storing information. And it’s actually made up of a three open components: Open Publication Structure basically is about the formatting, how it looks; Open Packaging Format is how it’s tied together using navigation and metadata; and Open Container Format is a zip-based container format for the file, where you get the .epub file extension. When you toss those three components together, you have the EPUB ebook format.

While we’ve only see EPUB on black-and-white e-ink-based readers so far, like Sony’s Readers or the B&N Nook, the capabilities of the file format go way “beyond those types of things,” says Nick Bogaty, Adobe’s senior development manager for digital publishing. Unlike PDF, which is a fixed page, EPUB provides reflowable text, a page layout that can adjust itself to a device’s screen-size. With EPUB, content producers can use cascading style sheets, embedded fonts, and yes, embed multimedia files like color images, SVG graphics, interactive elements, even full video—the kind of stuff Steve promised in the iPad keynote. So, we haven’t seen the full extent of EPUB’s capabilities, and won’t, until at least April 3 and presumably much later. Even if the books you buy from Apple iBook store worked on other devices—and as you will soon see, there’s little chance of that—don’t count on the coolest stuff, like video, to be somehow compatible with current-generation black-and-white e-ink readers.

D-D-D-DRM!

But let’s not get too excited seeing the words “free” and “open” so much in conjunction with EPUB. It’s like MP3 or AAC, and not only because it’s become a semi-universal industry standard. Make no mistake, these files can be totally unencrypted and unmanaged, or they can be wrapped up in any kind of digital rights management a distributor wants.

So far, according to Bogaty, the DRM every EPUB distributor currently uses is Adobe Content Server, which conveniently also wraps around PDF files. Sony and Barnes & Noble both use it on their readers, though since Adobe’s DRM doesn’t allow for sharing books between accounts, B&N actually uses a slightly custom version, and manages the Nook’s lending feature using their own backend. (Adobe is working on a sharing provision.) It does, however, support expiration, which is how Sony’s vaunted library lending feature works.

The plus side of all this compatability that it’s actually possible to move files from a Sony Reader to a Nook, using Adobe Digital Editions to authorize the transfer. (Though according to some reviewers, that would be like moving pelts from a dead horse to a rotting bear.)

Apple, on the other hand, chose EPUB as the preferred file format, but will be wrapping DRM’d files from its iBooks Store in the FairPlay DRM, which is used to protect movies and apps (and formerly music) in the iTunes Store. As always, expect them to be the only company using it.

(There’s a precursor to EPUB’s dilemma: Audible downloads. You can buy Audible audiobooks from an enormous number of sources, but the ones you buy from iTunes aren’t going to play on any other Audible-capable device, no matter how many logos they slap on the box.)

You may be thinking that it’s just a matter of time before ebook stores all go DRM free. That would be wishful thinking at best. While ebooks might seem a lot like digital music circa 2005, you can’t rip a book, so the only way to get a bestseller on your reader is to buy it legally, or to steal it. It’s pretty much that simple. There will be free books, there will be unencrypted books, and the torrents will rage with bestsellers (as they already do). Still, DRM’s gonna be a hard fact of life with every major bookstore, since they’re going to at least try to keep you from stealing it. You don’t see Hollywood giving up DRM, do you?

Kindle, Barnes & Noble, and How The Dead PDA Business Affects the Live Ebook War

Did you know that Amazon owns Mobipocket, which mainly targeted ebooks for PDAs and smartphones, and had its own file format that with roots in the PalmDOC format? The Mobipocket format, consequently, has two extensions: .mobi and .prc. I bring it up, not because you should care about Mobipocket—you really shouldn’t—but because the Kindle’s preferred AZW format is actually a very slightly modified version of MOBI, which is why it’s easy to convert files from one format to the other. Unprotected AZW files can be renamed to the MOBI or PRC format and simply work with MobiPocket readers.

The problem with Mobipocket is that it’s not a very capable format, since it was originally designed for ancient-ass PDAs and all. So there’s another special Amazon format that’s a little more mysterious, called Topaz, which is more capable than MOBI, with powers like the ability to have embedded fonts. It’s used for fewer books, and carries the file suffix .tpz or .azw1. For what it’s worth, some people complain books in the Topaz format are less responsive than the standard AZW files. In truth, none of this may matter if and when the Super Kindle arrives.

In terms of DRM, Amazon uses its own DRM on both formats. Both have been cracked, though it apparently took longer with Topaz. This may be good news for pirates, but matters not at all from a cross-platform point of view, since that format is completely proprietary, and nothing but the Kindle or Kindle software will read it anyway.

But the old PDA legacy crap doesn’t stop with Amazon. Palm once owned its own ebook platform, which it sold to a company who called it eReader. Eventually, the format and the software platform came to be owned by Barnes & Noble. I’m only dragging you into this because Barnes & Noble actually still sells many books in this format, even while they transition to the more popular and “open” EPUB format. You can spot an eReader format because the file ends in .pdb—but you only see that after you bought the damn thing. That is to say, even if you care enough about formats to go with the reader that supports the one you like, you still might get stuck with a limited, if not completely proprietary, stack of books.

PDF, I Still Love You

In comparison to EPUB, PDF is simple. Developed over 15 years ago by Adobe, the portable document format has been an open standard since 2008. You’re probably pretty damn familiar with it, but the main thing about it versus these other formats is that everything is fixed—fonts, graphics, text, etc.—so it looks the same everywhere, versus the reflowable format that adjusts to the screen size. Hence, Amazon offers PDF without zoom on its Kindle DX, which has the screen real estate to (usually) not muck it up too much. With smaller screens than the PDF’s native size, it requires some pan-and-zoom voodoo, and it still usually looks pretty disgusting.

Zoom issues notwithstanding, having a fixed format has advantages. For instance, a lot of “electronic newspapers” were transmitted via PDF back in the day, because it retained their design. It’s really nice for comics. (Consequently, you can bet scanned-comic piracy to explode when the iPad arrives, unless Marvel and DC come up with killer strategies to get their comics on a device that’s clearly begging for it.) Wikipedia covers a lot of the technical ground, surprisingly thoroughly, even if the usual Wiki caveats apply. As mentioned above, it can be protected with Adobe Content Server DRM, just like EPUB.

The Great Shiny Hope: Apps

The other path for digital publishers: Build an app to hold your books and magazines. This is the route magazines are taking, because they’re envisioning some fancy digital jujitsu. With Adobe AIR, which is what Wired and the NYT are using in various incarnations for their respective rags, they’re able to do more advanced layouts, more rich multimedia, Flash craziness, and other designer bling that EPUB can’t handle, says Adobe’s Bogarty. Also, importantly you can dynamically update content, like when new issues arrive, which you can’t really do with EPUB.

Interestingly, the publisher Penguin is also taking the app route for their books, building apps using web technologies like HTML5 for the iPad, so their books are in fact, way more like games and applications than mere books. So it’s another tack publishers could take.

But the app business can help with the openness of the big ebook file formats, too. Many people read Amazon’s proprietary formats on their iPhone, because Amazon wants to sell books, and Apple wants people to use apps. Barnes & Noble has a reader app, too; while not great, it at least somewhat helps get over the PDB/EPUB confusion. It’s pretty likely that these and many other ebook apps will turn up on the iPad, unless Jobs decides that they “duplicate” his “functionality.” Since iBooks itself is an app you have to download, it probably won’t be an issue. Here’s hoping.

The Upshot

The idea of an open ebook format that works on any reader sounds really nice. And in some cases, if you pay really really close attention, it’s true. That open format thing actually can work for you. But the reality? You’re pretty much going to be stuck with the books you buy in one device working only in that same ecosystem, or at least hoping and praying for an assortment of proprietary reader apps to appear on all your devices. Now, where’d I put that copy of Infinite Jest? Was it in my Kindle library, my B&N library or my iBooks library?

Still something you wanna know? Send questions about ebooks, bookies or horse heads here with “Giz Explains” in the subject line.

The Bloom Box is the latest energy miracle that sounds too good to be true: Debuting with a wide-eyed segment on 60 Minutes, it promises to be clean, cheap and backyard-friendly, the solution to our energy problems. What is it?

The heart of the box is a fuel cell. Though Bloom Energy‘s CEO K.R. Sridhar—a former NASA scientist—says it’s a new kind of fuel cell. And though it’s cleaner than any combustion engine out there, it still relies on fossil fuels and biofuels—not just hydrogen, like some other kinds of fuel cells do. Nevertheless, the folks at Bloom are doing something that could help make reduced emissions a reality for big businesses first, and then later, for homes.

To get a good grip on why we should care about this thing, let’s first look at the basics of fuel cell technology.

Fuel Cell Basics

Like a battery, a fuel cell is an electrochemical cell, basically meaning it derives electricity from chemical reactions. Sandwiched between two electrodes—an anode and a cathode—is an ion-conducting material called an electrolyte. Fuel flows in one side, over the anode. An oxidant flows into the other side, over the cathode. What happens, very basically, is that the fuel and the oxidant react, like strangers locking eyes across a room. The metaphorical sparks that fly from that encounter are actual electrons, which flow into the fuel cell’s circuit. Bingo, electricity. As with any molecular reaction, the recombination of atoms produces some waste as well—like water or carbon dioxide. So while it’s cleaner, there’s definitely a byproduct.

To be clear, a fuel cell’s not like a battery; it’s like a power plant. Once it converts fuel to energy, it sends that energy out the door. And as such, it requires some peripheral way to physically storing the fuel ingredients, and some way to capture produced electricity—such as a battery.

There are a several different kinds of fuel cells—unsurprisingly, since they were invented in the 1830s. Generally, they are categorized based on what their electrolyte is made out of, but sometimes they’re referred to by their fuel and oxidant, which varies too. You’re probably most familiar with “hydrogen fuel cells,” like for cars and small electronics. These are in fact proton exchange membrane fuel cells, which happen to use hydrogen as a fuel and oxygen as an oxidant. (The PEM fuel cell is what is specifically diagrammed above.)

Solid Oxide Fuel Cells

Bloom Energy’s Energy Servers are of the solid oxide variety of fuel cell. There’s two ways to do up an SOFC: A tubular design, which you can see above, or a planar design, which is what Bloom uses, as you can see below, since it allows them to be stacked into very neat boxes.

A solid oxide fuel cell is made out of all solid state materials—that is, every major component is made out of ceramic-like stuff. Bloom Energy claims their fuel cells are made out of “sand” baked into ceramic squares, and that’s just what an SOFC is. The exact material is a slightly secret sauce as are the black and green “inks” that coat the ceramic plates. Bloom’s got a pretty nice little Flash animation showing the basic process.

The major thing about an SOFC versus other fuel cells is that the material composition means they can run crazy hot—up to 1800ºF, says the US Department of Energy—and have to, since the ceramic materials don’t become active until they reach a certain temperature. Only at this temperature can they perform the chemical reactions with the fuel and oxidant we talked about above. The problem with the high operating temperatures is that traditionally it has lead to higher maintenance costs. You know, stuff breaks down. The goal for this technology is to have an “uptime” of 99.99%, as cited by cited by Scott Samuelsen, who’s the director of the National Fuel Cell Research Center at the University of California-Irvine. Bloom’s own trial at Google cites a 98% uptime.

The types of fuel cells you hear more about—the “methanol” ones that can already power laptops—do their business at a much lower temperature. Toshiba has one that typically runs at 120º to 200ºF. Though Bloom’s is obviously not a tech that could be a laptop’s power source, the Bloom Box’s higher operating temperature is a big advantage over “legacy” fuel cell technology. Bloom Energy VP of Marketing and Products Stu Aaron told me it gives them “fuel flexibility.” They can use biogases from land waste or fossil fuels like propane—so far in demos it’s been an even split between biogases and natural gas—whereas low-temp fuel cells require hydrogen in a much purer state that has to basically be refined or extracted via chemical processes.

While some other SOFCs use the hot exhaust generated by the reaction kind of like a cogen—a means of capturing heat emitted by a power generator, so that it too can be converted to electricity—Bloom’s Energy Servers simply recycle the heat within the cell, since the temperature generated by the reaction is almost exactly the heat needed for the reaction to happen. The rated efficiency spec for their current energy server is greater than 50%, compared to around 10% to 15% for solar (though University of Delaware-led researchers did recently hit a world’s record of 42.8% for solar).

Again, to be clear, the energy generated isn’t emission-free: These servers generate a small amount of CO2 when converting natural gas or bio-gas. It is less than what would get released if the same fuel was combusted, however. Customers can pick which of the two kinds of fuel they’d like to use; the trade off is between “optimizing for cost or carbon reduction,” depending on the company’s priorities, says Aaron.

Electricity In Bloom

Right now, the only box that Bloom is selling is a 100-kilowatt-hour energy server, which you can check out there. Inside are thousands of solid oxide fuel cells—each one able to power a light bulb. The cells are arranged in stacks, which are aggregated into modules, and so on, with a common fuel input. Right now, they’re just for corporations—like Google and Coca-Cola—and run about $700,000 to $800,000 each. The goal’s to get them down to three grand, where they’d be suitable for home use. That may still sound expensive, but they pay for themselves in 3-5 years, says Aaron, with an energy cost of 8-9 cents per kW hour vs. the 13-14 cents it typically costs in California. (It saved eBay $100,000 on their power bill.)

But cost is where the real skepticism comes in. Fuel cells aren’t a voodoo technology. They work. They produce energy. What analysts, and others, are wondering is whether Bloom’s really cracked the secret to making them cheap, at least some day. The critic that CBS trotted out on 60 Minutes, Green Tech Media’s Michael Kanellos, says that while there’s a 20 percent chance we’ll have a fuel cell box in our basements in 10 years, but “it’s going to say GE.” Which is fine with me, actually, because that means another season of 30 Rock jokes.

Still something you wanna know? Send questions about fuel cells, terrorist cells or Boom Blox here with “Giz Explains” in the subject line.

In 1975, the first digital camera took 23 seconds to record a 100-line black-and-white photo onto cassette tape. Today, a Nikon D3s takes photos with 12 million pixels at 1/8000 of a second. And it can see in the dark.

The conventional wisdom is that the romp-stomp-stomp of progress in digital imaging has proceeded on the mostly one-way track of ballooning pixel counts. Which wasn’t always a pointless enterprise. I mean, 1.3-megapixel images, like you could take in 1991, aren’t very big. The Nikon D1, introduced in 1999, was the digital camera that “replaced film at forward-looking newspapers.” It was $5,000 and shot 2.7 megapixel images using a CCD sensor, large enough for many print applications. But still, there was room to grow, and so it did. Now pretty much every (non-phone) camera shoots at least 10-megapixel pictures, with 14 megapixels common even in baseline point-and-shoots. Cheap DSLRs from Canon are now scratching 18MP as standard. Megapixels were an easy-to-swallow specification to pitch in marketing, and became the way normal people assessed camera quality.

The now-common geek contrarianism is that more megapixels ain’t more better. The new go-to standard for folks who consider themselves savvy is low-light performance. Arguably, this revamped arms race was kickstarted by the D3, Nikon’s flagship DSLR that forsook megapixels for ISO. (Rumor had it that the D3 and D300 led Canon to shitcan their original, middling update to the 5D, pushing full-steam-ahead for a year to bring us the incredible 5D Mark II.) However it began, “amazing low-light performance” is now a standard bullet point for any camera that costs more than $300 (even if it’s not true). Nikon and Canon’s latest DSLRs have ISO speeds of over 100,000. Welcome to the new image war.

How a Camera Sees

The name of the game, as you’ve probably gathered by now, is collecting light. And in fact, the way a digital camera “sees” actually isn’t all that different from the way our eyeballs do, at one level. Light, which is made up of photons, enters through a lens, and hits the image sensor (that boring looking rectangle above) which converts it into an electrical signal, sorta like it enters through an eye’s lens and strikes the retina, where it’s also converted into an electrical signal. If nothing else after this makes sense, keep this in mind: The more light an image sensor can collect, the better.

When a camera is spec’d at 10 megapixels, it’s not just telling you that its biggest photos will contain about 10 million pixels. Generally, it’s also telling you the number of photosites, or photodiodes on the image sensor; confusingly, these are also often referred to as pixels. Photodiodes are the part of the sensor that’s actually sensitive to light, and if you remember your science, a photodiode converts light (photons) into electricity (electrons). The standard trope for explaining photosites is that they’re tiny buckets left out in a downpour of photons, collecting the light particles as they rain down. As you might expect, the bigger the photosite, the more photons it can collect at the moment when it’s exposed (i.e., when you press the shutter button).

Image sensors come in a range of sizes, as you can see in this helpful diagram from Wikipedia. A bigger sensor, like the full-frame slab used in the Canon 5D or Nikon D3, has more space for photosites than the thumbnail-sized sensor that fits in little point-and-shoots. So, if they’re both 12-megapixels, that is, they both have 12 million photosites, the bigger sensor can obviously collect a lot more light per pixel, since the pixels are bigger.

If you’re grasping for a specification to look for, the distance between photosites is referred to as pixel pitch, which roughly tells you how big the photosite, or pixel, is. For instance, a Nikon D3 with a 36mm x 23.9mm sensor has a pixel pitch of 8.45 microns, while a Canon S90 point-and-shoot with a 7.60 mm x 5.70 mm sensor has a pitch of 2 microns. To put that in less math-y terms, if you got the same amount of light to hit the image sensors the D3 and the S90—you know, you took the exact same exposure—the bigger pixels in the D3 would be able to collect and hold on to more of the light. When you’re looking for low-light performance, it’s immediately obvious why that’s a good thing.

Catch More Light, Faster, Faster

Okay, so that’s easy enough: As an axiom, larger photodiodes result in more light sensitivity. (So with the 1D Mark IV, Canon kept the same photodiode size, but the shrunk the rest of the pixel to fit more of them on the same-size chip as its predecessor). There’s more to an image sensor than simply photosites, though, which is why I called up Dr. Peter B. Catrysse from the Department of Electrical Engineering at Stanford University. The “ideal pixel,” he says, would be flat-just an area that collects light-nearly bare silicon. But even at a basic level, a silicon photodiode sits below many other structures and layers including a micro lens (which directs light onto the photodiode), a color filter (necessary, ’cause image sensors are in fact color blind) and the metal wiring layers inside each pixel. These structures affect the amount of light that the photodiode “sees.” So one way manufacturers are improving sensors is by trying to make all of these structures as thin as possible-we’re talking hundreds of nanometers-so more light gets through.

One major way that’s happening, he says, is with back-illuminated sensors, which move the wiring to the back-side of the silicon substrate, as illustrated in this diagram by Sony. It’s currently still more expensive to make sensors this way, but since more light’s getting through, you can use smaller pixels (and have more of them).

In your basic image sensor construction, there’s an array of microlenses sitting above the photosites to direct light into them. Previously, you had gaps between the microlenses, which meant you had light falling through that wasn’t being directed onto the actually light-sensitive parts of the sensor. Canon and Nikon have created gapless microlenses, so more of the light falling onto the sensor is directed into the diode, and not wasted. If you must persist with the bucket metaphor, think of it as putting a larger funnel over the bucket, one that can grab more because it has a wider mouth. Here’s a shot of gapless microlens architecture:

A chief reason to gather as much light as possible is to bring up your signal-to-noise ratio, which is the province of true digital imaging nerds. Anyways, there are several different sources and kinds of noise. Worth knowing is “photon shot” or just “shot” noise, which occurs because the stream of photons hitting the image sensor aren’t perfectly consistent in their timing; there’s “read” noise, which is inherent to image sensors; and “dark current” noise, which is basically stray electrons striking the sensor that aren’t generated by visible light—they’re often caused by heat.

Taken with a Nikon D3s at ISO 102,400
Back in the day, when people shot photographs on this stuff called film, they actually bought it according to its light sensitivity, expressed as an ISO speed. (A standard set by the International Organization for Standardization, confusingly aka ISO. The film speed standard is ISO 5800:1987.) With digital cameras, you also can tell your camera how sensitive to light it should be using ISO, which is supposed to be equivalent to the film standard.

The thing is, whether you’re shooting at ISO 100 or ISO 1600, the same number of photons hit your sensor—you’re just boosting the signal from the sensor, and along with it, all the noise that was picked up on the way. If you’ve got more signal to work with—like in a camera whose sensor has some fat photon-collecting pixels, you get a higher signal-to-noise ratio when you crank it up, which is one reason a photo taken D3 at ISO 6400 looks way better than one from a teeny point-and-shoot, and why a 1D Mark IV or D3s can even think about shooting at an ISO of over 100,000, like the photo above. (Another reason is that a 1D Mark IV-level camera possesses vastly superior image processing, with faster processors that can crunch complex algorithms to help reduce noise.)

Sensor Shake and Bake

There are two kinds of image sensors that most digital cameras use today: CCD (charge-coupled device) sensors and CMOS (complementary metal-oxide-semiconductor) sensors, which are actually a kind of active-pixel sensor, but the way they’re made have become a shorthand name. “Fundamentally, at least physics-wise, they work exactly the same,” says Dr. Catrysse, so one’s not intrinsically more awesome than the other. CCD sensors are the more mature imaging technology. So for a long time, they tended to be better, but now CMOS sensors are taking over, having almost completely crowded them out of cellphones and even high-end DSLRs (Leica’s M9 is an exception). Dr. Catrysse suspects CCD sensors will be around for some time, but perhaps more likely in scientific and niche applications where high-level integration, speed and power usage are less of an issue as compared to mainstream mobile applications.

A “CMOS sensor” is one that’s made using the CMOS process, the way you make all kinds of integrated circuits—you know, stuff like CPUs, GPUs and RAM—so they’re actually cheaper to make than CCD sensors. (The cheap-to-make aspect is why they’ve been the sensor of choice in cameraphones, and conversely, DSLRs with huge chips.) And, unlike a CCD sensor, which has to move all of the electrons off of the chip to run them through an analog-to-digital converter, with a CMOS sensor, all of that happens on the same integrated chip. So they’re faster, and they use less power. Something to think about as well: Because they’re made pretty much the same way as any other semiconductor, CMOS sensors progress along with advances in semiconductor manufacturing. Smaller transistors allow for more circuits in a pixel and the potential to remove more noise at the source, says Dr. Catrysse, bringing us closer to fundamental physical limits, like photon noise, and performance that was once the prerogative of CCD sensors. And then we’re talking about using small features in advanced semiconductor manufacturing technology for controlling light at the nanoscale.

The Point

We’ve reached, in many ways, a point of megapixel fatigue: They’re not as valuable, or even as buzzy as they used to be. Not many of us print billboard-sized images. But the technology continues to progress—more refined sensors, smarter image processors, sharper glass—and the camera industry needs something to sell us every year.

But that’s not entirely a bad thing. Our friend and badass war photographer Teru Kuwayama says that while “increasing megapixel counts are mostly just a pain in the ass, unless you happen to be in the hard drive or memory card business, skyrocketing ISOs on the other hand, are a quantum leap, opening up a time-space dimension that didn’t exist for previous generations of photographers. I’d happily trade half the megapixels for twice the light sensitivity.”

Better images, not just bigger images. That’s the promise of this massive shift. The clouds to this silver lining are that by next year, ISO speeds will likely be the headline, easy-to-digest spec for consumers. And like any other spec, just because the ISO ratings go higher doesn’t mean low-light performance will be better. Remember, “more” isn’t more better.

Still something you wanna know? Send questions about ISO, isometric exercise or isolation here with “Giz Explains” in the subject line.

The beardier parts of the web-o-sphere have been abuzz about HTML5, the next version of the language that powers our internet. Will it revolutionize web apps? Will it kill Flash video? Will it fix our gimpy iPads? Yes… and no.

The tech press has transformed HTML5 from a quiet inevitability to an unlikely savior: When YouTube and Vimeo started testing it, it’s was invoked as a Flash-killer, and the emancipator of web video. When Google used it to design a new Google Voice web app, among others, it was framed as the murderer the of the OS-specific application. When the iPad was announced with no Flash support, HTML5 was immediately pegged as a salve, not to mention a way to get around the “closed system” of Apple’s App Store.

It doesn’t take much imagination to draw these stories into an appealing narrative about how the app-less, plugin-free, totally web-based future is just a browser update away. The thinking goes, somewhere in this impenetrable 125,000-word published standard, you’ll find the answer to the internet‘s every ailment: its clunky, proprietary plugins, its stunted web apps, its fundamental shortcomings as a platform for rich media. At the heart of each of these theories lies a grain of truth, but none of them are totally—or even mostly—true.

Here’s what’s really going on. HTML 5 is already working its way into the underpinnings of web apps you use every day, making them faster and more stable than those relying on Java or other plugins. They’re more like real apps. It’s helping us inch closer to the dream of having real applications available at all times, on any platform.

HTML is also setting forth a vision of media—specifically video—that doesn’t rely on crashy, resource-intensive proprietary plugins. Look in your plugins folder, you will probably see four video plugins at a minimum. HTML is a standard with an optimistic view of the future: You launch your browser, and whatever site you visit, whatever media you choose to play, your browser just magically supports it, without the frustration, confusion and added instability of a plug-in.

But at heart HTML is just a framework, a glimpse, and an ideal: Its real effect on the internet continues to be defined by the companies and web developers who choose to adopt its many pieces—and it is further shaped by those who don’t.

The Basics

Before we get into what HTML5 means, we have to talk about what it is, and to talk about what it is, we need to talk about what it’s built upon.

Hypertext markup language, or HTML, is the language underneath every web page you’ve ever been to. The language, along with its various complementary technologies (see: CSS, Javascript), has become immensely complex over the years, but the concept is simple. HTML is what turns this:

<u><em><strong><a href=”http://gizmodo.com”>Hello!</a></strong></em></u>

Into this:

Hello!

It’s basically a set of instructions that a website hands to a browser, which the browser then reads and converts into a formatted page, full of text, images, links and whatever else.

Here, try this: Right-click anywhere on this webpage, and click “View Page Source,” or “View Source,” or something to that effect. Your eyes will be assaulted with a wall of inscrutable text. You’ll see evidence of syntax, but your brain won’t be able to parse it. Your eyes will glaze over, and you will close the window. This, my friends, is HTML. But you probably already knew that, because it’s 2010, basic web languages are basically in our drinking water. So what’s this “5″ business?

Somewhere in the central command center basement of the internet, there’s a group of guys who maintain the standard, or the rules, of HTML. In the case of HTML5, the buck stops with the Web Hypertext Application Technology Working Group (WHATWG), and to a lesser extent, the World Wide Web Consortium (W3C). It is through these independent standards organizations that new features are codified and presented to the public, and later—in theory—supported by various browsers, no matter what company is behind them.

In the early nineties, the W3C and a few influential torchbearers would collect various new web features thought up by different browser makers, publishing these standards with the hope that we didn’t end up with different internets for different browsers. By the mid to late nineties, the standards had grown in both size and stature, then serving as the de facto guide for browser makers and developers alike. (If this sounds a bit rosy, the reality was far grimmer—just ask any seasoned web developer about Internet Explorer, version 6 or earlier.)

Despite an occasionally rocky road, HTML standards went beyond being just a record of changes in web technology; eventually they became the blueprint to push them forward. Still, standards are guides, not laws, and no browser maker has to adopt each and every revision.

The last major revision of the HTML standard, version 4.01, was published in 1999. HTML5 hasn’t yet been formally codified, but it was born in 2004 and has been undergoing steady work and maintenance since. In the ’90s, HTML discussion centered around topics like font coloration, or tables, or buttons, or something more esoteric. Today, a new HTML version means deep-down support for the modern web, namely web apps and video.

The New Features

The HTML5 spec is more than just new tags and tools, but for users and developers, they’re what matter most. Specifically, I’m talking about APIs, or application programming interfaces. It’s because of these APIs (usually manifested as tags like <VIDEO> or <IMG>) that we’ll soon be treated to a richer internet. And it’s because of these APIs that when work on HTML5 started, it was called “Web Applications 1.0.” Today, if you pick apart HTML5, these are the biggest pieces:

• Video. If you watch video on the internet, you’re watching it through a plugin—a piece of software that works within your browser, but which isn’t technically a part of it. A decade ago, this plugin may have been clunky RealPlayer software, semi-reliable Windows Media Player controls, or a QuickTime plugin that you were better off skipping altogether. Today, it’s probably Flash or Microsoft Silverlight, or a newer, subtler Quicktime or Windows Media plugin. Whether you’re playing a YouTube movie embedded on a web page, or just viewing a .mov file as you download it, your browser has to use the plugin.

HTML5 includes support for a simple tag that lets developers embed video in a page just like they’d embed a JPEG or other image, with a pointer to a file on a server. Packed along with the ability to read that video tag are a few rendering engines, which would decode the video without any kind of plugin. Embedding a video with HTML5 is as easy as embedding an image, provided the video codec is compatible with the browser’s rendering engine. In terms of code, it can be as simple as this:

<video src=”video.mp4″ width=”320″ height=”240″></video>

Boom. Video. Here’s what some of the current rudimentary players look like:

-SublimeVideo (Safari 4, Chrome)
-YouTube (Safari 4, Chrome)
-Vimeo (Safari 4, Chrome)
-DailyMotion (Firefox, Safari 4, Chrome, Opera)

In theory, eliminating the video plugins means no extra CPU overhead, fewer crashes, and wider compatibility—if HTML 5 video was standard now, we wouldn’t be stuck waiting for Adobe to port their plugin to our mobile phones, and Mac users wouldn’t bring their systems to a crawl every time they tried to watch a YouTube video in HD. As a general rule, playing a video file through an extra plugin like Flash is going to be slower, buggier, and more resource-intensive than playing it through a browser’s native decoder. That’s why people are excited about HTML5 video.

• Offline storage: Remember Google Gears? It was a set of plugins for various browsers that let web apps, like Gmail or Zoho Writer (an online text editor), store content locally on your computer, so they could behave more like native apps. Gmail, for example, could then work without an internet connection. It wouldn’t retrieve your new emails while offline, obviously, but it’d at least have a working interface and a database of your old emails, just like Outlook or Mail.app would. Well, Google abandoned Gears, because HTML5 basically supports the same thing, again, without a plugin.

-Here’s a basic demo (Firefox 3.6, Safari 4, Chrome, Opera)
-And a more complex one, including lots of other tricks (Firefox 3.6, Safari 4, Chrome, Opera)
-Or, try Gmail on your iPhone or Android phone

• Drag-and-Drop Elements, and Document Editing. You know how you can drag and drop emails in Gmail? And how you type into text boxes, to post or send everything from Tweets to emails to forums posts? As it stands, these systems are built on a delicate, complicated stack of ad-hoc code tricks, which have worked fine up until now, but which could stand to be simplified. Even if you’re not a developer, just know that this, in theory, translates to increased stability. And that’s exactly what HTML5 proposes: Super-simple implementations of editable documents boxes, drag-and-drop page elements, and drawing surfaces.

-A helpful, ugly demo(Firefox 3.6, Chrome, Safari, Opera)
-And an exceedingly pretty one(Firefox 3.6, Chrome, Safari

• Locations services. Now a web app can tell where you are, if you choose to let it. Here‘s how that works. (Firefox 3.6, Chrome, Safari 4, Opera, iPhone)

There’s a clear trend here. HTML5 is about video, and it’s about far more stable yet complex web apps. These are the sources of excitement right now, but they’re also the sources of confusion.

Hopes and Dreams

On the desktop, the transition to HTML5 will be largely seamless, though you’ll notice an uptick in the quality, speed and richness of some apps you use all the time—think webmail, document editors, and text entry applications for starters. On mobile, the results will definitely be more pronounced. Remember Google’s new Voice web app for the iPhone and Pre? Take away the browser controls, and it’s almost indistinguishable from a native app.

The hope—and it’s a realistic one—is that certain categories of web apps will supplant native apps. The advantages are obvious: If your document editor is online, it’ll work consistently whether you’re on an iPad or a Windows desktop; if your email client is a website, your messages are always available, and your read/unread status is always in sync. Web apps like Google Documents will get faster, more consistent, and more universally compatible. Still, you’re not going to see Photoshop or Final Cut in your browser window anytime soon. If this dream sounds familiar, it’s because it’s very old, and already realized in many ways: Ancient services like Hotmail mark its genesis, and the app-less Chrome OS is its eventual, if limited, endpoint.

The second dream, and the one you’ve probably been hearing the most about lately, is that HTML5 video could kill Flash. As in, render Adobe’s plugin, which most internet-connect computers already have installed, completely obsolete, simultaneously making Apple’s iPad and other mobile devices more capable of getting at all the media the web has to offer.

Vimeo, DailyMotion and YouTube (YouTube!) have all recently launched pilot programs for HTML5 video technology. On the surface this is very exciting. Their players are basic, but they work, and there are some rather spectacular demos of more advanced HTML5 video players doing the rounds right now. The latest builds of the WebKit rendering engine, which comprises the guts of both Mac OS and iPhone/iPad (mobile) Safari, Google’s Chrome OS, the Pre’s browser and the Android browser, among others, support full-screen HTML5 video. The iPad notoriously won’t ship with Flash, but Apple’s desktop (Mac OS) Safari is one of the first browsers to fully support the HTML5 video discussed here, the natively rendered video used by YouTube and Vimeo in their tests. So the stars are aligning for an HTML5 video takeover, right? No, they’re really not.

Managing Expectations

As I mentioned, the WHATWG and W3C can publish as many standards as they want, but in order for any to actually matter, browsers have to support them—and by browsers, I mean all major browsers, from nimble, rapidly-developed apps like Opera and Chrome to Internet Explorer, which, by the way, is still globally the most popular dashboard to the internet. Take the <VIDEO> tag as an example: Safari and Chrome do support it, both the HTML code and the native rendering of a couple of associated video formats. Firefox supports the tag, but doesn’t support decoding of the key video format currently used by YouTube and Vimeo. Internet Explorer doesn’t support it at all without a plugin, and isn’t the whole point of HTML5 to get rid of plugins?

Just as different browsers update their rendering engines at different speeds, users of browsers update their software even less predictably, and some don’t update at all. Despite Microsoft’s aggressive IE8 evangelism, IE6 was only just bumped from being the Number One browser in the world. It was released in 2001, when HTML 4 was just learning to walk and HTML5 was but a glint in the W3C’s eye. IE6 will never work with HTML5 video. But it plays video just fine with Flash.

Even on the cutting edge, there are serious roadblocks to widespread adoption of HTML5 video, the most dangerous being video codecs. Because HTML5 supports video embedding natively, browsers will have to be able to decode embedded video files in lieu of the plugin that use to do it for them. The current working HTML5 standard doesn’t explicitly define a video format to be used with the tag—and as luck would have it, there are now two formats vying for the job.

• Ogg Theora is a free codec standard—free as in open source—which most browsers that support HTML5 video support right now. It’s an attractive option on paper, because browser companies don’t have to pay any licensing fees to include the ability to decode it in their software. The trouble is, it’s notoriously inefficient, and, perhaps because of this, it’s not too popular. Google’s standards guru Chris DiBona infamously said:

If [YouTube] were to switch to Theora and maintain even a semblance of the current quality, it would take up most available bandwidth across the internet.

True or not, as a codec standard Ogg Theora isn’t gonna cut it, even though from a business point of view, it’s ideal.

• h.264 video suffers from pretty much the opposite situation. Based on a codec standard that’s natively supported in many mobile phones, it’s what Vimeo and YouTube are running in their respective experiments. These video sites’ already store their mobile-quality libraries in h.264—what do you think streams to your iPhone YouTube app, since Flash isn’t supported? So enabling h.264 streaming is as simple as developing a player interface, which takes no time and even less resources. It’s also efficient—that’s why it’s popular in the first place. One problem though: It’s proprietary.

Yes, if you want to build a browser that plays back h.264-based video with HTML5, you need to be prepared to pay millions of dollars to the companies that own the format’s patents. Beyond the basic cost issue, some deem it risky to put the internet’s entire video ecosystem into the hands of some obscure rightsholders, whose whims could change down the road. (Who, exactly? These guys!)

Google and Apple have so far been okay with the royalties, but Mozilla, creator of Firefox, is taking a more conservative longview. As Mozilla’s Chris Blizzard insists, there’s a precedent for these worries:

Because it’s still early in H.264′s lifespan it’s extremely advantageous to lightly enforce the patents in the patent pool. MP3 and GIF both prove that if you allow liberal licensing early in a technology’s lifespan, network effects create much more value down the road when you can change licenses to capture value created by delivering images and data in those formats. Basically wait for everyone to start using it and then make everyone pay down the road.

So, while h.264 is a shoo-in for the job, it would probably be unbelievably perilous to sign it up.

If this seems like a lot to digest, don’t worry! Despite the thousands of urgent words spilled on this subject, it doesn’t really matter. Flash is here for a while, because nobody can get their act together.

First let’s talk about DRM, a sore subject, but something you can’t not talk about. Flash video supports it. HTML5 video doesn’t, as it stands. Could you imagine a Hulu on which every video is a right-click away from saving to your computer? A Netflix where you keep what you stream? I mean, sure, you can imagine this, but there’s not enough Tums in Los Angeles for Hollywood execs to stomach that discussion. No DRM, no movies or TV shows. Simple as that. And if the fight over a basic HTML5 video standard is fraught, just imagine how tough it’d be to get Mozilla, Apple, Google, Opera and Microsoft to agree on DRM.

Meanwhile, the test runs show, in reality, how little weight is being thrown behind HTML5 video at the moment. This is how YouTube describes their HTML5 initiative, which caused such a fuss last week:

In the last year our community has made it clear that they want YouTube to do more with HTML5. To meet this demand we recently rolled out HTML5 support in TestTube, a destination on YouTube where we routinely experiment with different products. Some of the products in TestTube are successful and rolled out to the wider community. Others, however don’t make it beyond TestTube. We’re still in the early stages, but our hope is to continue this active and ongoing discussion around emerging Web standards.

Can you feel the enthusiasm? YouTube’s HTML5 test is just that, a test. There’s no convincing evidence of idealistic shift in the works. YouTube’s future hinges on the ability to integrate ads into their videos, to sell access to DRM’d content, and to reach the largest audience possible. Until HTML5 video can pull this off, Google and YouTube are going to keep on doing what they’ve been doing—using Flash.

Lastly, Adobe has interests in this discussion too, and is working frantically to push Flash to virtually all mobile smartphone platforms that don’t already have it. Meanwhile HTML video tag support on smartphones is barely the discussion phases—it’s plagued with as many problems, if not more, than desktop HTML 5 video.

And we haven’t even talked about the other holes in the HTML5 Murders Flash! narrative. What about the spec’s glaring lack of ability to replace Flash’s other, non-video functions? Sure, increasing browser support for scaled vector graphics and HTML5′s Canvas tag go a short way to creating vivid, visual web applications without plugins, as does the wide array of Javascript tools already available to web developers.

But what about games? And more importantly for developers who like paychecks, what about animated, interactive ads (some which are overlaid on the aforementioned YouTube videos)? The internet’s not going to give up on those anytime soon, and the non-Flash web technologies we have now aren’t going to cut it for years.

What’s Really Going to Happen to Your Internet

As I said way back at the beginning, part of the job of an HTML spec is to codify what’s already being done by developers and browser makers. As such, there’s a very good chance that HTML5 is partially supported by your desktop browser. If you have a smartphone with a WebKit-based browser, you already use web apps that leverage the technology. This will simply become more common, in a mundane, linear way: Google, Apple, WebKit, Mozilla, Opera, and yes, even Microsoft will continue to include new features in their software, and developers will begin to leverage it as soon as they can. Web apps will get smarter, faster and more powerful, even if you don’t really notice it. You’ll worry less about having a constant internet connection, and you’ll probably install few native applications on your phone or laptop.

For the foreseeable future, video on the internet is going to remain almost exactly as-is. If anything, Flash will become more entrenched in the short term, as the YouTubes and Hulus of the world expand their catalogs with more DRM’d content, and continue building their desktop content platforms around the plugin. As for mobile devices like the iPhone and iPad, for whom Flash seems eternally out of reach, video delivery will move increasingly toward apps, which content companies can tightly control, and not toward HTML5 video, which—all other problems aside—they really can’t.

HTML5 has a place in online video, and I expect companies to continue testing it, playing with it, and expanding their uses for it. I expect browsers to continue increasing support for it—hey, maybe even mobile Safari!—but don’t stake your hopes, or a specific gadget purchase, on its immediate promise. An internet where native web languages have killed all plugins, including Flash, is just too far away to talk about coherently.

HTML5 is infiltrating the web, not tearing it down and building it back up. Like the standard itself, the HTML5 web will evolve slowly, with web technologies gradually supplanting tools you use now. You’ll notice it, but you’ll have to watch closely.

Hat tip to Lifehacker, for noticing—and explaining—the groundswell all the way back in December

Still something you wanna know? Does some other tech term have your fleshy processing unit in a tangle? Send questions, tips, addenda or complaints to [email protected], with “Giz Explains” in the subject line