UI3

Proposal

Background

Macromedia Flash has become the defacto standard for creating user interface's for the web, with a rich development environment to create an appealing UI, and an efficient player to do the rendering with lot's of eye candy and visual effects. Media centric applications with 10' user interfaces ("MCE apps") have the same needs, but there really does not exist any software that can do this for desktop applications, like Flash does for web applications.

A few MCE apps try to use Flash for their user interfaces. But this does not work very well since this isn't Flash's intended purpose. For example, the UI in a MCE app cannot be stand alone and isolated, rather it must be a UI that floats on top of whatever media the MCE app is delivering. The Flash player is not able to blend it's rendered output on top of the media application, like Xine or MythTV. So although you could create a really attractive EPG program guide in Flash, you cannot blend that guide on top of the live TV in MythTV.

Therefore, nearly all the MCE applications, like Xine, Myth, Windows MCE, etc., typically code their own UI's. This is inefficient; it is like re-inventing the wheel each time. And, most of the time, the developers are more focused on their core application, not the UI, and cannot create a rich UI engine. For example, Xine, MythTV, Mplayer, Freevo, etc., all have their own UI's, but none are rich with eye candy and visual effects. In general, only Microsoft has been able to develop a really eye-candy rich UI with lots of visual effects, and most other media center applications have fairly boring, static UI's, even if the core media application itself is rich. For example, while MythTV developers began implementing a rich OpenGL UI engine over two years ago, this is still only used for some of the menu system and had not been extended to the video overlay or plug-ins, and because there is no UI editor the capabilities of it's graphics engine go largely unused.

The goal, therefore, is to create (1) a UI player, like the Flash player, which is efficient and can render high-res, 3d, user interfaces with lots of eye candy on a variety of platforms (x86 Linux/Windows, PDA's, cell phones, etc.), and (2) a comprehensive development environment, like the one Macromedia developed for Flash, that is well documented and not overly technical so creative designers can use it to build innovate UI's.

The intention is that this can be used to create UI's for a variety of MCE apps, like MythTV, MPlayer, etc., and that unlike Flash which is designed to run as an isolated application, this UI will be designed from the ground up to bolt on top of the core MCE app. In other words, you can create a beautiful EPG, like you could with Flash, but it can be blended on top of MythTV's live tv feed and used to control MythTV. The UI tool should be modular and flexible so graphics designers can create interesting UI's without needing the help of the coders of the core MCE app. For example, a designer could create, say, an innovate 3d TV program guide on a rotating sphere and add it to MythTV without needing the MythTV developers to write new code. Having a unified UI will also allow us to address some of the vexing issues, like text entry using remote controls of various types, with new UI widgets which only need to be learned once for the entire class of 10' applications.

It is understood that this is a big, complicated task which hasn't really been done before. However, we feel it is an important gap that needs to be filled. The first thing most end users notice is the eye candy in the UI and how pretty it is, and only Apple and Windows Vista have had the resources to create such UI's. This will allow other MCE apps to have UI's that are just as catchy and the developers can focus on the core functionality.

Target user / financial model

Naturally this tool would be an asset to the FOSS MCE apps, like MythTV. However, there are many commercial applications as well. For example, only TiVo and Moxi have created rich UI's for their PVR's. There are tens of millions of PVR's out there from Motorola and Scientific Atlantic distributed by the cable operators, but they cannot justify the huge expense of creating a rich UI, so in general they have ugly, plain, static UI's. If there were an inexpensive, easy way to add a beautiful UI to these PVR's it would be worth it. Also, most TV's have a user interface for on screen menu's that is static and boring even though the TV's themselves have graphics processing chips in them that have the power to render a rich UI. There are numerous DMA's and set top boxes as well that could benefit from a rich UI as well. Mobile phones and PDA's also have media players and mini MCE apps.

In general these UI's are powered by a few single chip solutions, such as Sigma Designs, Broadcom and Marvell, and specialty chips for TV's, like AMD's Xillion. Macromedia Flash's player has been ported to many of these platforms so you can, for example, navigate a flash UI on a mobile phone. But, again, since Flash doesn't work well with MCE apps, in general these MCE apps all have crude, basic UI's, or they look to specialized middle-ware providers like Syabase, Mediabolic, Digion, etc., to create a UI for them.

The UI player will first be developed for X86, probably Linux and Windows in parallel. The software would be developed in such a way that the UI itself was subject to the GPL license, so there is an immediate potential to license the UI player for some commercial MCE apps. There's concern, for example, that SageTV will have a hard time remaining viable since it just doesn't have the pretty UI that Windows MCE does. They would be a potential customer to license the UI player outside the GPL since they would not want to release their UI under the terms of the GPL.

Over time the UI player could be ported to other platforms, like the Sigma, Broadcom, etc. Then the makers of set top boxes could use the UI generator to build rich UI's which could be played back on those set top boxes. By being cross-platform, we would have a strong advantage because the UI developer could target any platform for which the player existed, like Flash developers do now. Also, by being open source, the barrier of entry would be quite low. Anybody could start using the tool set, build it upon it if needed to add new visual effects, and only need to license it when they wanted to deploy a commercial application. And there would not be a concern of being locked into a closed, proprietary solution like the existing middle ware providers.

This relates to Pluto's core business of licensing an interoperability platform. Once a company adopted the use of the UI, Pluto would be able to offer a variety of modules, like lighting control, telephony, etc., that could easily 'bolt on' to the UI delivering a lot of extra functionality without writing any specialized code. If SageTV, for example, used the UI as their front end, Pluto could then offer lighting control modules, some to view security cameras, make phone calls, etc. which could be added to the SageTV package without them writing any new code. Pluto will be willing to sponsor the development of this new UI in exchange for receiving a license allowing Pluto to commercially license the UI software outside the GPL.

If QT is providing the underlying graphics library and hardware abstraction, then this benefits Trolltech as well because every commercial user who didn't want to release the UI under the GPL would also need to obtain a license to QT. The hope is that QT would also have an interest in furthering the development of this software.

How this relates to LinuxMCE

The UI in LinuxMCE, Orbiter, was developed to accomplish this same task. It has succeeded in some ways. For example, LinuxMCE's UI can be played on both Linux and Windows pc's, web pads, pda's, mobile phones, desktop phones, and web browsers. All those platforms are rendering the same UI and same set of screens created with a single design tool, called Designer. The problem is that Orbiter and Designer were created nearly 5 years ago, and were not very well designed. They are still quite crude. Designer is nearly impossible to use and does not appeal to graphic designers. The limited eye candy and visual effects in LinuxMCE, like the 3d cube media browser, are hardcoded into the player, Orbiter, rather than being part of the design as they should be. So, Orbiter and Designer are not general purpose enough to be useful to other MCE apps.

Skill sets needed

Here are 4 groups we need:

1. First we need people who really understand how to design a great UI. These could be Flash designers who have created rich UI's before. They need to outline what the new UI software must do to appeal to the designers that will be building the UI's. That is the most essential thing since, if designers are creating, say beautiful, rich TV program guides with this software, it will be easy to approach the commercial PVR makers and show them why they need to use this UI in their product. So this team is more composed of graphic designers, not C++ coders.
2. Next we need to know how to develop the UI design tool. There will be a lot of overlap with #1 since the people in #1 will be the users of the code written by this group. The UI design tool will not need to be cross platform. Windows/Linux x86 is acceptable. But it will have to be logical and similar to other design tools, like Macromedia's flash development tools, so designers can start using it without a steep learning curve.
3. Next we need people who create MCE apps and know what the player must do to integrate well with those apps. For example, what are the low level C-language hooks the player must have to handle alpha blending the UI on top of MythTV. What is the socket-based control needed to allow the player and MythTV to seamlessly communicate as though the player were developed as part of MythTV, so the user is not aware that the UI is in fact a separate piece of software.
4. The player needs to be highly efficient using a lot of the tricks that the video game industry uses to create high res 3d effects with smooth animation. For this we need some people who understand the Windows Direct X architecture, as well as the OpenGL fragment programs used by Linux and OS X, and some of the single chip solutions, so the player can be designed in such a way that as much code as possible is shared across all platforms, yet the player takes advantage of the unique hardware acceleration tricks each platform offers to be lean and efficient.

Proposed next steps

Some of the people needed will be in the Bay Area anyway for the KDE/Google event on Jan 18. We propose having a mini developers conference on Jan 19th. Pluto has agreed to sponsor this and provide airfare and accomodations to some key people, as well as pay change fees for some already at the KDE/Google event who want to stay an extra day.

The goal is to get representative people from the four skill sets to brainstorm some of the general concepts and come up with an organizational structure and delegate responsibilities. Also, key developers could be chosen to be sponsored full time.

We would also like to finalize an agreement with Trolltech so that if a committment is made to use QT as the underlying library for this new UI, it can be determined how Trolltech and Pluto will work together on the commercial licensing, and what resources Trolltech is willing to commit to the project.

Community Discussion

There is a discussion of this UI3 proposal in the developers forum, topic UI3 Discussion.