UI3

Proposal

Join the UI3 brainstorming meeting via live web conferencing

All are invited to attend the meeting in person or via web cam on Saturday, January 19, 2008, which will start at 9:00am at the Sheraton Hotel, 1108 N. Mathilda Avenue, Sunnyvale, CA 94089 in the Daffodil Room.

You can view, hear, chat and participate in the discussion live by visiting "http://go.megameeting.com/", select "guest", "plutohome" as meeting name, provide your name, and choose "default(host)".

Optionally call (408) 556-9826 or skype call/chat to id: linuxmce

Minutes of the meeting

9:30am -- We're very close to getting the conference up. There's been some networking problems at the hotel.

As the meeting progresses we will add notes here. Issues we want to resolve:

1. Will it be a general purpose design suite, like Flash, with predefined effects, or a suite for creating UI's, which is easier, but means the user can't do anything he wants unless it's implemented in code. Can the former be as efficient as the latter? If we do the former it almost certainly has to be based on gnash/flash if we want it to run on lots of platforms and processors.

2. If it's not flash based, is it based on Plasma, Qt, or something else? If on Qt we have cross platform tools. But it sounds like Plasma adds a ton of useful stuff, like the animator, however could Plasma run on Windows or a mobile phone?

3. Will flash be used? Possibilities are: 1) Design everything in Flash so designers can use Adobe's tools. Challenge: We need extensions to flash, and can't extend Adobe's CS anyway. So we'd still need a separate design tool to add the extra data in a way that Adobe's CS wouldn't delete them. 2) Use flash to create the object 'containers' and handle animating them, but what the objects do handled by custom flash extensions.

4. If there are complex tasks needed, like custom sorting and filtering of 1,000 song titles, is action script efficient enough to do this?

5. Do we use VG or bitmaps or both?

6. Off the wall idea... What if we wrote our own design tool and the 'file format' was actually C++; ie it was a code generator. And all the code it generated linked to plasma. That means the binary executable actually is the UI (rather than playing the UI from a data file). The design tool could include a compiler under windows/linux for the designer to preview, and when it comes to deploying, we could supply an online compilation system that built all the other platforms (symbian, etc.)

Meeting

There is a big KDE event at Google in Mountain View on January 18. It is expected that some people who would be interested in the new LinuxMCE UI which is intended to become an add-on to KDE will be in the area for this event already. Therefore, a local meeting room has been booked for all day on Saturday, January 19, 2008 to have a mini dev conference to brainstorm and plan the new LinuxMCE UI at the Sheraton Hotel, 1108 N. Mathilda Avenue, Sunnyvale, CA 94089 in the Daffodil Room. All are invited to attend. Pluto has agreed to sponsor the event and will also provide food for everyone. Pluto will also cover the air fare and accommodations for Daniel K, as well as up to 2 members of the LinuxMCE community who would be able to attend and could be valuable contributors. Post comments and recommendations on the developers mailing list.

If there is interest from people in attending virtually via web cam or call in, post comments on the developers mailing list so we can set that up.

Background

Macromedia Flash has become the defacto standard for creating user interface's for the web, with a rich development environment to create an appealing UI, and an efficient player to do the rendering with lot's of eye candy and visual effects. Media centric applications with 10' user interfaces ("MCE apps") have the same needs, but there really does not exist any software that can do this for desktop applications, like Flash does for web applications.

A few MCE apps try to use Flash for their user interfaces. But this does not work very well since this isn't Flash's intended purpose. For example, the UI in a MCE app cannot be stand alone and isolated, rather it must be a UI that floats on top of whatever media the MCE app is delivering. The Flash player is not able to blend it's rendered output on top of the media application, like Xine or MythTV. So although you could create a really attractive EPG program guide in Flash, you cannot blend that guide on top of the live TV in MythTV.

Therefore, nearly all the MCE applications, like Xine, Myth, Windows MCE, etc., typically code their own UI's. This is inefficient; it is like re-inventing the wheel each time. And, most of the time, the developers are more focused on their core application, not the UI, and cannot create a rich UI engine. For example, Xine, MythTV, Mplayer, Freevo, etc., all have their own UI's, but none are rich with eye candy and visual effects. In general, only Microsoft has been able to develop a really eye-candy rich UI with lots of visual effects, and most other media center applications have fairly boring, static UI's, even if the core media application itself is rich. For example, while MythTV developers began implementing a rich OpenGL UI engine over two years ago, this is still only used for some of the menu system and had not been extended to the video overlay or plug-ins, and because there is no UI editor the capabilities of it's graphics engine go largely unused.

The goal, therefore, is to create (1) a UI player, like the Flash player, which is efficient and can render high-res, 3d, user interfaces with lots of eye candy on a variety of platforms (x86 Linux/Windows, PDA's, cell phones, etc.), and (2) a comprehensive development environment, like the one Macromedia developed for Flash, that is well documented and not overly technical so creative designers can use it to build innovate UI's.

The intention is that this can be used to create UI's for a variety of MCE apps, like MythTV, MPlayer, etc., and that unlike Flash which is designed to run as an isolated application, this UI will be designed from the ground up to bolt on top of the core MCE app. In other words, you can create a beautiful EPG, like you could with Flash, but it can be blended on top of MythTV's live tv feed and used to control MythTV. The UI tool should be modular and flexible so graphics designers can create interesting UI's without needing the help of the coders of the core MCE app. For example, a designer could create, say, an innovate 3d TV program guide on a rotating sphere and add it to MythTV without needing the MythTV developers to write new code. Having a unified UI will also allow us to address some of the vexing issues, like text entry using remote controls of various types, with new UI widgets which only need to be learned once for the entire class of 10' applications.

It is understood that this is a big, complicated task which hasn't really been done before. However, we feel it is an important gap that needs to be filled. The first thing most end users notice is the eye candy in the UI and how pretty it is, and only Apple and Windows Vista have had the resources to create such UI's. This will allow other MCE apps to have UI's that are just as catchy and the developers can focus on the core functionality.

Use case

Here's an example of how the design suite could work, and how it's different than a normal design program:

1. Someone is creating a new screen in the design tool. In the tool, there exists style sheets with all the font styles. The style "Mid-size button" is defined as Times Roman 10 point. I create a button called 'play'. It has a blue background that pulsates with an animated effect 3 times per second, and has the text "PLAY" on top of it in that "Mid-size button" style. I cluster this together with a "pause" and "stop" button in a container called "Media Controls".

2. A different designer doesn't like the blue background, and instead wants something with wood grain. He doesn't want to recreate the screen, or change the layout or have to duplicate the commands associated with the buttons. Just change the background. So, he creates a new skin called 'wood' in the design tool, and creates a wood grain bitmap, and in the design tool associates this new bitmap with the skin. Now, when the user selects the 'wood' skin he sees the same button with 'play' in Times Roman 10 point, but the background is wood grain. The only thing is the pulsating animated effect doesn't look good on top of the wood, so he chooses another effect called "scroll" which makes the background wood grain image scroll a bit. There may be 100 buttons the original designer created with the blue background and the pulsating effect. This designer shouldn't have to open the properties for each button one at a time, he should be able to make a global change that says for this new skin, use this wood grain bitmap and scroll effect instead.

3. Now there's a user running the UI on a PDA. He finds that with the original blue skin it looks ok except that the pulsating effect is using all the cpu, and with the wood skin, the scrolling is just too much for the pda. So he opens the 'play' button in the design tool, clicks on the 'effects' tab, chooses the 'blue' skin and sees the 'pulsate' effect, he clicks 'criteria' and under "target device" selects "PDA", and then moves to the property field "times per second" and changes it from 3 to 1 so it goes slower on the pda. Then he chooses the 'wood' skin, again sets the criteria to 'target device=pda', and this time sets 'effect' to 'none'. This doesn't effect either user #1 or #2 since they don't use a pda.

4. Now there's another guy using a Cisco VOIP phone. He sees that on his phone all the effects are too much for the phone and need to be disabled. And although the 'Times Roman 10 point' text is legible with the 'blue' skin, when he choose the 'wood' skin which has more contrast, the text is unreadable because the Cisco phone's lcd display isn't that sharp. So, he opens the design tool, selects 'platform = cisco voip phone' and checks off "turn off all effects". Then he opens the style sheet tool and selects the "mid-size button" style, and clicks on "criteria" and sets "skin" = "wood", and changes the font size from 10 point to 12 point. Except, he notices, when the language is German. Then the words are just too big for 12 point font so the user has to live with 10 point, so he edits his criteria and adds to "skin=wood" a "and language<>german".

5. Another user decides he wants to create a new variation called "sideways" that aligns the clusters vertically rather than horizontally. He doesn't want to change any of the other stuff. He just wants to open "Media Controls" cluster, which contains the play/pause/stop buttons in a horizontal group, and he re-arranges them so they are vertical. They retain all their current properties; the play button still pulsates when it's the blue skin, except on the Cisco phone, and the font size is still 12 point instead of 10 with the wood skin on a cisco phone. It's just arranged differently.

Target user / financial model

Naturally this tool would be an asset to the FOSS MCE apps, like MythTV. However, there are many commercial applications as well. For example, only TiVo and Moxi have created rich UI's for their PVR's. There are tens of millions of PVR's out there from Motorola and Scientific Atlantic distributed by the cable operators, but they cannot justify the huge expense of creating a rich UI, so in general they have ugly, plain, static UI's. If there were an inexpensive, easy way to add a beautiful UI to these PVR's it would be worth it. Also, most TV's have a user interface for on screen menu's that is static and boring even though the TV's themselves have graphics processing chips in them that have the power to render a rich UI. There are numerous DMA's and set top boxes as well that could benefit from a rich UI as well. Mobile phones and PDA's also have media players and mini MCE apps.

In general these UI's are powered by a few single chip solutions, such as Sigma Designs, Broadcom and Marvell, and specialty chips for TV's, like AMD's Xillion. Macromedia Flash's player has been ported to many of these platforms so you can, for example, navigate a flash UI on a mobile phone. But, again, since Flash doesn't work well with MCE apps, in general these MCE apps all have crude, basic UI's, or they look to specialized middle-ware providers like Syabase, Mediabolic, Digion, etc., to create a UI for them.

The UI player will first be developed for X86, probably Linux and Windows in parallel. The software would be developed in such a way that the UI itself was subject to the GPL license, so there is an immediate potential to license the UI player for some commercial MCE apps. There's concern, for example, that SageTV will have a hard time remaining viable since it just doesn't have the pretty UI that Windows MCE does. They would be a potential customer to license the UI player outside the GPL since they would not want to release their UI under the terms of the GPL.

Over time the UI player could be ported to other platforms, like the Sigma, Broadcom, etc. Then the makers of set top boxes could use the UI generator to build rich UI's which could be played back on those set top boxes. By being cross-platform, we would have a strong advantage because the UI developer could target any platform for which the player existed, like Flash developers do now. Also, by being open source, the barrier of entry would be quite low. Anybody could start using the tool set, build it upon it if needed to add new visual effects, and only need to license it when they wanted to deploy a commercial application. And there would not be a concern of being locked into a closed, proprietary solution like the existing middle ware providers.

This relates to Pluto's core business of licensing an interoperability platform. Once a company adopted the use of the UI, Pluto would be able to offer a variety of modules, like lighting control, telephony, etc., that could easily 'bolt on' to the UI delivering a lot of extra functionality without writing any specialized code. If SageTV, for example, used the UI as their front end, Pluto could then offer lighting control modules, some to view security cameras, make phone calls, etc. which could be added to the SageTV package without them writing any new code. Pluto will be willing to sponsor the development of this new UI in exchange for receiving a license allowing Pluto to commercially license the UI software outside the GPL.

If QT is providing the underlying graphics library and hardware abstraction, then this benefits Trolltech as well because every commercial user who didn't want to release the UI under the GPL would also need to obtain a license to QT. The hope is that QT would also have an interest in furthering the development of this software.

How this relates to LinuxMCE

The UI in LinuxMCE, Orbiter, was developed to accomplish this same task. It has succeeded in some ways. For example, LinuxMCE's UI can be played on both Linux and Windows pc's, web pads, pda's, mobile phones, desktop phones, and web browsers. All those platforms are rendering the same UI and same set of screens created with a single design tool, called Designer. The problem is that Orbiter and Designer were created nearly 5 years ago, and were not very well designed. They are still quite crude. Designer is nearly impossible to use and does not appeal to graphic designers. The limited eye candy and visual effects in LinuxMCE, like the 3d cube media browser, are hardcoded into the player, Orbiter, rather than being part of the design as they should be. So, Orbiter and Designer are not general purpose enough to be useful to other MCE apps.

Skill sets needed

Here are 4 groups we need:

1. First we need people who really understand how to design a great UI. These could be Flash designers who have created rich UI's before. They need to outline what the new UI software must do to appeal to the designers that will be building the UI's. That is the most essential thing since, if designers are creating, say beautiful, rich TV program guides with this software, it will be easy to approach the commercial PVR makers and show them why they need to use this UI in their product. So this team is more composed of graphic designers, not C++ coders.
2. Next we need to know how to develop the UI design tool. There will be a lot of overlap with #1 since the people in #1 will be the users of the code written by this group. The UI design tool will not need to be cross platform. Windows/Linux x86 is acceptable. But it will have to be logical and similar to other design tools, like Macromedia's flash development tools, so designers can start using it without a steep learning curve.
3. Next we need people who create MCE apps and know what the player must do to integrate well with those apps. For example, what are the low level C-language hooks the player must have to handle alpha blending the UI on top of MythTV. What is the socket-based control needed to allow the player and MythTV to seamlessly communicate as though the player were developed as part of MythTV, so the user is not aware that the UI is in fact a separate piece of software.
4. The player needs to be highly efficient using a lot of the tricks that the video game industry uses to create high res 3d effects with smooth animation. For this we need some people who understand the Windows Direct X architecture, as well as the OpenGL fragment programs used by Linux and OS X, and some of the single chip solutions, so the player can be designed in such a way that as much code as possible is shared across all platforms, yet the player takes advantage of the unique hardware acceleration tricks each platform offers to be lean and efficient.

Proposed next steps

Some of the people needed will be in the Bay Area anyway for the KDE/Google event on Jan 18. We propose having a mini developers conference on Jan 19th. Pluto has agreed to sponsor this and provide airfare and accomodations to some key people, as well as pay change fees for some already at the KDE/Google event who want to stay an extra day.

The goal is to get representative people from the four skill sets to brainstorm some of the general concepts and come up with an organizational structure and delegate responsibilities. Also, key developers could be chosen to be sponsored full time.

We would also like to finalize an agreement with Trolltech so that if a committment is made to use QT as the underlying library for this new UI, it can be determined how Trolltech and Pluto will work together on the commercial licensing, and what resources Trolltech is willing to commit to the project.

Community Discussion

There is a discussion of this UI3 proposal in the developers forum, topic UI3 Discussion.