This page last changed on Feb 15, 2012 by ebariaux.

I would like to use this thread to discuss the UI elements (and their properties) that should be present in the future version of the modeler (and thus in the different consoles as well).
Here is the current wish list (items in bold are currently available in the public release).

Label:

  • text
  • wrap or truncate : general support for multiple lines of text, what is required ?
  • font : (css font-family) need to be careful as what selection is available. Can we "transfer" fonts to the different consoles? If a font is not found at rendering, how do we fall back ?
  • font size (css font-size)
  • text effect : bold, underline, italic (css font-style : normal | italic | oblique; font-weight : normal | bold | bolder | lighter | 100, 200, ..., 900; text-decoration: none | underline | overline | line-through | blink)
  • text alignment : 9 possible values to take horizontal and vertical alignment into account
  • color
  • background color
  • hidden
  • sensor, link to a sensor so text value can be dynamically updated

Image:

  • image
  • alignment : 9 values, works both when images is smaller or bigger than display region
  • resizing : no, fit with distortion, fit no distortion (image will usually ends up smaller than display area), fill no distortion (image usually ends up bigger than display area), resize widget (so "container" does resize to accommodate the provided image and not the other way around)
  • background color : used if the image does not fill the entire display area
  • hidden
  • sensor, link to a sensor so image can be dynamically updated (sensor will return name of image, that should already be part of configuration)
  • fallback label, link to a Label used when no images displayed

Button

  • name
  • image
  • press image
  • enabled
  • hidden
  • some commands and navigation options are available but they do not influence the display of the button

Switch

  • image on
  • image off
  • enabled
  • hidden
  • switch command, in fact a switch "model" that determines the on/off state of the switch and the commands to perform on user action

Slider

  • vertical
  • min image
  • track image min
  • thumb image
  • track image max
  • max image
  • inverted, usual direction is to have min on the left / bottom, when inverted, min is on right / top
  • value bubble display, indicates how the real-time value is displayed as the user moves the slider: above, below, left, right, none, label (displayed in a separate label only when user moves slider, label is blank when no action), label_always (real-time when the user moves the slider, current slider value otherwise)
  • value bubble label : link to a label in the UI where the real-time slider value must be displayed
  • enabled
  • hidden
  • slider command, in fact a slider "model" that determines the min, max and current value and the commands to perform on user action

Web

  • url
  • username
  • password
  • refresh rate : indicates that the console must reload the URL every x seconds
  • enabled
  • hidden
  • link to a sensor so url value can be dynamic

Colorpicker, some form a visual "wheel" or graphical representation allowing the user to pick an RGB color

  • background image
  • command

Text input widget

Some form of list ? Something that could be used for media browsing ?
Video ?

For "layout", in addition to having multiple screens that you can navigate between with slide gestures, add support for a "surface" bigger than the screen size that you can pan and zoom.
Must define all available gestures and screen "navigation" mechanism, which are exclusive and which can work together (e.g. if there are multiple screens defined in a group, "pagination" navigation is automatically provided and it should not be possible to define left-to-right or right-to-left gestures).

Also in general, the concept of "z axis" or ordering of elements must be introduced. And with it, the concept of how to manage transparency with regards to user actions. For instance, I have an image with some transparency in front of a button. Does all pixels where alpha is not 0 a pass the event through to the underlying button ?

From the discussion New Object model I assume we also would like something like enable/disable and show/hide for all UI elements.

Web could have a reload interval for images which are calculated on the fly. This way graphs can automatically be updated. This could also be used for IP cameras where you can retrieve a new picture every few seconds.

New element colorpicker for RGB leds. The object model could reflect this with a object for lights that support color and provide not only on/off but also RGB/HSB values. This could also include brightness.

A modified version of Slider which can be used for blinds and can reflect the blind state in the icon (10%, 20%, 30%, ...). The element could have a Up button, then the state image and then a down button.

Floorplanes can probably be achieved via Images and transparent buttons.

I also vote for a list object (single and multi-selection should be possible).

Schedules and timer based actions can probably be handled by the rule engine. But the modeler would need a possibilty to edit rules.

The modeler is missing an import off a complete configuration (upload of the downloaded zip file).

Posted by mredeker at Sep 29, 2011 15:34

From the discussion New Object model I assume we also would like something like enable/disable and show/hide for all UI elements.

Yes, will add that above

Web could have a reload interval for images which are calculated on the fly. This way graphs can automatically be updated. This could also be used for IP cameras where you can retrieve a new picture every few seconds.

Yes, might be easier. Original idea was that there is a sensor for the URL and you could update through this. But it might not even work because the sensor update would be ignored if there is no change in value. Will add refresh rate.

Will add color picker.

A modified version of Slider which can be used for blinds and can reflect the blind state in the icon (10%, 20%, 30%, ...). The element could have a Up button, then the state image and then a down button.

Not sure this requires a dedicated UI element. I'm usually in favor of limiting the number of UI elements available and make sure you can re-create them using the basic blocks.
So in this case, buttons do cover the up/down scenario.
An image could be used for the state icon if there is a mechanism to define the link between the value and the image. There is already such a thing for custom sensor, so extending that to say something like having ranges of values mapping to image names could do the trick.

For the import, definitely something required. I started this thread to discuss improvements to the modeler that are not directly related to adding more UI widgets.

Posted by ebariaux at Sep 29, 2011 16:04

For both label and image, I would start the wishlist by simply taking the text and box properties already available in CSS3 – this will also yield 1:1 mapping to the HTML5 web console. I'd add these to the schema design even if at this point not all the native clients support all the features, just so we don't have to reiterate on this part of the schema so soon.

The open question I have with regards to this is whether the we want the specify those properties with our own XML schema or simply link to a CSS definition?

Latter is obviously simple for web console but requires extra effort on native panels.

For text labels all the currently proposed properties already exist in CSS, plus some extras that have not been mentioned yet (mostly around visualizing the 'box' around the label where desired).

Thoughts?

Posted by juha at Sep 29, 2011 16:52

We should have both list and table, as you suggest.

Also I think we will need dedicated streaming components for audio and video?

Posted by juha at Sep 29, 2011 17:17

Reusing something that is already there would make sense.
But we probably need the properties in our schema to be able to have the properties available in our model on the controller side?

Posted by mredeker at Sep 29, 2011 17:21

For the latter (audio and video) can we lift something from the current HTML5 spec?

The list and tables are obviously already there. They need to have similar ability to update from the controller side (same as image refresh rate). This is a generic mechanism that should be introduced to all view components (e.g. change label text color to red). We would need to define the Controller REST API update for this.

Posted by juha at Sep 29, 2011 17:22

We need the properties in the object model but what source we use to read them is still open. If we include in the schema, we probably still end up generating the CSS file just for web console. Or we can all parse the CSS artifact from the openremote.zip for view properties (including the controller which needs to build the object model).

I've no idea which way is better at this point.

We will also want to leave the view properties open for extension, so for example it will be possible to include some HTML rendering specific properties (CSS properties specific for webkit or mozilla, for example), just so those view properties can be tweaked and used by the HTML rendering even if they're ignored by the native ones. They could also be used for 'hints' for specific Android or iOS functionality.

Posted by juha at Sep 29, 2011 17:32

Guys,

Would love to have the chance to have the following functionality:

  • Buttons:
    enabled/disabled: by greying out the button, i.e. you see it but it is overlaid with a grey to indicate it is disabled.
    color: standard grey, but offering choice similar to text colors.
  • switches:
    enabled/disabled: by greying out the button, i.e. you see it but it is overlaid with a grey to indicate it is disabled.
    color: standard grey, but offering choice similar to text colors.
  • images:
    fix size: so clip image when larger than indicated area, or
    dynamic size: resize automagically
    sensor: either sends updated value, forcing to reload image from controller (which might be updated). Current implementation expects the name of the image (???) from the sensor, never got it to work, would love to see example to check wether it fits what reload is required
    sensor: sends back with update the name of the changed jpg/png/gif file to replace the existing one.

support for mjpeg, i.e. to update with a video feed up to every second or other second.

  • sliders:
    position: text not in bubble but fixed position above/in front/behind the slider

Hope this is useful. And if you have an example how the sensor for image works with filename update, please point me to this, as I must be doing something wrong.

regards,
Rick

Posted by rickcn at Oct 02, 2011 16:21

On Slider, was pointed out elsewhere that there's a desire to control the min/max location on the axis, so right-left or up-down instead of the opposite.

Posted by juha at Oct 10, 2011 03:10

This is not related to the UI elements themselves but more about the view part of the xml that is sent back to the panels, so adding it to this discussion:

The panel's XML definition should include information about the screen dimensions. This will help with the rendering, among other things. Richard Turner made the point in email discussion, so I'm quoting the relevant part here:

the reason it would be useful is that on desktops I could size the console unit emulator to match the dimensions of the selected panel, at present I'd have to iterate through the screen elements and determine the limits from the widget positions, this is just a nicety and could possibly allow for a setting in the console to only show panels with a compatible size.

This could be as simple as property values associated with the panel design (or meta-data if you will) – as generic mechanism to add information that could include screen dimension but also if panel design was targeted for a specific class of devices (iPhone, iPad, Galaxy Tab, etc.)

The point related to usability further implies changes to the REST API – this could be as simple as returning a filtered (technically transformed) version of the XML based on distinct REST requests, once the full model is in a single XML schema. For example, first REST request would only have a return document with XML elements containing the panel root with name + the properties (metadata). This can then be further filtered by the panel to expose to user only the panel identifiers that match a specific panel type or the panels resolution. Once the user choice has been made, the full view definition is returned for the chosen identifier.

Should we start a separate topic for REST API changes or did you start one already?

Posted by admin at Oct 16, 2011 17:44

Another relevant use case for metadata/properties on the schema document and REST API is for public/commercial installations where it's desired to bypass the panel selection completely to reduce the steps the user is required to take – essentially locking the panel design down to a single choice.

The panel selection could be somewhat parameterized on the controller side if the panel device sends information about its properties as part of the initial REST call (such as iPhone device which can be matched to an appropriate design based on panel resolution/device category or device ID properties). The controller in this case makes the decision to pick the suitable panel design and returns the panel definition for that device category (or if more fine-grained, the device ID), rather than servicing a panel definition request where the ID is set by user choice.

Posted by juha at Oct 16, 2011 17:58

support for mjpeg, i.e. to update with a video feed up to every second or other second.

The web element will be used for that, not the image one.

Posted by ebariaux at Oct 20, 2011 16:03

To make the UI as feature rich as possible does anyone else feel there is a need for some inter widget communication/binding mechanism?

Enabling / disabling UI elements, updating labels without using sensors (e.g. drag a slider and a label updates with the value)

In the new Web Console I am trying to create all system screens using a panel definition (obviously we don't have all the required widget types yet but I think this would be a good test for what is possible). Based on this the current settings screen would become the default screen for the system panel definition and the tab bar along the bottom would perform various tasks: -

  • Run Controller Discovery
  • Navigate to 'Manually Add Controller' screen
  • Navigate to 'Security Config' screen

The above is along the lines suggested by Juha in this thread. Whether or not this is a good idea (or even feasible I am not sure but it seems like a good way to develop a richer UI with a valid use case). How data input would work etc. I am not sure at the moment but some form of binding is required whether or not this can be defined within the xml I am not sure.

Comments/suggestions gladly welcomed.

Posted by kurrazyman at Oct 27, 2011 19:59

In the process of implementing the current widgets within the new Web Console I have got to the Image Component and am curious as to when the fallback label takes effect?

I have created an image component linked to a sensor with a fallback label defined and I then tried removing the image files from the controller webapps directory so the images couldn't be found and on Android and iOS I just see empty space where the image should be, should I be seeing the fallback label text in the place of the image?

Posted by kurrazyman at Oct 27, 2011 20:08

Regarding graphs, something to put on the table at least:

Graphs as images is the basic version – and we got a lot of mileage out of that model on the web.

Now there are richer graph components, mostly flash and I can only assume HTML5 is already used somewhere or is not far off (obvious candidate for web console then).

How about native graph rendering on Android/iOS – anything readily available (I would definitely default to third party component here)? If not, the image model may be sufficient for now, except would lead with a richer HTML5 component on web console at least.

Thoughts?

Posted by juha at Oct 27, 2011 20:24

Yeah this has been briefly discussed. Javascript fills the need on web console for DOM tree modifications. And forms for data input.

There are certain types of layouts/designs that would require this.

On iOS/Android side we're dealing with XML – that can make things a bit easier. What's the choice there, XSLT? I would at least initially strictly define which types of modifications are supported (e.g. visibility, positioning). I don't think the full power of JavaScript is necessary for the basic use cases. Although for web console I wouldn't limit it (just as I wouldn't limit the full use of CSS for web console).

Also the wish list is getting pretty long – we may want to start putting votes/priorities on these, plus effort estimates (iOS probably being the most critical).

Posted by juha at Oct 27, 2011 20:38

I see dealing with XML as just an artifact for me in the iOS console. It is just a transport format to get the UI description from the server to the client. The XML is used to build up a "UI Object Model" that is used for rendering.
I would then allow manipulating that object model to influence "in real time" the rendering on the console. So very much like the DOM in a browser.

Manipulation on that object model could occur from the server side (where some kind of proxy mechanism might be used or some other way to send "UI OM" manipulating command to the panel), or it could occur locally (e.g. using javascript or Lua - I mention Lua because I know it's fairly easy to integrate with iOS and there are applications in the AppStore that provide Lua interpreters and have been approved).

Posted by ebariaux at Oct 28, 2011 10:28

Ok makes sense. Lua may then be the right choice. We'll definitely want something that's already approved and done by third party here.

Yes, we'll have use cases for both controller-side driven modifications (something happened to device, modify UI widget properties in response) and panel-side modifications (user interacted with UI widget, modify UI object model in response – controller doesn't care about these).

Posted by juha at Oct 28, 2011 10:56

Briefly discussed the fallback label semantics with Eric. Will let him answer in detail – he's touching this part of code on iOS.

Posted by juha at Oct 28, 2011 10:57

So semantics currently implemented is:

  • if there is a mapping defined between the sensor value and an image (i.e. you have associated an image with a sensor value in the modeler), then the image is used. If the image is not found, it does NOT fallback to the label
    (Maybe we should change that so that if no image is found if also falls back)
  • if there is no mapping, use the label
    Again here, if there is a mapping between a sensor value and a text, use the text to populate the label
    If there is no mapping, just use the sensor value as is to populate the label
Posted by ebariaux at Oct 28, 2011 15:43

On iOS there is core-plot. Never used but I've met one of the main developers and it looks good.
Should not be too difficult on integrate in iOS console.

Posted by ebariaux at Oct 28, 2011 15:51
  • The ability to hide/display a button name. When a button uses an image, sometimes the text overlay is useful. At other times it gets in the way. Anonymous buttons can be used, but I prefer to attach a name to them. In the mean time, I modified the Android app to hide names that start with an underscore.
  • Ability to stretch a button image to fit the button, or to align the image with respect to the button.
  • Ability to make use of the rowspan and colspan cell attributes.
  • Include panel dimensions in panel.xml so that panels of similar sizes can reuse the same design by scaling things appropriately.
Posted by gburca at Nov 20, 2011 04:13

Don't know if it fits in this thread, cause you are concentrating on (G)UI items. Don't know if there are right API's in place on the different platforms neither but how about supporting some kind of speech recognition element's. For instance Siri support on IOS.
Would be great to have a scenario like : 'Please switch the lamp on in kitchen' or 'start kino environment' which forces to switch on my TV, close down the blinds and starts a lighting scene for a romantic evening
That would be really a multimodal interface, giving verbal input and getting the actual states out visual. Definitely worth to look deeper into it.

Posted by maja at Jan 24, 2012 12:21

An input widget would be nice. Youst a plain textfield which will be passed to the controller.
Maybe attributes can be used to tweak the looks like DatePicker or TimePicker.

Posted by mredeker at Feb 10, 2012 09:54

Guys, I've been playing with OpenRemote for a few months, programming a few of my own protocols to add. One thing I'd like to see in the UI is a scrolling list so a user can select an item from the list. Initially it could be just a list of text items, but then it could be expanded to include multiple columns, images, images with text, multiple items per row and ultimately different layouts like Windows Explorer.

The main use would be for media browsing, but it could also be used for other things such as selecting an AV receiver's sound field or selecting a lighting scene and so on.

You could then link other buttons on the screen to perform some action based on the selected element (for example, add to playlist, play now). Each element in the list would need some sort of alias or ID value that the button commands would use as a parameter.

Posted by jst01 at Mar 09, 2012 21:42

Would HighCharts be something? Highly interactive. http://www.highcharts.com/

Posted by reesen at Oct 13, 2012 16:12

@Marcus
It would definitely be good to have some input possibilities. But where would that information go in Building Modeller? Shouldn't date/time picker be something on its own with button to move it up and down? That would beeasier in the interface. For the time being I would be happy with just a plain text input that I can access in the rule engine

Posted by pz1 at Dec 21, 2012 08:37

A feature I feel eseential in order to be able to create a less bloated GUI is drop-down menus, if they could be made "feature rich" according to the below list the GUI would feel a lot more intuitive and for my sake, a ton of buttons could be grouped in one drop-down list (scene selection/input selection/room selection/navigation to some extent).

Drop-down list

  • Text
  • Images (not as important but nice from a look and feel perspective)
  • Sensor, link to a sensor so selected value can be dynamically updated
  • Each item works as a button in terms of what it can do (ie, command and/or navigation)

I think this would benefit the GUI look and feel a LOT, and also make it feel more modern.

Hopefully, you guys agree

Posted by tobo at Dec 21, 2012 12:24

Hi Tobo,

You can simulate a drop down menu system by using different screens (no animation of drop down but a drop down effect none the less). On screen 1 you have the menu collapsed (single button) set the button action to navigate to screen 2 and on screen 2 have the exact same layout but with the menu expanded (grid layouts are good for this).

Not ideal but it works.

Posted by kurrazyman at Dec 22, 2012 21:23

Off the bat, not really a maintainable workaround I'd say. I'd have to create as many screens as I have different options in the list if I want it to reflect last or currentlt selected. Also, can a sensor change the page you're at? Otherwise I won't be able to reflect status changes made outside of OR (ie, changing input source/current scene etc) in the GUI. Also, if I'm in a screen that's not displaying the sensor I don't want to automatically move to that page which I guess would be the case if I could actually have a rule that switched pages.

I'll give it a try at some point just to see how responsive it is and what the user feel is but for my information, are you guys planning on adding something like this or would I have to look elsewhere for this functionality?

I realize that my comment can be taken as rude and that's not the point at all, it's just that I've gotten so far with my HA I'm trying to compile a list of pros and cons between different solutions. Even though OR is my choice at the moment I'm getting far enough along the implementing line to realize som cons in my book. I'd be happy to put in loads of time tweaking and tuning my OR-installation, just need to know that I won't hit a major obstacle in the middle of the journey.

Would I be getting more functionality if I payed for the pro version? As far as I can tell there were some more functionality in there when looking at the code but not sure if everythings working etc. Also, I don't think I saw a drop down list.

Still, a great job guys, I just hope it's about to get pieces of the puzzle that are missing for me personally And as mentioned, I'd be fine with dishing out the pro money if there was gain in doing that?

Posted by tobo at Dec 25, 2012 17:50

Tried this out somewhat and ran into a couple of snags. I tried to mimick the example from another HA software provider and here are my initial comments.

1. It's possible to get the transition smooth as long as you have the screens in different groups so it doesn't scroll.
2. Very time consuming. Not the biggest issue ever but can get out of hand pretty quickly.
3. Figured I'd use a text label as the activator for the drop down, this way I could connect a sensor and see current state, a press would then then get the menu. The drawback here is that a label has no navigation tied to it. So a label and also an "invisible" button I guess is the solution...
4. The area outside of the drop down has to be a huge button (or several depending on layout) in order to cover the use case "don't select anything from drop down". This invisible button just backs one screen.
5. Borders around drop down areas aren't possible unless using buttons with custom images. Nothing major, just looks better with some sort of delimeter (borders or similar) since the original label isn't tied into the drop down without major graphics work.

Hmmm, it might be possible to get an ok solution with the fake solution but the work associated with setting it up and maintaining it if things are moved around etc is huge. Considering the amount of time I've put into including KNX/XBMC/receiver etc in OR I won't go anywhere right now but rather implement a workaround of some sort. But in order for OR to be even more kick ass and flexible in the GUI department I'd LOVE to see this feature.

Let me know if there's anything I can do to help.

Posted by tobo at Dec 25, 2012 19:32

Layering of images. Seems like the different consoles have different image layering abilities. I just tried out android and iphone. If I put an image overtop a button:

ios: the image blocks access to the button (i.e. button not clickable)
android: the image allows access to the button (i.e. button is still clickable)

Also, how is Z-order calculated? Is the last gui element added, the one on top?

The layering of transparent images (coupled with sensors) can make for some nice dynamic graphical abilities ( I was thinking of doing the iPhone badge thing with custom sensor and an image per badge number.... no easy way to scale it up over 10 or so numbers... too much work)

Posted by mdarwin at Mar 21, 2013 16:40

Hi Martin,

Image click through was something I specifically added to the web console (provided browser supports pointer-events property) as this is a common thing to do, so would have thought this requirement would have come up for iOS as well but Eric will be able to comment more on that.

Regarding Z-order; as there is no concept of z-order in the modeller the order they appear on the console will depend on the order from the panel.xml file (I cannot confirm if this ordering is always chronological but from my experience it seems to be).

Rich

Posted by kurrazyman at Mar 21, 2013 20:42

It would be a really nice feature with an opportunity to set the Z-order in the designer.

Posted by kenta at Mar 22, 2013 13:04

There is no explicit way to manage z-index in the designer but the notion of z-index is there: the order the elements are included in the generated xml file should be identical to the order the elements are displayed in the designer and should be identical to the way they are rendered in the consoles.
That's a lot of should and clearly there is work required on making that explicit, documented and validate it's indeed implemented consistently everywhere.

In the current modeler implementation, based on selection / moving around, you can make widgets move below / above other ones.

It's on the (long) TODO list to have explicit z-index management with standard: move to back, move backwards, move forward, move to top functions.

Posted by ebariaux at Mar 22, 2013 15:44

As Eric said, this is something that has been on the list for a while but his TODO list is already far too long. If there's anyone with programming skills willing to take on the task and help him out on this, please let us know.

Best regards,

– Juha

Posted by juha at Mar 23, 2013 18:34

Z-order is probably okay as is now. It seems to be consistent between consoles.

The fact that images don't pass through "clicks" on iOS is probably the only inconsistency with respect to z-order.

MD

Posted by mdarwin at Mar 23, 2013 19:11

Recognising that this list is already far to long, I have one more suggestion for the slider:
Could we have pressed/unpressed buttons for the increase/decrease arrows?

Because I mostly fine tune my thermostats with one or two degrees, I have reduced my sliders to just the up and down arrows. Some visual feedback would be nice

Posted by pz1 at Jun 13, 2013 08:41
Document generated by Confluence on Jun 05, 2016 09:32