This page last changed on Nov 08, 2010 by trondk.

Hi,

I have some doubts about the scalability of how devices are added in the Openremote building modeller.

E.g. for a KNX installation you typically would want to have on/off and dimming for all your dimmer zones.

If I have figured things correctly you would then create:

  • a "switch on" command
  • a "switch off" command
  • a "status" command to see on/off state
  • a switch device
  • a sensor device (for the switch)
  • a "dim" command to set dimmer value
  • a "dim" status command to see current dimmer value
  • a slider device
  • a sensor device (for the dimmer value)

In other words, you quickly get 9 "entries" in your building modeller for each dimmer zone.

For a "whole house" installation I would claim that it is not unheard of to have more than 30-50 dimmer zones, which then yields 270-450 entries in your building modeller, and this for light control alone...

Am I missing something here, am I to hang up in the "KNX way" of doing light control, or is this somewhat of an issue?

Additionally, for the building modeller I do not see the big advantage of doing these definitions by means of a GUI, as there is not really any "WYSIWYG" benefit like there is for the UI designer.

From my point of view it would be much better if it was possible to produce the controller.xml file directly (then I could make some python script etc, and parameterize things to the point where adding a new dimmer zone would be a one-liner...), and then import the building model into the Openremote Designer GUI.

Regards,
Trond

First of all, congratulations. For having gotten this far and figured out all of the steps. I couldn't create a slider to test your bug in android yesterday that easily. It is clear that our software is v0.1 so far and needs A LOT OF IMPROVEMENT. It is not a finished product by any stretch of the imagination, it is just a start.

I think you are exactly right that there is little benefit to a wysiwyg for building declaration.

Question is what are the ways to simplify this?

1- ETS import? With ETS 4 I think we can come up with something that gives you a palette from file definition. With ETS 3 it is trickier and involves a hack. Are you based on ETS4? Alternatively Olivier Gandit was saying we could build a plug-in for ETS and export the data with a controller.xml file.

2. templates?

3. scripts? we are open to that as well...

Any ideas are welcome. If this is something you want to help with we can talk offline. Are you a consultant in the field? an installer? obviously you know your way around a keyboard...

Posted by marcf at Nov 08, 2010 09:01

Hey Trond,

Yeah makes sense. I will look at adding the import utility in as soon as possible. Should help with local scripting and then upload to model and use designer for the UI.

– Juha

Posted by juha at Nov 08, 2010 09:31

Hi,

That was a quick reply

I am a consultant/contractor working with SW development, primarily in the embedded space. I am also a homeowner with a growing KNX installation.

So, I am not really working with KNX or home automation in a professional capacity (at least not yet , but I have done the programming part of the KNX installation in my home, using ETS3.

As you write the Openremote SW is still in its infancy, but I think you have come a long way already.

I think that being able to import the building model from ETS in some way would be a great long term goal for the building modeller, and and I would guess that its probably something you would want/need to be able to do to get the professional KNX installers on board.

As I mentioned, being able to import the building model (as far as I understand basically controller.xml) into the Openremote Designer would also be a great improvment I think, and possibly one which would not be to difficult to add (I guess depending on how much validation/sanity checks you would want to add.)

Also, in the building modeller, I think it would have been great if there was a possibility to "collapse" the definition of command(s), status, sensor, and switch/slider for a given dimmer (or simmilar physical device) into one object instead of having them as separate objects could maybe be nice. Or, at least being able to define them all in one go in the same "dialog", without having to define them separately, would be an improvement.

At the moment I have just deployed the released packages both for the ORB and for the Android panel, so I have not really got the full toolchain verified to build everything from scratch myself. I am hoping to find time to do that soon, both for the ORB and for the Android app. When I have that in place I would be glad to contribute whatever functionality time and competence allows for.

One feature I am really looking forward to is the addition of some sort of scriptable rules engine. Currently I am rolling my own very simple one, based on python scripts and XML configuration files, but it would be great to get something more integrated and powerful. Visualization features is also something that would be a great addition.

In other words, when Openremote is an acceptable replacement for the outrageously priced Gira HomeServer products I will be a happy man

Regards,
Trond

Posted by trondk at Nov 08, 2010 10:20

Thanks for the props on Designer and saying it has come a long way it has, but it can be frustrating as well.

ETS4 supports an export function so that can be straightforward.

Thanks for the offer to contribute on the android front. I know juha is planning on scripting for the 3.0 release. It will take some time but it is on the roadmap. Also if juha has you on the scripting then you can be looking forward to that.

yeah replacing GiraHomeServer is a valuable goal.

Posted by marcf at Nov 08, 2010 11:10

I've hacked together a simple Python script that lets me generate a controller.xml and a matching panel.xml with something like this:


if __name__ == "__main__":

    cfg = config()
    gui = gui()

    #
    # List the devices in the configuration
    #
    cfg.addDimmerChannel("LigthZone 1", "2/6/6")
    cfg.addDimmerChannel("LigthZone 2", "1/1/7")

    #
    # Then specify the panel(s)
    #
    scr = screen("Starting Screen")
    scr.addSlider(cfg.getSlider("LigthZone 1"), 51, 60, 198, 44)
    scr.addLabel(cfg.getLabel("LigthZone 1"),
                 "Label Text", 51, 140, 198, 44)

    grp = group("Default Group", scr)

    pnl = panel("TestPanel", grp)
    
    gui.addScreen(scr)
    gui.addGroup(grp)
    gui.addPanel(pnl)

    #
    # Lastly, write the XML files..
    #
    cfg.write(open("controller.xml", "w"))
    gui.write(open("panel.xml", "w"))

The script would then generate a slider, a switch, and a label bound to the slider sensor for each of the dimmer channels.

The panel stuff is just to be able to make really simple panel configurations until it is possible to import a generated controller.xml into the online designer.

Have not had time to test the generated config yet, but plan to post the complete script once I am sure it generates a working config.

One thing though, the panels.obj, which as far as I have figured out is "Java serialization data" (whatever that means), is that something that is specific to the config as well, or is it just related to the format?

I guess what I am asking is if I need to generate this panels.obj file as well for my generated panel.xml file.

Posted by trondk at Nov 11, 2010 12:23

Panel.obj IIRC is legacy from 1.0 – although related to import. It stores the designer state.

You shouldn't need it for 'first-time' import where your design doesn't exist in the first place, the designer just defaults to normal start state. Will see later if necessary to reintroduce it for export/import cycle for other designer related state. Ignore for now.

Posted by juha at Nov 11, 2010 13:14

Ok, I have tested the script now, and it generates a configuration that at least works with my setup.

Incuded below if anybody else have a need for it.

Regards,
Trond

(edited Thu Nov 11 21:49:32 CET 2010: Added support for different group addresses for the dimming and the status)

#!/usr/bin/python
#
# Quick and dirty script to generate controller.xml and panel.xml
# for the Openremote controller.
#
# Configure server settings at the top, and devices and panels at the bottom.
#
# 2010-11-11 Trond Kjeldaas
#
import sys
import xml.etree.cElementTree as ET

#
# General controller config
#
controller_config = [

    { "name"  : "controller.roundrobin.tcpserver.port",
      "value" : "20000" },
    { "name"  : "multicast.address",
      "value" : "224.0.1.100"},
    { "name"  : "controller.roundrobin.multicast.port",
      "value" : "10000"},
    { "name"  : "multicast.port",
      "value" : "3333"},
    { "name"  : "controller.groupname",
      "value" : "floor20"},
    { "name"  : "Macro.IR.Execution.Delay",
      "value" : "500"},
    { "name"  : "controller.roundrobin.multicast.address",
      "value" : "224.0.1.200"},
    { "name"  : "lircd.conf.path",
      "value" : "/etc/lircd.conf"},
    { "name"  : "controller.groupmember.autodetect.on",
      "value" : "true"},
    { "name"  : "webapp.port",
      "value" : "8080"},
    { "name"  : "copy.lircd.conf.on",
      "value" : "true"},
    { "name"  : "irsend.path",
      "value" : "/usr/local/bin/irsend"},
    { "name"  : "resource.upload.enable",
      "value" : "true"},
    { "name"  : "controller.applicationname",
      "value" : "controller"}
    ]


# Global "ID" allocator...
nextid = 0
def allocId():
    global nextid
    nextid += 1
    return nextid

def prettyPrint(e, indent = 0):
    "Add tail-content to elements so that we get a nice printout"
    e.tail = "\n" + " "*indent
    if len(e) > 0:
        if not e.text:
            e.text = "\n" + " "*(indent+2)
        for c in e:
            prettyPrint(c, indent+2)
        e[-1].tail = "\n" + " "*indent

class component(object):

    def getType(self):
        return self.type

    def getId(self):
        return self.id

    def getName(self):
        return self.name

    def getSensorId(self):
        return self.sensorId
    
class command(object):

    def __init__(self, value, groupAddr, DPT):

        self.id        = allocId()
        self.value     = value
        self.groupAddr = groupAddr
        self.DPT       = DPT

    def getId(self):

        return self.id
    
    def xml(self):

        c = ET.Element("command", id="%s"%self.id, protocol="knx")
        c.append(ET.Element("property", name="command", value=self.value))
        c.append(ET.Element("property",
                            name="groupAddress", value=self.groupAddr))
        c.append(ET.Element("property", name="DPT", value=self.DPT))

        return c
        

class sensor(component):

    def __init__(self, name, typ, command):

        self.type  = "sensor"
        self.id    = allocId()
        self.name  = name
        self.type  = typ
        self.cmdId = command.getId()

    def xml(self):

        s = ET.Element("sensor", id="%s"%self.id,
                       name=self.name, type=self.type)
        s.append(ET.Element("include", type="command", ref="%s"%self.cmdId))

        return s
        
class label(component):

    def __init__(self, name, sensor):

        self.type     = "label"
        self.id       = allocId()
        self.name     = name
        self.sensorId = sensor.getId()

    def xml(self):

        s = ET.Element("label", id="%s"%self.id)
        s.append(ET.Element("include", type="sensor", ref="%s"%self.sensorId))

        return s

class slider(component):

    def __init__(self, name, command, sensor):

        self.type     = "slider"
        self.id       = allocId()
        self.name     = name
        self.cmdId    = command.getId()
        self.sensorId = sensor.getId()

    def xml(self):

        s = ET.Element("slider", id="%s"%self.id)
        sv = ET.Element("setValue")
        sv.append(ET.Element("include", type="command", ref="%s"%self.cmdId))
        s.append(sv)
        s.append(ET.Element("include", type="sensor", ref="%s"%self.sensorId))

        return s

class config(object):

    def __init__(self):

        self.components = []
        self.sensors    = []
        self.commands   = []
        
    def addDimmerChannel(self, name, groupAddr, sensGroupAddr=None):

        # If sensor group address not specified, use the same
        # group address as for the set dimmer value
        if sensGroupAddr == None:
            sensGroupAddr = groupAddr
            
        dimCmd    = command("Dim", groupAddr, "5.001")
        dimStat   = command("status", sensGroupAddr, "5.001")
        dimSensor = sensor(name, "level", dimStat)
        dimLabel  = label(name, dimSensor)

        dimSlider = slider(name, dimCmd, dimSensor)
        
        self.components.append(dimSlider)
        self.components.append(dimLabel)
        self.sensors.append(dimSensor)
        self.commands.append(dimCmd)
        self.commands.append(dimStat)

    def getComponent(self, typ, name):
        # print "finding: %s/%s" %(typ, name)
        for c in self.components:
            # print "  %s/%s" %(c.getType(),  c.getName())
            if c.getType() == typ and c.getName() == name:
                return c

        # not found...
        raise TypeError

    def getSlider(self, name):

        return self.getComponent("slider", name)

    def getLabel(self, name):

        return self.getComponent("label", name)

    def write(self, file):

        root = ET.Element("openremote")
        root.set("xmlns", "http://www.openremote.org")
        root.set("xmlns:xsi", "http://www.w3.org/2001/XMLSchema-instance")
        root.set("xsi:schemaLocation","http://www.openremote.org "
                 "http://www.openremote.org/schemas/controller.xsd")

        components = ET.Element("components")
        sensors    = ET.Element("sensors")
        commands   = ET.Element("commands")

        for c in self.components:
            components.append(c.xml())
            
        for s in self.sensors:
            sensors.append(s.xml())

        for c in self.commands:
            commands.append(c.xml())
        
        root.append(components)
        root.append(sensors)
        root.append(commands)

        # Add the "static" configuration stuff at the bottom...
        
        c = ET.Element("config")
        for cc in controller_config:
            c.append(ET.Element("property", name=cc["name"], value=cc["value"]))
        root.append(c)
        
        prettyPrint(root)
        
        tree = ET.ElementTree(root)
        tree.write(file, "UTF-8")

class gui(object):

    def __init__(self):

        self.panels  = []
        self.screens = []
        self.groups  = []

    def addGroup(self, group):

        self.groups.append(group)

    def addPanel(self, panel):

        self.panels.append(panel)

    def addScreen(self, screen):

        self.screens.append(screen)
    
    def write(self, file):

        root = ET.Element("openremote")
        root.set("xmlns", "http://www.openremote.org")
        root.set("xmlns:xsi", "http://www.w3.org/2001/XMLSchema-instance")
        root.set("xsi:schemaLocation","http://www.openremote.org "
                 "http://www.openremote.org/schemas/panel.xsd")

        panels = ET.Element("panels")
        for p in self.panels:
            panels.append(p.xml())
        root.append(panels)

        screens = ET.Element("screens")
        for scr in self.screens:
            screens.append(scr.xml())
        root.append(screens)

        groups = ET.Element("groups")
        for g in self.groups:
            groups.append(g.xml())
        root.append(groups)

            
        prettyPrint(root)
        
        tree = ET.ElementTree(root)
        tree.write(file, "UTF-8")


class panel(object):

    def __init__(self, name, group):

        self.id      = allocId()
        self.name    = name
        self.groupId = group.getId()

    def xml(self):

        s = ET.Element("panel", name=self.name, id="%s"%self.id)
        s.append(ET.Element("include", type="group", ref="%s"%self.groupId))

        return s

class group(object):

    def __init__(self, name, screen):

        self.id         = allocId()
        self.name       = name
        self.screenRef  = screen.getId()

    def getId(self):
        return self.id

    def xml(self):

        s = ET.Element("group", name=self.name, id="%s"%self.id)
        s.append(ET.Element("include", type="screen", ref="%s"%self.screenRef))

        return s


class screen(object):

    def __init__(self, name):

        self.id      = allocId()
        self.name = name
        self.components = []

    def getId(self):
        return self.id
    
    def addSlider(self, slider, posX, posY, width, height):

        self.components.append(("slider",
                                (posX, posY, width, height),
                                (slider.getId(), slider.getSensorId())))

    def addLabel(self, label, initialText, posX, posY, width, height):

        self.components.append(("label",
                                (posX, posY, width, height),
                                (label.getId(), label.getSensorId(),
                                 initialText)))

    def xml(self):
        
        s = ET.Element("screen", name=self.name, id="%s"%self.id)

        for c  in self.components:

            typ, position, params = c

            x,y,w,h = position
            
            p = ET.Element("absolute", left="%s"%x, top="%s"%y,
                           width="%s"%w, height="%s"%h)
            
            if typ == "slider":

                sid, sref = params
                
                sl = ET.Element("slider", id="%s"%sid,
                                vertical="false", passive="false")
                sl.append(ET.Element("link", type="sensor", ref="%s"%sref))
                sl.append(ET.Element("min", value="0"))
                sl.append(ET.Element("max", value="100"))

                p.append(sl)

            elif typ == "label":

                lid, sref, ltext = params
                
                sl = ET.Element("label", id="%s"%lid,
                                fontsize="14", color="#FFFFFF", text=ltext)
                sl.append(ET.Element("link", type="sensor", ref="%s"%sref))

                p.append(sl)

            s.append(p)
            
        return s

        
if __name__ == "__main__":

    cfg = config()
    gui = gui()

    #
    # List the devices in the configuration
    #
    cfg.addDimmerChannel("LightZone 1", "1/1/6")
    cfg.addDimmerChannel("LightZone 2", "1/1/7")
    cfg.addDimmerChannel("LightZone 3", "1/1/8")
    cfg.addDimmerChannel("LightZone 5", "1/1/9")
    cfg.addDimmerChannel("LightZone 6", "1/1/10")

    #
    # Then specify the panel(s)
    #
    scr = screen("Starting Screen")
    scr.addSlider(cfg.getSlider("LightZone 1"), 51, 20, 198, 44)
    scr.addLabel(cfg.getLabel("LightZone 1"),
                 "Label Text", 51, 70, 198, 44)
    scr.addSlider(cfg.getSlider("LightZone 3"), 51, 120, 198, 44)
    scr.addLabel(cfg.getLabel("LightZone 3"),
                 "Label Text", 51, 170, 198, 44)

    grp = group("Default Group", scr)

    pnl = panel("TestPanel", grp)
    
    gui.addScreen(scr)
    gui.addGroup(grp)
    gui.addPanel(pnl)

    #
    # Lastly, write the XML files..
    #
    cfg.write(open("controller.xml", "w"))
    gui.write(open("panel.xml", "w"))
Posted by trondk at Nov 11, 2010 15:08

Excellent, thanks!

This here precisely demonstrates the issue with designer today:

    #
    # List the devices in the configuration
    #
    cfg.addDimmerChannel("LightZone 1", "1/1/6")
    cfg.addDimmerChannel("LightZone 2", "1/1/7")
    cfg.addDimmerChannel("LightZone 3", "1/1/8")
    cfg.addDimmerChannel("LightZone 5", "1/1/9")
    cfg.addDimmerChannel("LightZone 6", "1/1/10")

Which says, once we know what protocol you're dealing with, we can derive a lot of defaults from limited set of information.

The challenge, and the headache today, is how to deal with this in a generic enough fashion so when someone comes up with a new device protocol, it can be supported easily in a similar fashion, without requiring hard-coded protocol support in the designer.

What your script implies to me is a flow of (leaving out the UI part):

Select protocol: KNX -> Create Slider -> Enter name and group address

Read/write commands and sensor can then be created with default values, and only modified if necessary (it seems to happen sometimes read and write come from different group addresses, etc.)

Top-down vs. bottom-up.

We could hard code this reverse flow for KNX (and maybe we should) but it still leaves the main issue open of how do we do the same for the next protocol without it all costing too much in terms of designer implementation per protocol. I don't have an obvious solution to it yet. Maybe ideas will pop up. At the moment, the bottom-up approach works for the "generic" protocol but leaves a lot of frustrating click-work to the user.

Maybe the solution then is to make user written import tools (such as your script) cheap to add to the designer, instead? Putting the cost back to individual protocol implementers.

And maybe the third major area where scripting is now sorely needed.

Hmm.

Posted by juha at Nov 11, 2010 20:13

Further thoughts on this, how do we make it simple to add new protocol support to designer without knowing the specific protocol details in advance?

The part that is customizable in designer now is the protocol dialog, ie. the dialog for KNX commands are not hard-coded but are defined in knx.xml

Part of the issue is that this XML UI definition works on the simple v1.0 vocabulary of "commands" - i.e the simple control abstraction of commands bound to buttons to control. Something Marc is now repeating with the serverless android experience.

What we did in 2.0 was increase this vocabulary to read/write commands, sensors, switches, and sliders. So now we have a larger vocabulary, but the protocol UI customizations still operate on the 1.0 vocabulary (how do I enter a command?)

So one way of addressing this is to extend the designer customization mechanism to deal with all the new constructs.

So for example, the use cases for KNX user might be:

1. Create KNX Switch
2. Create KNX Dimmer
3. Create KNX Scene
4. ....

In the designer flow, these would follow from:

New Model -> Which Protocol? KNX -> 1. Create KNX Switch, 2. Create KNX Dimmer ...

Where the dialogs and flow for particular KNX use cases can be specified and externalized in the designer configuration using xml and/or scripting. Today this only allows for command properties, and therefore all the other abstractions are built up from bottom-up (single vocabulary of "commands").

So this translates to a much richer mechanism to customize the designer UI flow:

1. Create KNX Switch

becomes

a. Designer Dialog for properties 'name' and 'group address'
b. Designer Invocation: Create 'Write KNX Switch On Command' with default name
c. Designer Invocation: Create 'Write KNX Switch Off Command' with default name
d. Designer Invocation: Create 'Read KNX Switch Status Command' with default name
e. Designer Invocation: Create 'KNX Switch Sensor with Read Command (d)' with default name
f. Designer Invocation: Create Switch with aggregate of (b), (c), (d), (e) using 'name'

Where (a) to (f) are specified in generic language that is not specific to individual protocols but our current abstractions of 'command', 'sensor', 'switch', 'slider'. And (a) to (f) are externalized as configuration of the designer.

This reduces the necessary click-heavy approach by presenting a dialog to new users that they can follow (without necessarily introducing them to a script immediately which they are expected to modify). It is not as low effort as the script, so for a large installation of dozens or even hundreds of dimmers, modifying a script to do the job may still be preferred. But would be an option for the power users, not the expected initial experience to the designer.

Posted by admin at Nov 12, 2010 05:57

Further note:

"Designer Invocations" in this case should be REST API Invocations against Beehive database to create and store these constructs on individual user accounts.

Something that was always intended to be there but was lost in the current 2.0 implementation quagmire.

It's been on the todo list for a while to correct this design mistake, this would be another reason to drive it home.

Posted by admin at Nov 12, 2010 06:02

Maybe a crazy thought here.. but I find the OR KNX app easy in configuring dimmers, lights and shutters.

Why not have the ability to read the config file that one stores on google apps into the OR BOSS designer interface?

Rick

Posted by rickcn at Dec 03, 2010 19:32
Document generated by Confluence on Jun 05, 2016 09:31