This page last changed on Dec 07, 2013 by juha.

Gentlemen (and Ladies),

First let me commend all the great folks that have created and contributed work to Openremote. I am moving from the 'Mote' app to Openremote as it is much more powerful. Thanks everyone.

Now, here is my problem...I have a sensor that is uses a HTTP call using the GET method, it returns data that I am parsing using regex. (I have the same problem with HTTP/GET returning JSON formatted data and parsing using the JSON parser). The regex works great except when the returned data contains a '&', such as the following...

The HTTP/GET returns the following data, I want just the artist and song (basically skip the first line). This is from a status.txt request from SqueezeboxServer.

Now Playing (299 of 369)
Starlight Orchestra & Choir - Silver Bells

I am using the following regex parser syntax,

\n([a-zA-Z0-9\s-\.\&,()'.]){1,}

This works for all types of things except for when the data contains a '&', then the sensor breaks until the data returned does not contain that character.

Can anyone suggest a solution?

BTW, a JSON formatted return from a DirecTV call for current program also has the same problem, if the data contains a '&' the sensor breaks, for instance for the show 'Law & Order'.

Thanks,
Erik

When you say the sensor breaks, is there an error on the logs, or does it just mean the regex fails to parse to a desired result?

Posted by juha at Dec 07, 2013 11:52

Hi Juha,

Thanks for your reply. I have looked at the logs and there are no errors, in fact the regex works as well according to the log file...

2013-12-07 09:50:08,089 INFO [Polling thread for sensor: Squeezebox_ArtistSong]: received message: Now Playing (360 of 365)
Starlight Orchestra & Choir - We Three Kings Of Orient Are

2013-12-07 09:50:08,089 INFO [Polling thread for sensor: Squeezebox_ArtistSong]: result of regex evaluation:
Starlight Orchestra & Choir - We Three Kings Of Orient Are

2013-12-07 09:50:08,089 TRACE [Polling thread for sensor: Squeezebox_ArtistSong]: Processed '
Starlight Orchestra & Choir - We Three Kings Of Orient Are
', received 'Starlight Orchestra & Choir - We Three Kings Of Orient Are'

The label that I am trying to update using the sensor data is not updating when there is a '&' in the message. That is, the previous label data remains on the label while the sensor is returning text with a '&', when the sensor returns text that does not contain a '&' the label updates to the new data.

In summary, there is no error in the sensor log, and the regex seems to work correctly, but the sensor cannot update the label when the data contains a '&'.

BTW, I just got my first slider switch to work with my Veralite, totally awesome, thanks guys you rock.

Erik

Posted by skogen75 at Dec 07, 2013 17:02

Ok, is this on iOS, Android or web console?

Posted by juha at Dec 07, 2013 17:19

This is using...
iOS
OpenRemote-Controller-2.1.0_SNAPSHOT-2013-06-17
Designer (Markab)
running controller on a win7 OS

have never tried the web console, but I just tried and don't know how to get the web console to work. I'll keep trying though and report back if I learn anything new.

Thanks,
Erik

Posted by skogen75 at Dec 07, 2013 19:17

Sure, if you can check if the behavior is the same with the web console, might be good for comparison. There's some instructions to use web console here: Web Console

Posted by juha at Dec 07, 2013 19:35

Okay, I got the web console working...

The web console works fine until an artist/song name contains the '&' and then the following error message pops up...

Web Console
Error!

The following error occurred: -

XML to JSON Parsing Error

At that point the web console crashes and the panels need to be reloaded. I checked the dev.log and no error is listed, and actually indicated successful parsing of the regex like what I showed in the post above.

Strange...

Thanks,
Erik

Posted by skogen75 at Dec 07, 2013 19:55

Ok thanks,

I think Eric is following the thread and may check if he can reproduce it on iOS (when he has time). That both iOS and web console would fail in the same issue seems a bit strange since they are independently developed codebases. But once he has time to try to verify the issue, should be able to test the same case for both iOS and web consoles.

– Juha

Posted by juha at Dec 08, 2013 05:04

I did reproduce this, testing on iOS.
Issue is that & is not escaped / enclosed in CDATA in the XML providing the sensor values, e.g.

<?xml version="1.0" encoding="UTF-8"?>
<openremote xmlns="http://www.openremote.org" 
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
            xsi:schemaLocation="http://www.openremote.org 
            http://www.openremote.org/schemas/controller.xsd">
<status id="232163">Header & Trailer</status>

This is not valid XML and the parsers in the consoles will just not process that.
So first fix is to ensure the XML generated by controller is valid, then assess again how the consoles behave.

Posted by ebariaux at Dec 09, 2013 09:16

The W3 consortium maintains a Markup validation service at: http://validator.w3.org/#validate_by_input
There you can test the validity of "web documents" using either URL's, uploaded files or just pasted text.
Has been very useful for me over the years

Posted by pz1 at Dec 09, 2013 10:35

Eric,

Yea, I kind of figured something like that was going on. I'm a little confused on what the fix is though. Will you do something in the controller, perhaps replace all '&' with '/&', or are you saying I should do something to prevent the controller from ever getting the '&'? Of course I don't want to bother you guys, you're working very hard as it is, so thanks for looking into this for me. I also saw on the forum that someone else has posted a similar problem with the '&'.

I have one more question, maybe this is a new thread, is there a character limit in a reply from a device on which the regex parser will operate? I have run into a problem with a long return string from a device and I cannot get a valid parsed result within the Openremote environment. My regex expression works in various testers I have tried.

Once again, thanks very much, I am so glad I found Openremote.

Erik

Posted by skogen75 at Dec 09, 2013 14:28

For my music collection in Squeezebox I used the MP3tag "action command" to globally change in the Titles all "&" into "and".
Took a while for the whole collection.

Posted by pz1 at Dec 09, 2013 14:48

Controller code needs to be fixed to produce valid XML, ideally writing a unit test first to exhibit the problem and prove it's gonna be fixed.
If somebody wants to have a go at it, I can point in the right direction, otherwise it'll have to wait a bit as I need to finish some higher priority tasks on my TODO list.

Posted by ebariaux at Dec 09, 2013 15:59

@Eric, great thanks. I suppose that this will be included in the update log when a new snapshot of controller comes out? (If you get to it before a new version comes out, that is).

@Pieter, thanks for the suggestion and yea, I started to update my library tags. But, I also return program information from DirecTV and there are a surprising number of television shows that have '&' in the name, this is something I cannot work around.

Thanks all.

Posted by skogen75 at Dec 09, 2013 16:56

Took a quick glance at the code and it's very local, so I tried a fix and on a 1min test it does seem to work.

Code that generate this XML output is in StatusPollingServiceImpl.java and StatusCommandServiceImpl.java (for poll and status calls respectively) and generates the XML by manually building the string.

I did use StringEscapeUtils.escapeXml() from Apache lang framework (already used by controller) to escape the sensor values and that did the trick.
So e.g. for StatusPollingServiceImpl, I end up with this line

sb.append(StringEscapeUtils.escapeXml(changedStatuses.get(sensorID)));

If you can compile the controller yourself, then this should be pretty easy to apply.

Although this works, as longer term fix, I'm thinking going to an XML library (jdom or dom4j) to build the DOM and generate the string should be the way to go. We can also have a schema validate that the generated XML is per our specs.
Juha, a thought on that ?

BTW, I did create issue JIRA-416 to track this.

Posted by ebariaux at Dec 10, 2013 08:06

I think we already use JDOM. The one liner fix is fine. The validation will be relevant to add later when the response documents contain more than individual sensor values (e.g. property changes driven by server-side rules).

Posted by juha at Dec 11, 2013 07:51

I'll create a patch branch with the quick fix.

Posted by ebariaux at Dec 11, 2013 08:50

Gentlemen,

I would be more than happy to give the fix a long term test in my environment; if you would be so kind as to direct me to the patch and give me a few instructions as to how to implement/install the patch on my system.

I would report back with any results.

Thanks,
Erik

Posted by skogen75 at Dec 12, 2013 13:28

The patch branch is here : svn+ssh://svn.code.sf.net/p/openremote/code/patches/individual/Controller_2_0_2_ORCJAVA-416

If all the features you need from the controller are included in v2.0.2, then you can check out that branch using an svn client and build (just typing ant will build a controller package in output folder).

Posted by ebariaux at Dec 13, 2013 08:51
Document generated by Confluence on Jun 05, 2016 09:34