This page last changed on Sep 15, 2015 by pz1.

I do have a problem with this OpenRemote Http-command and my ZWay (RaZberry) installation. Apparantly something is going wrong with the authentication. I do get a Permission denied message in my dev.log

If I do issue the following curl command in my raspberrypi console, it does authenticate, and retruns the correct status:

pi@quadras ~ $ curl -v -u JohnDoe:SeCret http://raspberrypi:zzzz/OpenRemote/SwitchBinaryStatus/9/0
* About to connect() to raspberrypi port zzzz (#0)
*   Trying 192.168.xxx.yyy...
* connected
* Connected to raspberrypi (192.168.xxx.yyy) port zzzz (#0)
* Server auth using Basic with user 'JohnDoe'
> GET /OpenRemote/SwitchBinaryStatus/9/0 HTTP/1.1
> Authorization: Basic Sm9obkRvZTpTZUNyZXQ=
> User-Agent: curl/7.26.0
> Host: raspberrypi:zzzz
> Accept: */*
>
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< Connection: keep-alive
< Content-Type: application/json
< Content-Length: 4
< Transfer-Encoding: chunked
<
* Connection #0 to host raspberrypi left intact
"on"* Closing connection #0

What am I missing in my Http call specification as in the image (top right)

Thank you for your attention


authenticate.PNG (image/png)

What exactly does the dev.log show?
PermissionDenied could be something else. Do you see a HTTP error code?
-Marcus

Posted by mredeker at Sep 16, 2015 00:07

Sorry I should have been more precise with my wording. This is what the log says:

    2015-09-15 19:04:54,938 ERROR [Polling Sensor Thread ID = 248217, Name ='Duwi']: ClientProtocolException when executing HTTP method
    org.apache.http.client.HttpResponseException: Forbidden
       at org.apache.http.impl.client.BasicResponseHandler.handleResponse(BasicResponseHandler.java:67)
       at org.apache.http.impl.client.BasicResponseHandler.handleResponse(BasicResponseHandler.java:54)
       at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:735)
       at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:709)
       at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:700)
       at org.openremote.controller.protocol.http.HttpGetCommand.requestURL(Unknown Source)
       at org.openremote.controller.protocol.http.HttpGetCommand.read(Unknown Source)
       at org.openremote.controller.model.sensor.Sensor$DeviceReader.read(Unknown Source)
       at org.openremote.controller.model.sensor.Sensor$DeviceReader.run(Unknown Source)
       at java.lang.Thread.run(Thread.java:745)

The Zway developers suspect that OpenRemote does send the url in the form of http://user:passwd@host instead of sending a "proper" http-header (see this post)
I am using Pro 1.2.0

Posted by pz1 at Sep 16, 2015 07:51

We do send proper HTTP header for basic authentication and do not include user and password in URL.
Do you see anything in z-way logs?
Is the machine you executed the curl command from and the machine running the controller the same machine?

Posted by mredeker at Sep 16, 2015 09:43

We do send proper HTTP header for basic authentication and do not include user and password in URL.

Thanks that brings me one step further

Is the machine you executed the curl command from and the machine running the controller the same machine?

Just did a successfull curl on my Synology DS212+, where the OR controller resides

Do you see anything in z-way logs?

Yes, it is a cryptic error message from the OpenRemoteHelpers API module. As said without authentication on both sides this works just fine.

[2015-09-15 20:50:18.677] [E] [8083] Callback execution error: TypeError: Cannot read property 'data' of undefined
    at OpenRemote (automation/modules/OpenRemoteHelpers/index.js:155:65)
    at WebServer.document_root (automation/Webserver.js:27:11)

I am going to do some experiments with their JSON-API, and the vDev API to see how those two behave. I'll be back

Posted by pz1 at Sep 16, 2015 10:03

I did get the OR authentication working for ZWaveAPI. So I think there is something wrong with the authentication via OpenRemoteHelpers

Posted by pz1 at Sep 16, 2015 12:16

Marcus,
Was to quick with my conclusion. It appears that ZWaveAPI does not (yet?) require authentication, so it was obvious it worked. But also the virtual device interface (ZAutomation) did not authenticate, so I can not yet pinpoint where the problem is.

2015-09-16 14:18:44,903 ERROR [Polling Sensor Thread ID = 248580, Name ='Duwitruth']: ClientProtocolException when executing HTTP method
org.apache.http.client.HttpResponseException: Unauthorized
	at org.apache.http.impl.client.BasicResponseHandler.handleResponse(BasicResponseHandler.java:67)
	at org.apache.http.impl.client.BasicResponseHandler.handleResponse(BasicResponseHandler.java:54)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:735)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:709)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:700)
	at org.openremote.controller.protocol.http.HttpGetCommand.requestURL(Unknown Source)
	at org.openremote.controller.protocol.http.HttpGetCommand.read(Unknown Source)
	at org.openremote.controller.model.sensor.Sensor$DeviceReader.read(Unknown Source)
	at org.openremote.controller.model.sensor.Sensor$DeviceReader.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:745)

Do you happen to have some recent dump of the http header that I can show as an example? I have looked into using wireshark, but the 200+ page manual is rather frightening, so I'd rather avoid that.

Posted by pz1 at Sep 16, 2015 13:32

I assume this is an issue on Zway side. You receive two different responses. The first is "403 Forbidden" and now you receive "401 Unauthorized".
The first actually means that the user was authorized but has no rights wheres the 401 means user or password is wrong.

Posted by mredeker at Sep 16, 2015 21:56

I did check again. Kept user and pw, but changed url. Apparantly the different interfaces do give different responses. At least something seems to be wrong at the Zway side.

Posted by pz1 at Sep 17, 2015 09:46

I did run TCPDUMP (howto) on the RaspberryPi that hosts ZWay and on Synology DS212+ (OR controller) to check if authentication data is being send. That seems not to be the case. (Pro 1.2.0)

I gave the following command:

 sudo tcpdump -A -v -s 10240 'tcp port 8083 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 10240 bytes
16:39:54.401620 IP (tos 0x0, ttl 64, id 28065, offset 0, flags [DF], proto TCP (6), length 217)
    DS212.fritz.box.51536 > raspberry.fritz.box.8083: Flags [P.], cksum 0x5f84 (correct), seq 1900776448:1900776613, ack 17874579, win 183, options [nop,nop,TS val 361883232 ecr 16443814], length 165
E...m.@.@.H....!.....P..qK.........._......
...`....GET /OpenRemote/SwitchBinaryStatus/9/0 HTTP/1.1
User-Agent: OpenRemoteController
Content-Type: application/text
Host: raspberrypi:8083
Connection: Keep-Alive


16:39:54.423896 IP (tos 0x0, ttl 64, id 22326, offset 0, flags [DF], proto TCP (6), length 271)
    raspberry.fritz.box.8083 > DS212.fritz.box.51536: Flags [P.], cksum 0x848f (incorrect -> 0x4413), seq 1:220, ack 165, win 470, options [nop,nop,TS val 16443816 ecr 361883232], length 219
E...W6@.@._%.......!...P....qK.............
.......`HTTP/1.1 403 Forbidden
Connection: close
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Type: text/plain
Content-Length: 17
Transfer-Encoding: chunked

11
Permission denied
0

Same command issued on Synology DS212+ (OR controller):

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 10240 bytes
19:06:39.556139 IP (tos 0x0, ttl 64, id 60154, offset 0, flags [DF], proto TCP (6), length 217)
    DS212.49681 > raspberry.fritz.box.8083: Flags [P.], cksum 0x8459 (incorrect -> 0xff89), seq 2526527256:2526527421, ack 1226417713, win 183, options [nop,nop,TS val 362763759 ecr 17324330], length 165
E.....@.@......!............I..1.....Y.....
..U...Y*GET /OpenRemote/SwitchBinaryStatus/9/0 HTTP/1.1
User-Agent: OpenRemoteController
Content-Type: application/text
Host: raspberrypi:8083
Connection: Keep-Alive


19:06:39.578540 IP (tos 0x0, ttl 64, id 65308, offset 0, flags [DF], proto TCP (6), length 271)
    raspberry.fritz.box.8083 > DS212.49681: Flags [P.], cksum 0xe418 (correct), seq 1:220, ack 165, win 470, options [nop,nop,TS val 17324332 ecr 362763759], length 219
E.....@.@..>.......!....I..1...............
..Y,..U.HTTP/1.1 403 Forbidden
Connection: close
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Content-Type: text/plain
Content-Length: 17
Transfer-Encoding: chunked

11
Permission denied
0

I do not see any authentication information in the HTTP header. I must admit that I am totally new to this. So this may not be the right tool and/or command parameters

Posted by pz1 at Sep 17, 2015 15:43

Unfortunately the http.log file does not give any more information about authentication TRACE mode:

INFO 2015-09-18 16:14:06,763 (HTTP): received message:
DEBUG 2015-09-18 16:14:06,763 (HTTP): *** Out of read method:
DEBUG 2015-09-18 16:14:11,769 (HTTP): read sensor: Sensor (Name = 'Duwi', ID = '248217', State Mappings: {null=${state.getName()}})
ERROR 2015-09-18 16:14:11,797 (HTTP): ClientProtocolException when executing HTTP method
org.apache.http.client.HttpResponseException: Forbidden
at org.apache.http.impl.client.BasicResponseHandler.handleResponse(BasicResponseHandler.java:67)
etc.

Posted by pz1 at Sep 18, 2015 15:21

I assume this is an issue on Zway side. You receive two different responses. The first is "403 Forbidden" and now you receive "401 Unauthorized".
The first actually means that the user was authorized but has no rights wheres the 401 means user or password is wrong.

These different responses were an error on the ZWay side. Both those APIs have been harmonised in the meantime.
Had nothing to do with my problem, which seems to be that no user credentials are being sent

Posted by pz1 at Sep 18, 2015 15:26

Does the controller.xml show the correct information?
You can find it in the controller/webapps/controller folder-

Posted by mredeker at Sep 18, 2015 17:58

This is the Zway sensor that I am polling

   <command id="2625" protocol="http">
      <property name="username" value="JohnDoe" />
      <property name="pollingInterval" value="5s" />
      <property name="method" value="POST" />
      <property name="contentType" value="application/html" />
      <property name="password" value="SeCret" />
      <property name="url" value="http://raspberrypi:8083/OpenRemote/SwitchBinaryStatus/9/0" />
      <property name="name" value="Duwi Status" />
      <property name="urn:openremote:device-command:device-name" value="Power Switches" />
      <property name="urn:openremote:device-command:device-id" value="304" />
    </command>

During the test above I had contentType application/text.

update: I had checked that already since I do get these repeating error messages in my boot.log (I did sent a mail to Eric about that)

TRACE 2015-09-18 19:32:57,677 : Unable to retrieve controller identity
org.openremote.controller.exception.ConnectionException: The required password for user 'XXXXXXXXX' was not found. Password manager error : Password for user 'org.openremote.security.PasswordManager$PasswordNotFoundException: Implementation Error: password alias 'pieter' does not correspond to secret key entry in the keystore.' was not found
	at org.openremote.controller.service.BeehiveCommandCheckService$BeehiveCommandChecker.connect(Unknown Source)
	at org.openremote.controller.service.BeehiveCommandCheckService$BeehiveCommandChecker.httpRequest(Unknown Source)
	at org.openremote.controller.service.BeehiveCommandCheckService$BeehiveCommandChecker.httpPost(Unknown Source)
	at org.openremote.controller.service.BeehiveCommandCheckService$BeehiveCommandChecker.getControllerIdentity(Unknown Source)
	at org.openremote.controller.service.BeehiveCommandCheckService$BeehiveCommandChecker.getRemoteCommandService(Unknown Source)
	at org.openremote.controller.service.BeehiveCommandCheckService$BeehiveCommandChecker.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.openremote.controller.service.Deployer$PasswordException: Password for user 'org.openremote.security.PasswordManager$PasswordNotFoundException: Implementation Error: password alias 'XXXXXXX' does not correspond to secret key entry in the keystore.' was not found
	at org.openremote.controller.service.Deployer.getPassword(Unknown Source)
	... 7 more
INFO 2015-09-18 19:32:57,678 : Unable to retrieve controller identity, retrying in 1,000 milliseconds...
INFO 2015-09-18 19:32:57,678 : Unable to retrieve controller identity, retrying in 1,000 milliseconds...
TRACE 2015-09-18 19:32:58,958 : Unable to retrieve controller identity
Posted by pz1 at Sep 18, 2015 18:08

The BeehiveCommandChecker has nothing todo with the HTTP command for polling the sensor.
You also use the pro 1.2.0 controller, right?
I will try and check the HTTP command to make sure the authentication header is send.

Posted by mredeker at Sep 18, 2015 23:40

You also use the pro 1.2.0 controller, right?

Yes

Update I did install a controller on a Win7 PC. TCPDUMP results are the same; no login credentials
Update 2 Similar http authentication error is reported here in this thread

Posted by pz1 at Sep 19, 2015 08:10

I did some investigation and the situation is as follows:

1) If the username is not given then no authentication will be performed at all (empty username is not supported)

2) Our library first tries the connection without basic authentication. If that fails (HTTP 401 given by server) then the authentication header is added and the request is performed a second time. I confirmed this with tcpdump.

It looks like razberry is not following the HTTP specs. They send a "403 forbidden" even if no authentication information is provided.
They first need to send a "401 unauthorized" and only if the wrong user is given they are allowed to send "403 forbidden".

Regards,
-Marcus

Posted by mredeker at Oct 04, 2015 23:54

@marcus,
Thanks for taking the time. Did you see username/passw being sent in your TCPDUMP?

Posted by pz1 at Oct 05, 2015 10:18

Yes, it was sent correctly.
But only the second time after the server returned the 401 error.
The server has to return a 401 error if no authentication header is present but the resource is protected.
It seems that in your case the server already returns a 403 error if not authentication header is present. That is not correct.

Posted by mredeker at Oct 05, 2015 10:30

server already returns a 403 error

I think they fixed that 403 error in the meantime. I will check it there

Posted by pz1 at Oct 05, 2015 11:12

Marcus and I have been working hard on this authentication problem in the past days.
We came to the conclusion that Zway does not comply with the relevant IETF standard

Posted by pz1 at Nov 25, 2015 16:29
Document generated by Confluence on Jun 05, 2016 09:33