This page last changed on Jan 19, 2015 by aktur.
As my design on ebox starts to become too big I'm hitting limits after a while. First symptoms are falling shell commands:
ERROR 2015-01-19 20:51:51,545 (ShellEXE): Could not execute shell command: /bin/ping
java.io.IOException: Cannot run program "/bin/ping": java.io.IOException: error=12, Cannot allocate memory
What will be the best strategy to recover from this situation? Top shows that my controller java process is using 50-60% of the memory. I'm doing a brutal reboot now, but perhaps there is a more clever way?
You can try and increase the heap space of the Java VM.
Posted by mredeker at Jan 24, 2015 16:50
Increasing heap size did not help. What did help was:
echo 1 > /proc/sys/vm/overcommit_memory
but I'm not sure if this wont have some negative side effects. So far so good. What I read about overcommit_memory is:
Controls overcommit of system memory, possibly allowing processes to allocate (but not use) more memory than is actually available.
0 - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slighly more memory in this mode. This is the default.
1 - Always overcommit. Appropriate for some scientific applications.
2 - Don't overcommit. The total address space commit for the system is not permitted to exceed swap plus a configurable percentage (default is 50) of physical RAM. Depending on the percentage you use, in most situations this means a process will not be killed while attempting to use already-allocated memory but will receive errors on memory allocation as appropriate.
Posted by aktur at Apr 18, 2015 09:33
Never heard of it. A JVM is only using what the heap size has configured.
I don't see why that should bring anything even since it says "but not use".
If you think it does help even better
Posted by mredeker at Apr 18, 2015 21:17
Well, it helps but to the point of 'some scientific applications'... It is possible to brick an eBox to the point that only a clod boot can help. So obviously, this is not a solution for regular people.
I've dig the issue a bit more. The root of the problem lies in the implementation of Runtime.getRuntime().exec() method. It uses fork, and fork first allocates the amount of memory of original process (thus openremote including tomcat) and fails if it does not find it free in the physical memory. Therefore, DECREASING instead of INCREASING of the heap space actually helps a bit. However, the side effect is that the heap space is much lower now and yajsw needs to reboot the controller even more frequently. Therefore this solution is a pity of memory resources. At this point I see 2 options for better solving this issue:
1) As I've noticed, on the root cause is lies a fact that my shell command exec sensors are called few at the time, each one within its own thread, correct? Therefore serializing threads with semaphore should help. Limiting that only one thread can call Runtime.getRuntime().exec() at the time and hold other calls till this one frees the memory will obviously help.
2) Use a different method of calling shell commands, without the fork(). One such possibility is to use ssh to localhost and with RSA keys preallocation this method would be quite transparant in usage. Thre are some native ssh libraries available for Java.
Now my question would be, which solution would be preferable? In my opinion adding ssh protocol would be a better solution. It would not only help to solve the shell execution protocol memory problems, but add a new way of executing commands on remote hosts. Quite a nice feature, think off:
ssh root@somehost reboot
ssh someuser@somehost ls -lt rss/
Posted by aktur at May 07, 2015 14:38