B4J Question b4j server down. which log file to watch

[juvanum]

I installed my application (B4J SERVER using ABMATERIAL) on a DEBIAN GNU / Linux vps server.
every 15 days (more or less) unexpectedly my application goes down.

I looked in the nohup.out file but found nothing of significance.

What other logs should I look at to find the reason for the crash?

These are the files in the / var / log directory

thanks

 

[Erel]

every 15 days (more or less) unexpectedly my application goes down.

What exactly happens? The process is killed? Check it with ps -ef |grep java

 

[juvanum]

What exactly happens? The process is killed? Check it with ps -ef |grep java

The process is killed.
When I check it with "ps -aux" the abj-server (alfa.jar) does not appear in the list.

I activate 2 abj-servers: alfa.jar and beta.jar
alfa.jar responds on port 5000
beta.jar responds on port 80

the app alfa.jar crashes
the app beta.jar run normally

 

[juvanum]

more info
file log messages.1

Nov 16 06:25:28 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="24536" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 16 08:45:10 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="24536" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 16 08:45:10 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="27703" x-info="http://www.rsyslog.com"] start
Nov 16 18:20:13 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="27703" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 16 18:20:14 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="31312" x-info="http://www.rsyslog.com"] start
Nov 17 03:58:55 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="31312" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 17 03:58:55 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="2417" x-info="http://www.rsyslog.com"] start
Nov 17 06:25:05 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="2417" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 17 14:03:34 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="2417" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 17 14:03:34 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="5996" x-info="http://www.rsyslog.com"] start
Nov 18 01:28:08 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="5996" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 18 01:28:08 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="9924" x-info="http://www.rsyslog.com"] start
Nov 18 06:25:04 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="9924" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 18 09:49:14 webapp1 kernel: [16161023.721181] VM Periodic Tas invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0
Nov 18 09:49:15 webapp1 kernel: [16161023.733987] VM Periodic Tas cpuset=/ mems_allowed=0
Nov 18 09:49:15 webapp1 kernel: [16161023.739756] CPU: 0 PID: 18459 Comm: VM Periodic Tas Not tainted 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Nov 18 09:49:15 webapp1 kernel: [16161023.749512] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Nov 18 09:49:15 webapp1 kernel: [16161023.759096] 0000000000000000 ffffffffaef29964 ffffb3c9405c7c28 ffff932cc1e55100
Nov 18 09:49:15 webapp1 kernel: [16161023.767351] ffffffffaedfeb70 0000000000000000 0000000000000000 0000000c024213ca
Nov 18 09:49:15 webapp1 kernel: [16161023.775579] ffffffffaed84c47 00000042c1adb140 0000000000000000 0000000000000001
Nov 18 09:49:15 webapp1 kernel: [16161023.783924] Call Trace:
Nov 18 09:49:15 webapp1 kernel: [16161023.786877] [<ffffffffaef29964>] ? dump_stack+0x5c/0x78
Nov 18 09:49:15 webapp1 kernel: [16161023.792628] [<ffffffffaedfeb70>] ? dump_header+0x78/0x1fd
Nov 18 09:49:15 webapp1 kernel: [16161023.798551] [<ffffffffaed84c47>] ? get_page_from_freelist+0x3f7/0xb40
Nov 18 09:49:15 webapp1 kernel: [16161023.805445] [<ffffffffaed80bda>] ? oom_kill_process+0x21a/0x3e0
Nov 18 09:49:15 webapp1 kernel: [16161023.811819] [<ffffffffaed8085d>] ? oom_badness+0xed/0x170
Nov 18 09:49:15 webapp1 kernel: [16161023.817852] [<ffffffffaed81071>] ? out_of_memory+0x111/0x470
Nov 18 09:49:15 webapp1 kernel: [16161023.824059] [<ffffffffaed862e7>] ? __alloc_pages_slowpath+0xbb7/0xbf0
Nov 18 09:49:15 webapp1 kernel: [16161023.830972] [<ffffffffaed8651e>] ? __alloc_pages_nodemask+0x1fe/0x260
Nov 18 09:49:15 webapp1 kernel: [16161023.837988] [<ffffffffaedd6981>] ? alloc_pages_current+0x91/0x140
Nov 18 09:49:15 webapp1 kernel: [16161023.844550] [<ffffffffaed7f070>] ? filemap_fault+0x300/0x5a0
Nov 18 09:49:15 webapp1 kernel: [16161023.850689] [<ffffffffc01cd7e1>] ? ext4_filemap_fault+0x31/0x50 [ext4]
Nov 18 09:49:15 webapp1 kernel: [16161023.857740] [<ffffffffaedaf6a1>] ? __do_fault+0x81/0x170
Nov 18 09:49:15 webapp1 kernel: [16161023.863521] [<ffffffffaedb3e20>] ? handle_mm_fault+0xd80/0x12d0
Nov 18 09:49:15 webapp1 kernel: [16161023.869980] [<ffffffffaec24761>] ? __switch_to+0x2c1/0x6d0
Nov 18 09:49:15 webapp1 kernel: [16161023.876042] [<ffffffffaec5ee5c>] ? __do_page_fault+0x25c/0x500
Nov 18 09:49:15 webapp1 kernel: [16161023.882383] [<ffffffffaf208b58>] ? page_fault+0x28/0x30
Nov 18 09:49:15 webapp1 kernel: [16161023.888157] Mem-Info:
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] active_anon:127679 inactive_anon:1903 isolated_anon:0
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] active_file:725 inactive_file:1562 isolated_file:0
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] unevictable:0 dirty:1 writeback:0 unstable:0
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] slab_reclaimable:1923 slab_unreclaimable:3906
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] mapped:1350 shmem:1955 pagetables:1189 bounce:0
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] free:7792 free_pcp:30 free_cma:0
Nov 18 09:49:15 webapp1 kernel: [16161023.925504] Node 0 active_anon:510716kB inactive_anon:7612kB active_file:2900kB inactive_file:6248kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:5400kB dirty:4kB writeback:0kB shmem:7820kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:33805 all_unreclaimable? yes
Nov 18 09:49:15 webapp1 kernel: [16161023.955076] Node 0 DMA free:3008kB min:788kB low:984kB high:1180kB active_anon:12448kB inactive_anon:0kB active_file:0kB inactive_file:132kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:40kB kernel_stack:80kB pagetables:32kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 18 09:49:15 webapp1 kernel: [16161023.986635] lowmem_reserve[]: 0 555 555 555 555
Nov 18 09:49:15 webapp1 kernel: [16161023.992272] Node 0 DMA32 free:28160kB min:28208kB low:35260kB high:42312kB active_anon:498268kB inactive_anon:7612kB active_file:2900kB inactive_file:6104kB unevictable:0kB writepending:4kB present:612340kB managed:590876kB mlocked:0kB slab_reclaimable:7692kB slab_unreclaimable:15584kB kernel_stack:2432kB pagetables:4724kB bounce:0kB free_pcp:120kB local_pcp:120kB free_cma:0kB
Nov 18 09:49:15 webapp1 kernel: [16161024.026162] lowmem_reserve[]: 0 0 0 0 0
Nov 18 09:49:15 webapp1 kernel: [16161024.031581] Node 0 DMA: 8*4kB (UM) 2*8kB (U) 3*16kB (U) 1*32kB (U) 11*64kB (UM) 11*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3008kB
Nov 18 09:49:15 webapp1 kernel: [16161024.047958] Node 0 DMA32: 138*4kB (UME) 71*8kB (UE) 22*16kB (UE) 90*32kB (UME) 102*64kB (UM) 67*128kB (UM) 20*256kB (UM) 7*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 28160kB
Nov 18 09:49:15 webapp1 kernel: [16161024.067172] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 18 09:49:15 webapp1 kernel: [16161024.076373] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 18 09:49:15 webapp1 kernel: [16161024.085300] 4259 total pagecache pages
Nov 18 09:49:15 webapp1 kernel: [16161024.089482] 0 pages in swap cache
Nov 18 09:49:15 webapp1 kernel: [16161024.093181] Swap cache stats: add 0, delete 0, find 0/0
Nov 18 09:49:15 webapp1 kernel: [16161024.098961] Free swap = 0kB
Nov 18 09:49:15 webapp1 kernel: [16161024.102514] Total swap = 0kB
Nov 18 09:49:15 webapp1 kernel: [16161024.105899] 157083 pages RAM
Nov 18 09:49:15 webapp1 kernel: [16161024.109265] 0 pages HighMem/MovableOnly
Nov 18 09:49:15 webapp1 kernel: [16161024.113494] 5387 pages reserved
Nov 18 09:49:15 webapp1 kernel: [16161024.117004] 0 pages hwpoisoned
Nov 18 09:49:15 webapp1 kernel: [16161024.120448] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Nov 18 09:49:15 webapp1 kernel: [16161024.129485] [ 416] 0 416 13954 381 24 3 0 0 systemd-journal
Nov 18 09:49:15 webapp1 kernel: [16161024.139368] [ 436] 0 436 11485 305 22 3 0 -1000 systemd-udevd
Nov 18 09:49:15 webapp1 kernel: [16161024.149222] [ 466] 0 466 1054 24 8 3 0 0 acpid
Nov 18 09:49:15 webapp1 kernel: [16161024.158493] [ 470] 0 470 9495 121 23 3 0 0 systemd-logind
Nov 18 09:49:15 webapp1 kernel: [16161024.168448] [ 471] 108 471 11283 116 26 3 0 -900 dbus-daemon
Nov 18 09:49:15 webapp1 kernel: [16161024.177974] [ 495] 0 495 7409 62 19 3 0 0 cron
Nov 18 09:49:15 webapp1 kernel: [16161024.187495] [ 604] 0 604 5120 276 12 3 0 0 dhclient
Nov 18 09:49:15 webapp1 kernel: [16161024.196753] [ 652] 0 652 3634 34 12 3 0 0 agetty
Nov 18 09:49:15 webapp1 kernel: [16161024.205847] [ 653] 0 653 3578 36 12 3 0 0 agetty
Nov 18 09:49:15 webapp1 kernel: [16161024.214941] [ 674] 106 674 24470 132 22 3 0 0 ntpd
Nov 18 09:49:15 webapp1 kernel: [16161024.224283] [ 758] 109 758 163380 14985 104 4 0 0 mysqld
Nov 18 09:49:15 webapp1 kernel: [16161024.233381] [ 817] 0 817 13692 3111 31 3 0 0 google_ip_forwa
Nov 18 09:49:15 webapp1 kernel: [16161024.243501] [ 818] 0 818 13729 3139 33 3 0 0 google_accounts
Nov 18 09:49:15 webapp1 kernel: [16161024.253358] [ 819] 0 819 13729 3158 32 3 0 0 google_clock_sk
Nov 18 09:49:15 webapp1 kernel: [16161024.263222] [ 825] 0 825 17489 213 40 3 0 -1000 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.272120] [30829] 0 30829 11208 97 27 3 0 0 sudo
Nov 18 09:49:15 webapp1 kernel: [16161024.281202] [30830] 0 30830 532390 50302 166 6 0 0 java
Nov 18 09:49:15 webapp1 kernel: [16161024.290274] [18449] 0 18449 11208 96 27 3 0 0 sudo
Nov 18 09:49:15 webapp1 kernel: [16161024.299192] [18450] 0 18450 522507 39168 136 6 0 0 java
Nov 18 09:49:15 webapp1 kernel: [16161024.308095] [ 9924] 0 9924 62529 140 29 4 0 0 rsyslogd
Nov 18 09:49:15 webapp1 kernel: [16161024.317341] [11724] 0 11724 1072 20 7 3 0 0 apt.systemd.dai
Nov 18 09:49:15 webapp1 kernel: [16161024.327190] [11728] 0 11728 1072 33 8 3 0 0 apt.systemd.dai
Nov 18 09:49:15 webapp1 kernel: [16161024.337086] [11764] 0 11764 42919 10514 88 3 0 0 unattended-upgr
Nov 18 09:49:15 webapp1 kernel: [16161024.347224] [12532] 0 12532 15889 82 32 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.356263] [12533] 0 12533 15889 82 31 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.365246] [12535] 0 12535 15889 82 32 4 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.374502] [12536] 0 12536 15889 82 31 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.383422] [12537] 0 12537 15889 81 31 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.392499] [12538] 0 12538 15885 74 31 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.406436] [12539] 0 12539 15353 232 30 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.415332] [12540] 0 12540 11888 55 24 4 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.424228] [12541] 0 12541 809 22 6 3 0 0 sshd
Nov 18 11:38:05 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="9924" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 18 11:38:05 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="13159" x-info="http://www.rsyslog.com"] start
Nov 18 22:49:45 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="13159" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 18 22:49:45 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="17187" x-info="http://www.rsyslog.com"] start
Nov 19 06:25:03 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="17187" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 19 06:25:03 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="17187" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 19 09:06:22 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="17187" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 19 09:06:22 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="21250" x-info="http://www.rsyslog.com"] start
Nov 19 21:01:43 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="21250" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 19 21:01:43 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="24876" x-info="http://www.rsyslog.com"] start
Nov 20 06:25:02 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="24876" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 20 07:44:02 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="24876" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 20 07:44:02 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="28734" x-info="http://www.rsyslog.com"] start
Nov 20 19:15:25 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="28734" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 20 19:15:25 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="32165" x-info="http://www.rsyslog.com"] start
Nov 21 06:18:22 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="32165" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 21 06:18:22 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="3477" x-info="http://www.rsyslog.com"] start
Nov 21 06:25:03 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="3477" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 21 17:26:12 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="3477" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 21 17:26:12 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="6999" x-info="http://www.rsyslog.com"] start
Nov 22 02:50:55 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="6999" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 22 02:50:55 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="10305" x-info="http://www.rsyslog.com"] start
Nov 22 06:25:03 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="10305" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

 

[agraham]

I don't pretend to understand all that but it looks a bit like an out of memory problem. There are various references to "page fault", "oom" and "out of memory" and I particularly noted "oom_kill_process". But then I could be wrong!

 

[juvanum]

I don't pretend to understand all that but it looks a bit like an out of memory problem. There are various references to "page fault", "oom" and "out of memory" and I particularly noted "oom_kill_process". But then I could be wrong!

I think you are right. thanks.
I found answers similar to yours on stackexchange.
(My server is an entry level vps.. I was hoping to see a warning from google).

 

[Erel]

Don't you see heap dump files in the project folder?