System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7
-
Starting to look for differences since the DB Maint should have run about an hour ago.
]$ cd /opt/screenconnect//App_Data/ [ App_Data]$ ls -l total 101304 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 25210880 Dec 17 01:59 Session.db -rw-r--r-- 1 root root 32768 Dec 17 02:04 Session.db-shm -rw-r--r-- 1 root root 78403632 Dec 17 02:04 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml -rw-r--r-- 1 root root 21908 Dec 16 17:24 User.xml -
]$ sar Linux 3.10.0-327.10.1.el7.x86_64 (dny-lnx-sc) 12/17/2016 _x86_64_ (2 CPU) 12:00:02 AM CPU %user %nice %system %iowait %steal %idle 12:10:01 AM all 1.05 0.00 0.24 0.06 0.55 98.10 12:20:02 AM all 1.16 0.00 0.22 0.04 0.49 98.09 12:30:01 AM all 0.84 0.00 0.20 0.04 0.40 98.53 12:40:01 AM all 1.05 0.00 0.19 0.05 0.40 98.31 12:50:01 AM all 0.83 0.00 0.19 0.04 0.42 98.52 01:00:01 AM all 1.06 0.00 0.19 0.04 0.37 98.34 01:10:01 AM all 1.03 0.00 0.22 0.04 0.46 98.25 01:20:02 AM all 1.16 0.00 0.20 0.06 0.47 98.12 01:30:01 AM all 0.96 0.00 0.20 0.04 0.52 98.28 01:40:01 AM all 0.97 0.00 0.19 0.05 0.44 98.35 01:50:01 AM all 0.86 0.00 0.16 0.04 0.44 98.51 02:00:01 AM all 1.00 0.00 0.18 0.05 0.43 98.34 Average: all 1.00 0.00 0.20 0.04 0.45 98.31 -
-
Maybe reboot post maintenance so that we know that we have a fresh, clean system and see what it does.
-
What I see right off compairing:
https://mangolassi.it/topic/11869/system-lockups-with-screenconnect-6-0-11622-6115-on-centos-7/33
https://mangolassi.it/topic/11869/system-lockups-with-screenconnect-6-0-11622-6115-on-centos-7/56
is that Session.db is larger, but Session.db-shm is smaller.
-
@scottalanmiller said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Maybe reboot post maintenance so that we know that we have a fresh, clean system and see what it does.
Done... awaiting system to come back online.
-
Session-db.wal went from 78,403,632 to 4,128,272 !!!
App_Data]$ ls -l total 28836 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 25284608 Dec 17 02:17 Session.db -rw-r--r-- 1 root root 32768 Dec 17 02:17 Session.db-shm -rw-r--r-- 1 root root 4128272 Dec 17 02:17 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e 6cea952.xml -rw-r--r-- 1 root root 21908 Dec 16 17:24 User.xml -
That's quite a change.
-
No heavy CPU or memory on restart either.
-
Of note - the only 'difference' was the selection of Maintenance Plan Actions:
Previously only the Delete Ended Support Sessions and Compact Database Files were selected. The boxed items were not.
-
-rw-r--r-- 1 root root 244,855,752 Dec 14 16:25 Session.db-wal-rw-r--r-- 1 root root 78,403,632 Dec 16 08:33 Session.db-walrw-r--r-- 1 root root 4,128,272 Dec 17 02:17 Session.db-walGoing back and doing some comparisons and noticed this -
(commas added for 'dramatic effect'
-
SAR
r this morning - near as I can tell, no one is on SC currently.$ uptime 10:36:37 up 22:38, 1 user, load average: 0.05, 0.17, 0.22 [ ]$ sar Linux 3.10.0-327.10.1.el7.x86_64 ( ) 12/18/2016 _x86_64_ (1 CPU) 12:00:01 AM CPU %user %nice %system %iowait %steal %idle 12:10:01 AM all 1.89 0.00 0.36 0.09 0.06 97.60 12:20:01 AM all 1.03 0.00 0.33 0.03 0.03 98.57 12:30:01 AM all 1.83 0.00 0.36 0.07 0.05 97.68 12:40:01 AM all 0.89 0.00 0.29 0.03 0.05 98.73 12:50:01 AM all 1.68 0.00 0.35 0.08 0.04 97.85 01:00:01 AM all 1.03 0.00 0.34 0.07 0.03 98.54 01:10:01 AM all 2.30 0.00 0.45 0.13 0.06 97.05 01:20:01 AM all 0.79 0.00 0.29 0.05 0.03 98.84 01:30:01 AM all 1.82 0.00 0.36 0.09 0.08 97.65 01:40:01 AM all 0.71 0.00 0.28 0.02 0.03 98.97 01:50:01 AM all 1.78 0.00 0.35 0.11 0.07 97.70 02:00:01 AM all 0.74 0.00 0.29 0.06 0.03 98.88 02:10:02 AM all 1.76 0.00 0.36 0.09 0.07 97.72 02:20:01 AM all 1.16 0.00 0.33 0.04 0.04 98.43 02:30:01 AM all 1.90 0.00 0.35 0.08 0.11 97.56 02:40:01 AM all 0.76 0.00 0.28 0.02 0.05 98.88 02:50:01 AM all 1.80 0.00 0.36 0.24 0.08 97.52 03:00:01 AM all 0.69 0.00 0.28 0.03 0.05 98.95 03:10:01 AM all 1.86 0.02 0.36 0.08 0.07 97.61 03:20:01 AM all 0.82 0.00 0.32 0.02 0.04 98.79 03:30:01 AM all 1.84 0.00 0.34 0.09 0.05 97.67 03:40:01 AM all 0.74 0.00 0.28 0.03 0.04 98.91 03:50:01 AM all 1.81 0.00 0.35 0.09 0.06 97.70 04:00:01 AM all 0.82 0.00 0.29 0.03 0.04 98.82 04:10:01 AM all 2.42 0.00 0.44 0.09 0.10 96.96 04:20:01 AM all 0.77 0.00 0.29 0.03 0.05 98.86 04:30:01 AM all 1.84 0.00 0.37 0.09 0.10 97.61 04:40:01 AM all 0.75 0.00 0.30 0.06 0.06 98.83 04:50:01 AM all 1.84 0.00 0.36 0.09 0.08 97.63 05:00:01 AM all 0.81 0.00 0.29 0.02 0.05 98.83 05:10:01 AM all 2.17 0.00 0.43 0.09 0.11 97.21 05:20:01 AM all 0.85 0.00 0.29 0.16 0.04 98.67 05:30:01 AM all 1.79 0.00 0.38 0.09 0.07 97.67 05:40:01 AM all 0.86 0.00 0.32 0.03 0.08 98.72 05:50:01 AM all 1.94 0.00 0.46 0.16 0.12 97.32 06:00:01 AM all 0.80 0.00 0.33 0.02 0.05 98.80 06:10:01 AM all 1.79 0.00 0.37 0.33 0.08 97.42 06:20:01 AM all 0.93 0.00 0.34 0.02 0.09 98.62 06:30:01 AM all 1.82 0.00 0.36 0.08 0.09 97.65 06:40:02 AM all 0.76 0.00 0.29 0.03 0.06 98.87 06:50:01 AM all 1.87 0.00 0.37 0.09 0.11 97.56 07:00:01 AM all 0.82 0.00 0.30 0.03 0.05 98.79 07:10:02 AM all 2.23 0.00 0.46 0.08 0.15 97.09 07:20:01 AM all 0.82 0.00 0.32 0.03 0.09 98.74 07:30:01 AM all 2.84 0.00 0.75 0.09 0.12 96.20 07:40:01 AM all 3.05 0.00 0.30 0.03 0.09 96.54 07:50:01 AM all 24.73 0.32 0.62 0.12 0.15 74.07 08:00:01 AM all 6.52 0.00 0.34 0.04 0.07 93.03 08:00:01 AM CPU %user %nice %system %iowait %steal %idle 08:10:01 AM all 25.18 0.00 0.58 0.07 0.16 74.01 08:20:01 AM all 7.17 0.00 0.34 0.03 0.10 92.37 08:30:01 AM all 25.70 0.00 0.52 0.12 0.23 73.42 08:40:01 AM all 6.75 0.00 0.34 0.03 0.13 92.75 08:50:01 AM all 24.80 0.00 0.50 0.07 0.17 74.45 09:00:01 AM all 7.47 0.00 0.34 0.02 0.08 92.08 09:10:01 AM all 28.54 0.00 0.63 0.09 0.48 70.27 09:20:01 AM all 7.15 0.00 0.36 0.17 0.17 92.16 09:30:01 AM all 26.84 0.00 0.54 0.09 0.34 72.20 09:40:01 AM all 8.76 0.00 0.36 0.07 0.27 90.54 09:50:01 AM all 26.93 0.00 0.53 0.07 0.31 72.15 10:00:01 AM all 8.60 0.00 0.35 0.03 0.18 90.84 10:10:01 AM all 28.29 0.00 0.63 0.08 0.47 70.53 10:20:01 AM all 9.19 0.00 0.33 0.08 0.28 90.12 10:30:01 AM all 27.27 0.00 0.52 0.08 0.32 71.81 Average: all 5.79 0.01 0.38 0.07 0.11 93.64 -
-
~]$ cd /opt/screenconnect//App_Data/ [gene@dny-lnx-sc App_Data]$ ls -l total 49048 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 24989696 Dec 18 10:26 Session.db -rw-r--r-- 1 root root 65536 Dec 18 10:39 Session.db-shm -rw-r--r-- 1 root root 25082592 Dec 18 10:39 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml -rw-r--r-- 1 root root 21908 Dec 17 10:41 User.xml -
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
-rw-r--r-- 1 root root 244,855,752 Dec 14 16:25 Session.db-wal-rw-r--r-- 1 root root 78,403,632 Dec 16 08:33 Session.db-walrw-r--r-- 1 root root 4,128,272 Dec 17 02:17 Session.db-walGoing back and doing some comparisons and noticed this -
(commas added for 'dramatic effect'
-rw-r--r-- 1 root root 25,082,592 Dec 18 10:39 Session.db-wal -
So it is climbing back up and fast. Show that to support.
-
Memory is up to 398M used now.
SAR is bouncing around as well -
08:00:01 AM CPU %user %nice %system %iowait %steal %idle 08:10:01 AM all 25.18 0.00 0.58 0.07 0.16 74.01 08:20:01 AM all 7.17 0.00 0.34 0.03 0.10 92.37 08:30:01 AM all 25.70 0.00 0.52 0.12 0.23 73.42 08:40:01 AM all 6.75 0.00 0.34 0.03 0.13 92.75 08:50:01 AM all 24.80 0.00 0.50 0.07 0.17 74.45 09:00:01 AM all 7.47 0.00 0.34 0.02 0.08 92.08 09:10:01 AM all 28.54 0.00 0.63 0.09 0.48 70.27 09:20:01 AM all 7.15 0.00 0.36 0.17 0.17 92.16 09:30:01 AM all 26.84 0.00 0.54 0.09 0.34 72.20 09:40:01 AM all 8.76 0.00 0.36 0.07 0.27 90.54 09:50:01 AM all 26.93 0.00 0.53 0.07 0.31 72.15 10:00:01 AM all 8.60 0.00 0.35 0.03 0.18 90.84 10:10:01 AM all 28.29 0.00 0.63 0.08 0.47 70.53 10:20:01 AM all 9.19 0.00 0.33 0.08 0.28 90.12 10:30:01 AM all 27.27 0.00 0.52 0.08 0.32 71.81 10:40:01 AM all 8.15 0.00 0.33 0.07 0.23 91.22 10:50:01 AM all 25.58 0.00 0.46 0.07 0.27 73.61 11:00:01 AM all 41.85 0.00 0.66 0.14 0.29 57.06 11:10:01 AM all 26.54 0.00 0.62 0.10 0.47 72.27 11:20:01 AM all 8.44 0.00 0.39 0.03 0.21 90.92 11:30:01 AM all 43.08 0.00 0.68 0.15 0.58 55.50 11:40:01 AM all 27.83 0.00 0.52 0.11 0.47 71.07 11:50:01 AM all 24.17 0.00 0.53 0.07 0.38 74.84 12:00:01 PM all 8.93 0.00 0.42 0.04 0.31 90.30 12:10:01 PM all 26.29 0.00 0.60 0.10 0.40 72.61 12:20:01 PM all 9.36 0.00 0.61 0.05 0.17 89.80 12:30:01 PM all 23.48 0.00 0.66 0.07 0.19 75.60 12:40:01 PM all 9.61 0.00 0.50 0.06 0.21 89.63 12:50:01 PM all 23.44 0.00 0.62 0.06 0.18 75.70 01:00:01 PM all 8.63 0.00 0.48 0.03 0.10 90.76 01:10:01 PM all 24.10 0.00 0.70 0.07 0.21 74.93 Average: all 8.90 0.00 0.41 0.07 0.15 90.46 -
Yup that CPU is WAY too high.
-
Curious - system did a restart this evening.
05:05:15 PM LINUX RESTART 05:10:01 PM CPU %user %nice %system %iowait %steal %idle 05:20:01 PM all 3.56 0.00 0.15 0.02 0.09 96.19 05:30:01 PM all 14.36 0.00 0.23 0.09 0.07 85.25 05:40:01 PM all 4.74 0.00 0.16 0.03 0.09 94.99 05:50:01 PM all 16.36 0.00 0.26 0.14 0.11 83.13 06:00:01 PM all 3.68 0.00 0.13 0.02 0.06 96.12 06:10:01 PM all 16.09 0.00 0.27 0.10 0.13 83.41 06:20:01 PM all 4.60 0.00 0.24 0.13 0.09 94.96 06:30:01 PM all 15.20 0.00 0.24 0.32 0.11 84.14 06:40:01 PM all 3.54 0.00 0.13 0.01 0.05 96.27 06:50:01 PM all 15.19 0.00 0.24 0.09 0.07 84.40 07:00:01 PM all 5.10 0.00 0.14 0.02 0.07 94.68 07:10:01 PM all 15.11 0.00 0.29 0.13 0.11 84.36 07:20:01 PM all 3.91 0.00 0.12 0.03 0.06 95.89 07:30:01 PM all 14.58 0.00 0.24 0.08 0.08 85.02 07:40:01 PM all 4.20 0.00 0.18 0.02 0.05 95.55 07:50:01 PM all 14.34 0.00 0.26 0.07 0.08 85.25 Average: all 9.65 0.00 0.20 0.08 0.08 89.98DB is still on the upside for 25MB
]$ ls -l total 49596 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 25583616 Dec 18 19:48 Session.db -rw-r--r-- 1 root root 32768 Dec 18 19:50 Session.db-shm -rw-r--r-- 1 root root 25082592 Dec 18 19:50 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml -rw-r--r-- 1 root root 21908 Dec 17 10:41 User.xml -
After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.
