System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7
-
Here is the paging activity. Yes the page file is being used, but not heavily. There was no excessing paging during the times of performance issues:
# sar -B Linux 3.10.0-327.10.1.el7.x86_64 (dny-lnx-sc) 12/15/2016 _x86_64_ (1 CPU) 12:00:02 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 12:10:02 AM 1070.88 475.19 313.52 18.61 303.18 229.64 2.30 227.53 98.10 12:20:02 AM 370.40 229.51 206.97 0.52 283.77 128.90 0.00 117.97 91.52 12:30:03 AM 739.50 639.48 115.19 0.78 350.09 271.70 0.00 259.53 95.52 12:40:02 AM 3.36 47.00 94.66 0.06 54.44 0.00 0.00 0.00 0.00 12:50:02 AM 0.00 9.23 93.99 0.00 54.01 0.00 0.00 0.00 0.00 01:00:02 AM 0.04 23.73 130.96 0.00 75.68 6.19 0.00 6.16 99.50 01:10:02 AM 138.80 24.04 202.75 0.38 240.10 106.09 0.00 93.01 87.67 01:20:01 AM 0.35 8.15 96.68 0.00 52.07 0.00 0.00 0.00 0.00 01:30:02 AM 0.27 7.73 111.38 0.00 58.43 0.00 0.00 0.00 0.00 01:40:02 AM 8.53 11.16 289.74 0.20 126.59 0.00 0.00 0.00 0.00 01:50:01 AM 0.40 8.22 116.28 0.00 72.54 0.00 0.00 0.00 0.00 02:00:02 AM 0.26 7.31 90.02 0.00 49.75 0.00 0.00 0.00 0.00 02:10:02 AM 63.54 9.81 188.55 0.26 156.20 32.83 0.00 24.31 74.04 02:20:02 AM 9.54 8.48 115.26 0.07 67.49 0.00 0.00 0.00 0.00 02:30:01 AM 0.36 8.91 116.85 0.00 72.74 0.00 0.00 0.00 0.00 02:40:01 AM 0.27 7.98 94.15 0.00 53.94 0.00 0.00 0.00 0.00 02:50:02 AM 11.25 11.48 169.47 0.01 81.96 0.00 0.00 0.00 0.00 03:00:02 AM 0.26 9.15 149.14 0.00 73.15 0.00 0.00 0.00 0.00 03:10:02 AM 11.69 8.54 136.34 0.10 70.33 0.00 0.00 0.00 0.00 03:20:02 AM 52.71 21.43 164.83 0.21 152.25 25.04 0.00 17.44 69.64 03:30:01 AM 0.20 14.92 170.39 0.00 81.22 0.00 0.00 0.00 0.00 03:40:01 AM 4.14 11.31 128.74 0.01 63.29 0.00 0.00 0.00 0.00 03:50:02 AM 0.15 7.17 85.33 0.00 53.95 0.00 0.00 0.00 0.00 04:00:02 AM 0.37 4.52 43.55 0.00 37.19 0.00 0.00 0.00 0.00 04:10:02 AM 0.92 10.29 72.12 0.00 35.06 0.00 0.00 0.00 0.00 04:20:03 AM 62.63 11.58 149.51 0.27 148.09 40.53 0.00 30.23 74.57 04:30:02 AM 15.06 8.68 123.79 0.03 73.41 0.00 0.00 0.00 0.00 04:40:01 AM 0.36 7.96 92.66 0.00 55.38 0.00 0.00 0.00 0.00 04:50:02 AM 0.44 8.41 90.09 0.00 50.81 0.00 0.00 0.00 0.00 05:00:01 AM 0.31 7.99 99.26 0.00 50.50 0.00 0.00 0.00 0.00 05:10:02 AM 9.71 10.33 141.26 0.10 74.02 0.00 0.00 0.00 0.00 05:20:02 AM 48.96 14.03 164.96 0.08 159.73 30.39 0.00 22.40 73.70 05:30:01 AM 2.17 7.82 90.30 0.00 51.34 0.00 0.00 0.00 0.00 05:40:02 AM 15.94 6.61 101.93 0.01 51.49 0.00 0.00 0.00 0.00 05:50:02 AM 0.88 7.12 127.64 0.01 72.83 0.00 0.00 0.00 0.00 06:00:02 AM 3.55 6.36 90.42 0.05 56.44 0.00 0.00 0.00 0.00 06:10:02 AM 95.21 8.28 203.59 0.34 185.54 47.92 0.00 39.88 83.23 06:20:02 AM 0.03 5.98 102.36 0.00 61.96 0.00 0.00 0.00 0.00 06:30:01 AM 0.00 6.31 109.01 0.00 72.76 0.00 0.00 0.00 0.00 06:40:02 AM 0.00 5.99 99.20 0.00 71.66 0.00 0.00 0.00 0.00 06:50:02 AM 0.13 6.12 93.71 0.00 59.18 0.00 0.00 0.00 0.00 07:00:04 AM 469.46 186.17 211.96 0.67 187.31 141.09 0.00 121.64 86.21 07:10:02 AM 189.35 14.65 238.19 0.67 278.61 43.89 0.00 34.50 78.60 07:20:02 AM 0.01 6.17 95.28 0.00 69.84 0.00 0.00 0.00 0.00 07:20:02 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 07:30:01 AM 0.00 5.77 89.42 0.00 58.34 0.00 0.00 0.00 0.00 07:40:02 AM 0.00 5.86 89.96 0.00 58.88 0.00 0.00 0.00 0.00 07:50:02 AM 11.22 6.36 116.90 0.01 72.79 0.00 0.00 0.00 0.00 08:00:02 AM 2.25 5.50 88.66 0.03 67.45 0.00 0.00 0.00 0.00 08:10:02 AM 83.13 6.81 166.69 0.25 78.10 0.00 0.00 0.00 0.00 Average: 71.30 40.58 132.09 0.48 99.66 22.51 0.05 20.27 89.88 08:11:58 AM LINUX RESTART 08:20:01 AM pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff 08:30:01 AM 6460.45 1452.55 860.54 10.95 2203.45 2695.77 1.84 1968.47 72.97 08:40:02 AM 3843.51 992.95 1509.57 23.78 2091.99 1544.96 5.08 1063.32 68.60 08:50:01 AM 178.13 106.49 398.48 0.75 419.55 94.83 0.00 63.27 66.72 09:00:01 AM 18.67 52.13 185.53 0.17 230.14 9.80 0.00 5.19 52.98 09:10:01 AM 244.59 246.77 245.09 0.73 323.11 201.56 0.13 109.10 54.09 09:20:01 AM 15.81 44.15 113.35 0.10 87.69 0.00 0.00 0.00 0.00 09:30:02 AM 0.68 37.97 94.69 0.02 77.13 0.00 0.00 0.00 0.00 09:40:01 AM 0.62 43.77 104.97 0.01 79.69 0.00 0.00 0.00 0.00 09:50:01 AM 4.18 40.15 135.47 0.10 101.44 0.00 0.00 0.00 0.00 10:00:01 AM 81.33 44.61 172.65 0.16 200.52 0.00 0.00 0.00 0.00 10:10:02 AM 208.11 56.65 211.51 0.62 314.71 112.92 0.35 101.51 89.62 10:20:01 AM 52.47 44.80 433.22 0.19 345.24 0.00 0.00 0.00 0.00 10:30:01 AM 0.95 44.14 186.85 0.00 262.01 0.00 0.00 0.00 0.00 Average: 823.62 239.63 350.69 2.78 505.34 345.80 0.55 245.66 70.93 -
Here is the disk IOPS during the period:
# sar -b Linux 3.10.0-327.10.1.el7.x86_64 (dny-lnx-sc) 12/15/2016 _x86_64_ (1 CPU) 12:00:02 AM tps rtps wtps bread/s bwrtn/s 12:10:02 AM 37.78 35.51 2.26 2141.75 950.39 12:20:02 AM 60.72 59.03 1.69 740.81 459.02 12:30:03 AM 49.76 44.11 5.65 1478.99 1278.96 12:40:02 AM 4.00 0.10 3.91 6.73 94.01 12:50:02 AM 1.01 0.00 1.01 0.00 18.47 01:00:02 AM 2.50 0.01 2.49 0.07 47.46 01:10:02 AM 6.86 4.90 1.96 277.61 48.09 01:20:01 AM 1.09 0.08 1.01 0.70 16.29 01:30:02 AM 1.00 0.07 0.94 0.54 15.46 01:40:02 AM 1.48 0.34 1.14 17.05 22.31 01:50:01 AM 1.06 0.08 0.98 0.80 16.44 02:00:02 AM 0.95 0.07 0.88 0.52 14.62 02:10:02 AM 2.72 1.71 1.01 127.08 19.63 02:20:02 AM 1.43 0.40 1.03 19.09 16.96 02:30:01 AM 1.13 0.09 1.04 0.71 17.83 02:40:01 AM 1.02 0.07 0.95 0.54 15.96 02:50:02 AM 1.60 0.45 1.15 22.49 22.96 03:00:02 AM 1.14 0.07 1.07 0.52 18.29 03:10:02 AM 1.87 0.87 0.99 23.37 17.08 03:20:02 AM 3.42 1.64 1.78 105.42 42.86 03:30:01 AM 1.48 0.05 1.43 0.41 29.83 03:40:01 AM 1.67 0.44 1.23 8.28 22.61 03:50:02 AM 0.87 0.04 0.83 0.29 14.34 04:00:02 AM 0.59 0.04 0.55 0.74 9.05 04:10:02 AM 1.26 0.23 1.03 1.85 20.58 04:20:03 AM 2.61 1.76 0.84 125.26 23.16 04:30:02 AM 1.56 0.51 1.06 30.12 17.36 04:40:01 AM 1.03 0.09 0.94 0.73 15.92 04:50:02 AM 1.10 0.11 0.99 0.89 16.81 05:00:01 AM 1.03 0.08 0.95 0.63 15.98 05:10:02 AM 2.18 1.03 1.16 19.42 20.66 05:20:02 AM 4.03 2.23 1.80 97.92 28.06 05:30:01 AM 1.33 0.39 0.95 4.33 15.65 05:40:02 AM 1.11 0.29 0.82 31.88 13.22 05:50:02 AM 0.94 0.04 0.90 1.75 14.25 06:00:02 AM 0.92 0.11 0.81 7.09 12.71 06:10:02 AM 3.43 2.45 0.98 190.42 16.55 06:20:02 AM 0.76 0.01 0.75 0.06 11.95 06:30:01 AM 0.78 0.00 0.78 0.00 12.62 06:40:02 AM 0.78 0.00 0.78 0.00 11.97 06:50:02 AM 0.83 0.01 0.82 0.26 12.23 07:00:04 AM 18.36 11.59 6.77 938.93 372.35 07:10:02 AM 8.82 7.30 1.52 378.69 29.30 07:20:02 AM 0.81 0.00 0.81 0.01 12.34 07:20:02 AM tps rtps wtps bread/s bwrtn/s 07:30:01 AM 0.78 0.00 0.78 0.00 11.55 07:40:02 AM 0.78 0.00 0.78 0.00 11.72 07:50:02 AM 0.93 0.12 0.81 22.45 12.71 08:00:02 AM 0.81 0.06 0.75 4.49 11.00 08:10:02 AM 5.67 4.84 0.83 166.27 13.62 Average: 5.09 3.73 1.35 142.61 81.15 08:11:58 AM LINUX RESTART 08:20:01 AM tps rtps wtps bread/s bwrtn/s 08:30:01 AM 162.30 151.14 11.15 12920.88 2905.10 08:40:02 AM 129.66 120.51 9.15 7687.04 1985.90 08:50:01 AM 10.27 6.22 4.05 356.26 212.99 09:00:01 AM 5.04 1.12 3.92 37.33 104.27 09:10:01 AM 9.50 6.04 3.46 489.18 493.55 09:20:01 AM 5.15 1.93 3.22 31.62 88.30 09:30:02 AM 2.77 0.03 2.74 1.36 75.94 09:40:01 AM 3.22 0.06 3.16 1.24 87.54 09:50:01 AM 3.14 0.28 2.87 8.37 80.30 10:00:01 AM 7.41 4.23 3.18 162.67 89.22 10:10:02 AM 7.35 4.22 3.13 416.22 113.30 10:20:01 AM 5.79 2.59 3.20 104.94 89.60 10:30:01 AM 3.10 0.02 3.08 1.90 88.28 10:40:01 AM 15.26 12.06 3.20 637.15 92.53 Average: 25.61 21.40 4.21 1574.71 451.49 -
HTOP when things are healthy:
-
TOP when healthy:
-
GLANCES when healthy:
-
We know that the process that takes the system resources is mono which is screenconnects parent container.
-
Looks like i'm running the same version:-
On CentOS7 ( I think it's all up to date with patches etc)Not seeing this problem.
Running on XenServer 7 - 2vCPU 2GB RAM 40GB HD Space -
@hobbit666 take a look at your SAR output, any chance that you have one threading engine hitting the wall and one that is not?
-
We are resizing the instance up to 2 vCPU and 2GB RAM right now to see what happens with more resources available at ScreenConnect support's request. But it is not using the resources available to it now, we are afraid of masking the issue and getting the system limping along rather than fixing it.
-
@DustinB3403 said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Any chance you can clone this VM, and downgrade the SC version to the previous version. And monitor both, putting them side by side?
No, can't downgrade that easily as the Client version would be different and not likely to re-sync
-
Resized and the issue remains. Look at that CPU load for an idle box!!
-
Sorry @scottalanmiller i'm a Linux newb
How do I run the SAR stuff?
-
@hobbit666 said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Sorry @scottalanmiller i'm a Linux newb
How do I run the SAR stuff?
If it is installed, form CLI just run
sar -
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@hobbit666 said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Sorry @scottalanmiller i'm a Linux newb
How do I run the SAR stuff?
If it is installed, form CLI just run
sarIf it is not installed
$ yum install sysstat -
Example of
$ sar$ sar Linux 3.10.0-327.10.1.el7.x86_64 (dny-lnx-sc) 12/15/2016 _x86_64_ (2 CPU) 10:56:35 AM LINUX RESTART 11:00:02 AM CPU %user %nice %system %iowait %steal %idle 11:10:01 AM all 22.30 0.00 0.85 0.16 3.05 73.64 Average: all 22.30 0.00 0.85 0.16 3.05 73.64Or as above from @scottalanmiller with
$ sar -b -
This is all i'm getting from SAR
[root@screenconnect6 ~]# sar
Linux 3.10.0-327.28.3.el7.x86_64 (screenconnect6) 15/12/16 x86_64 (2 CPU)16:20:43 LINUX RESTART
-
@scottalanmiller said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Resized and the issue remains. Look at that CPU load for an idle box!!
How do you get that pretty picture??
Funny I was about to start a thread about tuning Linux systems for resources so I can free some up for more VM's
-
@hobbit666 said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
This is all i'm getting from SAR
[root@screenconnect6 ~]# sar
Linux 3.10.0-327.28.3.el7.x86_64 (screenconnect6) 15/12/16 x86_64 (2 CPU)16:20:43 LINUX RESTART
Takes ten minutes to accumulate the first line of data.
-
@hobbit666 said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@scottalanmiller said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Resized and the issue remains. Look at that CPU load for an idle box!!
How do you get that pretty picture??
Command is htop, it's in the EPEL. Then I just screenshot it. It's the default terminal on Ubuntu 16.10.
-
OK so i'll leave my VM running over night as i'll be heading home soon.
I've put a sticky note on my monitor to remind me to run the commands and post screenshots
