CloudatCost Issues

Yum -y update took 15 minutes

thanksajdotcom

Yum -y update took 15 minutes

Depending on how many updates it had, that's quite possible.

Danp

I'm also having trouble with my C@C Dev1 box. I'm unable to connect via SSH, Yum install via web console throws errors, etc.

Danp

I wonder if there's a DDoS in progress. Maybe I pissed of the folks in HK when I blocked all of their SSH attempts in iptables?

@thanksajdotcom There was 166 updates, but it did this same thing yesterday, and the updates took less then a minute, both Dev1 boxes. Something is up,

scottalanmiller

@Aaron-Studer said:

Yum -y update took 15 minutes

That's slow but not crazy. Which OS are you using? It is a big operation with a lot of stuff to download, then to setup in the database, then to roll out one package at a time. Updates really can take a bit of time. Fifteen minutes seems long, but if there is any IO issues, then it would slow down a lot.

scottalanmiller

@Aaron-Studer said:

@thanksajdotcom There was 166 updates, but it did this same thing yesterday, and the updates took less then a minute, both Dev1 boxes. Something is up,

Could be the IO issues that they list on their site. But it could be more network problems with Rogers, too.

scottalanmiller

@Danp said:

I'm also having trouble with my C@C Dev1 box. I'm unable to connect via SSH, Yum install via web console throws errors, etc.

Did SSH work before? Or is this a new build?

scottalanmiller

Oh you know what, is everyone having login problems using Ubuntu instead of CentOS? I've seen that issue too.

scottalanmiller

I got some huge delays just typing to the console. Like 30 seconds for a line that I was typing to continue.

scottalanmiller

I'm checking the Ubuntu instances and am seeing issues around passwords failures in the auth.log.

scottalanmiller

Looks like you just need to run pwconv after creating a user on Ubuntu 14.04.02. That's all. Works fine once you do that.

scottalanmiller

As I suspected, it's IOWait problems. Always check SAR first for performance problems.

Linux 3.13.0-32-generic (ubuntu)        03/09/2015      _x86_64_        (1 CPU)

02:05:11 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
02:15:01 AM     all      0.10      0.00      0.31      7.35      0.00     92.24
02:25:01 AM     all      0.02      0.00      0.21      5.97      0.00     93.81
02:35:01 AM     all      0.00      0.00      0.17      2.01      0.00     97.82
02:45:01 AM     all      0.19      0.00      0.28     20.81      0.00     78.73
02:55:39 AM     all      1.63      0.00      0.46     28.88      0.00     69.03
03:05:01 AM     all      0.22      0.00      0.14      9.61      0.00     90.02
03:15:01 AM     all      0.00      0.00      0.12      1.73      0.00     98.15
03:25:01 AM     all      0.00      0.00      0.14      7.41      0.00     92.45
Average:        all      0.28      0.00      0.23     10.63      0.00     88.86

I'm seeing rather large IO delays here too.

scottalanmiller

So far I've only seen the IO issues on the Dev1 instances.

@scottalanmiller said:

So far I've only seen the IO issues on the Dev1 instances.

I wonder why?

scottalanmiller

@Aaron-Studer said:

@scottalanmiller said:

So far I've only seen the IO issues on the Dev1 instances.

I wonder why?

I'm sure the over-commit rate on those is much higher. And the IO needs are higher as there are so many OSes doing things, even when idle, in the same IO space.

Danp

What is a "good" average? On my Dev1 box, I'm seeing 8.41%. On my Dev3 box, I'm seeing 4.45%.

scottalanmiller

@Danp said:

What is a "good" average? On my Dev1 box, I'm seeing 8.41%. On my Dev3 box, I'm seeing 4.45%.

Average "should" approach zero. Definitely way under 1%.

scottalanmiller

For example, the MangoLassi database, which you can imagine is relatively busy, produces only .03% IOWait state average. It almost never spikes above .08% in any ten minute period.

That's point ZERO three. So 8.41 is 280 times higher!!