Pros/Cons Dual Best Effort ISP vs Fiber/MPLS.
-
@Dashrender said:
Going Colo (other than possibly saving me on power and cooling) wouldn't save me anyplace else.... I'd still need the exact same highly available or dual ISP setup as my original post.
Do you ever need to physically be there off hours? Colo gives you 24x7 physical support. It also increases reliability my a dramatic amount.
-
@scottalanmiller said:
@Dashrender said:
Going Colo (other than possibly saving me on power and cooling) wouldn't save me anyplace else.... I'd still need the exact same highly available or dual ISP setup as my original post.
Do you ever need to physically be there off hours? Colo gives you 24x7 physical support. It also increases reliability my a dramatic amount.
You're right, reliability would be potentially better (at least the servers wouldn't suffer power loss when our building does), but unless the cost is exactly what we pay today, or less, they'd never go for it. I'd have show that the power use in my room and the heating/cooling cost would reduce by the amount of the rent at the Colo to even consider it.. because the reliability we have today is adequate.
Internet access is the single most important thing to us. If all of our internal servers just died.. yet we could still access the internet and our cloud based EHR, we would be able to continue to function.
-
@Dashrender said:
HUH? How did you come to this conclusion? Sure eventually I'll probably push email offsite and to O365, then I'm left only with file and print onsite - no apps.
What's email then? That's a serious driver of traffic, and a main reason you need bandwidth locally now. Why not move it to a box in the sky with massive redundant links that you could only dream about having locally?
Going Colo (other than possibly saving me on power and cooling) wouldn't save me anyplace else.... I'd still need the exact same highly available or dual ISP setup as my original post.
Nopes, this would reduce your need for bandwidth and could even get you to a single loop. A pipe with your stuff in colo if it goes offline isn't that big of a deal. Your services will still run, they will "never" go down because of problems in the main site. Site goes down for internet? Oh well, just smurf it out with cell data or failover to an el-cheapo pipe until the other pipe is back online.
Even if I go hosted today (never going to happen, the boss is anti remote - doubly admitted to me just yesterday), I'd still want/need very reliable fast links to the internet for my EHR with is my daily driver of an app that is already in the 'cloud'.
The only 100% reliable link I can guarantee is my LAN. How often does an internal LAN link go down?
Your EHR is your driver, but you are choking it with cheap bad pipes on hot sweaty hardware stuck in a closet for your users to access the internet. If it's already cloud based, setting up a terminal server or VDI farm on the colo cage would make sure anyone can work anywhere. BYOD, further reducing costs, by eliminiating the need to buy equipment for those guys to do their work or having to buy bleeding edge or newer equipment. A dumb terminal is cheap as shit, buy a bunch of Pi's and go to town! See, I've saved you even more money
And your boss is a dumbass.
-
@Dashrender said:
You're right, reliability would be potentially better (at least the servers wouldn't suffer power loss when our building does), but unless the cost is exactly what we pay today, or less, they'd never go for it.
Why worry about downtime if that's not a factor? Seems like the consensus is that cost, not uptime, is the only important factor. This suggests that the powers that be see the operational situation as having low value.
Colocation is cheaper in cases that we have measured. Better uptime, cost savings, less work for you. Pretty big win.
-
@Dashrender said:
.. because the reliability we have today is adequate.
"good enough" is an odd way to measure uptime. It should be a cost to risk scale. Not a "good enough" or "not good enough" scale. How does the CFO determine what is good enough without it being tied to money?
-
@PSX_Defector said:
The only 100% reliable link I can guarantee is my LAN. How often does an internal LAN link go down?
Had a customer lose their LAN last month
-
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
Something that PSX is clearly not understanding is that Email does not matter, local files do not matter, printing really doesn't matter. Two things in our environment really matter for operations to continue.
- phones. Yesteryear I'd say if the phones weren't working we'd close the office because we couldn't reach 911 if there was an emergency, but today that's not a problem because 80% or more of our staff have cellphones in their pockets and can call 911.
- Internet access. Our main app (EHR) that everyone in the company uses (except me, and I do to support them... soooo... ) is part of a service that we subscribe to. It's hosted, managed, etc all by them. So, as long as I have internet access, I can access my EHR.
PSX mentioned BOYD, my $11/hr employees are lucky if they have a desktop at home, let alone a laptop they can bring into the office. Furthermore, I can be assured that they do and will have constant problems (I know because they are always asking for help on their personal machines). Additionally, management does not feel right asking employees to use their personal equipment to do company business - and employees that have volunteered to use their personal equipment because it will make their lives easier have nearly demanded reimbursement for said usage.
Our external email usage is around 100 Megs a day, moving email to the cloud would actually drastically increase our internet usage because now all internal email would have to traverse our internet connection, granted it probably wouldn't be much, maybe increase to 500 megs a day, but it would be much worse.
We had a remote site that lost internet about a month ago, clinic was scheduled for one physician from 2-5. When the internet was not back in service at 2:15, the rest of the clinic was canceled, and the patients rescheduled. those patients that were already onsite were seen in a severally reduced capacity, and possibly rescheduled again if specific information wasn't available at the time. Fortunately this has only happened at our remote clinics about 3 times in 7+ years and has been considered acceptable. That said, there have been many more than 3 outages, there have been well over 15 between all of the remote locations, fortunately it seems the HFC plants don't all die at once in this city.
Now Scott will definitely tell me that I'm the wrong person to be driving redundant ISPs. The business should be telling me - hey it costs us x millions of dollars per hour that we are down, so it makes sense for you to spend y thousand dollars to help us reduce the downtime risk.
Unfortunately I'm not sure I can give you the lost revenue, and the reality is that 95% of canceled appointments that were going to happen (we have a noticeable no show rate) will definitely reschedule in the near future, and the cost is the physician's personal time while they squeeze those patients in to already booked days, usually at little or no staff time cost. Of course there's staff time lost during the outage itself which is not spent doing work because the system is down, but there is enough slack time in their schedules to do any additional work needed without paying overtime (yeah that means we are probably overstaffed - another topic).Have we been over paying for internet on fiber for the past 7+ years, well maybe not the whole time, but for most.. probably. The dual ISP solution has been available for quite some time I'm sure, but hasn't been a consideration until I started looking at my phone situation.
Things are getting a little disjointed.. hope you can just make due.
I feel comfortable telling the BOD that Best Effort ISP solutions WILL give them at least 3 downtime periods per year, probably averaging 4 hours. This seems to be the experience over all of our sites the last several years. With that in mind, and considering that our current Fiber ISP costs $880, and the new 50/10 connection will be $180, I feel comfortable given that information, that the BOD would tell me to get a second ISP installed at $120/month as a failover (12/2 best thing we can get at Best Effort at this location). The inconvenience factor alone would be worth saving to them, even if the clinic wasn't losing more than $1440/yr (plus taxes and the cost of the Peplink).
-
@Dashrender said:
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
If you don't have that info, who is making the financial decisions on technology spends?
-
@scottalanmiller said:
@Dashrender said:
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
If you don't have that info, who is making the financial decisions on technology spends?
Really, truly, me. I do what feel is the best solution at the cheapest price, I'm providing best effort considering the situation. That said, large expense, my phone project, this internet change, are run past the BOD. Sometimes they ask some questions, once they ask me to fine more information, but generally they say, what do you think, OK, do that.
Has it lead to overspending - clearly it has, we've had this expensive fiber for this whole time, perhaps never needed it, but like SMBs who get suckered into a SAN, I probably did get suckered into this Fiber because I was/am concerned about keeping us online as much as possible.
On a side note, when we cancel a clinic at a remote site, we're canceling no more than 10 patients, when we have problems at our main location that could could more like 40 in AM or PM, not to mention our ambulatory surgical center (ASC) (outpatient surgery), which if we are done, we finish the surgeries we are in, and stop until it's restored, or cancel the rest of the day. This is were the big dollars can be lost (I really should have mentioned this side earlier) there is only so much time available to book the ASC, and while I'm sure there is a small amount of slack time, it's generally pretty well booked, so we could definitely end up seeing some pretty substantial losses if we have to cancel those surgeries. So the justification of $1500/yr for a backup ISP is an easy one just considering the ASC.
-
@Dashrender said:
@scottalanmiller said:
@Dashrender said:
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
If you don't have that info, who is making the financial decisions on technology spends?
Really, truly, me. I do what feel is the best solution at the cheapest price, I'm providing best effort considering the situation. That said, large expense, my phone project, this internet change, are run past the BOD. Sometimes they ask some questions, once they ask me to fine more information, but generally they say, what do you think, OK, do that.
Has it lead to overspending - clearly it has, we've had this expensive fiber for this whole time, perhaps never needed it, but like SMBs who get suckered into a SAN, I probably did get suckered into this Fiber because I was/am concerned about keeping us online as much as possible.
On a side note, when we cancel a clinic at a remote site, we're canceling no more than 10 patients, when we have problems at our main location that could could more like 40 in AM or PM, not to mention our ambulatory surgical center (ASC) (outpatient surgery), which if we are done, we finish the surgeries we are in, and stop until it's restored, or cancel the rest of the day. This is were the big dollars can be lost (I really should have mentioned this side earlier) there is only so much time available to book the ASC, and while I'm sure there is a small amount of slack time, it's generally pretty well booked, so we could definitely end up seeing some pretty substantial losses if we have to cancel those surgeries. So the justification of $1500/yr for a backup ISP is an easy one just considering the ASC.
Some items, like the ISP, is probably pretty easy as you have so few choices to work with. It's more when you have to decide on things like colocation, system architecture, hardware purchases, support contracts, etc. where there are far more granular choices and the decisions tend to come down to careful financial comparisons.
-
I've definitely learned a lot and seen mistakes I've made where I could have saved the company a ton of money recently. I will take those lessons with me to the next project and hopefully continue to make wiser choices for us.
That said, overall considering my experience I think I've done a pretty good job. We've had no major outages/issues since I've taken over IT here.
-
@Dashrender said:
Something that PSX is clearly not understanding is that Email does not matter, local files do not matter, printing really doesn't matter.
If that's the case, then your job is redundant. If phones and internet access are the ONLY driving factors in your situation then smurf it out with a set of phones. You get internet access and phones that will always work, and if one provider doesn't, another one will.
Your business case is cost is the only driving factor, to hell with everything else. Well, given that a bunch of cell phones will run ya ~$40 a line for some MVNOs, you are going even cheaper.
Here's the rub, you are so hyper-focused on cost you are lost on reliability, scalability, and the actual bandwidth usage of your environment. There is no way you are only using 100MB of emails a day unless you have two people using emails. If that's the case, then KILL THE MAIL SERVER and move to a cloud environment. And if internet is absolutely required, then going cheap is NOT the way to go. I would be calling some CLECs to get me multiple pipes into the environments. TWTelecom is always dropping pipes in, Covad/Megapath services almost every CO in the country.
Risk is something you get with cheap pipes. If your bosses are so risk adverse that they cannot stand to have even the millisecond of downtime, then it's gonna cost them not only in proper loops with SLAs and such but having multiple loops with SLAs and such. You gotta get real with them, lest they get screaming at you because they had a blip on these el-cheapo pipes.
-
Yes I'm (and my management) are extremely price conscious. But clearly we are willing to pay when we think (rightly or wrongly) we need expensive services. i.e. the fiber link we currently have.
We really don't use email at this company. Outlook is more about calendars for the physicians to sync to their mobile devices than any other purpose. If not for the need for calendar access, we'd go from 88 people having email to around 20.
Internet is critical, but we can suffer some downtime, we are a slightly higher requirement than average SMB (yeah I know "all SMBs think they can afford to suffer no down time" lol). I've already mentioned that we've suffered down time at the remote locations and yet, the BOD has not come to me saying - You need to make sure we don't have downtime. That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
At this point I also have to ask myself - are you saying these things just to rial me up, like JaredBusch seemed to be to Scott the other day?
-
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
-
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
Because the vendors won't provide failure rates on Best Effort solutions, the best you can go by is past experience. I can look at my tickets and see how many failures we've had on average over the past three years at each remote location, then make an educated guess as to what will probably happen in the future. Of course it's not perfect, it COULD be completely wrong, past performance can't predict future situations. But it's not simply a gut feeling though.
I know that on average we have 3 outages a year at each remote location, so assuming that to be the case, I hope and anticipate that to be what happens in the future. So now I consider options to mitigate those 3 failures, which is the purchase of a second line.
Let's leave the colo stuff out of this as it's not important to the current needs, which is protecting internet access for access to our EHR.
-
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
-
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
-
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
The starting point is knowing the cost of downtime. IT exists within a business context, so knowing the business is critical to understanding IT's place within it. Knowing what the impact of slowness, flakiness and outages is important (as they vary from company to company.) Once you have a good financial understanding of how telecommunications relates to the business then you can do risk analysis and cost analysis of the options and determine which make the most sense based on the business' unique needs.
-
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
Because the vendors won't provide failure rates on Best Effort solutions, the best you can go by is past experience. I can look at my tickets and see how many failures we've had on average over the past three years at each remote location, then make an educated guess as to what will probably happen in the future. Of course it's not perfect, it COULD be completely wrong, past performance can't predict future situations. But it's not simply a gut feeling though.
I know that on average we have 3 outages a year at each remote location, so assuming that to be the case, I hope and anticipate that to be what happens in the future. So now I consider options to mitigate those 3 failures, which is the purchase of a second line.
Let's leave the colo stuff out of this as it's not important to the current needs, which is protecting internet access for access to our EHR.
This doesn't really answer the question. You are looking a failure rates of a line not at the numbers associated with reasonable downtime.
Do you have good cell reception where you are?
-
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't feel that "reasonable" downtime is a thing. It's reasonable at what cost. Reasonable changes based on the costs and options. It's a curve, not a singular value.
Example: Manager says "reasonable downtime is 15 minutes."
But what if halving the risk only cost $1? Would having 15 minutes of risk for $800 a month be reasonable if a risk of 7 minutes was only $801? Probably not, what was stated as reasonable would almost certainly be reckless when the whole picture was taken account.
