Pros/Cons Dual Best Effort ISP vs Fiber/MPLS.
-
@Dashrender said:
@scottalanmiller said:
@Dashrender said:
I understand that 'good enough' isn't a scale that we should be using. But I'm not privy to the financials so I have no idea what we make per hour.
If you don't have that info, who is making the financial decisions on technology spends?
Really, truly, me. I do what feel is the best solution at the cheapest price, I'm providing best effort considering the situation. That said, large expense, my phone project, this internet change, are run past the BOD. Sometimes they ask some questions, once they ask me to fine more information, but generally they say, what do you think, OK, do that.
Has it lead to overspending - clearly it has, we've had this expensive fiber for this whole time, perhaps never needed it, but like SMBs who get suckered into a SAN, I probably did get suckered into this Fiber because I was/am concerned about keeping us online as much as possible.
On a side note, when we cancel a clinic at a remote site, we're canceling no more than 10 patients, when we have problems at our main location that could could more like 40 in AM or PM, not to mention our ambulatory surgical center (ASC) (outpatient surgery), which if we are done, we finish the surgeries we are in, and stop until it's restored, or cancel the rest of the day. This is were the big dollars can be lost (I really should have mentioned this side earlier) there is only so much time available to book the ASC, and while I'm sure there is a small amount of slack time, it's generally pretty well booked, so we could definitely end up seeing some pretty substantial losses if we have to cancel those surgeries. So the justification of $1500/yr for a backup ISP is an easy one just considering the ASC.
Some items, like the ISP, is probably pretty easy as you have so few choices to work with. It's more when you have to decide on things like colocation, system architecture, hardware purchases, support contracts, etc. where there are far more granular choices and the decisions tend to come down to careful financial comparisons.
-
I've definitely learned a lot and seen mistakes I've made where I could have saved the company a ton of money recently. I will take those lessons with me to the next project and hopefully continue to make wiser choices for us.
That said, overall considering my experience I think I've done a pretty good job. We've had no major outages/issues since I've taken over IT here.
-
@Dashrender said:
Something that PSX is clearly not understanding is that Email does not matter, local files do not matter, printing really doesn't matter.
If that's the case, then your job is redundant. If phones and internet access are the ONLY driving factors in your situation then smurf it out with a set of phones. You get internet access and phones that will always work, and if one provider doesn't, another one will.
Your business case is cost is the only driving factor, to hell with everything else. Well, given that a bunch of cell phones will run ya ~$40 a line for some MVNOs, you are going even cheaper.
Here's the rub, you are so hyper-focused on cost you are lost on reliability, scalability, and the actual bandwidth usage of your environment. There is no way you are only using 100MB of emails a day unless you have two people using emails. If that's the case, then KILL THE MAIL SERVER and move to a cloud environment. And if internet is absolutely required, then going cheap is NOT the way to go. I would be calling some CLECs to get me multiple pipes into the environments. TWTelecom is always dropping pipes in, Covad/Megapath services almost every CO in the country.
Risk is something you get with cheap pipes. If your bosses are so risk adverse that they cannot stand to have even the millisecond of downtime, then it's gonna cost them not only in proper loops with SLAs and such but having multiple loops with SLAs and such. You gotta get real with them, lest they get screaming at you because they had a blip on these el-cheapo pipes.
-
Yes I'm (and my management) are extremely price conscious. But clearly we are willing to pay when we think (rightly or wrongly) we need expensive services. i.e. the fiber link we currently have.
We really don't use email at this company. Outlook is more about calendars for the physicians to sync to their mobile devices than any other purpose. If not for the need for calendar access, we'd go from 88 people having email to around 20.
Internet is critical, but we can suffer some downtime, we are a slightly higher requirement than average SMB (yeah I know "all SMBs think they can afford to suffer no down time" lol). I've already mentioned that we've suffered down time at the remote locations and yet, the BOD has not come to me saying - You need to make sure we don't have downtime. That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
At this point I also have to ask myself - are you saying these things just to rial me up, like JaredBusch seemed to be to Scott the other day?
-
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
-
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
Because the vendors won't provide failure rates on Best Effort solutions, the best you can go by is past experience. I can look at my tickets and see how many failures we've had on average over the past three years at each remote location, then make an educated guess as to what will probably happen in the future. Of course it's not perfect, it COULD be completely wrong, past performance can't predict future situations. But it's not simply a gut feeling though.
I know that on average we have 3 outages a year at each remote location, so assuming that to be the case, I hope and anticipate that to be what happens in the future. So now I consider options to mitigate those 3 failures, which is the purchase of a second line.
Let's leave the colo stuff out of this as it's not important to the current needs, which is protecting internet access for access to our EHR.
-
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
-
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
-
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
The starting point is knowing the cost of downtime. IT exists within a business context, so knowing the business is critical to understanding IT's place within it. Knowing what the impact of slowness, flakiness and outages is important (as they vary from company to company.) Once you have a good financial understanding of how telecommunications relates to the business then you can do risk analysis and cost analysis of the options and determine which make the most sense based on the business' unique needs.
-
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
Because the vendors won't provide failure rates on Best Effort solutions, the best you can go by is past experience. I can look at my tickets and see how many failures we've had on average over the past three years at each remote location, then make an educated guess as to what will probably happen in the future. Of course it's not perfect, it COULD be completely wrong, past performance can't predict future situations. But it's not simply a gut feeling though.
I know that on average we have 3 outages a year at each remote location, so assuming that to be the case, I hope and anticipate that to be what happens in the future. So now I consider options to mitigate those 3 failures, which is the purchase of a second line.
Let's leave the colo stuff out of this as it's not important to the current needs, which is protecting internet access for access to our EHR.
This doesn't really answer the question. You are looking a failure rates of a line not at the numbers associated with reasonable downtime.
Do you have good cell reception where you are?
-
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't feel that "reasonable" downtime is a thing. It's reasonable at what cost. Reasonable changes based on the costs and options. It's a curve, not a singular value.
Example: Manager says "reasonable downtime is 15 minutes."
But what if halving the risk only cost $1? Would having 15 minutes of risk for $800 a month be reasonable if a risk of 7 minutes was only $801? Probably not, what was stated as reasonable would almost certainly be reckless when the whole picture was taken account.
-
@scottalanmiller said:
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't feel that "reasonable" downtime is a thing. It's reasonable at what cost. Reasonable changes based on the costs and options. It's a curve, not a singular value.
Example: Manager says "reasonable downtime is 15 minutes."
But what if halving the risk only cost $1? Would having 15 minutes of risk for $800 a month be reasonable if a risk of 7 minutes was only $801? Probably not, what was stated as reasonable would almost certainly be reckless when the whole picture was taken account.
No argument there. I use reasonable as a point of reference not necessarily an indication of cost.
-
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't see it as being an ass, it's just been asked before (though perhaps not in this thread) - I thought I had addressed it.
-
But getting management to give a "reasonable" number is a bad idea because reasonable doesn't exist until the market analysis and business needs are done. Really, only IT can determine what is reasonable and only the financial department can provide the necessary information for IT to make that call. Unless the two work together, IT can, at best, make wild ass guesses.
-
@scottalanmiller said:
But getting management to give a "reasonable" number is a bad idea because reasonable doesn't exist until the market analysis and business needs are done. Really, only IT can determine what is reasonable and only the financial department can provide the necessary information for IT to make that call. Unless the two work together, IT can, at best, make wild ass guesses.
Good point. I hadn't thought of it that way. I was thinking more along the lines of cost of downtime vs cost of a solution.
-
@scottalanmiller said:
@coliver said:
@Dashrender said:
@coliver said:
@Dashrender said:
That said, I do want to minimize it within reason. This is the whole reason for looking at dual ISPs.
How do you determine what is reasonable? Is it a gut feeling? Or do you have numbers behind what you think is reasonable?
I absolutely don't mean this to sound glib, mind telling me how you would determine something like this?
I would have to see what the business sees as reasonable downtime. That isn't something I can determine for the business, unfortunately. Once they tell me what they are see as reasonable then I can tailor a solution to fit that.
I didn't mean to sound like an ass (I have a hard time with that in written language) just a question out of curiosity to see how other people do it.
I don't feel that "reasonable" downtime is a thing. It's reasonable at what cost. Reasonable changes based on the costs and options. It's a curve, not a singular value.
Example: Manager says "reasonable downtime is 15 minutes."
But what if halving the risk only cost $1? Would having 15 minutes of risk for $800 a month be reasonable if a risk of 7 minutes was only $801? Probably not, what was stated as reasonable would almost certainly be reckless when the whole picture was taken account.
Great - how do you quantify the actual risk?
All I know right now is I pay $880 and in 7+ years I've had 15 mins of unplanned downtime.
I pay $120 at another location and I 3 different days at 4 hours per day of downtime.I won't post publicly dollar values of lost revenue.. not that I know what it is anyhow, the power that be won't share with me.
-
@Dashrender said:
At this point I also have to ask myself - are you saying these things just to rial me up, like JaredBusch seemed to be to Scott the other day?
Nah, if I wanted to do that, I would do other things. Got better things to do with my time than just rile you up.
I've dealt with this kind of mentality with bosses before. Really, you gotta break out of their mold and start breaking it down into a much more concise set of requirements.
Your cloud based EHR is your most important application, and if they are saying that nothing else matters then its time to ignore anything with onsite resources. That shit can be set it in the colo cage and is not the problem here. You need bandwidth to the cloud but cannot abide by any downtime. Since it's public traffic, there is no need for MPLS, fancy fiber loops between sites, or even the VPN solution I mentioned with Peplink. Your sites don't care if they can talk with each other, so any interoffice communication is pointless and unnecessary. You need interwebs and you need it to be working when you need it working. It sounds as though bandwidth is a pointless consideration, you can probably get away with 1.5Mbps loops for the amount of traffic you are probably generating.
There are two ways to handle this. One way is to add an additional high quality loop into the mix. This can be from a third party vendor, so you wouldn't have any layer 3 issues upstream, like if Cogent decided to stop routing traffic from one ISP. This is gonna cost you money because quality loops are not cheap. But this doesn't eliminate any layer 2 issues, like a cut fiber or central office burning to the ground. Even if you went with different mediums, like cable plant versus fiber versus traditional telco, you will still have layer 2 potential issues. The setup would be fairly simple, even if you didn't have fancy load balancers behind it. Worst thing that would happen would be you reload your ASA or whatever firewall you have with a config for the other ISP.
The other way is to go cheap but go many. Swarm the problem with super cheap pipes from multiple locations. A mix of commodity pipes with MVNOs and any other wireless vendor you can find. Sure, one pipe might go down, but odds are the Sprint, AT&T, and Verizon towers won't and you would still have other pipes to fall back on. This is a much more complex setup, but could be much cheaper for what you have now or move to with a second quality pipe. Failover would require more complex equipment, like the Peplink, but would keep downtime to a minimal.
You need clear directives from the bosses. If they are gonna shoot down anything that cost more than their current setup or focus and spend money on things that don't matter, then this is an exercise in futility because I wouldn't dare guarantee a cheaper solution because I know what it will entail. If you are requiring more bandwidth, the costs are going to increase no matter what. If ANY other items come in, like local services, then internet services are only a small part of the issue here. Then it's time to take a look at all your services, the business workflow, and what can be done to streamline everything and start doing things right. Moving to a colo with better pipes locally would really help.
-
@Dashrender Well you can make another wild {censored} guess like the rest of us do, lol.
But Here, I think is where the rubber meets the road. If what you are doing has the ability to improve internet speeds and (both cut cost and improve reliability [through use of a backup connection]) , how is that a bad business decision?
-
@Dashrender said:
Great - how do you quantify the actual risk?
All I know right now is I pay $880 and in 7+ years I've had 15 mins of unplanned downtime.
I pay $120 at another location and I 3 different days at 4 hours per day of downtime.Risk is difficult to assess, especially in a case where you have to consider more than technology. The biggests risks that you have to address here are around scope (you are dealing only with service risk, but what if your building burns down or your own equipment fails!!), risk around businesses going under or changing policies, risk that you will be considered a pain to fix and traditional downtime could suddenly become insane downtime, lock in risk, risk because of service quality issues, potential downtime, likely downtime.... it's hard, very hard. That's why risk analysis is one of the absolute top skills for IT decision makers. It is complex and often the most important value that we provide to the business - without risk analysis they could just pick a solution themselves without IT being needed.
Risk is a bit of a soft science, but understanding the range of risks, a rough idea of likelihood, what risks apply how heavily to the business in question, how well mitigation factors can or should work, etc. is what IT provides and while far from perfect, it can be pretty decent.
And it is important to remember that we are only coming up with risk. And this is multidimensional risk not just a 20% kind of number.
-
@Dashrender said:
I won't post publicly dollar values of lost revenue.. not that I know what it is anyhow, the power that be won't share with me.
Ok, that answers part of it... you have no way to calculate the financial risk of downtime if those people won't work with you.
Just some basic math on the links alone (hopefully I did them correctly) The first one costs you ~$1.21/hour. Meaning you lost ~$0.30 on those 15 minutes of downtime. The second one costs you ~$0.16/hour. You've lost $1.92 for all those 12 hours you were down. In total though you've spent $73,920 on link 1 and $10,080 on link 2. Just something I thought was interesting.
