@scottalanmiller said in Is this server strategy reckless and/or insane?:
And that's on SATA. Go to PCIe and you can breach a million per drive!
Technically the "Million IOPS" card is NVMe (also it's reads only and that is most CERTAINLY reporting numbers coming from the DRAM on the card and not the actual NAND).
@tim_g said in Is this server strategy reckless and/or insane?:
x4 SSD any RAID level will outperform x4 15k HDD in any configuration
ehhhhh There are some low end SSD's that are "read optimized" that if I throw steady state throughput writes at they will fall over and implode (latency, shoot to the roof especially when Garbage collection kicks in, or the drive starts to get full).
If you don't have NVDIMM's doing write Coalesce or a 2 tier design (write endurance to absorb the writes) you can get really unpredictable latency out of the lower tier SATA SSD's.
@scottalanmiller said in Is this server strategy reckless and/or insane?:
@creayt said in Is this server strategy reckless and/or insane?:
@scottalanmiller said in Is this server strategy reckless and/or insane?:
@creayt said in Is this server strategy reckless and/or insane?:
@travisdh1 said in Is this server strategy reckless and/or insane?:
Has anyone mentioned going OBR5 instead of split arrays yet?
Also, I'd spend the little extra money for the Pro edition of the Samsung 850 drives if you want to use commodity parts rather than Dell supplied ones.
People did suggest OBR5, yep. The benchmarks I ran ( see the large Crystal DiskMark grid below ) made me feel like I'm going to be giving up a lot of performance for not that much additional peace of mind w/ a 5 given my set up and the ability of either server to temporarily take over duties in a pinch. My overarching goal is for most requests to be as close to perceptibly instant as possible most of the time, w/ some downtime being acceptable.
The drives are all Pros, good tip, thanks.
The big question is... is it performance that affects the application? Benchmarks and raw numbers don't matter all that much. What matters is how the app is impacted. That's why people are asking about the WAN and other components. Getting that kind of performance on such a small web app typically is all throw away performance. Not necessarily, but often.
It's heavily realtime-oriented, by which I mean I'm going to be attempting to stream the presence and actions of users to other users in real time and let them see what the other is doing Google Docs style. The ability to retrieve a good handful of information from MySQL per request in as close to 0 ms as possible is very important for the effect to work correctly, hence wanting to keep the app server and database on the same machine for example. Every little ms counts.
This is where it feels to be like MySQL was a bad choice. I don't know your details, but MySQL seems at odds with all of your other requirements.
Yah, you don't need consistency isn't needed and mySQL will scale like shit no matter how much you shard it.
@creayt said in Is this server strategy reckless and/or insane?:
@storageninja said in Is this server strategy reckless and/or insane?:
@creayt said in Is this server strategy reckless and/or insane?:
Ideally more than that, but it'll be a gradual climb. Right now it's in private alpha w/ ~ 100 users and they post stuff all the time. Once I make it public I imagine the content volume will skyrocket.
Why not use Cloud/PaaS? There are some systems where you pay by the transaction so you're not out capital for hardware that will not scale where you need to go a long time, and you will not waste money on hardware if this project goes nowhere.
Pricing out equivalent horsepower on Amazon I think came to something like $50k a month, this whole set up cost me under $10k I believe. By the time I exhaust the capabilities of this hardware/investment, I hope, I'll be at the venture capital phase and and can redeploy into a fully cloud strategy, grinning shit-eatingly at how well that original $10k investment served me.
Will also mention that colocation where I live is a dirt-effing-cheap $55-per-U/month.
There are far cheaper IaaS providers than Amazon (I assume you are looking at EC2, when you should be looking at RDS if you're doing AWS). I'm partial to Softlayer these days, but to each their own.
Deploying and managing your own infrastructure for a startup is a nightmare as if/when your product "Blows up" and goes from 100, to 100K users it will implode and crash on the weekend before you can get new hardware in and scale it, or refactor for a platform with real scalability. If your worried about cloud lock-in use abstraction systems that allow for multi-cloud strategies (although honestly in the early phase I'd just accept the lockin as that's easier to refactor than trying to refactor the platform AND scale at the same time).
If you can't maintain growth and have large hiccups in engineering VC gets spooked easily.
Also If you're really looking to scale one thing is trying to limit your dependency on RDMS in general. 9/10 times I see a startup using one, they should have used object storage or a No-SQL system.
Then again, I'm just a Palo Alto Serf working for "the man" and not feeling the wind in my hair of founding the next big thing in the garage.
@scottalanmiller said in SIP Pricing: How much are 11.338 milliseconds worth?:
@rojoloco said in SIP Pricing: How much are 11.338 milliseconds worth?:
A delay becomes audible at 28-30ms. Not sure if that translates the same to ping times...
That's a local delay. A telephone delay is closer to 200ms. This is because there is always a huge delay in the other person responding. If it is a delay where you hear your own voice, yeah, 30ms will do it.
Jitter is worse as you are at the mercy of what the buffer can fix on the other side. Standard DSP's I think cap out at 120ms, after that it all goes to hell.
@creayt said in Is this server strategy reckless and/or insane?:
Ideally more than that, but it'll be a gradual climb. Right now it's in private alpha w/ ~ 100 users and they post stuff all the time. Once I make it public I imagine the content volume will skyrocket.
Why not use Cloud/PaaS? There are some systems where you pay by the transaction so you're not out capital for hardware that will not scale where you need to go a long time, and you will not waste money on hardware if this project goes nowhere.
@dashrender said in Is this server strategy reckless and/or insane?:
UREs are probably pretty low on these SSDs, but not zero, so something else to consider, what are the chances of a URE killing your RAID 0? (now Scott will educate me that these don't matter
- seriously don't know if do or not)
The failure mode that you should be afraid of isn't a URE, but data loss on power loss that is OUT of order with acknowledged writes. This breaks standard data loss recovery you get from a Journal Log on MySQL and other database apps.
If it would just cleanly loose the last write that would be fantastic, sadly it's how the Samsung consumer drives tend to recover lost data. Normally they are used in laptops that have a giant battery attached, are not running RAID (Which will see this out of order recovery as a failed drive when it fails their ECC check). This can/will catastrophically fail with multiple drives dropping on something as simple as a controller or host failure or hard reboot.
@brianlittlejohn said in I am going to start an ISP:
What area of West Texas? I am in Midland/Odessa area
In Rocksprings, I can get 20Mbps down in my west pasture by the sinkhole on 4G. Cell providers are going to increasingly threaten WISPs.
@scottalanmiller said in I am going to start an ISP:
You can mitigate nearly all of that by splitting them between Vultr locations. But that still leaves account exposure primarily (I need to do a video on that!!)
I've seen Cloud Providers and ISP's f*** [moderated] up their external BGP announcements and get black holed by everyone they peer with....
@bigbear said in I am going to start an ISP:
3.) Plenty of Tier 1 internet to contract with to your main tower site. Start with 2 connections, they will be redundant as your ip range will be hooked to a BGP. Any teir 1 internet provider will gladly walk you through all this to get your business
You are joking, right? Level3 used to basically tell you to go @#$@ yourself if you don't know eBGP and you call them to do a link turn up.
@nerdydad said in I am going to start an ISP:
I would try to keep the business itself as paperless as possible, but would give the sub the option either way.
This one is simple. Charge $5 more a month for paper billing and outsource it to someone who does paper billing.
@nerdydad said in I am going to start an ISP:
Vultr will host the uCRM, UNMS, FreePBX (internally to company only), and email (again, internal to company only
Don't host your public website, PBX and email on the same provider as your back end. If VULTR has an outage you don't want to have ALL communication systems go down hard.
@psx_defector said in Looking for virtualization advice:
@storageninja said in Looking for virtualization advice:
Something I learned from being a group admin for Spiceworks for years is that enterprise types lurk heavily in SMB forums.
Hey, shush your mouth. I don't lurk, I troll.
WAIT, PSX IS STILL A THING?!?
@dashrender said in Looking for virtualization advice:
@tim_g said in Looking for virtualization advice:
@jaredbusch said in Looking for virtualization advice:
@tim_g said in Looking for virtualization advice:
@john-nicholson said in Looking for virtualization advice:
That $20 hr was for a contractor (yah, that's insanely low labor considering someone's skimming something off it). Note I lice in Houston, a fairly cheap city and that rate is what you'd pay a supervisor at a gas Station. It's unskilled cannon fodder here at that rate. Sometimes you get lucky (I hired a guy who moved furniture at IKEA for that for helpdesk, he ended up being smart and had to give him a raise to 56K at the end of the year though to keep him around.
In this case It was an unskilled typist, with no formal equation we taught to do the copy pasta cleanups from some scripts we wrote to try to accelerate it.
How I got out of making $20 an hour (hell slightly less than that when I started in this field) was identifying the lowest skilled things I did and then finding the cheapest resource who could do it for me. If your the god of Oracle RAC but still changing printer toner management is going to pay you like a printer serf. Stop doing cheap labor and you'll get paid more (assuming there are more valuable things to do in the day, if not find a new job).
I don't judge the value of a human being based on how much they are paid (I just spent July mostly in countries where a lot are on less than $2 a day). I do judge the value of labor (I was a hiring manager and had to know fair market rates). If your un-happy getting paid $20 an hour that's honestly your issue. If your unhappy that I say it's a cheap rate for labor in the US in a metro area for someone handling work on an IT department that's just disagreeing with a fact. I hear that's popular these days, but I've never understood it as a concept.
I completely understand. I'm not saying I make that little. My issue lies with how you say things. You seem to always say it in a degrading manner when referencing someone making less than you or boasting how many millions you make. It's not about the 20/hr, or any specific number, or any specific job or task.
Since it does not seem that way to me, then I can only assume that you are the one with the hangup on things here.
You know what they say about assumption.
I do tend to agree that @John-Nicholson posts often mention money and how much he's getting. This particular post isn't one such post. And the benefits post was really asking about compensation, so it's kind of expected there.
But the reality is that this is an SMB forum for the most part. Sure there is a small handful of people in the Enterprise, or have been in the enterprise making hundreds of thousands or millions, but those dollars just aren't the norm around these parts.
Something I learned from being a group admin for Spiceworks for years is that enterprise types lurk heavily in SMB forums. They PM people, and when they post it's often under pseudonyms as they are a bit more easily spooked about revealing who they are, largely for fear of vendors deal registering things or stalking them once they learn they have the budget or need. When you control even something small like 100K in annual capital spend the vendors can get kinda crazy. While they may not purposely come to places like this, Google and odd questions will lead them here. For the longest time "HDS vs. Netapp" led you to a thread on Spiceworks I started. HDS and Netapp both have MASSIVE forums, but they are either largely ghost towns, or hidden from public view (or just have Awful SEO). Also just because a place is a SMB don't assume they have no budget or pay people peanuts. I consulted across plenty of SMB's who paid individual contributors six figures and spent millions a year on IT purchases. I actually kind of hate using the term "SMB" (and some vendors don't even use it internally in account classifications) because it is often associated with a pejorative image of rock farmers, in a 6 man office with 1 IT guy who removes virus's all day. This just isn't reality. Those rock farmers have real time bidding systems on their rocks, and drones that map their piles of rocks several times a day to update their inventory of their rock piles (I'm not joking, an actual company here locally does this). Inversely I consulted in enterprises where they clung to NT4, paper processes, and 10-year-old servers. I've seen companies with 10,000 employees require the CFO's signature to buy a brand new $2000 MacBook Pro they were so cheap on capex!
As Scott's mentioned - If you are thigh well paid and are in IT, it's likely you would never visit a forum like ML as part of your day job because we have little to nothing to offer you. You area already probably as knowledgeable as most support staff at the vendor for whatever thing you're supporting, so it's likely there would be little to gain here.
Even if your not in SMB IT, there's reasons to be at places like ML and SPiceworks.
The lolz. There is some funny content here.
SMB overlaps with home IT on vendors sometimes (Ubiquit was a great choice for my home router, and wifi and you will not find tips on how to configure them in more enterprise places)
It's a forum. Most of the more enterprise conversations I have online are on Twitter, or Telegram or Slack. Forums have some old school charm in their permanence.
To give back. Some people got their start in the SMB space in forums like this. Personally, If it wasn't for forums like this my career would have been vastly different (It's been something like 7 years since I met Scott at the first Spiceworks user group).
An opportunity to argue with Scott. It's fun, you learn things in forming your arguments and it helps you sharpen your rhetoric.
There's a lot more consistency of actors here than you'll find on places like Reddit. You can learn to understand who/what people are about etc. There's a fair amount of people on this community I've had a beer with over the years.
I'm curious, is @John-Nicholson even in IT anymore? I can't quite tell from the conversations. It seems that he's more in a leadership role, but that might be a misunderstanding of what he's posting?
He's still around the field. he stopped touching production a year and a half ago. He's clearly an individual contributor. If you go to VeeamOn, Dellworld, VMworld he'll buy you a drink and explain.
That $20 hr was for a contractor (yah, that's insanely low labor considering someone's skimming something off it). Note I lice in Houston, a fairly cheap city and that rate is what you'd pay a supervisor at a gas Station. It's unskilled cannon fodder here at that rate. Sometimes you get lucky (I hired a guy who moved furniture at IKEA for that for helpdesk, he ended up being smart and had to give him a raise to 56K at the end of the year though to keep him around.
In this case It was an unskilled typist, with no formal equation we taught to do the copy pasta cleanups from some scripts we wrote to try to accelerate it.
How I got out of making $20 an hour (hell slightly less than that when I started in this field) was identifying the lowest skilled things I did and then finding the cheapest resource who could do it for me. If your the god of Oracle RAC but still changing printer toner management is going to pay you like a printer serf. Stop doing cheap labor and you'll get paid more (assuming there are more valuable things to do in the day, if not find a new job).
I don't judge the value of a human being based on how much they are paid (I just spent July mostly in countries where a lot are on less than $2 a day). I do judge the value of labor (I was a hiring manager and had to know fair market rates). If your un-happy getting paid $20 an hour that's honestly your issue. If your unhappy that I say it's a cheap rate for labor in the US in a metro area for someone handling work on an IT department that's just disagreeing with a fact. I hear that's popular these days, but I've never understood it as a concept.
@garyp said in Looking for virtualization advice:
@dashrender said in Looking for virtualization advice:
@scottalanmiller said in Looking for virtualization advice:
@dashrender said in Looking for virtualization advice:
@scottalanmiller said in Looking for virtualization advice:
@garyp said in Looking for virtualization advice:
We have equipment in 5 NA locations and over 500 phones, so management is not looking to move to anything else at this time.
Why would management be involved? IT should be like "we can save money, improve systems" and that's the end of it. Why would management have any say other than verifying cost savings and such? The bigger the network, the more money there is to be saved, right?
He's in the same situation as me. The hardware is already in place. Moving to FreePBX would likely require purchasing all new phones, or moving users to softphones on their computers, which would require purchasing headsets most likely.
In either case, there would be a substantial hardware outlay likely if they changed.
But they are looking at significant outlay to keep using what they have. They have to invest specifically in a VMware solution instead of what meets the needs of the business, they have to pay to keep the Avaya running and they have to take on the risks of using a solution from a non-viable or marginally viable vendor. That's all real costs that they are facing to NOT switch.
I'm not saying you're wrong - but 500 phones, even Yealink aren't cheap, not to mention the training to the staff, the IT time, etc.
If it really boils down to it, they can just leave it on the server it's currently on, and change nothing else about that one server. We assume there is already a backup solution in place - so that shouldn't be that bad to maintain.
Then the business can plan for this change over down the road.
We also have a few call centers, which accounts for a significant amount of revenue. So, aside from all the hardware expenses to rip and replace, we would need to reprogram all the routing (and the ability needs to be there, in whatever new system gets installed) along with retraining the call center staff. It would probably take an amazing ROI, before management would even begin to consider approving a project like this. I do appreciate the comments regarding the on-going expenses to manage Avaya.
I worked for a 30 seat call center. It took well over 6 man years to rewrite our scripts, port our IVR's, and migrate our crazy voicemail trees. Even using cheap as shit labor for some of this ($20Hr). Combined that during the transition we had to split operator pools between two systems, license both for the support it was a painful and expensive band-aid to get off old OmniCall, Tadaran, and VM3 systems.
In our case were had servers that were physically dying and lacked support during this process. We threw less than 50K at a VMware cluster to protect this stuff while we finished the migration, and leveraged the cluster for other platforms afterward. Considering that cluster ran for 5 years with minimal hassle, that's only 10K a year. That's $27 a day. (less than 1 hour of employee carry cost for the staff doing the migration).
@minion-queen said in At What Size Should a Business Have a PBX:
business
If I want to buy a business and it emails all to someone's personal email that's going to be a pain in the ass.
@dashrender said in Looking for virtualization advice:
@scottalanmiller said in Looking for virtualization advice:
@dashrender said in Looking for virtualization advice:
@scottalanmiller said in Looking for virtualization advice:
@garyp said in Looking for virtualization advice:
We have equipment in 5 NA locations and over 500 phones, so management is not looking to move to anything else at this time.
Why would management be involved? IT should be like "we can save money, improve systems" and that's the end of it. Why would management have any say other than verifying cost savings and such? The bigger the network, the more money there is to be saved, right?
He's in the same situation as me. The hardware is already in place. Moving to FreePBX would likely require purchasing all new phones, or moving users to softphones on their computers, which would require purchasing headsets most likely.
In either case, there would be a substantial hardware outlay likely if they changed.
But they are looking at significant outlay to keep using what they have. They have to invest specifically in a VMware solution instead of what meets the needs of the business, they have to pay to keep the Avaya running and they have to take on the risks of using a solution from a non-viable or marginally viable vendor. That's all real costs that they are facing to NOT switch.
I'm not saying you're wrong - but 500 phones, even Yealink aren't cheap, not to mention the training to the staff, the IT time, etc.
If it really boils down to it, they can just leave it on the server it's currently on, and change nothing else about that one server. We assume there is already a backup solution in place - so that shouldn't be that bad to maintain.
Then the business can plan for this change over down the road.
Consulting across mid-market and enterprise really exposed to me how much money it is to replace some systems no matter how old and shitty they are just not being worth the capital. I was helping one customer with Millienium PBX"s (yes that ancient) that were 20 years out of service. Did they need to upgrade? Sure. Did they have over 10K phones they would need to replace when they moved to a new system. yup. Sadly this system was designed with 100% American military spec capacitors and DSP's and was used on coast guard ships so it was hardened to hell and back against the elements so it wouldn't die.
Did it support VOIP trunks? No. Did we cludge in VOIP to PRI Adtrans to make up for this? Yup!
40-50K for a new VMware HA cluster vs. Millions for some nicer features, but a huge balance sheet hit for the quarter that will spook investors on the ROIC ratio? Thanks but no thanks.
add in risks of outages, disruption on retraining staff and that 40-50K to kick that can down the road looks REALLY damn atractive.
Remember why people ran AS400's and mainframes forever? Beacuse migrations were trivial and the incremital trade in cost was often low. IBM kept the "total value" vs. a full code re-write always SLIGHTLY better looking in the short to middle term and that's all most people care about.
@scottalanmiller said in Looking for virtualization advice:
@john-nicholson said in Looking for virtualization advice:
@scottalanmiller said in Looking for virtualization advice:
@john-nicholson said in Looking for virtualization advice:
Knowing list is honestly useless when you don't know what discounts will apply in a quarter, or for what products. A lot of vendors have a small handful products every quarter that get an extra 30% off "power play". If you get a quote that's good for the quarter but don't keep in touch with the VAR you might miss the discount (not that you need to buy things you don't need, but it may throw off your budgeting). This is why you gota talk to your VARs!
I don't agree. Because all vendors do this. So knowing list really does give you 90% of what you need. And if a vendor throws away business because they do stupid public pricing is their own loss. Most vendors know not to do that, this is basic IT. So while knowing list doesn't tell you what you will pay, it does give you reasonable basis for comparison because everyone will have a sale quarterly that you can account for.
There are vendors where discount off list is normally 10%, there are ones where it's normally 70% (Meraki, from a MSP who does a lot of volumes). Often those standard discount rates can vary between product lines for the same vendor, especially if they are not fully integrated into the same partner network.
True, but Meraki is a product you'd never use anyway. Of actual quality products and vendors, do you ever see this happening?
For wireless? Airnet (Best AP's for hostile, CleanAir's patents for this stuff are amazing in weird industrial environments), Ruckus, Aruba (The standard in conference centers for unmatched density) all have deal registration with heavy discount schedules (at least 50%). Wireless is notorious for it because even small companies can end up buying quite a few if they have remote offices, or warehouses needing coverage. A small older Hotel with walls that are RF dense can end up with some crazy dense on AP's also. Ubiquity networks is pretty much alone in doing retail style pricing, and while I love them (have the monster AC-PRO in my house) they lack quite a few enterprise features that will need to be kludged in with other systems or made up for in over-deployment of units.