@scottalanmiller said in Discovering FakeRAID in the Real World:

@pete-s said in Discovering FakeRAID in the Real World:

@dustinb3403 No, Broadcom 3008 has it's own integrated cpu, on board memory etc. It's just not enough performance for it to do all kinds of raid. It's the same controller as Dell H330 and a bunch of others make cards with it as well.

From the specs:

The SAS 3008 offered with the Avago Integrated RAID (IR) feature is a
low cost, high performance RAID solution designed for blade, entry and
mid-range servers that require redundancy and high availability but where
a full featured RAID implementation is cost prohibitive or not desired. The
Avago advanced Integrated RAID options include Integrated Mirroring
(IM), which is RAID 1, Integrated Mirroring Enhanced (IME), known as RAID
1E, Integrated Striping (IS), which is RAID 0, and Integrated Mirroring and
Striping (IMS) which is RAID 10. By simplifying the RAID configuration
options, Integrated RAID is easy to install and configure and meets the
needs of most internal RAID requirements.

https://mangolassi.it/topic/6375/examining-the-dell-perc-h310-controller/

The H3xx series is hardware. Terribly slow hardware, but hardware.

The H330 is a 3008 based product but it has a different firmware package, and some cut down internals so it has a much lower queue depth, and a bastardized mega-raid capability (no write cache). There is memory on the card, but it's for the queues.

The HBA330 isn't slow it @#%@% flies but it's pass through only. It is the full queue'd (exposes the raw possible queue based on the memory which is crazy high), and has no firmware option for a RAID firmware.

The H310 was a LSI2008 based ASIC package which had its queue crippled to something pathetic like 25.

Dell runs their own trunk for HBA firmware's rather than just grabbing various checkpoints on the tree is my understanding, so beyond these customizations, you will sometimes see different behavior. Sometimes they will selectively ship a fix, and avoid a feature they found problematic or didn't want (Tri-Mode RAID wasn't supported on the newest PERC's at first).

The controller industry is crazy opaque.

@dashrender said in Is Most IT Really Corrupt?:

One of the things here is not to simply look at cost savings, but cost parity. If the company gets all the same work done at a higher quality at the same cost, that's a win.

When I worked for a MSP, no one came to us to save money. (and I mean no one).

They came to us over money but it boiled down to...

1. They wanted to get more value for the same spend.
2. They wanted to spend more, but know where it was going.

I'd argue SaaS business have the same model. I can keep using my 12-year-old copy of ACT for CRM, and it's a TON cheaper than SalesForce (Sunk cost) but I see a hell of a lot of companies going to SalesForce.

If you think your primary job in IT is to control cost (and not to increase agility for the business, or mitigate risk) then at best you're going to be stuck at 30% below market rate for someone with your number of years in IT, and at worse you'll get pushed out by further automation, SaaS/Cloud/Simplification.

@JaredBusch said in Veeam Company Announcement:

Private equity firms are known to fuck things up.

Public companines responding to quarter to quarter results are known to fuck up.
CEO's are known to fuck things up.
Private companies are known to mis-manage money.

Everyone fucks up.

With De-duplication and Compression and RAID 5/6 Flash drives are cheaper than 10K RPM drives. We did the price comparisons with VSAN 6.2 came out and 10K is officially "dead" unless all your data is encrypted or something.

@stacksofplates said in Discovering FakeRAID in the Real World:

@scottalanmiller there isn’t a software based raid option on ESXi unless you pass it to a VM or use vSAN. The cheapest raid controllers supported are the BOSS modules M.2 mirroring things.

I think that's what our VxRails are coming with.

14Gen VxRAIL and ReadyNodes use's BOSS for the boot modules. VxRAIL keeps the factory re-install image I think on the SD Cards in case you need to factory nuke a node.

We did these at my last job (Think the last one I did was ~40K). Really I wanted the report to stand on its own and have enough value that we could walk away and do nothing more, but they would still have value.

Discounts against implementation create bad incentives (to spike the project costs enough that you can still maintain a P&L for the projects). I'd rather have simple, projects where the P&L for every interaction is fair pricing, clearly understood deliverables and value for every engagement, and everyone's value can clearly be understood. You also screw the P&L for your BA's if you do discounts like this (which lowers their book value to the company and generally means you'll end up paying them or the "shock troop" consultants who do the assessment less than they are worth).

@dashrender said in Is Most IT Really Corrupt?:

or a lot of standing around with not so trustworthy clients in their offices where their purses, phones, etc. are. I don't know if any of you deal with the public a lot or not, but our clientele are not to be trusted, as they have stolen from many of our employees before

I like central print servers where you walk up, enter your code, then it dumps the print queue for what you had queued (on some crazy fast printer). You can do secure printing in a central manner this way while still maintaining compliance. It also has the benefit of they can pick up their print jobs from ANY office.

@IRJ PayPal are space pirates on getting money back. Beyond that assuming he does what I do (link it to a checking account that is kept at zero $ and is a clearing account) you can't claw back anything even if they try (My bank will bounce the request). Escrow fights are just not worth the time for $150.

@Dashrender said in 4th Ammendment:

@StorageNinja said in 4th Ammendment:

@scottalanmiller said in 4th Ammendment:

"The Supreme Court has clearly and repeatedly confirmed that the border search exception applies within 100 miles of the border of the United States as seen in cases such as United States v. Martinez-Fuerte where it was held that the Border Patrol's routine stopping of a vehicle at a permanent checkpoint located on a major highway away from the Mexican border for brief questioning of the vehicle's occupants is consistent with the Fourth Amendment."

https://en.wikipedia.org/wiki/United_States_v._Martinez-Fuerte

I assume this is like the 4th amendment really doesn't apply much to game wardens for the purpose of looking in your freezer etc. Doesn't it only work for the purposes though of border enforcement, and only by border patrol?

Very likely - but meh, the cops will simply call in border patrol to do the searching, etc.

Couple things here..

that case involved a fixed internal checkpoint on a highway (Not my house, or an arbitrary checkpoint setup on my local street).

The court felt that any intrusion to motorists was a minimal one and that the government and public interest outweighed the constitutional rights of the individual - I don't see how searching my house is ever going to fall under this.

The court also ruled that the stops were Constitutional even if largely based on apparent Mexican ancestry - The courts cool with casual racial based policing when near a border.

one's expectation of privacy in an automobile and of freedom in its operation are significantly different from the traditional expectation of privacy and freedom in one's residence. United States v. Ortiz, 422 U.S. at 422 U. S. 896 n. 2; see Cardwell v. Lewis, 417 U. S. 583, 417 U. S. 590-591 (1974) Basically, again if it's in a car your expectations are a lot lower.

we hold that the stops and questioning at issue may be made in the absence of any individualized suspicion at reasonably located checkpoints - The checkpoint has be reasonable. I-35 North coming out of Larando? reasonable. I-35 north of San Antonio? yahhhh unlikely.

This isn't 100% removal of the 4th amendment within 100 miles of the border.
It IS still a questionable ruling.

@KOOLER Limits if I recall are 1TB file size max, no virtual machines, post process only, 32KB block size (but Variable Block at least right?) 2016 should raise the limit.

Advantage to doing data reduction on the back end is you can dedupe out common applications and OS files between Virtual machines. That said flash is so cheap (~55 cents per GB for enterprise grade storage) throwing hardware at the problem has its advantages...

@tirendir said in Is Most IT Really Corrupt?:

However, I will say that the acting as a partner versus a vendor, service provider, or a supplier has been very noticeable in the MSPs I've talked with, as most of the MSPs I've talked with seem to be all about what they can do for us instead of what we actually need or want them to do for instance. You know, marketing BS instead of just trying to do their jobs and sell us what we want/need, lol. It all sounded impressive to the others in management, as they didn't know enough to know why the marketing was just BS

To be fair, many in house IT staff don't get what the business needs are. (It's fairly common). I've seen many times where internal IT thought their goal was to cut costs (buying desktops or cheap heavy laptops for sales people) and missed out on what the business need was (Sales people who could work from anywhere and would benefit from SaaS Mobile apps, and high battery low weight ultrabooks). Don't conflate a recommendation to spend money on things you don't see value in, without things that DON"T actually have value to the business. Looking back to my time working in house at a SMB Dunning–Kruger effect was common in our department.

@wirestyle22 Just deploy Apple TV's and let them Airsync their content over.

They did this for the sole reason that this was the only way to continue providing their solutions based on the legacy vendors and their lock out and lack of access.

Not sure where this information came from.
PernixData, SanDisk’s Flashsoft, and ScaleIO have all used kernel modules with vSphere...

The reason these vendors use VSA's is a combination of factors, largest of which writing kernel code is hard...

@scottalanmiller said in Is Most IT Really Corrupt?:

@tirendir said in Is Most IT Really Corrupt?:

My one big gripe with our current Executive is that the he is very anti-remote access for some reason.

Emotions. SMBs will often punish their workers simply because they dislike people more than they like profits. Willing to sabotage the organization for some personal emotional benefit.

That's not to say that working remotely is best in all cases. But being against it emotionally instead of considering it as a business decision is actively not doing his job as an executive and exactly the kind of problem I see in the SMB - either hurting the company through incompetence or outright overt corruption.

had a hilarious meeting with a VP who was over an IT department at a SMB. He was explaining why he wanted to fire the department manager.

"Some days he comes in late and works from home. He sets an awful example for the younger staff! How can they ever expect to have any work ethic like that!"

Had to calmly explain that work from home and flex time is common in our industry, and if they wanted to make people work 8-5 always be on call and only have 1 week of vacation they needed to double their staff to absorb the on call, risk higher attrition or give everyone a 40% pay raise they wanted to keep. Being a consultant was fun some days.

Equity owners in SMB's often expect everyone to work like they have equity even when they don't is a common issue I see. the "We view our company like family" I often find translated to "We wish we could claim the labor exemptions for overtime and wages that farmers can for their 13 yr old kids!"

@scottalanmiller EA's and audit requirements have huge variables depending on industry, requirements, the country its originated in, the countries it is used in. The language varies so much (and you can ask for things to be waved, changed, or added based on your needs). EA's are fundamentally driven by both parties liking the numbers, and what the lawyers will approve. There is no "standard language" as what the DOD will accept is different from a hosting company is different from a oil company.

@donaldlandru said in ZFS Based Storage for Medium VMWare Workload:

Ok, so a little background. the storage situation at my organization is our weakest link in our network. Currently we have a single HP MSA P2000 with 12 spindles (7200 rpm) serving two separate ESXi clusters.

This is not a lot of IOPS I have easily 100x more IOPS in the laptop I'm typing this on, than this disk configuration.

It is not uncommon for us to max out the disk i/o on 12 spindles sharing the load of almost 150 virtual machines and everyone is on board that something needs to be changed.

Yep, Go all Flash.

Here is what the business cares about the solution: Reliable solution that provides necessary resources for the development environments to operate effectively (read: we do not do performance testing in-house as by the very nature, it is much a your mileage may vary depending on your deployment situation).

If your a VMware shop doing QA testing, there's some workflows you can do with Linked Clones, and Instant Clones (No IO overhead, 400ms to clone a VM with zero memory or disk as it runs ajournal log for both) to reduce disk load, speed up process's and in general make everyone's life easier.

In addition to the business requirements, I have added my own requirements that my boss agrees with and blesses.

1. Operations and Development must be on separate storage devices

This isn't necessary at 150 VM's. Just get something with all flash and if it is that big of a deal that people are running IO burner something that has QoS as an option to smack them down.

2. Storage systems must be built of business class hardware (no RED drives -- although I would allow this in a future Veeam backup storage target)

Think strongly about using reds with Veeam. Reverse Incremental or roll ups use random IO and your backup windows will hate you.

Requirements for development storage

• 9+ Tib of usable storage
• Support a minimum of 1100 random iops (what our current system is peaking at)
• disks must be in some kind of array (zfs, raid, mdadm, etc)

Proposed solutions:

#1 a.k.a the safe option
HP StoreVirtual 4530 with 12 TB (7.2k) spindles in RAID6 -- this is our vendor recommendation. This is an HP renew quote with 3 years 5x9 support next-day on-site for ~$15,000

Wait is this a single node storevirtual? ALso the IOPS on this will be awful. (7.2K drives are slow).

Less performance than solution #2 out of the box
More expensive to upgrade later (additional shelves and drives at HP prices)
All used hardware
Its worse than that, as you have to not just buy HP parts but licensing.

#2 ZFS Solution ~$10,000
24 spindle 900Gb (7.2k SAS) in 12 mirrored vdevs

To my knowledge no one makes 900GB 7.2K SAS drives.

Based on Supermicro SC216E16 chassis
X9SRH-7F Motherboard
Intel E5-1620v2 CPU
64 GB of RAM
No L2ARC or ZIL planned

Then why the hell would you use ZFS?

Dual 10gig NICs

Pros
Better performance out of the box (twice the spindle count)
Non-vendor specific parts means upgrades require less investment

Cons
Alright, tear me apart tell me I am wrong or provide any other useful feedback. The biggest concerns I have exist in both platforms (drives fail, controllers fail, data goes bad, etc) and have to be mitigated either way. That is what we have backups for -- in my opinion the HP gets me the following things:

1. The "ability" to purchase a support contract
2. Next-day on-site of a tech or parts if needed

Be careful with NBD Parts, as a failure on Thursday afternoon really means Monday afternoon is within SLA as its based on when it was isolated.

With the $4000 saved from not buying the HP support contract I can buy a duplicate Supermicro system, and a couple extra hard drives, and have the same level of protection.

Note: this is my first time posting an actual give me feedback topic, I tried to include all information I felt was relevant. If more is needed I can provide.

Your a VMware shop, curious if you looked at Using VSAN? You could go all flash, and get inline Dedupe and Compression which makes all flash cheaper than 10K drives at that point.

@dashrender said in Is Most IT Really Corrupt?:

@storageninja said in Is Most IT Really Corrupt?:

@tirendir said in Is Most IT Really Corrupt?:

However, I will say that the acting as a partner versus a vendor, service provider, or a supplier has been very noticeable in the MSPs I've talked with, as most of the MSPs I've talked with seem to be all about what they can do for us instead of what we actually need or want them to do for instance. You know, marketing BS instead of just trying to do their jobs and sell us what we want/need, lol. It all sounded impressive to the others in management, as they didn't know enough to know why the marketing was just BS

To be fair, many in house IT staff don't get what the business needs are. (It's fairly common). I've seen many times where internal IT thought their goal was to cut costs (buying desktops or cheap heavy laptops for sales people) and missed out on what the business need was (Sales people who could work from anywhere and would benefit from SaaS Mobile apps, and high battery low weight ultrabooks). Don't conflate a recommendation to spend money on things you don't see value in, without things that DON"T actually have value to the business. Looking back to my time working in house at a SMB Dunning–Kruger effect was common in our department.

Yeah, this is one I have to remind myself of often.

In consulting I would always invite the software developers, and a few random users and the operations management at customers to lunch and leave the Sysadmins behind. You learned what the REAL value of performance, uptime, and a given application was talking to them. It was comical how out of touch in house sysadmins get (Hey, I used to be one too). Oddly enterprises tend to do a better job at this because they have dedicated non-technical IT management functions (CIO's, Directors) who's jobs are to bridge with other BU's and departments.

@scottalanmiller said in Vmware Audit:

@scottalanmiller EA's and audit requirements have huge variables depending on industry, requirements, the country its originated in, the countries it is used in. The language varies so much (and you can ask for things to be waved, changed, or added based on your needs). EA's are fundamentally driven by both parties liking the numbers, and what the lawyers will approve. There is no "standard language" as what the DOD will accept is different from a hosting company is different from a oil company.

I understand that it is very hard. It's also tough because the OP is saying that this is from a EULA, not from the EA. Hopefully he will chime in soon. It seems like crazy audit stuff.

I don't believe auditing is in the standard EULA on the website. I have NEVER heard of a non-EA customer being audited.

@Dashrender IF you put enterprise grade in a REAL datacenter its scary how long it will run without failure. Now if this stuff is going in some Nolo Telco Dataslum, or his closet yah, stuff dies all the time.

@jmoore said in Where Does University Need to Focus for IT Students:

@storageninja I went to Baylor and thought they had a good program. It was mostly business with a bit of programming mixed in. I often "had" to help the girls figure out visual c++

I didn't say Baylor's CS was a bad program, just for working in IT infrastructure management, the MIS program was a bitter better rounded. If I wanted to do software engineering obviously MIS is a crap degree compared to CS. My personal issue with Baylor's CS program was that they had a lot of brilliant CS people, but beyond Bill Booth (Who had a master on the side in education) very few were good teachers. This isn't unique to Baylor and is a common issue in an engineering discipline at research universities where it is graduate students who do most of the actual education. Ultimately I wandered out of CS wasn't grades (made A's in my intro classes) it was because I watched office space, and visited a development office that seemed terribly similar and realized I didn't want to do that with my life.

I ended up with a really broad/rounded education (International Studies, minor business, Honors College (Baylor Interdisciplinary Core) that has served me well for my pursuits.

If I had my way, it would be impossible to graduate college without taking ancient and modern rhetoric. So many people in IT (and everywhere) make awful arguments.