@dustinb3403 No writes are lost, it's handled on your VM level (VM OS wait for "ack" of virtual HDD but it's not answering, so it waits). Basically, cluster said: "writes command won't be answered as long as we figured it out".

So it's safe

@fateknollogee said in XenServer hyperconverged:

Are the hosts shown in your example using HW or software RAID?

What is preferred, HW or software RAID?

@dustinb3403 said in XenServer hyperconverged:

@fateknollogee said in XenServer hyperconverged:

Are the hosts shown in your example using HW or software RAID?

What is preferred, HW or software RAID?

Based on the blog post I'm guessing HW raid

It's not that easy to answer. Phase III will bring multi-disk capability on each host (and even tiering). So it means you could use any number of disks on each hosts to make inception-like scenario (replication on host level + on cluster level). But obviously, hardware raid is perfectly fine too

@r3dpand4 said in XenServer hyperconverged:

@olivier We're talking about node failure where you're replacing hardware correct?
"On a 2 node setup, there is an arbiter VM that acts like the witness. If you lose the host with the 2x VMs (one arbiter and one "normal"), you'll go in read only."
So are Writes suspended until the 2nd node is brought back online and introduced to the XOSAN? Obviously that's going to be a lot longer than a few seconds even if you're talking about scaling outside of a 2 node cluster. Do you mean that Writes are suspended or cached in the event of a failure while Host is shutting down and then Writes resume as normal on the Active node? If this is the case when the new Host is introduced to the cluster the replication resumes, correct?

Nope, writes are suspended when a node is down (time for system to know what to do). If there is enough nodes to continue, writes are resumed after being paused few secs. If there isn't enough nodes to continue, it will be then in read only.

Let's imagine you have 2x2 (distributed-replicated). You lose one XenServer host in the first mirror. After few secs, writes are back without having any service failed. Then, when you'll replace the faulty node, this fresh node will "keep up" the missing data in the mirror, but your VM won't notice it (healing status).

@dustinb3403 said in XenServer hyperconverged:

@olivier said in XenServer hyperconverged:

@dustinb3403 said in XenServer hyperconverged:

So with XOA Free you can get XOSAN?

Yes.

@fateknollogee said in XenServer hyperconverged:

When will pricing be available?

It's a matter of weeks.

So then when will the community edition get XOSAN?

Not like this. We'll open source the gluster driver, to allow people to make their own solution (hyperconverged or not). But it won't be turnkey.

@dustinb3403 said in XenServer hyperconverged:

So with XOA Free you can get XOSAN?

Yes, XOA Free + paid XOSAN will work.

@fateknollogee said in XenServer hyperconverged:

When will pricing be available?

It's a matter of weeks.

@dustinb3403 said in XenServer hyperconverged:

@fateknollogee said in XenServer hyperconverged:

Where will XOSAN exist in this pricing structure?
https://xen-orchestra.com/#!/pricing

I believe it was mentioned as being a separate price entirely. So you can get XOA with or without XOSAN.

XOA Free at least to get XOSAN (you need XOA to deploy and manage it).

@r3dpand4 said in XenServer hyperconverged:

@olivier So you only gain in capacity scaling in 2's? Again as I mentioned earlier I'm curious how the Write suspension functions during failure....

Write are suspended few secs, works fine. Tested on Linux and Windows VMs.

@dashrender said in XenServer hyperconverged:

@olivier said in XenServer hyperconverged:

@dashrender As far you don't lose a complete mirror (2 hosts on the same pair), no problem. So on 8 nodes, it means up to 4 hosts if there in one by mirror.

and what happens if you do loose an entire mirrored pair?

If you lose an entire mirror pair, it's like in RAID10: you will lose data.

@fateknollogee said in XenServer hyperconverged:

Where will XOSAN exist in this pricing structure?
https://xen-orchestra.com/#!/pricing

Nope, it will be a dedicated pricing, per pool, without number of hosts or disk space limitations.

@dashrender As far you don't lose a complete mirror (2 hosts on the same pair), no problem. So on 8 nodes, it means up to 4 hosts if there in one by mirror.

@fateknollogee yes

@dustinb3403 Scaling at one host: going on "triplicate" (same data on each node, you have 1/3 of total disk capacity, but you can lose 2 hosts and still have data access).

Scaling by adding 2 hosts at the same time: it's adding a subvolume, so more space.

@fateknollogee If you are in replicated-2, you can add:

• 2 hosts to create a new extra RAID1 (bottom of a RAID0), creating a distributed-replicated (2x2)
• 1 host to go from replicated 2 replicated 3 (data is copied on three hosts, so it's a 1x3)

I asked to modify the picture to draw a host with the disk inside to avoid confusion Thanks for the feedback on that guys

@dustinb3403 I suppose you speak about the 2 XS host setup. It's only in this case you need an arbiter, and it will be on one of the 2 host (to avoid split brain).

If only your arbiter VM is down, quorum is still met (2 on 3). If you lose more than 1/3 of nodes (arbiter or not), you are on read only (protecting against split brains).

And no, you are not correct: each disk picture is a XenServer host. There is a XOSAN VM controller on each host (and an extra arbiter ONLY in 2 hosts scenario, if you grow the XOSAN from 2 to 4, no more arbiter, it's automatically removed)

The question is (i think) if you lost all hosts in the orange group, would XOSAN and the operating VM's in the entire pool still be functional Read/Write until those servers are brought back online?

Because data are spread on all subvolumes (think like RAID10), you'll be in read only on the whole thing.

You can avoid that if you decide to NOT stripe files on all subvolumes (which is the default behavior in Gluster by the way), but it's NOT a good thing for VMs (because heal time would be horrible, and subvolumes won't be balanced)

@dashrender said in XenServer hyperconverged:

In this picture

how many systems have the data on it? only 2?

This is a part of distributed-replicated setup. In this "branch", you have 2 XS hosts with 100GiB data on each, in "RAID1"-like.

The others branches (not in the picture you displayed) are like a RAID0 on top.

@dustinb3403 said in XenServer hyperconverged:

@olivier I think this is a 2 node setup that @Dashrender is discussing.

Can the system scale to more than 2 nodes?

This IS meant to scale (up to max pool size, 16 hosts, which is more a XenServer limit than ours )

@Dashrender The picture is not clear maybe: each "disk" icon is not a disk, but a XenServer host. So you have 6 hosts there. When you lost "enough" hosts (2 replicated hosts in the 6 setup, in the same "mirror"), it will be in read only. As soon one of this 2 is back online, R/W is back.

On a 2 node setup, there is an arbiter VM that acts like the witness. If you lose the host with the 2x VMs (one arbiter and one "normal"), you'll go in read only. No split brain possible.

@Dashrender Can you be more specific on the total number of nodes from the start?

1. If you shutdown all XOSAN VMs, your SR is unreachable. When you start it, as soon the minimal number of nodes is up, it will work again (even if there is auto healing, it will be slower but work in R/W)

2. If you replace one node, you can do it live (we have a "replace" button in the UI!). The new node will replace the old one and get all the missing pieces via the heal process.

In the end, nothing to do to make it work again

Article is out (as 5.12): https://xen-orchestra.com/blog/improving-xenserver-storage-performances-with-xosan/

Doc updated: https://xen-orchestra.com/docs/xosan.html

@fateknollogee 1. Because KVM APIs are really light (like "pure" Xen APIs).
2. This will require hiring 10 people approx for 1y.