The XFS real-time subvolume in Linux

> Eric Sandeen <[hidden email]> writes:
>
>>That is basically true, yes.  There is a non-free GRIOV2 product in
>>use with CXFS, but for your purposes, I think it is safe to say that
>>there is no standalone GRIO equivalent on Linux.
>
>
> It's not. In fact it's a standard feature now.
>
> The CFQ2 IO scheduler has IO priorities settable with ionice, including
> a RT class with 8 priorities.
>
> It's not available in SLES9 though, only in newer kernels (2.6.13+)
> and SUSE releases (like SL10.0)
>
> -Andi
>

> Andi Kleen wrote:
> > Eric Sandeen <[hidden email]> writes:
> >>That is basically true, yes.  There is a non-free GRIOV2 product in
> >>use with CXFS, but for your purposes, I think it is safe to say that
> >>there is no standalone GRIO equivalent on Linux.
> >
> > It's not. In fact it's a standard feature now.
> >
> > The CFQ2 IO scheduler has IO priorities settable with ionice, including
> > a RT class with 8 priorities.
> >
> > It's not available in SLES9 though, only in newer kernels (2.6.13+)
> > and SUSE releases (like SL10.0)
> >
> > -Andi
>
> Ah, but is there a bandwidth reservation system to go with it? That is
> the missing link here, being able to say 'I need 10 Mbytes/sec until
> further notice'.

> On Tuesday 04 October 2005 17:41, Steve Lord wrote:
>
>>Andi Kleen wrote:
>>
>>>Eric Sandeen <[hidden email]> writes:
>>>
>>>>That is basically true, yes.  There is a non-free GRIOV2 product in
>>>>use with CXFS, but for your purposes, I think it is safe to say that
>>>>there is no standalone GRIO equivalent on Linux.
>>>
>>>It's not. In fact it's a standard feature now.
>>>
>>>The CFQ2 IO scheduler has IO priorities settable with ionice, including
>>>a RT class with 8 priorities.
>>>
>>>It's not available in SLES9 though, only in newer kernels (2.6.13+)
>>>and SUSE releases (like SL10.0)
>>>
>>>-Andi
>>
>>Ah, but is there a bandwidth reservation system to go with it? That is
>>the missing link here, being able to say 'I need 10 Mbytes/sec until
>>further notice'.
>
>
> Indirectly there is. The RT priority defines how many time slots you
> get and the length of the timeslots are configurable using sysfs.  If you know
> the bandwidth of the disk you can use that to define an approximation
> of the guaranteed bandwidth for a specific RT priority.
>
> On the other hand I don't really see how you can get real bandwidths.
> e.g. on most disks the bandwidth varies greatly depending on where
> the blocks are allocated and how much seeking it does. If you take
> that all in account you'll probably get a pretty slow worst case
> as baseline to divide.

> Andi Kleen wrote:
> >On Tuesday 04 October 2005 17:41, Steve Lord wrote:
> >
> >>Andi Kleen wrote:
> >>
> >>>Eric Sandeen <[hidden email]> writes:
> >>>
> >>>>That is basically true, yes.  There is a non-free GRIOV2 product in
> >>>>use with CXFS, but for your purposes, I think it is safe to say that
> >>>>there is no standalone GRIO equivalent on Linux.
> >>>
> >>>It's not. In fact it's a standard feature now.
> >>>
> >>>The CFQ2 IO scheduler has IO priorities settable with ionice, including
> >>>a RT class with 8 priorities.
> >>>
> >>>It's not available in SLES9 though, only in newer kernels (2.6.13+)
> >>>and SUSE releases (like SL10.0)
> >>>
> >>>-Andi
> >>
> >>Ah, but is there a bandwidth reservation system to go with it? That is
> >>the missing link here, being able to say 'I need 10 Mbytes/sec until
> >>further notice'.
> >
> >
> >Indirectly there is. The RT priority defines how many time slots you
> >get and the length of the timeslots are configurable using sysfs.  If you
> >know the bandwidth of the disk you can use that to define an approximation
> >of the guaranteed bandwidth for a specific RT priority.
> >
> >On the other hand I don't really see how you can get real bandwidths.
> >e.g. on most disks the bandwidth varies greatly depending on where
> >the blocks are allocated and how much seeking it does. If you take
> >that all in account you'll probably get a pretty slow worst case
> >as baseline to divide.
>
> If you get into this stuff seriously you have to dedicate hardware
> all the way from the cpu to the disks, make worst case estimates of how
> fast your disks will go. Multiply all this out and say that to record n
> streams of HD video without dropping a frame you need this much hardware.
> It is a bit of a black magic art, and not something you just go out and
> buy a PC to do. It tends to waste a lot of the potential bandwidth of
> the hardware, but you try explaining to CNN why their satellite feed just
> went black ;-)

> Jens Axboe wrote:
> >That's exactly why the Linux ioprio stuff has been designed the way it
> >is right now - it's not overengineered for something we cannot support
> >anyways. The CFQ io priorities will work well enough for general use, if
> >you are basing your business on GRIO it's a different game completely. I
> >don't want to add kernel infrastructure for something that is very
> >specialized, especially because the code to do so would be 10 times
> >bigger and more complex that the current stuff..
>
> Jens, I didn't mean to imply that you -should- have done a GRIO-type
> design (and I doubt that Steve did, either.)  My only point was that
> GRIO and ioprio are two different IO control mechanisms.

> Jens Axboe wrote:
> > That's exactly why the Linux ioprio stuff has been designed the way it
> > is right now - it's not overengineered for something we cannot support
> > anyways. The CFQ io priorities will work well enough for general use, if
> > you are basing your business on GRIO it's a different game completely. I
> > don't want to add kernel infrastructure for something that is very
> > specialized, especially because the code to do so would be 10 times
> > bigger and more complex that the current stuff..
>
> Jens, I didn't mean to imply that you -should- have done a GRIO-type
> design (and I doubt that Steve did, either.)  My only point was that
> GRIO and ioprio are two different IO control mechanisms.

> On Wednesday 05 October 2005 16:11, Eric Sandeen wrote:
> > Jens Axboe wrote:
> > > That's exactly why the Linux ioprio stuff has been designed the way it
> > > is right now - it's not overengineered for something we cannot support
> > > anyways. The CFQ io priorities will work well enough for general use, if
> > > you are basing your business on GRIO it's a different game completely. I
> > > don't want to add kernel infrastructure for something that is very
> > > specialized, especially because the code to do so would be 10 times
> > > bigger and more complex that the current stuff..
> >
> > Jens, I didn't mean to imply that you -should- have done a GRIO-type
> > design (and I doubt that Steve did, either.)  My only point was that
> > GRIO and ioprio are two different IO control mechanisms.
>
> I suspect for most people they will be pretty much equivalent.

> On Wednesday 05 October 2005 16:58, Jens Axboe wrote:
>
> > There are still unknowns, the HD still being the biggest one of course.
> > The problem is that you don't know the worst case HD performance, it
> > might be doing all sorts of rewriting, calibration, error correct etc
> > that can still screw you. So I think without definitely knowledge of
> > what the HD will do in case of errors (or a way to control that which
> > you definitely can on some drives), it's still pretty hazy. It gets
> > better, but if you are looking for complete guarantees I don't think
> > it's good enough.
>
> Yes, but GRIO has exactly the same problem. I assume they need custom
> calibration for each IO subsystem.

> On Wed, Oct 05 2005, Andi Kleen wrote:
>
>>On Wednesday 05 October 2005 16:58, Jens Axboe wrote:
>>
>>
>>>There are still unknowns, the HD still being the biggest one of course.
>>>The problem is that you don't know the worst case HD performance, it
>>>might be doing all sorts of rewriting, calibration, error correct etc
>>>that can still screw you. So I think without definitely knowledge of
>>>what the HD will do in case of errors (or a way to control that which
>>>you definitely can on some drives), it's still pretty hazy. It gets
>>>better, but if you are looking for complete guarantees I don't think
>>>it's good enough.
>>
>>Yes, but GRIO has exactly the same problem. I assume they need custom
>>calibration for each IO subsystem.
>
>
> Indeed it does, and yes if they really want to provide the type of
> guarantees that Steve listed, then that needs a custom box with either
> custom or known disk firmware options. If not you cannot give absolute
> guarantees and expect to always honor them. That's in addition to
> anything you may need to change in software, if using Linux you would
> need to audit/fix lots of things in the io path.
>

> Jens Axboe wrote:
> >On Wed, Oct 05 2005, Andi Kleen wrote:
> >
> >>On Wednesday 05 October 2005 16:58, Jens Axboe wrote:
> >>
> >>
> >>>There are still unknowns, the HD still being the biggest one of course.
> >>>The problem is that you don't know the worst case HD performance, it
> >>>might be doing all sorts of rewriting, calibration, error correct etc
> >>>that can still screw you. So I think without definitely knowledge of
> >>>what the HD will do in case of errors (or a way to control that which
> >>>you definitely can on some drives), it's still pretty hazy. It gets
> >>>better, but if you are looking for complete guarantees I don't think
> >>>it's good enough.
> >>
> >>Yes, but GRIO has exactly the same problem. I assume they need custom
> >>calibration for each IO subsystem.
> >
> >
> >Indeed it does, and yes if they really want to provide the type of
> >guarantees that Steve listed, then that needs a custom box with either
> >custom or known disk firmware options. If not you cannot give absolute
> >guarantees and expect to always honor them. That's in addition to
> >anything you may need to change in software, if using Linux you would
> >need to audit/fix lots of things in the io path.
> >
>
> Definitely, from my memory, grio had a tool for measuring a disk subsystem.
> I think reality is closer to overspecing the hardware than to providing an
> absolute guarantee of bandwidth. Being able to prioritize individual I/O
> calls
> is just part of the picture.