Super Resolution using MVTools

[Efenstor]

It's possible to use Fizick's MVTools to double video resolution and actually improve detail (i.e. bring out the barely visible). This technique is called "super resolution" and can really work miracles on videos shot with photo cameras and cell phones. Of couse, it can be also used to upscale SD to HD.

Updated on 2009-05-15
Required: MVTools 2.4.2, NNEDI 1.3, MT 0.7 (for mult-core CPU's, if yours is not, remove the SetMTMode lines)

SetMTMode(5)
AVISource("test.avi")
ConvertToYUY2()
SetMTMode(2)

MDeNoise(100)
MSR()


function MDeNoise(clip s, int th)
{
blkH = 8
blkV = 8
ovl = 2
dct = 0
srch = 4
pel = 2

super = MSuper(s, pel=pel)
vec1 = MAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb=true, delta=2)
vec2 = MAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb=true, delta=1)
vec3 = MAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb=false, delta=1)
vec4 = MAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb=false, delta=2)
MDegrain2(s, super, vec1, vec2, vec3, vec4, thSAD=th)
}



function MSR(clip s)
{
blkH = 16
blkV = 16
ovl = 8
srch = 3
dct=0
pel=2

bc = BilinearResize(s, Width(s)*2, Height(s)*2)
super = MSuper(bc, pel=pel)
vec1 = MAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb=true, delta=2)
vec2 = MAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb=true, delta=1)
vec3 = MAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb=false, delta=1)
vec4 = MAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb=false, delta=2)

s = nnedi(s, field=0, dh=true)
s = TurnLeft(s)
s = nnedi(s, field=0, dh=true)
s = TurnRight(s)

s = Sharpen(s, 1)

super = MSuper(s, pel=pel)
c1 = MCompensate(s, super, vec1)
c2 = MCompensate(s, super, vec2)
c3 = MCompensate(s, super, vec3)
c4 = MCompensate(s, super, vec4)

t1 = Overlay(s, c1, opacity=.5)
t2 = Overlay(s, c2, opacity=.5)
t3 = Overlay(s, c3, opacity=.5)
t4 = Overlay(s, c4, opacity=.5)

f1 = Overlay(t1,t2, opacity=.5)
f2 = Overlay(t3,t4, opacity=.5)
Overlay(f1,f2, opacity=.5)
}

[Efenstor]

You can also try

vec1 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, delta=-2)
vec2 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, delta=-1)
vec3 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, delta=1)
vec4 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, delta=2)

instead of

vec1 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb = true, delta=2)
vec2 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb = true, delta=1)
vec3 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb = false, delta=1)
vec4 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, isb = false, delta=2)

I've discovered it by chance. I don't know for what reason but it gives more details! (though a bit more noise too).

[Efenstor]

Multithreaded version (the second method, with negative deltas):

SetMTMode(5)
AVISource("11.avi")
SetMTMode(2)
ConvertToYUY2()

blkH = 16
blkV = 16
ovl = 4
srch = 3
dct=0
pel=2

bc = BilinearResize(Width()*2, Height()*2)
super = MVSuper(bc, pel=pel)
vec1 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, delta=-2)
vec2 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, delta=-1)
vec3 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, delta=1)
vec4 = MVAnalyse(super, blksize=blkH, blksizeV=blkV, overlap=ovl, search=srch, dct=dct, delta=2)

Lanczos4Resize(Width()*2, Height()*2)
Sharpen(1)
super = MVSuper(pel=pel)

c1 = MVCompensate(super, vec1)
c2 = MVCompensate(super, vec2)
c3 = MVCompensate(super, vec3)
c4 = MVCompensate(super, vec4)

t1 = Overlay(c1, opacity=.5)
t2 = Overlay(c2, opacity=.5)
t3 = Overlay(c3, opacity=.5)
t4 = Overlay(c4, opacity=.5)

f1 = Overlay(t1,t2, opacity=.5)
f2 = Overlay(t3,t4, opacity=.5)
Overlay(f1,f2, opacity=.5)

[Terka]

can you put some pictures?

[Sagekilla]

Sample images:

Downsized from 2160p source:


Downsized to 512x288 and upsized to 1024x576 with MVTools:


Downsized to 512x288 and upsized to 1024x576 with only Spline16:

[Didée]

Erm, that's old hats, no? Put short, what's happening is

sharpen().MCDenoise()

Nothing wrong with that, basically. It's just way too simple. The mere spatial step of [upsize().sharpen()] introduces halos and ringing, which won't be cancelled out by the temporal averaging.

You need to figure which parts of the input signal need emphasizing (increase signal amplitude), and which don't need emphasizing, but only final refinement. *That's* the clue.

Hint: one piece of the puzzle is named "median".


Consider:

"Downsized to 512*288, upsized 300%, sharpened, sharpened, resized to 1024*576"



Compare to your examples.

[Efenstor]

You mean that adjacent frames contain no additional information? Yes the idea is simple but there's more than that you supposed: I take four adjacent frames, motion compensate them, sharpen them and mix them all together with the central one, so that all exaggregated noises disappear and only real details become more visible. Moreover, since I motion compensate an upscaled frame, the edges of objects in the combined frame will be <=4 times more exact to their real-world shape. That's the method of super resolution, I didn't invent one.

I'll post my examples in a while and you'll see it's better. Not too much, but still better than simple sharpen, whichever it is.

[Efenstor]

Here they go. I used the "Multithreaded version" posted above. It gives the best quality of all.

Your method:

My method:


Note that the ear shape is much more accurate, more detail around the mouth are visible, you can see faint rims on the windowpane and black dots in the corners of the windowpane are less smeared. This is not achievable using standard methods.

I don't claim it's "hell that good" but it's a real wonder for lo-fi sources, especially those where shapes are slightly distorted from frame to frame due to heavy compression. Try some actual lo-res examples, not downscaled from HD, and you'll see it's better.

[*.mp4 guy]

.... How can you reproduce Didée's results when he never posted the script.


Dude, all its doing is removing noise its not recovering detail lost during interpolation, its recovering detail lost during compression, if its doing anything at all, which is debateable. I'm not saying temporal super resolution doesn't work on some sources, but mvtools isn't designed to do anything but denoise, it by design will not work on high contrast, low SAD areas of a frame (aliased lines fall well within this generalization).

[Efenstor]

Okay, then this method sucks and I'll not use it to improve quality of my lo-fi videos. Please suggest better.

[*.mp4 guy]

I have no idea what your trying to do, or what your videos look like, how am i supposed to suggest something else?

look, what your doing right now is interpolating them to make them bigger, sharpening to make them look less blurry, and denoisng to make them look less crappy, there isn't anything wrong with that. I'm just trying to tell you that mvtools isn't going to help with the interpolation step.

Their are plenty of high quality sharpening and interpolation methods available on the forums. If you don't want to use search to find them, you could post a representative sample from one of the videos you are working with, post what you want to do with it (destination resolution, denoising preferences, etc.) and ask for advice.

[Efenstor]

I'm trying to upscale by 2, at least in general and want super resolution in particular. As far as I know there's nothing better than EEDI2 but it looks too blurry to me and eats faint details. I tried Video Enhancer and Topaz Enhance that are both said to use super resolution but they are far worse even than my "method" (especially Topaz Enhance). Will there ever be a hero who'll make a plugin for TRUE video super resolution?

[Terka]

put to google IMARE superresolution
if you want, try matlab application (free for non-commercial)
http://zoi.utia.cas.cz/download
email to sroubekf@utia.cas.cz for download, but dont expect miracles.

[Efenstor]

I'll try and write back the results. Still I don't understand why Fizick won't try to implement real super-resolution using his motion tracking algorithms?

[Nightshiver]

Because imo, super-resolution is pointless.

[Dark Shikari]

Originally Posted by Efenstor

I'm trying to upscale by 2, at least in general and want super resolution in particular. As far as I know there's nothing better than EEDI2 but it looks too blurry to me and eats faint details. I tried Video Enhancer and Topaz Enhance that are both said to use super resolution but they are far worse even than my "method" (especially Topaz Enhance). Will there ever be a hero who'll make a plugin for TRUE video super resolution?

Try NNEDI + LimitedSharpenFaster.

[g-force]

Efenstor,

I think you are only going to find a lot of cynicism about "super-resolution", which is a misnomer at best, pie-in-the-sky at worst. The reason really lies in the question "what is super-resolution?" The answer is: "good upsizing with detail bolstering of moving detail from previous and subsequent frames". We already have both of those in the form of NNEDI, and MVTools, and when you put them together, nothing magical happens. It's just good upsizing practices. Nothing more. Definitely nothing to warrant the term "super-resolution".
Now these supposed "super-resolution" packages are just snake-oil. Show the potential customer some still frame, show them what it looks like when it is upsized poorly (point-resize), show them what it looks like upsized correctly, and voila, "super-resolution!"

-G

[mikeytown2]

Here's a good start
http://hdl.handle.net/2014/39112 (http://trs-new.jpl.nasa.gov/dspace/handle/2014/39112)
http://en.wikipedia.org/wiki/Richard..._deconvolution
http://en.wikipedia.org/wiki/Spitzer_Space_Telescope

[Efenstor]

I don't think super resolution is really pointless, I'm using a program called Photo Acute Studio and it really improves resolution. In theory, suppose we have a single point visible in two adjacent frames: in the first frame it's between the pixels and thus it's smeared 50/50 between them, but in the second frame it falls approximately to the pixel center and so it's much sharper. Go further: in the first frame points surrounding that point are sharp but in the second are smeared. Why can we use sharp information from both frames? There are other, more AI-like approaches such as analyzing which pixel of the two smeared closer represents the original point brightness/color or equalizing both smeared pixels depending on their subpixel position so that we can approximately reveal the two original non-interpolated values. We can just think of pixel grid as of another kind of noise, basically it is noise when we look not at single frame but at several where the grid constantly moves over the identical picture details. It's not pointless, but rather interesting I think. And very complex to implement.

[Fizick]

Originally Posted by Efenstor

Still I don't understand why Fizick won't try to implement real super-resolution using his motion tracking algorithms?

me tried

i even created supplement plugin to fill holes, and have some script, but i did not release it, and its development is paused.