Thread: help de-noise algorithm (fixed form threshold)

hi, im working on signals, here is an introduction to the context where i am.

im my work there is a 1-dimension camera that takes pictues in black and white.

so when i take a picture , i graphicate it and i receive a signal very similar to the waveform of an audio track.
but in my graphic, X AXIS is the vertical value in pixels and Y AXIS is Gray level detected. so you have all the colors taken on a vertical line.

we are using this to count an ammount of sheets stacked, and what i receive is a tooth style signal, but with a lot of noise because of the material of the sheet.

i`ve been searching a lot of algorithms to take out the noise of my signal, so then i can apply a "peaks-valley detection--counting algorithm" (thats another chapter upcoming soon)

what ive had tried is MCMASTER'S algorithm that makes a singal smoother, but is not useful for me, just doesnt work.

then i looked in a lot of papers about the fixed form threshold (being the best one among others) and many other algorthms that take out noise and you actually dont loose important points.


after hours of searching the code...but instead finding nice screenshots and descriptions of it like if the reader knew the algorithm....i ended with no succes...so i come here asking if someone can explain me (or give a link) the method in a way so i can program it.

thanks for reading, i hope someone can help me.

bye
cristobal

More particulars would be very helpful. You may want to post a pic of your signal which shows the noise problem at photobucket, and link it in your post in img tags so we can see what you're talking about, in more detail.

What happened when you tried McMaster's algorithm? Details are needed here. Perhaps your implementation of McMaster's algorithm just needs some tweaking to work correctly.
Can you post your coded efforts and show the results from that? Again, a picture would be helpful. It's hard to describe a signal pattern in just words.

If you can describe the steps needed to clean up the signal wave, in simple enough terms to allow a computer to take those steps, then it should be quite possible to code it up.

you are right, i just took some screenshots from my application.

Here is the 1-Dimension Photo (black and white), and below each graphic you see how it would look in 2D.

1.0) IMAGE NOT PROCESED
http://www.upload-images.net/imagen/545bb24098.jpg

1.1) IMAGE NOT PROCESED ( ZOOM IN )
http://www.upload-images.net/imagen/ebcbc52080.jpg

2.0) IMAGE PROCESED (MCMASTERS... 24 ITERATIONS)
http://www.upload-images.net/imagen/3ce5c67d13.jpg

2.1) IMAGE PROCESED (MCMASTERS... 24 ITERATIONS) ZOOM IN
http://www.upload-images.net/imagen/ef89f337e5.jpg


in this example taken randomly just a minute ago, mcmaster's smoothen method does it very well. but sometimes i loose some sheets for the counting, because mcmasters takes the average point from a sequence, so sometimes that means to move the point up.

i`ve also tried some of my home-made algorithms but mcmasters is the best i have now. i dont know whats the logic for the de-noise algorithms remaining but i would like to test them ...having their logic is all we need.

any of these are useful

-fixed form threshold
-heuristic SURE
-SURE

This part is unclear:

in this example taken randomly just a minute ago, mcmaster's smoothen method does it very well. but sometimes i loose some sheets for the counting, because mcmasters takes the average point from a sequence, so sometimes that means to move the point up.

Is "the point" represented by the single red dot in the graph of each wave?

How much can "the point" be raised before your counter begins mis-counting?

Does McMasters program have a way to use a lower average, or "squash" the line downward to smooth it, instead of choosing an average value.?

If the line could be lowered in that way, would it still be usable, or would the loss of the highest point value make it useless?

the red dots are part of the counting method(we have an algorithm there running for testing), you can ignore those red points now.

mcmasters algorithm logic is about picking five points one after another, calculate the average point from it, then move it half the distance to the third one. one iteration consist on doing this starting with the 3º point from the curve as center of the five (so point 1 and 2 to the left and point 4 and 5 to the right) and ending at N-2 as center of the five(same ... ).

link, with a very good explanation.
http://www.geogra.uah.es/gisweb/1mod...ualisation.htm

the valley of each wave shouldnt be raised, but a little is allowed (maybe 5% of the distance peak to valley), its a relative value for each input of data.

lossing the highest value is not so much of a problem, the important is the valley so that the tooth concept still can be applicable. its a good idea i should try add that part of code and see how it changes.

thanks for trying to help, and for your time. ill keep searching meanwhile other posibilities.

bye

cristobal

Look in any basic DSP book. You are trying to pick out a repetitive underlying signal. Take a single cycle of the ideal wave, reverse it, and convolve it with the signal. This produces a correlation signal. Count the peaks of this new signal.

i`ve been searching and found that there are 2 types of threshold. here is what i learned

-Hard threshold (known by most people)
-Soft Threshold (makes the new curve smoother)

both of them have an IF ELSe condition very trivial.1

but the important part is when selecting the threshold value,
there are a lot of ways to calculate it and fixed form threshold is one of them:

FIXED FORM: sqrt(2*log(length(y)))

source: http://www.mathworks.com/products/wa...alsdemo.html#5


the only problem is, if length(Y) is the length of the function... where is the relation with the y values which im going to compare later to apply the threshold. dont know if you guys see the ambiguity of that? i could have f(x) values going from 1 to 10 with a curve length of 50, or i could i have f(x) values going from 1000000 to 2000000 and curve length 50.... i would get the same threshold and thats not correct.

thats my only question now...
(sorry for english mistakes)

Originally Posted by neoideo

the only problem is, if length(Y) is the length of the function... where is the relation with the y values which im going to compare later to apply the threshold. dont know if you guys see the ambiguity of that? i could have f(x) values going from 1 to 10 with a curve length of 50, or i could i have f(x) values going from 1000000 to 2000000 and curve length 50.... i would get the same threshold and thats not correct.

The reason the method is confusing is because it sucks. In real world applications problems of this kind are solved by correlation, which is probably what you should do as well.

im also checking the correlation method which seems professional,
its god to have more than one option ill try to test them all and keep the best one for my situation.

thanks brewbuck

Originally Posted by neoideo

that seems a very interesting idea

im going to code in those conditions
lets see how it works!

pd: about the median value. you meant calculate it from the 5 input points in the algorithm? or from whole the signal?

thanks in advance

cristobal

From the 5 input points that the McMasters algorithm uses.