Bored again: Looking into Gestures

July 24th, 2013

So I was bored, and this time, after playing TRAUMA (great game you should get it, but be aware that there is no “real” linux client, “just flash”), I felt like looking into gestures & Qt.

The assistant instantly pointed me to QGestureRecognizer and I was like “Yay! This is gonna be so damn easy!”. But I quickly realized thats not the case. You have to overwrite QGestureRecognizer::recognize and then do all the detection by yourself, then I suddenly was like “hm.. not great, but I can do it”.

For the following attempt(s) I am talking about a mouse-turn gesture.
turn-rounded

Try 1: breaking things down

A beautiful turn, doable with your mouse and your finger, but difficult to describe in a plain x/y world, as you don’t know the end(&start) yet. But then it suddenly hit me, I could break it down to very basic directions. (To keep it easy let us use the svg 0,0 point at the left-bottom corner.) The turn-gesture for example can be split into 2 parts. The first part only goes up, x&y only raise, and the second part goes down, y still raises but x goes down.

turn-breakdown

Depending on how well you want to describe it, you also can break it down into several small moves. This “breaking down” is really possible with all gestures I can imagine, even with the “lift” gesture from TRAUMA.

lift-rounded

This is pretty easy to implement, as for the turn, you only need 4 states.

  1. Start: Set when the mouse is pressed, only x >= 0 && y>= 0 moves are allowed, after the first move we set the State to RightUp.
  2. RightUp: x >= 0 && y >= 0 moves are allowed, but after getting the first x>= 0 && y <= 0 move we set the state to RightDown.
  3. RightDown: Only x >= 0 && y <= 0 moves are allowed, but after getting the first move we set the state to Done.
  4. Done: Only x >= 0 && y <= 0 moves are allowed, but whenever the mouse is released we can trigger the event.

NOTE: Qt sometimes sends 0,1 or 1,0 move events when moving right-up, so >= 0 is needed.
The 2 extra states are needed to make sure we actually moved in that direction (at least 1 px), so it gets a little bit more complex, but still very easy to implement. You can improve the checks a lot, but it shows the basic concept.

And the best thing about it is… it works! If you move in a good half-circle, or one right-top&&right-down line it triggers the turn event. This is a really quick to implement and needs very less resources.

But now to the disadvantages… with the mouse I have huge trouble to really just make a right-top&right-down move, it is possible, but you really need to concentrate, that is not something you can do if you just barely can reach the mouse. Especially when you expect to do a half-circle. At least my half-circles look a bit different.
quick-turn
That’s how my half-circles look if I draw them quickly. The first one maybe passes, but the other two fail right at the start, as the move goes in the wrong direction. This method is very strict, too strict for complex gestures.

Try 2: The universal path

As the first attempt was not very good for complex gestures… I had to think of something else. Describing the gestures by move-directions was very easy, but not very good. So this problem must be solved first.

After browsing the assistant for a while I found QPainterPath, does not sound helpful huh? ;-)
But be not fooled! QPaintedPath has some amazing functions, for example arcTo, now you can finally describe a real circle and that with only one line. The only problem is that QPainterPath works with absolute pixel positions, so this would define the height&width of the gesture, not good. QPainterPath does not limit it, but luckily I can make it relative by myself, so describing it is only allowed from 0,0 to 1,1, qreal makes it possible.

    QPainterPath path;
    path.moveTo(0,1);
    path.arcTo(0.0, 0.0, 1.0, 2.0, 180, -180);

You might wonder why I use 2.0 as height-bounding-box, keep in mind that the bounding-box describes a full circle, I only want the upper half and fill the full 0,0 to 1,1 with it.

So far so good, now I can describe it. But it only describes a very thin line, noone is abled to do make a circle like that. That gets us to the matching part. There must be some kind of allowed variance from the line. It is easy to measure the distance from a line, QPainterPath::lineTo, but complicated for arcTo or curveTo. And it should be relative to the size of the gesture overall, this will allow a greater variance for big gestures and a smaller variance for a small one. Lets call this variance the “threshold”, it is definable for every new gesture.
Now it gets a bit performance problematic, we have to record ALL mouse moves (while the button is pressed), this is the only way we can actually get the width&height of the gesture AND scale the QPainterPath to the gesture to compare it. Scaling the QPainterPath itself is a huge PITA and does still not give us information if the points are in range. But lets not forget we can paint with the QPainterPath, using QPainter::scale +QBitmap+QRegion::contains makes this very easy. At this point we also know the size of the gesture and with the threshold value we can set the Pensize.

Let’s see what we have so far…
line-hit
The dynamic threshold and the line, they fit perfect, hurray!
Here have some more quick drawings.
line-hit-2
Also fits, how nice :)

While it sounds like we are done here at this point… we are not. We have a huge problem now, we can no longer tell the direction of the gesture. As we only compare points, so a simple move to the right, is the same for us as a simple move to the left.
The QPainterPath has a direction itself and also from the recorded mouse moves we can tell which was the first and which was the last move, it is not hopeless at all! For a simple line it is enough to check if the first point is close to the first QPainterPath point and likewise for the last. For complex gestures this will not work, as we might be to strict or to relaxed.
For example if you imagine we have two gestures, a line, from the right to the left and a y-mirrored “z”. You might think they are completely different, but if you draw the y-mirrored “z” with a very small height, it almost gets a thick line and has all characteristics of a line from the right to the left. The first QPainterPath point matches the first mouse move point and so does the last.
small-z

But the QPainterPath tells us even more, it tells us the count of the painter-actions aka points, NOTE: arcTo generated more than just 1 point, depending on the size it gets like 4 to 10 (or more). So the first check is to see if I can find a mouse move point to every QPainterPath point depending on the order. But as the mouse movement can be very “unsteady” this is not a easy task, so instantly canceling the detection if we move further away from the point is not an option. If we check every point we loose the direction again.
After some tests the best solution seemed to be splitting the QPainterPath list in 2 parts, for the first half we only check for the closest QPainterPath in max 2/3 of the mouse-move list, without ever going back. The second part checks from behind but only the last 2/3 of the mousemove list. This gives good results in even very narrow situations. I choosed 2/3 over 1/2 because the mouse is moved at different speed in different points in the gesture.
small-z-checkpoints
And to be mega sure, lets pass the found points back “to the user” so he can compare the important logic of it.

bool BCutGestureRecognizer::pointCheck(const QVector< QPoint >& list)
{
    bool ok =  list.size() == 4 &&
             list.at(0).x() > list.at(1).x() &&
             list.at(1).y() < list.at(3).y() &&
             list.at(2).x() > list.at(3).x();
    return ok;
}

But as this point and direction finding turned out to be cause a bit more load then I wanted to… I added minWidth and minHeight to gesture, for example minHeight 20px for the y-mirrored “z”.

So lets sum up, what do I need to get a full new complex gesture working?

      The QPainterPath aka the structure of the gesture.
      The Threshold, very important!
      OPTIONAL: minWidth
      OPTIONAL: minHeight
      OPTIONAL: pointCheck to check the logic of the found points

Sounds doable? :)
Wouldn’t it be nice if something like this (maybe more advanced) would be in Qt itself? Or maybe is there something easier to detect complex gestures?
My Implementation works pretty good, even if you have MANY similar complex gestures, so I am a bit proud, but currently (without cleanup) it super ugly and maybe way to heavy for low-power devices. But anyway in worst case it just filled boredom. (And by the way this is my super first time dealing with gestures, I may have overlooked something.)

Another boredom chapter closed.

kjs now with more emscripten-qt fun!

April 12th, 2013

Just a quick note, I finally got Emscripten-qt working with khtml/kjs!

kjs-emscripten4

transformations

kjs-emscripten5

painterpaths

You can find more demos on
http://vps2.etotheipiplusone.com:30176/redmine/projects/emscripten-qt/wiki/Demos.
Well not all demos currently work with khtml/kjs and the speed is… slow, but some work! ;-)

kjs finally under 1000 fails!

January 30th, 2013

Hello again planet,

I finally got time to konquer the kjs world once more. I was kinda busy in the past month and I still am. Many month have past and kjs was still over 1000(1004) fails in the ECMAScript Testsuite. But finally I broke the 1000 fails border, with implementing Function.prototype.bind, there are now under 1000 fails, even better also under 900! „Only“ 895 fails now.

ecma-under-1000-min

But if you look more closely at those 895 fails… ~700 are from strict testcases, which is like no gain for the user, ~100 are some not deletable properties of the standard builtin Objects, minor gain in some special situations for the user, and ~50 are because of some special call usage with null and undefined, maybe minor gain. So overall kjs is not in a bad shape…. function wise…

But sadly… I couldn’t commit any patches, they are still waiting for review, so don’t be surprised if your konq still shows 4147 fails.
Anyway… hurray! personal goal reached! My patch-bomb makes the fails go down from 4147 to 895 :)

Why kjs can not rule the web

August 31st, 2012

Dear Planet,

I feel like writing a bit about what I did for kjs, only about the new features (ignore bug fixes for now).

Lets start with the things I already got in a released version:
– Array.isArray
– Object.prototypeOf
– Object.keys
– Object.getOwnPropertyNames
– String.trim, trimLeft, trimRight
Not that much hm?

Now lets look at the Stuff I am trying to get in for 4.9.2:
– JSON.parse
– JSON.stringify
still not much…

And the rest with is implemented but still needs cleanup, or review, or something else to get in:
– Object.getOwnPropertyDescriptor
– Object.defineProperty
– Object.defineProperties (blocked by Object.defineProperty)
– Object.isExtensible (mainly blocked by Object.defineProperty)
– Object.preventExtensible (mainly blocked by Object.defineProperty)
– Object.seal (mainly blocked by Object.defineProperty)
– Object.isSealed (mainly blocked by Object.defineProperty)
– Object.freeze (mainly blocked by Object.defineProperty)
– Object.isFrozen (mainly blocked by Object.defineProperty)
– Object.create (blocked by Object.defineProperty)
– Date.toISOString
– Date.toJSON
Ok, thats a bit more.

With the already present functions that would nearly complete ECMAScript Edition 5. Doesn’t sound bad huh?
But still, with all those new functions (and some bugfixes), kjs has no chance to rule the web-world.
Why? Its not because it is to slow, or some other major bugs that keep websites from working. The reason is much simpler…
Let me show you the problem by (I think) simple js code, taken from battle.net (reduced).

var version;
if (browser == 'ff')
version = /firefox\/([-.0-9]+)/.exec(userAgent);
else if (browser == 'ie')
version = /msie ([-.0-9]+)/.exec(userAgent);
else if (browser == 'chrome')
version = /chrome\/([-.0-9]+)/.exec(userAgent);
else if (browser == 'opera')
version = /opera\/([-.0-9]+)/.exec(userAgent);
else if (browser == 'safari')
version = /safari\/([-.0-9]+)/.exec(userAgent);

UserAgent.version = version[1].substring(0, 1);

“version” is still undefined for khtml/kjs. And trying to access undefined[1] will cause an exception, so thats it for kjs, the exception is not caught and all the following javascript code will never be executed.

So the main reason is, kjs/khtml fails at the browser detection…
This could be solved if they add a sane “else”. Talking about sane “else”, there are really many websites that have an else in this case. But many of them have an insane else, which means hitting non-standard-javascript code, code that only one browser ever supported.

Maybe you are thinking “oh come on.. that only happens on some exotic websites”, and I wish that was the case. But its not. I looked at SOME websites why they don’t work with khtml/kjs, from the javascript point of view, and for the last 30 of them this was the case, and thats for some pretty big websites.

NOTE: Not for all websites its so easy to find the core problem, sometimes it looks very different at the beginning.

So thats it, no matter how fast or how bugfree kjs/khtml will become, it will never work this those websites.

P.S.: I know you can change the browser detection in konqueror, but that is not a solutions for the users.

When I get bored…

June 18th, 2012

Its Saturday night, Germany just won again Denmark (soccer), and I am bored, so what can I do?

Lets code something fun, like I used to do back then, just to see if I can do it. And there is something I always wanted to do but never did. A webcam-ascii-dcc-chat.
And here it is:

As I always do the konversation DCC stuff this was no problem, but even the webcam and ascii part, thanks to aalib and HAsciiCam, was really easy. But this will never end up in a real konversation, or at least I won’t do it, because beside from the fun fact its utterly useless. So no gain, only pain maintaining it. If you still want to try it
git://anongit.kde.org/clones/konversation/buschinski/konv-hacks.git

Thanks to my weak git rebase –interactive skills I managed to the change the author of my
“Implement DCC HasciiCam” commit, I didn’t even know that this was possible but if you can tell me how to change it again I would be happy :)

In other more sane konversation news. Konversation(currently only git master) support SASL for quite a while now and we have a very nice userbase page about how to use it.
http://userbase.kde.org/Konversation/Configuring_SASL_authentication. There are more new features, but this might be the most important one for all mobile-freenode users.

kjs, its over 9000!!

March 25th, 2012

Its over 9000!!
Over 9000 tests passes in the ecmascript test :)

Yes, I am still working on kjs, not as fast as I expected to be but… it’s coming along :)

Plus Bonus: We have more than 100 tests that kjs passes and firefox-11 & rekonq(with Qt 4.8.1 webkit) fail. Most of them are related to Object.defineProperty(array, “length”, ..) which firefox simply does not really support. I am not really sure if it is used in real world anyway.
NOTE: this is not array.length = foo;

But yes, as you can see kjs still has over 2500 fails and we are not faster. So kjs is not yet teh javscript thing, but please grant me this tiny moment of joy :)

Yay for kjs/khtml

February 28th, 2012

Just a quick note: kjs/khtml can run the ecmascript tests :D
previously it would just freeze in test S.15.4.4.12_A3_T1 (Bug 276629)
Fix is currently in review, so not yet commited and may need more cleanup.

Here the screenshot you are all waiting for

The results are currently “not so good”(compared to other browser), but I hope I can improve them :)
And now time for a *happy dance*

My quick wish for 2012

January 11th, 2012

My wish for 2012 is that every developer of <insert your project here>, assuming you use your own project, should use its own project with debugging symbols and enabled asserts.
If you feel sporty enough you can even enable asserts in your underlying lib.

That’s all, make real use of asserts, detect bugs early and we can all live in a happy world!

HINT: debugging symbols need not always mean that asserts are enabled, and please note that Q_ASSERT behaves different from assert.
Even if you are sure that you already do that, please do me a favor and double check.

I am writing this because some projects seem to ignore asserts completely. And I am not talking about super special situations… more like start with nearly default config -> hit assert + crash.

My personal top programs of 2011

January 9th, 2012

In case you didn’t notice it’s already 2012!
Time to honor my personal top programs of 2011. Yeah I am a bit late, this is usually done before the end of the year.. but I had to think about it very carefully! *caught*

Top Dominator – KDevelop
KDevelop really is teh IDE. And now even for python!

Top Coloriser – Krita
Krita, what a wonderful painting and illustration suite. I am not an artist but this program really feels like this is how a painting program should be. I even brought a graphics tablet for it. And even if you don’t know what to draw, or can’t draw at all, playing around with the different brushes is fun enough :)

Top Underdog – Calligra Words
Calligra Words, it was so much fun watching how much it has improved since koffice. It’s not perfect yet, but it already displays *.docx way better than LibreOffice (compared to LibreOffice 3.4.3, “for my needs, may be different for you”). The editing part currently doesn’t seem as good, but I am sure it will improve 2012 a lot!

Top Playground – Blender
Yeah not a KDE program, but really worth mentioning. Blender, also improved a lot since 2.4, I would say since 2.5 its finally “useable” for me. I only use it for playing around, first for physic simulations and since 2.61 it offers more nice features like cycles. Again I am no artist, but I am sure a real artist could do amazing things with blender. Here is a little evening project (with the help of a nice tutorial) which was fairly easy to do in blender 2.61 (blender internal render, no cycles).

If you don’t like my personal top XYZ of the year, or have some more, feel free to leave a comment :)
HINT: they are moderated

Konversation 1.4 Beta 1 released

November 4th, 2011

Yeah! It’s been quite a while but finally Konversation 1.4 Beta1 is here! The first pre-release of our next major release, with many changes and fixes.

A brief selection of highlights:

  • URL and email detection in text views has been rewritten from scratch, greatly improving the handling of various types of URLs and the contexts they might appear in.
  • Extensive improvements to IRC formatting code handling, including the return of background color support.
  • Extensive, sometimes full rewrites of user interface elements such as nearly all context menus, the URL Catcher and the Warning Dialogs system for a long list of user interface improvements and bug fixes.
  • Improved SSL connection behavior.
  • Translation support and various other improvements in several bundled scripts.
  • Expanded Python scripting support via the introduction of an API support package.
  • Support for more IRC numerics.
  • Various bugfixes to input line command handling and connection behavior.

A full list of all new features and changes is available on http://konversation.kde.org/.
I really really love those long and detailed changelogs. :)

In case you are still using Konversation 1.3.1 and experience strange crashes since Qt 4.7.4 update, I highly recommend you to use this 1.4 Beta 1. Or if you really want to stay with 1.3.1 you could fetch the latest commit from the 1.3.1 branch.
Please also note that we have at least one minor style fix for Qt-4.7, so again update is recommend.