Page MenuHome

Input Mouse or Tablet events are too sparse when input device moves very fast in Windows
Open, NormalPublic

Description

Basically, the problem we have detected is the mouse/pen events are missing when you move the mouse very fast. We have checked all the events in wm_event_add_mousemove() function and if you move very fast, you get only a fraction of the mouse positions. We have tested in Linux and Windows and the problem is mainly in Windows.

This is not a problem specific to 2.8x, it’s a general problem in all versions.

Example image:

In the image above, I tried to draw a curve very fast with the mouse (artists draw these curves very fast to get gesture drawing)… as you can see, the distance increases when the mouse moved fast (in the curve section). Of course, we tested MOUSEMOVE and INBETWEEN_MOUSEMOVE events, but the problem is we never received an INBETWEEN_MOUSEMOVE event. The tested operator is using OPTYPE_BLOCKING flag.

For other tools to have all events is a nightmare, but for drawing in grease pencil this is critical. We have added a lot of tricks to smooth and try to simulate the missing events and we get a reasonable result, but the “real” solution is to get the events because any method used always “change” slightly the artist gesture.

@Daniel Martinez Lara (pepeland) was testing and Linux hasn’t this problem or is less noticeable, and the “feeling” is better in that OS.

Details

Type
To Do

Event Timeline

Copied from @Brecht Van Lommel (brecht) mail:

You'll need to dive into GHOST, and investigate what is happening there. Either the mouse events coming from the Windows API are not enough, or something goes wrong passing events from GHOST to Blender. If it's the former, search the Windows API documentation for information.
There's also old commits like this:
https://developer.blender.org/rB7aa8ae37819b4a95910089c27fa1e6d704a0f907
The situation might also be different for tablet and mouse.

Yes, the diference between windows and linux is sooo huge in Blender in fast drawing movements.
In windows is a real issue have this poor perfomace in Blender for fast drawing movements.

Test file, just open it and draw a curve very fast:

You'll need to dive into GHOST

I neved looked at that code before, so I will try, but not sure I will be able to fix/found the problem.

On macOS via Sidecar, developers can get access to the raw Apple Pencil data, which is recorded at 256hz (!), That should really help produce very smooth lines.

I have been debug with Ghost and the events received in the operator.

==== Init =======
GHOST:Time:5250 (729,241)       MOUSEMOVE (729, 580)
GHOST:Time:5264 (720,241)       MOUSEMOVE (720, 580)
GHOST:Time:5272 (708,241)       MOUSEMOVE (708, 580)
GHOST:Time:5278 (687,242)       MOUSEMOVE (687, 579)
GHOST:Time:5288 (657,244)       MOUSEMOVE (657, 577)
GHOST:Time:5305 (572,245)       OTHER (572, 576)
GHOST:Time:5306 (516,252)       MOUSEMOVE (516, 569)
GHOST:Time:5321 (462,266)       MOUSEMOVE (462, 555)
GHOST:Time:5338 (370,290)       MOUSEMOVE (370, 531)
GHOST:Time:5355 (314,339)       MOUSEMOVE (314, 482)
GHOST:Time:5373 (301,380)       MOUSEMOVE (301, 441)
GHOST:Time:5388 (306,417)       MOUSEMOVE (306, 404)
GHOST:Time:5405 (330,446)       MOUSEMOVE (330, 375)
GHOST:Time:5421 (498,515)       MOUSEMOVE (498, 306)
GHOST:Time:5438 (670,529)       MOUSEMOVE (670, 292)
GHOST:Time:5455 (803,510)       MOUSEMOVE (803, 311)
GHOST:Time:5471 (830,499)       MOUSEMOVE (830, 322)
GHOST:Time:5488 (829,487)       MOUSEMOVE (829, 334)
======== End =======

All the events of Ghost (first column) are passed to the operator as events (2nd column). There are 18 events in Ghost and the same in the operator, and this is the resulted stroke with 18 points.

You can see the big gap between point 13th and 14th (330,446) and (498, 515).

Note: The diference in coords values is due the conversion done by event system to produce window coordinates.

It looks the problem is in Ghost because it's not receiving all events.

This seems fun, mind if i take it?

@LazyDodo (LazyDodo) I tested with these lines:

diff --git a/intern/ghost/intern/GHOST_EventCursor.h b/intern/ghost/intern/GHOST_EventCursor.h
index 7947b134f4b..ceff2249683 100644
--- a/intern/ghost/intern/GHOST_EventCursor.h
+++ b/intern/ghost/intern/GHOST_EventCursor.h
@@ -46,6 +46,7 @@ class GHOST_EventCursor : public GHOST_Event {
                     GHOST_TInt32 y)
       : GHOST_Event(msec, type, window)
   {
+    printf("GHOST:Time:%d (%d,%d)", msec, x, y);
     m_cursorEventData.x = x;
     m_cursorEventData.y = y;
     m_data = &m_cursorEventData;
diff --git a/source/blender/editors/gpencil/gpencil_paint.c b/source/blender/editors/gpencil/gpencil_paint.c
index f29e782c618..3649f00ae2b 100644
--- a/source/blender/editors/gpencil/gpencil_paint.c
+++ b/source/blender/editors/gpencil/gpencil_paint.c
@@ -2277,6 +2277,8 @@ static void gp_session_cleanup(tGPsdata *p)
 
 static void gp_session_free(tGPsdata *p)
 {
+  printf("======== End =======\n");
+
   if (p->rng != NULL) {
     BLI_rng_free(p->rng);
   }
@@ -3378,6 +3380,7 @@ static void gpencil_guide_event_handling(bContext *C,
 /* start of interactive drawing part of operator */
 static int gpencil_draw_invoke(bContext *C, wmOperator *op, const wmEvent *event)
 {
+  printf("==== Init =======\n");
   tGPsdata *p = NULL;
   Object *ob = CTX_data_active_object(C);
   bGPdata *gpd = (bGPdata *)ob->data;
@@ -3662,6 +3665,8 @@ static int gpencil_draw_modal(bContext *C, wmOperator *op, const wmEvent *event)
   GP_Sculpt_Guide *guide = &p->scene->toolsettings->gp_sculpt.guide;
   tGPspoint *points = (tGPspoint *)p->gpd->runtime.sbuffer;
 
+  printf(
+      "\t%s (%d, %d)\n", event->type == 4 ? "MOUSEMOVE" : "OTHER", event->mval[0], event->mval[1]);
   /* default exit state - pass through to support MMB view nav, etc. */
   int estate = OPERATOR_PASS_THROUGH;
Antonio Vazquez (antoniov) renamed this task from Mouse events are too sparse when mouse moves very fast in Windows to Inpunt Mouse or Tablet events are too sparse when input device moves very fast in Windows.Oct 13 2019, 11:44 AM
Antonio Vazquez (antoniov) renamed this task from Inpunt Mouse or Tablet events are too sparse when input device moves very fast in Windows to Input Mouse or Tablet events are too sparse when input device moves very fast in Windows.

We have been testing several options and even MS-Paint has the same problem. I'm thinking that maybe the solution is not at event side, but to generate a Bezier in the segments to get a smooth transition. Using Bezier, we keep event system, don't overpopulate events and get the same result in all OS's. I need to think, but I have a rough idea already.

Do gimp or krita have the same issues? I’m wondering if they’ve solved the problem.

Clip Studio Paint for comparison on windows.

It looks Clip Studio is doing the conversion to Bezier as Disney Meander does: https://www.disneyanimation.com/technology/innovations/meander

I will try to convert on the fly the stroke to Bezier and I hope to get very smooth strokes without ugly steps.

@LazyDodo (LazyDodo) I take the task to work in the Bezier idea inside grease pencil drawing operator, but you can investigate in the events system to check if we can improve in that area too.

Adding the calculation of points of the missing arc, I get this (wip).

Most 2D painting programs use pen tablet events from the pen tablet driver (e.g. wintab) instead of OS mouse cursor events for pointer position. This also allows subpixel precision and better accuracy. Perhaps this is where the problem lies?

I am sorry to insist so much, but the brake on the professional use of grease pencil in 2D animation is a design concept: the use of poly lines instead of splines. IMHO I think it would take a breakcoffe to redesign the GP evolution strategy.

Thanks for your great work.

Grease Pencil's sculpt tools are superior to other products.
I suppose the use of poly lines gives the feature of sculpting (and also animation) an advantage.

I think @s12a (s12a) is right. I don't think any other drawing application is using Beziers to generate samples from mouse events. Last year I had a problem with Photoshop, it was producing lines with artifacts the same way Grease Pencil does. It has a configuration file where you can change the pen input device to the tablet driver, and that solved the problem. That said, we should probably keep the bezier interpolation in case that a tablet device is not available.

This is what I'm doing. The arc ppoints are only created if the device is not able to provide it. In some OS's and/or deviced we get the "real" events data and don't need to create these missing points.

And It is not entirely correct that no application uses conversion to curves. Disney Meander, used to create Paperman shortfilm, used these techniques to get a very smooth strokes converting the mouse/pen movement to curves. We tested to use raw data from the device and the UI gets unstable because we are not reading this data directly but using all Blender operator system, so add raw data to GHOST is not only for this, but for all UI also and this is not good idea.

s12a (s12a) added a comment.EditedOct 15 2019, 3:57 PM

@Antonio Vazquez (antoniov)
For what it's worth, I have a Wacom Intuos5 L with the latest drivers on Windows 10, and on the latest Blender build, as well as all other previous builds, It doesn't appear that subpixel precision from the tablet driver is used. This is most apparent when removing stroke post-processing and smoothing options and attempting to write cursive text in very small size.

With a 2D painting program like Krita if I zoom back, write small text (also without stroke smoothing options enabled), then zoom in again, the text is readable. If I repeat the same test with Grease pencil on Blender, it looks as if it got constrained to the visible screen pixel grid. From this I assumed that raw mouse cursor events from the OS are used instead of coordinates and events from the tablet driver. Explicitly selecting WinTab/Windows Ink in program option makes no difference.

@s12a (s12a) you miss one point... all what you do in 2D is converted to 3D, so the limitation to pixels is not only due the device. I don't say it's not possible to do, but if we have to rewrite a lot of code, then it's out of scope now, and make comparison between Krita and Blender is not totally correct because their approach are totally different.. Grease Pencil is for animation, Krita is for drawing and painting pixels. I don't think we can get in Blender a tool for painting as Krita never.

@Antonio Vazquez (antoniov) But why are we not using screen subpixel precision from the tabled driver? It is going to be more noticeable in Grease Pencil, where the final data is not constrained by a pixel grid. Also, 2D paint now supports floating point precision in the brush position, so it will also benefit from that. I don't think we need to change anything in Blender tools to support that.

The mouse / pen event management is the code of the old 2.7x and I suppose that to use the sub-pixel, you must use float instead of ints, so this would require refactoring all the code using sbuffer strokes and any function related to that, To use this data we will need time to change that I don't have (or don't want to use in this) ... I think there are other areas more important than this.

And the question here is whether we will obtain a notable difference that is worthwhile for all these changes because the problem was not the accuracy, but the missing events when you move very fast.

Those use bezier strokes. mapped with bitmaps and vector textures:
Disney Meander, Moho ( Anime Studio), Toon Boom Harmony, CelAction

sorry to intrude.But I downloaded the blender 2.82 release from yesterday and I see the same issue.Here is some example:

Back when I started using grease pencil, I had an issue with attaching lazy nezumi to it. After some back and forth, the LN developer found some issue in his code. But he did mention that he wanted to write a guide for software developers on how to properly use the tablet drivers(wintab/windows ink). Maybe you guys can contact him if not already and collaborate on bringing direct tablet input.

We have doing all changes in a separate branch, not in master.