Page MenuHome

conflict with how Blender is sampling OSX input/stroke events
Open, Confirmed, MediumPublic

Description

System Information
Operating system: Mac OSX 10.14.3 (18D109)
Graphics card: Radeon Pro 570 4096 MB

Blender Version
Broken: 2.80beta
Worked: -

Short description of error
Blender registers wrong pressure values on stroke realease when input is given by OSX stroke events from iPad Pro (Astropad) or Force Touch Trackpad (inklet)

Exact steps for others to reproduce the error
We were recently testing a setup that replaces the Cintiq with iPad Pros. As you may know new macBook trackpads are pressure sensitive. This means that you can enable pressure recording in your apps using apple' integrated input/stroke events. This also means that there are applications out there that are enabling this feature for any osx software without having to code it.
One of the easiest app to use is called inklet and lets you use the trackpad as a graphic tablet .
The same feature is brought to iPad Pros with Astropad turning your iPad in a Cintiq for your Mac.
Unfortunately both apps (working fine in other software like photoshop or krita) are experiencing the same issue with blender and are easy to reproduce trying to draw with Grease Pencil: whatever you do the final point of the strokes records 100% pressure on release, drawing a bubble at the end.
I've been in touch with Astropad team and the say:
"With the latest, we were able to get pressure sensitivity and reproduce the high pressure end of stroke. But, we were able to get a similar result with just using a trackpad as well,
which makes us believe there is a possible conflict with how Blender is sampling input/stroke events in general"
is there something you can do to help us Mac people to have a working workflow with our hardware?

Details

Type
Bug

Event Timeline

William Reynish (billreynish) lowered the priority of this task from Needs Triage by Developer to Confirmed, Medium.EditedMar 14 2019, 11:26 AM

I can confirm the issue here, using an iPad Pro as input:

Blender w/ Grease Pencil:

Blender w/ Texture Paint:

In Blender, each line ends with full pressure.

Photoshop:

Photoshop and other apps act normal

Tested with Duet Display instead of Astropad. Same issue. So it seems not to be an issue with Astropad itself, but probably an issue in Blender itself.

2.79 also has the same behavior.

I'm new to blender from 2D land and have the same issue. I think I've added myself for updates?

Not sure why you assigned this, @Antonio Vazquez (antoniov) doesn't even have a mac for testing as far as I know.

I don't know of any developer with this kind of hardware, so no one currently.

If it becomes high enough priority we can try to get a setup like this, or deeply analyze the code to try to find the bug, but we have more important bugs to fix right now.

@Brecht Van Lommel (brecht) actually any macbook pro from 2016 to now has a forcetouch trackpad you can use to test it with inklet. This shares the same osx input and stroke event as astropad on ipad.
It’s very hard to believe that there isn’t any developer out there using a recent macbook pro.

Not sure why you assigned this, @Antonio Vazquez (antoniov) doesn't even have a mac for testing as far as I know.

Ah you're correct. I thought he fixed bugs with Apple before. but I must have mixed him up with someone else. Looking at an earlier post of his it seems he only has access to a Windows dev environment:

I cannot help here... I cannot test in Mac, only Windows.
If someone has a Mac, I can help him to debug/test but I cannot do it myself.

I can reproduce the issue with Inklet software on a trackpad.

Doesn't happen with Wacom Intuos btw, so it's something specific to that software and hardware.

@Sergey Sharybin (sergey) it works with almost any pressure enabled software out of the box through inklet and Astropad (you can easily test krita). Believe anyway it should work just like pinching, zooming and rotating with two fingers on the trackpad are working using the regular APIs.

please have a look here:

ForceTouchCatalog: Using the Force Touch Trackpad API

there are also js using it for web development here:

JavaScript library for handling Force Touch

@Brecht Van Lommel (brecht) searching on the bug tracker it seems this issue is also happening with other setups for non-wacom users.

Blender 2.78 Grease Pencil - pen pressure problem (Yiynova Tablet)

JoshBowman (JoshyB) added a comment.EditedMay 2 2019, 10:56 AM

The video at the end of that discussion shows where the issue lies and a possible fix
https://youtu.be/xkWECtX-ORs

Apparently it’s to do with a variable in gpencil_paint.c ‘pressure = 1.0f’ rather than getting changed to a default of ‘0.0f’ which solves the tablet issue but then drawing strokes with the mouse won’t work because the default is now 0.0

@JoshBowman (JoshyB), yeah thanks. That's why I was linking the topic. Hope some developer cares enough to have a look at it.

Feel your pain, I’ve been wanting this fixed for aaaaaaaaaaages.

@Sergey Sharybin (sergey) Wondering if this issue could just become a setting somewhere that allows us to switch between a mouse/tablet drawing mode which just changes the pressure variable between 1.0f and 0.0f? That would seem the most straight forward solution

@JoshBowman (JoshyB) @Sergey Sharybin (sergey)
can't this be a possible solution?

@Joshua Leung (aligorith) so I started blender with the --log-events flag and with that I could confirm your theory about the missing pressure information on the release event. It´s really weird that it just happens so randomly. I made another short video demonstrating whats happening with the console open: https://youtu.be/LRnzfMrVwmU
If possible I´d try to set the pressure value to 0f on every release-event. At least in my head it sounds like a plausible solution :)
Thanks a lot for taking your time everybody!

is there a reason why is not possible to set 0f on every release event?

@JoshBowman (JoshyB) Thanks for the tip, compiled blender with that option and works like a charm with both Astropad on iPad Pro/iPhone and Inklet with Trackpad.
At least works for grease pencil. For other modes I think some more files should be edited.

Hello people! I'm super-happy that my hack could help others to make their pens work correctly with blender! I don't know how to implement a clean solution which would let the mouse keep working. I still think that setting the pressure level to 0 at every release might work, but unfortunately I couldn't find out how to do that. I'd be even more happy if the real devs could find out a solution that helps everybody with not-wacom tablets to be using the GP, as a lot of other blender users might be working off finacial constraints. Let's hope everybody will be happily drawing very soon. Best regards everybody!

JoshBowman (JoshyB) added a comment.EditedJun 9 2019, 3:37 AM

@Konstantin Bukow (Kobuk) I've just followed the docs to compile my own version of 2.8 with the change you suggested and it works perfectly THANK YOU I'm super excited because it means i can continue development of my Grease Pencil based character rig!

BTW it's even more important to get this fixed with Apple announcing Sidecar to allow tablet drawing for Macbooks using the iPad, if Blender only recognises certain kinds of hardware tablets then this will turn into less of a niche issue.

@Antonio Vazquez (antoniov) Can you look at this issue? It seems like there is a solution for pen input - probably we just need to detect if you are using mouse or pen, which we already do anyway.

@William Reynish (billreynish) I would do if I could reproduce it. I have seen the problem is the line if (event->tablet_data) { (gpencil_paint.c) is not getting data when you use that type of tablets, but if we move the value of 0.0 in the else, then the mouse will not work.

Could you try to debug in you system what is the value of the event when use the tablets? maybe there are some way to detect that you have a tablet but it's not sending data... something like add in the else a code to verify if(mouse) p=1 else p=0

Probably painting operators should never use the release event mouse coordinates or pressure to draw anything? In principle there should be a mouse move event preceding it, and then the release event should merely stop the painting, not define the last part of the stroke.

It may be that GHOST does not work like this on all platforms, it would need to be tested.

Setting pressure to 0.0 for release events probably achieves the same effect in practice.

@Brecht Van Lommel (brecht) The events are filtered with the code below:

if (ELEM(event->type, MOUSEMOVE, INBETWEEN_MOUSEMOVE) || (p->flags & GP_PAINTFLAG_FIRSTRUN)) {
..
..
      gpencil_draw_apply_event(C, op

As it's the last part of the stroke, the First run is not used here, so we have an event of type MOUSEMOVE here.

EDIT: The release or press (any status change) of the mouse buttons is considered as end of stroke, but the wrong pressure in last point is generated in the code above, not in the finalization.

As I understand it, @JoshBowman (JoshyB) says this change in gpencil_draw_apply_event worked:

     /* No tablet data -> No pressure info is available */
-    p->pressure = 1.0f;
+    p->pressure = 0.0f;

So that would mean either some mouse move events are missing tablet that, and the release event is not the issue. Or the release event pressure is somehow used elsewhere and affecting the stroke. If someone can run blender with --debug-events from the command line and paints a single problematic stroke, then we can see in the output which it is.

@Brecht Van Lommel (brecht),
i have drawn 3 strokes. All affected by the pressure issue.
here is the console output for the events:

wm_event_do_handlers: Handling event
wmEvent  type:1 / LEFTMOUSE, val:1 / PRESS,
         shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
         mouse:(748,1034), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7fbfadd41ad8
 tablet: active: 1, pressure 0.3529, tilt: (0.5946 0.0222)

wm_event_do_handlers: Handling event
wmEvent  type:1 / LEFTMOUSE, val:2 / RELEASE,
         shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
         mouse:(626,1134), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7fbfad8cdb88

wm_event_do_handlers: Handling event
wmEvent  type:1 / LEFTMOUSE, val:1 / PRESS,
         shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
         mouse:(578,756), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7fbfadce7528
 tablet: active: 1, pressure 0.1961, tilt: (0.5800 -0.1339)

wm_event_do_handlers: Handling event
wmEvent  type:1 / LEFTMOUSE, val:2 / RELEASE,
         shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
         mouse:(786,514), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7fbfac9774d8

wm_event_do_handlers: Handling event
wmEvent  type:272 / TIMER, val:0 / NOTHING,
         shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
         mouse:(1752,326), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7fbfae830428

wm_event_do_handlers: Handling event
wmEvent  type:260 / WINDOW_DEACTIVATE, val:2 / RELEASE,
         shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
         mouse:(1636,910), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7fbfac9674e8

let me know if this helps

and these are the registered events for a single stroke in console

wm_event_do_handlers: Handling event
wmEvent  type:1 / LEFTMOUSE, val:1 / PRESS,
         shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
         mouse:(450,936), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7fbfadc86988
 tablet: active: 1, pressure 0.1373, tilt: (0.5712 0.3336)

wm_event_do_handlers: Handling event
wmEvent  type:1 / LEFTMOUSE, val:2 / RELEASE,
         shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
         mouse:(552,1326), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7fbfadd3af28

As I understand it, @JoshBowman (JoshyB) says this change in gpencil_draw_apply_event worked:

     /* No tablet data -> No pressure info is available */
-    p->pressure = 1.0f;
+    p->pressure = 0.0f;

So that would mean either some mouse move events are missing tablet that, and the release event is not the issue. Or the release event pressure is somehow used elsewhere and affecting the stroke. If someone can run blender with --debug-events from the command line and paints a single problematic stroke, then we can see in the output which it is.

one thing to be considered is this "fix" works only in grease pencil, all blender areas that can receive tablet pressure inputs (Texture paint, Weight paint, Vertex Paint, etc...) are affected though.

@Brecht Van Lommel (brecht) did the event debug data give any clue on how to handle this?

The same issue happens in the release version of Catalina with Sidecar and an iPad Pro/Apple Pencil.

I've got a partial answer for what's going on -- will try to pick this up again tomorrow night, but thought I'd pass on some notes.

I instrumented the macOS side of the code to verify when the various tablet routines were being called during a basic test using Sidecar and an iPad on Catalina. The short version is that the mouse up event is getting clobbered by a tablet proximity event.

In my sidecar test case, when the mouse up event arrives, it is marked with a subtype of NSEventTypeTabletPoint, and the code calls GHOST_SystemCocoa::handleTabletEvent(void *eventPtr, short eventType):

NSEventTypeTabletPoint found
Handle Tablet Event Enter
Pressure and Fill Set
Handled a Mouse Up Tablet Event

However, before that event is processed by Blender's internal event system, the code also processes a tablet proximity event, so the log continues with:

received a tablet proximity event in WindowViewCocoa
handle tablet event enter
NSEventTypeTabletProximity
pointer is leaving

And finally, the event is processed internally:

wm_event_do_handlers: Handling event
wmEvent type:1 / LEFTMOUSE, val:2 / RELEASE,

shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
mouse:(1158,712), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7faca5f52d18

If you look at the code near the "pointer is leaving" printf in the Cocoa code, handleTabletEvent sets the active mode to none.
ct.Active = GHOST_kTabletModeNone;

I think the second event is clobbering the mouse up event and clearing the tablet data. I don't know the rest of the codebase at all, but it looks like other code in Blender tests against GHOST_kTabletModeNone to determine whether any tablet data should be kept around. This code appears to propagate that into a different tablet format:
static void update_tablet_data(wmWindow *win, wmEvent *event)
{

const GHOST_TabletData *td = GHOST_GetTabletData(win->ghostwin);

/* if there's tablet data from an active tablet device then add it */
if ((td != NULL) && td->Active != GHOST_kTabletModeNone) {
  struct wmTabletData *wmtab = MEM_mallocN(sizeof(wmTabletData), "customdata tablet");

  wmtab->Active = (int)td->Active;
  wmtab->Pressure = wm_pressure_curve(td->Pressure);
  wmtab->Xtilt = td->Xtilt;
  wmtab->Ytilt = td->Ytilt;

  event->tablet_data = wmtab;
  // printf("%s: using tablet %.5f\n", __func__, wmtab->Pressure);
}
else {
  event->tablet_data = NULL;
  // printf("%s: not using tablet\n", __func__);
}

}

And that data structure is the one being passed into the earlier checks on this thread that found "no tablet data", setting the strength to 1.0.

I can confirm that if I comment out tabletPoint and tabletProximity in WindowCocoaView, the problem goes away. The mouse up messages have tablet information so pressure get released as appropriate. This was just a test; those events are important for tablets with proximity sensors. Also, I found that later events (like window deactivation) were showing tablet events, which would clearly not be the right thing.

It seems like the right behavior is for there to be two separate events. First the mouse up with tablet information, and then the tablet proximity event to clear all tablet information. This is where my knowledge of Blender's architecture mostly ends. The second event would mostly be synthesized just to clear the tablet event data, but I couldn't find a similar good example in the source. Perhaps a mouse moved message could be synthesized without actually moving the mouse position? Or would an "unknown event" be acceptable?

It would be a more radical change to this code, but setting up the tablet parameters before pushing events on the queue (rather than after) feels like a more natural model. It means that the tabletProximity event on the macOS side can clear the tablet information and then it just gets picked up by the next event.

Anyway, those are some thoughts for future discussion.

This may or may not be an acceptable fix, but adding dispatchEvents() when leaving proximity in handleTabletEvent() flushes out the event queue and fixes the issue for me in both grease pencil and texture paint modes:

         // pointer is leaving - return to mouse
+        dispatchEvents();
         ct.Active = GHOST_kTabletModeNone;

Good catch. I was looking for something like dispatchEvents() yesterday but couldn't find it.

I would put the dispatch events calls in GHOST_WindowViewCocoa.h instead.

  • (void)tabletPoint:(NSEvent *)event

{
systemCocoa->dispatchEvents();
systemCocoa->handleTabletEvent(event, [event type]);
}

  • (void)tabletProximity:(NSEvent *)event

{

systemCocoa->dispatchEvents();
systemCocoa->handleTabletEvent(event, [event type]);

}

My logic:
handleMouseEvent consistently spawns a new mouse event and then calls handleTabletEvent to add additional tablet information for that event. You can go from handleMouseEvent to handleTabletEvent(event) to handleTabletEvent(event,subevent) as a single chain of calls, and if proximity was an issue, it seems likely you'd want that to be a single event.

In contrast, the two handlers in GHOST_WindowViewCocoa.h are smashing their changes onto whatever happens to be on the queue - they don't generate new events on their own. So it might make sense for them to flush the changes out first.

I made the changes to my local copy and the problem doesn't trigger any more.

wm_event_do_handlers: Handling event
wmEvent type:1 / LEFTMOUSE, val:1 / PRESS,

shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
mouse:(1376,1234), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7fd5f1dd2878

tablet: active: 1, pressure 0.0784, tilt: (0.4111 -0.1333)

wm_event_do_handlers: Handling event
wmEvent type:1 / LEFTMOUSE, val:2 / RELEASE,

shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
mouse:(1380,1110), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7fd5f1c49828

tablet: active: 1, pressure 0.0000, tilt: (0.3333 -0.1667)

And the next event after this one correctly does not show any tablet info.

wm_event_do_handlers: Handling event
wmEvent type:260 / WINDOW_DEACTIVATE, val:2 / RELEASE,

shift:0, ctrl:0, alt:0, oskey:0, keymodifier:0,
mouse:(-804,-898), ascii:'', utf8:'', keymap_idname:(null), pointer:0x7fd631c68f88

So, I think that's the correct solution, unless there's an edge case somewhere that you really want the tablet events to override the mouse events. It probably requires someone with a tablet that regularly receives proximity events to test it though.

can conform this fix works also on astropad and duet.
thanks @Tim Lesher (tim) Carroll (codrus) for this!
@Brecht Van Lommel (brecht) Van Lommel (brecht) is there any reason why this fix can't go in master?

We have several cintiq here, the patch seems not affect cintiqs behaviour.

Which Mac tablet is supposed to use proximity events differently?

I'm not sure about flushing the queue where you suggest. handleTabletEvent() does not seem to modify the event in the queue, it just sets some tablet parameters (pressure, tilt) in some window context object, and then presumably some other code uses that info when it comes to processing the event. For drawing, then, you want the latest tablet info available when the event gets processed or else you're always using old pressure data. The only instance where you don't want tablet data smashing is on the leaving proximity event because it leaves tablet mode, and no drawing is going to happen anyway.

Also, flushing the queue on every tablet event may have performance implications. I haven't gone through to figure out how/when the queue normally gets handled.

Hi, don’t know every much about code and I'm fairly new to blender, can anyone help me out with an explanation on how to fix this issue or which blender scripts to change, I'd very much appreciate it. Thank you!