Pitched Audio renders incorrectly in VSE #50843

New Issue

Colin Levy · 2017-03-03T12:44:55+01:00

Colin Levy commented

2017-03-03 12:44:55 +01:00

In the Video Sequence Editor in 2.78, you can change the Pitch of sound clips.

This also stretches the clip's duration to compensate - from frame 1 of the source. (A bit annoying, this should be from frame 1 of what's in use - but that's not my bug.)

What I hear when I hit play in the VSE sounds good, and what I expect. But when I render it - either as an animation with video or just an audio mixdown by itself - the pitch shift doesn't happen, and therefore it renders a completely irrelevant part of the raw sound clip.

No workaround currently, other than capturing my system sound rather than exporting audio from Blender.

Thanks for your attention!

In the Video Sequence Editor in 2.78, you can change the Pitch of sound clips. This also stretches the clip's duration to compensate - from frame 1 of the source. (A bit annoying, this should be from frame 1 of what's in use - but that's not my bug.) What I hear when I hit play in the VSE sounds good, and what I expect. But when I render it - either as an animation with video or just an audio mixdown by itself - the pitch shift doesn't happen, and therefore it renders a completely irrelevant part of the raw sound clip. No workaround currently, other than capturing my system sound rather than exporting audio from Blender. Thanks for your attention!

Colin Levy commented

2017-03-03 12:44:55 +01:00

Changed status to: 'Open'

Colin Levy commented

2017-03-03 12:44:55 +01:00

Added subscriber: @ColinLevy

Aaron Carlisle commented

2017-03-03 15:27:19 +01:00

Added subscribers: @neXyon, @Blendify

Olly Funkster commented

2017-03-03 16:03:35 +01:00

Added subscriber: @Funkster-3

Olly Funkster commented

2017-03-03 16:03:35 +01:00

Hmm... can't confirm with released 2.78b, and I use the pitch control fairly often and thought I was familiar with its various foibles.

Any chance you're using keyframed values for pitch and / or have scenes-within-other-scenes?

Hmm... can't confirm with released 2.78b, and I use the pitch control fairly often and thought I was familiar with its various foibles. Any chance you're using keyframed values for pitch and / or have scenes-within-other-scenes?

Joerg Mueller commented

2017-03-03 19:07:49 +01:00

Hi Colin,

first of all: congratz to your success with the kickstarter campaign!

Could you please provide a small test file where this happens?

Cheers

Hi Colin, first of all: congratz to your success with the kickstarter campaign! Could you please provide a small test file where this happens? Cheers

Colin Levy commented

2017-03-03 22:18:19 +01:00

Thanks for the prompt replies - and appreciate the congrats, Joerg! :D

No keyframes on pitch, no nested clips. Here's a simple test file to illustrate. Compare the render with what you hear in the sequencer, and play around with the pitch values:

https://www.dropbox.com/s/ufvg7vkl4cyjgor/pitch_test.zip?dl=0

Thanks for the prompt replies - and appreciate the congrats, Joerg! :D No keyframes on pitch, no nested clips. Here's a simple test file to illustrate. Compare the render with what you hear in the sequencer, and play around with the pitch values: https://www.dropbox.com/s/ufvg7vkl4cyjgor/pitch_test.zip?dl=0

Olly Funkster commented

2017-03-04 13:50:54 +01:00

Confirmed.

Huh, well this is quite odd. I started a fresh blend from scratch using your audio, and everything pitch-wise was going fine UNTIL I moved the clip (with g). Then, all hell breaks loose and there is no way that I've found to get everything back in step.

I tried updating the audio animation cache (properties > scene > audio > update animation cache) but it turns out that also has a similar effect to moving a strip... having loaded a .blend with a pitched strip where everything is fine, updating the cache breaks the pitched strip's position in the VSE preview.

I realised that although I use pitch control a lot, I've got into the habit of this workflow:

make cuts at either end of the part I want to speed up (I'm always speeding things up but I don't think this matters)
change the pitch
adjust the strip length by typing "/3.5" or whatever multiple in the strip's length field, thus moving only the right-hand end of the strip
Never, ever, touch it again.

This avoids the problem, but isn't much use if you want to shuffle clips around.

I won't have a chance to do any more digging on this for a week or so, but I'll happily get into it then if no-one else has found it.

Confirmed. Huh, well this is quite odd. I started a fresh blend from scratch using your audio, and everything pitch-wise was going fine UNTIL I moved the clip (with g). Then, all hell breaks loose and there is no way that I've found to get everything back in step. I tried updating the audio animation cache (properties > scene > audio > update animation cache) but it turns out that also has a similar effect to moving a strip... having loaded a .blend with a pitched strip where everything is fine, updating the cache breaks the pitched strip's position in the VSE preview. I realised that although I use pitch control a lot, I've got into the habit of this workflow: 1) make cuts at either end of the part I want to speed up (I'm always speeding things up but I don't think this matters) 2) change the pitch 3) adjust the strip length by typing "/3.5" or whatever multiple in the strip's length field, thus moving only the right-hand end of the strip 4) Never, ever, touch it again. This avoids the problem, but isn't much use if you want to shuffle clips around. I won't have a chance to do any more digging on this for a week or so, but I'll happily get into it then if no-one else has found it.

blender-admin commented

2017-03-05 12:19:32 +01:00

This issue was referenced by f75b52eca1

This issue was referenced by f75b52eca1a54d6aba1e26dc0fc9d94c0ee43ecf

Joerg Mueller commented

2017-03-05 12:21:40 +01:00

Changed status from 'Open' to: 'Resolved'

Joerg Mueller closed this issue

2017-03-05 12:21:40 +01:00

Joerg Mueller commented

2017-03-05 13:43:07 +01:00

Hello again.

So today I had some time to look into this and indeed there was a bug in the intended behaviour, now the behaviour though maybe not optimal/intuitive should be at least consistent between editing and rendering. Let me explain the problem and the behaviour that is implemented:

Changing pitch changes the playback speed of the sound and thus all the positioning. Blender's VSE is not really equipped to deal with varying playback speed. When you seek (change the playback cursor's position, for example when clicking on the timeline), the current playback position in the sound file must be computed. This in theory works fine as long as the pitch of a sound is constant, for example if an uncut strip is 10 seconds long at pitch 2 and you seek to 2 seconds, the audio system actually would need to seek to 4 seconds into the file. However, what happens if you have the pitch animated during those first 2 seconds, for example linearly increasing from 1 to 2 in the first second. You would need to check everything before the actual point you are seeking too and compute the position based on that. Then there is the question, how accurately you do this: for every frame, is that too rough? For every audio sample then, is that too precise and thus takes forever? During actual playback the pitch actually changes depending on the buffer size, which is the reason why you in any case can get a different output if the playback buffer size is not the same as the render buffer size. To avoid having to compute this, which is in anyway inaccurate and expensive to do, the following behaviour is actually implemented: when you seek to 2 seconds into the strip, the audio system does so in the file assuming a pitch of 1, so the actual playback then will end up off (it will play back as if you sought to 1 and not to 2 seconds of the strip). This is the price we pay - the playback is only correct for a strip with pitch changes if it's played from the beginning.

Next up, there is another problem regarding changing pitch with respect to cutting/editing audio strips. Again the VSE is not equipped to handle playback speed changes properly. You can already observe this when you just change the pitch: the length of the strip doesn't change. Now the audio system gets two important pieces of information from blender: where in the file to start at and the pitch. You can check the start if you check the strip properties (- [x] panel) where it says "Time Duration", soft and hard are simply added. Now this number is in frames and divided by the frame rate gives the number of seconds to skip in the file. Now similar to the length of the file if you change the pitch of the strip and you want the start to stay in place, you would have to modify this value. Again this would be simple with a constant pitch, but what do you do when the pitch is animated? Therefore, again we follow the policy to consider a pitch of 1. Therefore if you move the start of the strip, everything behind changes as well, when the pitch is not 1. Unfortunately, also when you do cuts, the second strip does not start at the position it was at before, as again the exact position would need to be calculated from the first half.

To summarize: the start of an audio strip is the most important thing to consider when editing with pitch. Ideally you make sure you find the start of the audio that you want with cutting/editing and change the pitch afterwards. Also when you test playback, always start at the beginning of a strip. When you are done with editing a part with pitch changes and don't want to do that anymore, it's of course possible to mix down this part and then replace the strips with the result.

I'm really sorry that this is so problematic, but Blender's VSE just needs a complete rewrite with such things in mind - the current version is outdated, a Frankenstein monster and nobody seem to have the resources for a rewrite.

Hello again. So today I had some time to look into this and indeed there was a bug in the intended behaviour, now the behaviour though maybe not optimal/intuitive should be at least consistent between editing and rendering. Let me explain the problem and the behaviour that is implemented: Changing pitch changes the playback speed of the sound and thus all the positioning. Blender's VSE is not really equipped to deal with varying playback speed. When you seek (change the playback cursor's position, for example when clicking on the timeline), the current playback position in the sound file must be computed. This in theory works fine as long as the pitch of a sound is constant, for example if an uncut strip is 10 seconds long at pitch 2 and you seek to 2 seconds, the audio system actually would need to seek to 4 seconds into the file. However, what happens if you have the pitch animated during those first 2 seconds, for example linearly increasing from 1 to 2 in the first second. You would need to check everything before the actual point you are seeking too and compute the position based on that. Then there is the question, how accurately you do this: for every frame, is that too rough? For every audio sample then, is that too precise and thus takes forever? During actual playback the pitch actually changes depending on the buffer size, which is the reason why you in any case can get a different output if the playback buffer size is not the same as the render buffer size. To avoid having to compute this, which is in anyway inaccurate and expensive to do, the following behaviour is actually implemented: when you seek to 2 seconds into the strip, the audio system does so in the file assuming a pitch of 1, so the actual playback then will end up off (it will play back as if you sought to 1 and not to 2 seconds of the strip). This is the price we pay - the playback is only correct for a strip with pitch changes if it's played from the beginning. Next up, there is another problem regarding changing pitch with respect to cutting/editing audio strips. Again the VSE is not equipped to handle playback speed changes properly. You can already observe this when you just change the pitch: the length of the strip doesn't change. Now the audio system gets two important pieces of information from blender: where in the file to start at and the pitch. You can check the start if you check the strip properties (- [x] panel) where it says "Time Duration", soft and hard are simply added. Now this number is in frames and divided by the frame rate gives the number of seconds to skip in the file. Now similar to the length of the file if you change the pitch of the strip and you want the start to stay in place, you would have to modify this value. Again this would be simple with a constant pitch, but what do you do when the pitch is animated? Therefore, again we follow the policy to consider a pitch of 1. Therefore if you move the start of the strip, everything behind changes as well, when the pitch is not 1. Unfortunately, also when you do cuts, the second strip does not start at the position it was at before, as again the exact position would need to be calculated from the first half. To summarize: the start of an audio strip is the most important thing to consider when editing with pitch. Ideally you make sure you find the start of the audio that you want with cutting/editing and change the pitch afterwards. Also when you test playback, always start at the beginning of a strip. When you are done with editing a part with pitch changes and don't want to do that anymore, it's of course possible to mix down this part and then replace the strips with the result. I'm really sorry that this is so problematic, but Blender's VSE just needs a complete rewrite with such things in mind - the current version is outdated, a Frankenstein monster and nobody seem to have the resources for a rewrite.

Colin Levy commented

2017-03-06 22:10:47 +01:00

Thank you for your thorough investigation & explanation! oof, my head hurts. :P Yeah, it doesn't seem ideal how it works now, but this else me understand what's going on under the hood. Are you suggesting the mix-down method to render, or should at least be rendering work okay now?

Joerg Mueller commented

2017-03-06 22:24:41 +01:00

Well yeah, it's not so easy... I actually had to figure out again, why it's so problematic in detail, when I was writing this, that's a reason why it took so long...

I don't really get your question. I can say however that with the bug fix the behaviour should be the same everywhere, whether you render, mix down or play back (if you start from the beginning of the strip).

The mix-down method suggestion was just for when you are done editing with pitch changes and you are tired of playing from the beginning all the time. So you mix down and then import that mixed down audio file and use it instead of the pitched audio strips for further editing of your sequence.

Well yeah, it's not so easy... I actually had to figure out again, why it's so problematic in detail, when I was writing this, that's a reason why it took so long... I don't really get your question. I can say however that with the bug fix the behaviour should be the same everywhere, whether you render, mix down or play back (if you start from the beginning of the strip). The mix-down method suggestion was just for when you are done editing with pitch changes and you are tired of playing from the beginning all the time. So you mix down and then import that mixed down audio file and use it instead of the pitched audio strips for further editing of your sequence.

Colin Levy commented

2017-03-11 21:11:36 +01:00

Ah no worries, I understand now. My question was poorly worded! Excellent, thanks for this fix!

ChristopherAnderssarian commented

2018-01-26 12:23:03 +01:00

Added subscriber: @ChristopherAnderssarian

Richard Antalik commented

2018-06-23 16:22:52 +02:00

Added subscriber: @iss

Richard Antalik commented

2018-06-23 16:22:52 +02:00

@neXyon

I wrote patch D3496 to stretch waveforms of seq, so if you cut it based on waveform it behaves consistently with expectation.

I would propose to fix start of playback position, when pitch is set. Assuming, the pitch is constant.
I guess, that this will satisfy 90% of users

To improve further manipulation of sound strips, we (I) can modify operator, that changes pitch(so the start of seq wont move relative to waveform)
That would satisfy 99% of users

to satisfy 99.9% I can elaborate on method to go through keyframes and calculate offset by some integral, so it can be fast and precise. However I would rather not.
I animated 2s of sound in total in past 2 years... however I use sped up footage a lot, and editing sped up footage is pain mainly because sound is not in sync if I don't play from beginning.

@neXyon I wrote patch [D3496](https://archive.blender.org/developer/D3496) to stretch waveforms of seq, so if you cut it based on waveform it behaves consistently with expectation. I would propose to fix start of playback position, when pitch is set. Assuming, the pitch is constant. I guess, that this will satisfy 90% of users To improve further manipulation of sound strips, we (I) can modify operator, that changes pitch(so the start of seq wont move relative to waveform) That would satisfy 99% of users to satisfy 99.9% I can elaborate on method to go through keyframes and calculate offset by some integral, so it can be fast and precise. However I would rather not. I animated 2s of sound in total in past 2 years... however I use sped up footage a lot, and editing sped up footage is pain mainly because sound is not in sync if I don't play from beginning.

Richard Antalik commented

2018-06-24 03:26:22 +02:00

And adding a quick and dirty proof of concept: https://youtu.be/zB7wXox8VX0

--- a/source/blender/makesrna/intern/rna_sequencer.c
+++ b/source/blender/makesrna/intern/rna_sequencer.c
@@ -708,10 +708,17 @@ static void rna_Sequence_volume_set(PointerRNA *ptr, float value)
 static void rna_Sequence_pitch_set(PointerRNA *ptr, float value)
 {
        Sequence *seq = (Sequence *)(ptr->data);
+
+       if (seq->speed_fader == 0 || seq->anim_startofs != (int) seq->speed_fader) {
+               seq->speed_fader = ((seq->anim_startofs * seq->pitch) / value);
+               seq->anim_startofs = seq->speed_fader;
+       } else {
+               seq->speed_fader = ((seq->speed_fader * seq->pitch) / value);
+               seq->anim_startofs = seq->speed_fader;
+       }

        seq->pitch = value;
        if (seq->scene_sound) {
                BKE_sound_set_scene_sound_pitch(seq->scene_sound, value, (seq->flag & SEQ_AUDIO_PITCH_ANIMATED) != 0);
        }

And adding a quick and dirty proof of concept: https://youtu.be/zB7wXox8VX0 ``` --- a/source/blender/makesrna/intern/rna_sequencer.c +++ b/source/blender/makesrna/intern/rna_sequencer.c @@ -708,10 +708,17 @@ static void rna_Sequence_volume_set(PointerRNA *ptr, float value) static void rna_Sequence_pitch_set(PointerRNA *ptr, float value) { Sequence *seq = (Sequence *)(ptr->data); + + if (seq->speed_fader == 0 || seq->anim_startofs != (int) seq->speed_fader) { + seq->speed_fader = ((seq->anim_startofs * seq->pitch) / value); + seq->anim_startofs = seq->speed_fader; + } else { + seq->speed_fader = ((seq->speed_fader * seq->pitch) / value); + seq->anim_startofs = seq->speed_fader; + } seq->pitch = value; if (seq->scene_sound) { BKE_sound_set_scene_sound_pitch(seq->scene_sound, value, (seq->flag & SEQ_AUDIO_PITCH_ANIMATED) != 0); } ```

Joerg Mueller commented

2018-06-24 23:36:01 +02:00

Hey Richard,

thanks for your patch! I especially like the red coloring in case of clipping. Why wouldn't you let the waveform scale down if the volume is below 0 though? That would be the correct way to display it and it might confuse the user if it scales up above 1 but not down below it.

For pitch changes, animation of the pitch value is the worst, destroying everything of course. The reason the audio system doesn't consider the pitch when seeking (= starting playback in the middle of a strip) is that the same system is used for 3D audio, where the doppler effect results in pitch changes that also don't allow exact seeking without the knowledge of all data before the point in time that you're seeking to.

I tried your scaled drawing of the waveform and it works as long as the strip is not cut, but once either soft or hard trim start are changed the result is wrong even with constant pitch. It doesn't consider that the trim starts are considered before the pitch change, not after.

I don't get what you are trying to say in your suggestions. The playback position at the start of a strip currently doesn't change if you change the pitch. That's something the users can count on. The problem is that when you cut a strip with a different pitch, then the second half of the cut will not start correctly as the offset from the start of the previously whole strip is changed as if the first strip half has a pitch of 1. Your quick and dirty proof of concept doesn't seem to work properly at all. What is the speed_fader that you are using there?

Hey Richard, thanks for your patch! I especially like the red coloring in case of clipping. Why wouldn't you let the waveform scale down if the volume is below 0 though? That would be the correct way to display it and it might confuse the user if it scales up above 1 but not down below it. For pitch changes, animation of the pitch value is the worst, destroying everything of course. The reason the audio system doesn't consider the pitch when seeking (= starting playback in the middle of a strip) is that the same system is used for 3D audio, where the doppler effect results in pitch changes that also don't allow exact seeking without the knowledge of all data before the point in time that you're seeking to. I tried your scaled drawing of the waveform and it works as long as the strip is not cut, but once either soft or hard trim start are changed the result is wrong even with constant pitch. It doesn't consider that the trim starts are considered before the pitch change, not after. I don't get what you are trying to say in your suggestions. The playback position at the start of a strip currently doesn't change if you change the pitch. That's something the users can count on. The problem is that when you cut a strip with a different pitch, then the second half of the cut will not start correctly as the offset from the start of the previously whole strip is changed as if the first strip half has a pitch of 1. Your quick and dirty proof of concept doesn't seem to work properly at all. What is the speed_fader that you are using there?

Richard Antalik commented

2018-06-25 19:24:35 +02:00

Thanks for quick reaction!
I will try to explain myself a bit.

Why wouldn't you let the waveform scale down if the volume is below 0 though?

I had it this way, but I didn't like it. In my case I am using waveforms to guess, what footage I am looking at, to cut pauses and such. Therefore I like it "maximized" rather than realistic. At least I would like to have this as a option.

I tried your scaled drawing of the waveform and it works as long as the strip is not cut

That is what I "fixed" with code that I posted here, it works with hard cut only.
In video that I posted I suggested what behavior I would expect from a video/sound editor. That is that I have some strip, and when I change pitch, start of strip stays in place(unless I want to adjust that) and rest of footage(I would understand even end of sequence) moves.

The problem is that when you cut a strip with a different pitch, then the second half of the cut will not start correctly as the offset from the start of the previously whole strip is changed as if the first strip half has a pitch of 1.

Yesssn't? :)
I would say, that I wouldn't think of a cut strip like it has parent or first half. It is a separate entity and if we can calculate offset, we won the game.
In said video, when I cut sped up sequence and changed pitch, all (mostly...) waveforms are continuing as they should. This means that we can calculate offset.

What is the speed_fader that you are using there?

it's some unused property... I wasn't able to modify DNA and compile it. I mean I am noob and most of the time I don't know what I am doing :)
The reason why I am using that prop is, that I need a float variable to store precise value of cut position, so when I change pitch back to original value, calculated cut position also returns to it's original value.
In that video you can see, that trim_start was say 235 and temporary float held value say 235.123456.

Now what's my point:
I can lie to audaspace, but I wasn't able to find where(how) audaspace reads timeline.
Actually if there is no need to modify audaspace at all I can play with this and post a patch when I will be happy with it.
But I would be glad if you could point me at some file and function and I can "reverse" it from there

Edit: Nevermind I am blind... I guess it's BKE_sound_move_scene_sound and similar in bke/sound.c

Edit2: Oh I forgot, What needs to be done to audaspace is to play correct portion of audio when I do not start playing from the start of the sequence. If it is possible of course.

Also sorry for lengthy post...

Thanks for quick reaction! I will try to explain myself a bit. >Why wouldn't you let the waveform scale down if the volume is below 0 though? I had it this way, but I didn't like it. In my case I am using waveforms to guess, what footage I am looking at, to cut pauses and such. Therefore I like it "maximized" rather than realistic. At least I would like to have this as a option. >I tried your scaled drawing of the waveform and it works as long as the strip is not cut That is what I "fixed" with code that I posted here, it works with hard cut only. In video that I posted I suggested what behavior I would expect from a video/sound editor. That is that I have some strip, and when I change pitch, start of strip stays in place(unless I want to adjust that) and rest of footage(I would understand even end of sequence) moves. >The problem is that when you cut a strip with a different pitch, then the second half of the cut will not start correctly as the offset from the start of the previously whole strip is changed as if the first strip half has a pitch of 1. Yesssn't? :) I would say, that I wouldn't think of a cut strip like it has parent or first half. It is a separate entity and if we can calculate offset, we won the game. In said video, when I cut sped up sequence and changed pitch, all (mostly...) waveforms are continuing as they should. This means that we can calculate offset. >What is the speed_fader that you are using there? it's some unused property... I wasn't able to modify DNA and compile it. I mean I am noob and most of the time I don't know what I am doing :) The reason why I am using that prop is, that I need a float variable to store precise value of cut position, so when I change pitch back to original value, calculated cut position also returns to it's original value. In that video you can see, that trim_start was say 235 and temporary float held value say 235.123456. Now what's my point: I can lie to audaspace, but I wasn't able to find where(how) audaspace reads timeline. Actually if there is no need to modify audaspace at all I can play with this and post a patch when I will be happy with it. But I would be glad if you could point me at some file and function and I can "reverse" it from there Edit: Nevermind I am blind... I guess it's `BKE_sound_move_scene_sound` and similar in `bke/sound.c` Edit2: Oh I forgot, What needs to be done to audaspace is to play correct portion of audio when I do not start playing from the start of the sequence. If it is possible of course. Also sorry for lengthy post...

Joerg Mueller commented

2018-06-26 14:46:31 +02:00

Ok, so the first problem is, that the display of the pitch scaled waveform is wrong in D3496 when the start is not 0. Changing the start on pitch change is also wrong, since the start of the playback of that strip doesn't depend on the pitch. So to keep the sound start at the same position when changing the pitch, you shouldn't do anything, because that's what happens right now.

So what you should do is in your patch D3496 you need to consider the soft and hard starts correctly, i.e. startofs and anim_startofs should not be scaled by the pitch. The other thing you want to do is to change the computation of the start as it happens in the cut operator, not during the pitch change. During the cut you have to alter the start correctly.

Ok, so the first problem is, that the display of the pitch scaled waveform is wrong in [D3496](https://archive.blender.org/developer/D3496) when the start is not 0. Changing the start on pitch change is also wrong, since the start of the playback of that strip doesn't depend on the pitch. So to keep the sound start at the same position when changing the pitch, you shouldn't do anything, because that's what happens right now. So what you should do is in your patch [D3496](https://archive.blender.org/developer/D3496) you need to consider the soft and hard starts correctly, i.e. startofs and anim_startofs should not be scaled by the pitch. The other thing you want to do is to change the computation of the start as it happens in the cut operator, not during the pitch change. During the cut you have to alter the start correctly.

Richard Antalik commented

2018-06-27 22:27:16 +02:00

I understand what you want to do...
I was thinking that this will add a "layer of abstraction" for all functions that use sound strips. But this abstraction has to be present somewhere, so I don't really object to this variant.
I will try to make some patch during weekend.

I understand what you want to do... I was thinking that this will add a "layer of abstraction" for all functions that use sound strips. But this abstraction has to be present somewhere, so I don't really object to this variant. I will try to make some patch during weekend.

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Pitched Audio renders incorrectly in VSE #50843