Vixen 3 Lip Sync Overview

ebrady

ebrady

New member
Hi all, I have put together a set of overview videos for the Lip sync functionality I am working on for Vixen 3. This is the first time I have put together summary videos, so bear with my "newbie-ness" in this area. Here are the links and their table of contents.

Video #1
0:00 - Introduction
1:58 - setting up Display, new Template
4:34 - Setting up the preview
5:50 - New sequence items for Lipsync
6:55 - Lipsync to String/Pixel Mapping Overview
10:04 - Creating a Lipsync Map
13:20 - Default Mappings
14:55 - Setting Default Map

Video #2
0:00 - Introduction and recap of Video #1
1:05 - Sequencing a Lipsync element manually
4:54 - Overview of Papagayo
8:32 - Importing Papagayo Files
11:50 - Lyric to Phoneme Converter - Part #1

Video #3
0:00 Introduction and recap of Video #2
1:33 - Sequencing of Alvin and the Chipmunks
2:24 - Audio Processing to enhance Vocals
5:17 - Converting Lyrics to Phonemes with Text converter
7:01 - Text to Phoneme conversion using Marks
12:30 - Chipmunks Sequence with audio
16:55 - Other effect possibilities


Video #4
0:00 - Introduction
0:50 - Review some possible types of Lip Sync element configurations
3:35 - Review of setting up 8 "Dumb" String configuration
4:21 - New program options to simplify setup for smart pixel strings of varying length
14:38 - New Feature - Pixel Matrix / Mega Tree Mapping
20:10 - Import/Export of Images into Map Matrix Editor

(NOTE: In the section beginning at 0:50, I make several references to the "software" not supporting a Pixel Matrix while pointing at the Vixen3 Display Configuration.
I realize after posting the video that this will likely cause some confusion, "software" in this case means the LipSync module code, not Vixen3 itself).


Please note, the desktop recording software was unable to keep up when I tried recording the previews in videos #2 and #3 so the LipSync effect does not look as if it is working correctly even though it was.
 
Last edited:
Great job Ed

The videos are well done and the feature looks terrific. I love the 2 options, as you said there has already been a lot of work gone on to sequence songs so being able to easily bring them in is a bonus.

Thanks a heap!!

Kev
 
Thanks all!

I welcome all feedback and suggestions on the functionality, good or bad.
Also, it is likely you will stumble across a bug or two, so let me know if you do and I will make sure it gets corrected.

I will periodically update the links above as bugs are found and fixed.

Ed
 
WELL.. I kind of hate you right now ... as it look i will have to build more prop .... lol
quick question though .. will it be possible to copy /paste from 1 sequence to the other like in v2 ?? i understand with pixel element it it nearly impossible because not 2 person as same amount of light ect ..but with animation they could be almost standard ,,, and that would make my life way easier as I SU#$% at sequencing ...cheer
 
Sorry to cause you more work! :lol:

Do you mean copy between Vixen 3 sequences or something else?

If Vixen 3, yes it is possible to between sequences with different setups.

#1 - Copy between V3 sequences of the same profile/display setup - Original and target effects are duplicates of each other

#2 - Copy between V3 sequences created with different profiles/display setups
a - If the target sequence's profile/display has a mapping with the same name as the originating sequence, the pasted effects will use the mapping with the same name in the target profile.
b. - If the target sequence's profile does not have a mapping with the same name as the originating sequence, the pasted effects will use the default map in the target profile.

In both of the #2 cases, the target mappings could be significantly different from the original and it will work for both. (For example, if one sequence uses Props with 8 strings, and the other uses a Props with a 200 pixels, it will work for both ).

barbotte said:
WELL.. I kind of hate you right now ... as it look i will have to build more prop .... lol
quick question though .. will it be possible to copy /paste from 1 sequence to the other like in v2 ?? i understand with pixel element it it nearly impossible because not 2 person as same amount of light ect ..but with animation they could be almost standard ,,, and that would make my life way easier as I SU#$% at sequencing ...cheer
Click to expand...
 
quick question ,,,, is it or will it be possible to used that feature on a pixel tree ?? so i could have a mouth talking in the tree ?? or something like that ?? cheer
 
My intent was to allow the use of a Pixel Grid/Tree, so it should work but I have not tested it yet.

To do it, just add all pixels in your element to a Lip Sync map, and then specify how each individual pixel will map to the Phoneme.

(Side Note): To help with making large LipSync Maps for pixels, I implemented the ability to select a multiple check boxes on the Lip Sync Map Dialog and then toggle their state by pressing space bar. An example of this is shown in Video #1 section "Creating a Lip Sync Map". This will also work for setting multiple colors at once, you just need to select all colors you wish to change, hold shift, and double click the color cell.


barbotte said:
quick question ,,,, is it or will it be possible to used that feature on a pixel tree ?? so i could have a mouth talking in the tree ?? or something like that ?? cheer
Click to expand...
 
I have updated the links in post #1 to fix a couple of reported issues, with the 64-bit version.
 
I would think a really easy way to use this on your tree would be to add all the individual pixels that will be used as the outline under the outline group.
Do this for each Phoneme.
Then you still only have to create the lipsync map for 8 channels.

If this isn't possible then I think we would love to see it implemented.

Regards,
Rowan
 
Yes it can be done this way, but only after I make a small change to the form used to select channels for LipSync Map. I will see if I can get that implemented later today.

After having had the night to think about it, I plan to play a couple of days creating pixel matrix maps with the mapping dialog to assess its usability for managing a large number of pixels in a pixel matrix. More to follow...

battle79 said:
I would think a really easy way to use this on your tree would be to add all the individual pixels that will be used as the outline under the outline group.
Do this for each Phoneme.
Then you still only have to create the lipsync map for 8 channels.

If this isn't possible then I think we would love to see it implemented.

Regards,
Rowan
Click to expand...
 
ebrady said:
After having had the night to think about it, I plan to play a couple of days creating pixel matrix maps with the mapping dialog to assess its usability for managing a large number of pixels in a pixel matrix. More to follow...
Click to expand...
Great job on the Lip Sync stuff!!!

I'd love to have some control over individual pixels, like the LOR SuperStar module, so I could make things easily on my pixel mega tree like a hippo for my wife's favorite song - I want a hippopotamus for Christmas. The ability to make stuff like this outside of Nutcracker or another module has been a big sequencing hurdle for me in Vixen 3.
 
The code does allow you to control down to the pixel level, but to keep the videos from being too long, I only showed examples that used strings.

With that being said, the Lipsync map Editor is somewhat unfriendly to large pixel counts at the moment and configuring it for a large mega tree / matrix will be tedious. I am working on an improved version of the Map editor that is more friendly to this type of setup. I hope to have something ready to show in the next week if life doesn't get in the way. :)
 
Just what I was looking for! I had started playing around HLS for face animation however all my sequences thus far had been done using Vixen 3 so I am thrilled that all I had to do was add my face channels and install the module. So far working great, haven't tested it with the actual prop yet, I hope to do that later but it seems to be working fine in preview.

I do have one question / request. I did use the mark manager to mark where the words are, some are spot on and some are harder to get just right because of the way they are sung / held out, etc... It's not a matter of hearing the vocals, they are heard just fine in the song I am working on however in trying to get the phonemes into just the right spot for the entirety of a sung word, I think it would be easier we could make a dummy channel, maybe within the lipsync element that when you type your lyrics in to be converted it places the word in an effect bar that you can drag or shrink as need so we have a clear visual of when that word starts and ends and can adjust the phonemes as needed. Kind of like the "word" special control channel in HLS.

Just my two cents.

Love it, keep up the good work!
 
Thanks codec101!

You will have to forgive me, I have not used HLS very much and am not completely familiar with how it handles words. The control channel idea does seem like an interesting approach.

For Vixen3, I have been kicking around the idea of how to group phonemes into words. I figured the timeline would display the word and lock the phoneme group's start/end times into that of the word element. Dragging the word would cause cause all phonemes in the group to adjust accordingly, and resizing the word would shrink the phoneme group while maintaining a percentage of their original size. The individual phoneme properties could still be edited after being grouped as word via double-click.

So far I have tried a couple of approaches, both time I ran into architectural / code structure issues that caused me to abandon the effort. I think this type of feature can be accomplished, I just need to give it some more thought.

Ed
 
Ed,

Just throwing this little bug out there, don't know if it's known yet or not.

Using the 64bit version, the mark tab open on the left with a mark collection enabled and selected, with the lipsync text converter window open, while playing back a song and hit PAUSE due to trying to get words lined up, if I right click the timeline to ADD a mark than click the drop down menu to select the newly added mark, it erases the list of marks in the text converter window and if you click on the mark collection drop down menu, in erases the name of whichever one you select.

It does not seem to delete the marks collections or any marks as you can go to the mark manager and all the marks are still there, you just have to retype in the collection name in the mark manager when it happens and then you can keep going. Just a little hiccup, nothing major. I just need to remember to note where the mark needs to go and hit STOP, then add the mark. After doing it that way, the new mark can be found at the bottom of the marks list in the drop down.

I also broke my marks collections up into 2 - 3 collections in order to help keep them straight in the even something happens with the collection while adding new marks on the fly or when using the tapper function in mark manager.
 
Last edited:
You must log in or register to reply here.
Back
Top