Lip-Sync Ventriloquist Robot

Question

Hey Ez-B community! Newbie question here. I want to build a talking robot that will deliver speeches. Basically the idea is just to use my own voice and make mouth open and close when I speek like a Ventriloquist. The straight forward approach would just be to manually control the mouth with the computer or joy stick and try to make the lip-sync good. But is there a more elegant solution? For instance can the mouth move only when there is sound coming into a mic plugged in the computer? Are there these capabilities built into the SDK? Can somebody point me in the right dirrection. Thanks!

Jonny

DJ Sures · Answer

Hey there Jonnny... Using the SDK, you can do that. To capture audio from the Mic, I would recommend NAudio C# audio library... Get NAudio here It's quite simple to capture the audio. Also, there is an event on "Audio Received" but I think it is called Data Received. Anyway, the event is triggered when ever there is audio in the buffer. You can foreach through the byte array of audio and determine the volume of the sound. The data i think is PCM Wav by default, so that is pretty easy. There are plenty of source codes demonstrating how to determine the dB of a PCM. If you have trouble, we can help you out. Maybe i'll add a control to the ARC (after I'm done with the automation control) to do this. I can see benefits.

waldes · Answer

Thanks a lot for implementing this feature. It works works quite well for mouth movement! Progress:

DJ Sures · Answer

HAHA OMG That's creepy and awesome at the same time!!! Do you have any pics of how you attached teh mouth to the servo? And its neck too? Looks like there is a lot of motion there. Big project! you've been busy

waldes · Answer

Sure thing DJ. Pretty inelegant hack job, gobs of hot glue shored up with twist ties. 18 gauge steel wire pulls the bottom lip through a hole I drilled. This is an early gen Teddy Ruxpin than your v1.0 project. The hardest part was finding a Teddy Ruxpin without a broken mouth mechanism on fleabay. This is my third. Certainly kids abused the crap out of these. The rest of the project was EZ! Trying to decide how to do the arms. More servos are on order. I am leaning towards metal coat hangers. This guy will eventually host a comedic youtube channel, possibly political rants, definitely a sweet teddy bear voice and inappropriate subject matter. I'll keep you posted.

Jeremie · Answer

Pretty awesome job! A friend of mine thought the video was sped up until we heard the laughing in the background. Very quick response too, and with the blinking it makes it seem very much alive. Though your hacking may be a bit inelegant it gets the job done and the overall project looks great on the outside so it won't even matter!

DJ Sures · Answer

Fantastic! Elegant or not, it works ... We're all about making new ideas, so you fit right in there. Big time! I haven't slept in a few days. I have a big treat for everyone coming up very shortly.

waldes · Answer

Hmm video not working.  https://www.youtube.com/watch?v=oGFGv-JFlHc

DJ Sures · Answer

HAHAHAHAHAAAAAAAAAAAAAAAAAAAAA!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! THAT IS AWESOME I FELL OFF MY CHAIR!

Gunny · Answer

Haha that's awesome!

jamerperson · Answer

That's awesome!

sweetn8 · Answer

I am curious to know your progress! I would love to see more of the solutions you came up with as I am working on the same thing now three years later.

waldes

Upgrade to ARC Pro