Celemony Melodyne

Started by jon_bondy, March 13, 2008, 04:34:34 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

jon_bondy

I thought this might be interesting to some of you, although it is a bit off topic.  I know nothing more than what is available on the web site:

===

Direct Note Access is a technology that makes the impossible possible: for the first time in audio recording history you can identify and edit individual notes within polyphonic audio material. The unique access that Melodyne affords to pitch, timing, note lengths and other parameters of melodic notes will now also be afforded to individual notes within chords.

while editing single chord notes is common for MIDI, it is a world premiere for audio recordings. Patent pending.

http://www.celemony.com/cms/index.php?id=dna

vanceg

I have been using their software for a couple of years now, and have got a demo of this new technology in person. It is very, very, VERY impressive. Their existing system, without the Polyphonic abilities is head and shoulders above all of the other pitch and time altering technology I have ever tried to use.  Aces.  I can't recommend it any more highly!!!!

Quote from: jon_bondy on March 13, 2008, 04:34:34 AM
I thought this might be interesting to some of you, although it is a bit off topic.  I know nothing more than what is available on the web site:

===

Direct Note Access is a technology that makes the impossible possible: for the first time in audio recording history you can identify and edit individual notes within polyphonic audio material. The unique access that Melodyne affords to pitch, timing, note lengths and other parameters of melodic notes will now also be afforded to individual notes within chords.

while editing single chord notes is common for MIDI, it is a world premiere for audio recordings. Patent pending.

http://www.celemony.com/cms/index.php?id=dna

Elantric


feloniouspunk

That is amazing.  Just thinking of the possibilities makes the mind go numb.  That is really cool when he changes Chet Baker melody. Wow.
Lots of Gear. :)

scratch17

#4
I am contemplating the following setup and would like comments from anyone who's used
Melodyne Studio.

I want to use the six outputs of my RMC Fanout Box to feed Melodyne Studio on six separate tracks.
This will result in six mono tracks, one per guitar string.

According to Melodyne Studio's manual:

QuoteYou can use Melodyne as a flexible audio-to-MIDI converter and play back the detected
(or edited) melodies using an external MIDI synthesizer or a software instrument loaded into Melodyne.
Pitch bend and controller data are used to permit the continuous variation in pitch and amplitude of the
audio to be reproduced precisely by MIDI instruments. As an alternative to the realtime transmission
of the MIDI data, Melodyne can export it as a file that you can then load into a variety of other MIDI
programs for further editing.

The program will automatically detect a monophonic track and set it to what it calls "melodic mode".
Then it does analysis on the mono audio to determine audio parameters. The analysis process even
separates pitch data from tempo data.

The program uses audio analysis technology that works to great effect when the program is doing
pitch correction. The same analysis algorithms are used for audio to MIDI conversion. The analysis
is set to automatic by default, but certain parameters can be preset to help the analysis process.

For example the program lets you preset highest and lowest note (fundamental). Since you know the
note range on each string, you could set those data points on a per string basis and create a template
for each string.

Also, I believe that tracking latency which occurs with conventional pitch to MIDI converters will be zero
with this conversion process, because the program will perfectly synchronize the MIDI note data timing
with the original audio that it analyzed.

Note that there is no need to use tricks such as using light gauge strings, with pitch shifting to improve tracking.
Since the audio analysis does not have to be done in real time, it doesn't matter that lower frequencies on the D, A
and low E strings takes longer than that of the upper strings.

While this process would not work for live performance, it seems like a perfect studio tool.

Consider this:

I've always wanted to be able to play my guitar (on which I have decent chops) instead of keys
(on which I have really poor chops) to trigger synths and samplers. Zero latency pitch to MIDI
on the guitar makes this an ultimate composing tool.

I also like the idea of recording everything I play and then being able to save a magic moment as a MIDI
file. This way, I can load the MIDI file into a notation program so that I have a permanent record of what
I played. Instead of having to spend time trying to figure out what I played four weeks ago using only
an audio file I can print out the score and I am ready to rock.

Furthermore, as a composer, I can create a part for flute or piano using my guitar, turn it into a score, and
have a real flautist or keyboardist play a real instrument for my recording.

Comments are welcome.

Steven
Hamer Duotone, Brian Moore i213, Taylor 710 BCE 

VG-99, FC-300, RMC Fanout
RJM Mastermind GT10
Kemper Profiling Amp
Radial JDV Mk3, X-Amp
Mesa Recto Pre + 20/20
68 Fender Bandmaster (AB763)
Marshal AS80R

UA Apollo X6, Twin X, Logic Pro, Luna, Melodyne Studio

Elantric

#5
Get the fastest computer you can afford - to make this work effectively.
Melodyne is powerful - but its not a "real time Pitch to MIDi solution.

That means it will NOT work for Live use!

You might also wish to get involved in these other efforts below:.

Ricky Graham's Septar board and his hex guitar experiments in "Pure Data":
http://rickygraham.com/

http://rickygraham.com/septar-board-beta-kits-june-2011

Demo videos here:
http://www.youtube.com/user/rickygrahammusic


scratch17

Thanks, Elantric!

Very cool demo. I'm a little confused though. Is the software he's using available?

Steven.
Hamer Duotone, Brian Moore i213, Taylor 710 BCE 

VG-99, FC-300, RMC Fanout
RJM Mastermind GT10
Kemper Profiling Amp
Radial JDV Mk3, X-Amp
Mesa Recto Pre + 20/20
68 Fender Bandmaster (AB763)
Marshal AS80R

UA Apollo X6, Twin X, Logic Pro, Luna, Melodyne Studio

Elantric

Yes

Learn more about PD here

http://puredata.info/

Pd (aka Pure Data) is a real-time graphical programming environment for audio, video, and graphical processing. It is the third major branch of the family of patcher programming languages known as Max (Max/FTS, ISPW Max, Max/MSP, jMax, etc.) originally developed by Miller Puckette and company at IRCAM. The core of Pd is written and maintained by Miller Puckette and includes the work of many developers, making the whole package very much a community effort.

Pd was created to explore ideas of how to further refine the Max paradigm with the core ideas of allowing data to be treated in a more open-ended way and opening it up to applications outside of audio and MIDI, such as graphics and video.

It is easy to extend Pd by writing object classes ("externals") or patches ("abstractions"). The work of many developers is already available as part of the standard Pd packages and the Pd developer community is growing rapidly. Recent developments include a system of abstractions for building performance environments; a library of objects for physical modeling; and a library of objects for generating and processing video in realtime.




BBach

My experience with the fanout box is that there is considerable crosstalk between the first and sixth strings and that the sound quality coming from the gk pickup is specially eq'd to be processed by vg units and doesn't sound all that great. I guess the vg units somehow can separate out the crosstalk ,but going directly into melodyne, it will be there and unless enhanced in some way, will sound thin

scratch17

In this application, I am only using the fanout box to trigger the pitch to MIDI function in
Melodyne Studio. Therefore, I am not needing a quality guitar sound.

I do agree that as a guitar tracking sound, the magnetic hex pickups are really awful.

In fact, I have long wished for a standard humbucking or single coil pickup that had hex
outputs, so I could replace the stock bridge and neck picks in a custom built guitar.

I want to be able to use each sting's two pickup outputs in a matrix (12 total outputs), so I can set
up custom sets of pickup configurations. I'd patch the output of each string to a Switchblade GL,
and set up presets for each configuration which could be switched on the fly via a MIDI foot controller.

With this setup, you'd have 729 unique pickup configurations.

Another benefit would be the ability to reamp each string separately, without the restrictions
that traditional reamping imposes. Each string could have its own amp, effects, etc.
Hamer Duotone, Brian Moore i213, Taylor 710 BCE 

VG-99, FC-300, RMC Fanout
RJM Mastermind GT10
Kemper Profiling Amp
Radial JDV Mk3, X-Amp
Mesa Recto Pre + 20/20
68 Fender Bandmaster (AB763)
Marshal AS80R

UA Apollo X6, Twin X, Logic Pro, Luna, Melodyne Studio

Elantric

#10
QuoteIn this application, I am only using the fanout box to trigger the pitch to MIDI function in
Melodyne Studio.

Owning both Fanout Box and the Septar board, , I give an edge to the Septar board which has superior noise immunity and minimum crosstalk between adjacent strings.
http://rickygraham.com/septar-board-beta-kits-june-2011

QuoteIn this application, I am only using the fanout box to trigger the pitch to MIDI function in
Melodyne Studio. Therefore, I am not needing a quality guitar sound.

My opinion is unchanged - I believe you should ALWAYS use the best guitar you can get your hands on whenever you are recording.


QuoteIn fact, I have long wished for a standard humbucking or single coil pickup that had hex
outputs, so I could replace the stock bridge and neck picks in a custom built guitar.
Here's one source for Hex output "normal sized" guitar pickups @ $49 ea:

http://www.amazon.com/Power-Gig-SixString-Guitar-Playstation-3/dp/B003N1639A/ref=dp_cp_ob_vg_title_1
Details on "un-boxing" is here:
https://www.vguitarforums.com/smf/index.php?topic=3115.0

Others hex mag pickups are here:

UltraMagnetics:
http://www.ultramagnetics.org/


Copeland
http://www.marksmart.net/gearhack/jazzpedalboard/copeland.html

Paul Rubenstein
http://musicthing.blogspot.com/2008/08/six-output-pickups-for-stereo-guitars.html
http://ubertar.com/instruments/

Bartolini
http://www.joness.com/gr300/XJ-S.htm

http://www.premierguitar.com/Magazine/Issue/2010/Feb/Putting_the_Hex_On_the_Postmodern_Pickups.aspx
Putting the Hex On: the Postmodern Pickups
<by Wallace Marx Jr. >


All guitar pickups have the same basic components: a magnet, a coil, a bobbin and some sort of base or cover. While some pickups have a single blade magnet, most guitar pickups have individual pole pieces for each string. In either configuration, the vibrations of each of the six strings are captured and converted to an electrical signal, which is then sent to an amplification unit. Whether it's a strummed chord or a singlenote solo, the signal follows the same path. The sound produced is universally accepted as the pleasing sound of an electric guitar.

But back to those six strings. The normal output signal of our standard pickups is a combined signal of any or all of the six strings. Six pole pieces, six strings. Wouldn't it seem like a logical next step to separate the output signal of each string? In the long and varied history of guitar pickups it seems as if nearly every combination of materials has been tried and every shape and configuration explored. So how about this six-signal concept? It has been done, for various reasons and with varying levels of success. The beast in question is called the hexaphonic pickup.

The suffix hexa means six. Phonic means of, or relating to, sound. Six sounds. Just as a standard pickup does, the hexaphonic pickup captures the individual vibrations of each of a guitar's six strings. The difference is, rather than having a single output signal, the hexaphonic pickup has six. So who did it first, and what's it good for? And why aren't we all playing with hexaphonics?

In questions of firsts, I have two go-to sources: the US Patent Office and my friend and pre war-era guitar historian Lynn Wheelwright. An immediate problem in researching hexaphonics is that the word itself is a bit of a red herring in that it has not been the industry standard term. So in this particular case, I called Lynn first just to see if he remembered anything like it. He told me that the earliest pickup he had seen with anything similar to hexaphonic construction was on a Regal guitar from circa 1935–36. This particular axe had a pickup unit with six individual coils, one for each string. It was not actually hexaphonic, though, because even with the six poles the pickup produced just a single output signal. On to the Patent Office where the earliest patent I could find using the specific terms "hexaphonic pickup system" was awarded to Gibson in 1990. 1990 seemed pretty late. We know that Bartolini Pickups founder Bill Bartolini was producing hexaphonics for public sale as early as 1973. Additionally, experimentation with polyphonic pickups for the purpose of hooking up guitars to synthesizers began in the early days of synthesizer development in the 1960s.

A hexaphonic pickup can have a number of uses. The most obvious is to send separate signals to separate places. In the analog realm, this might take the form of sending the bass strings to one amp, the treble to another. That effect was achieved in the mid-1950s by Ray Butts, who designed a split-coil pickup for Chet Atkins. Theoretically, separate signals for each string could be sent to individual inputs in the same amp where each input had its own volume control, compression, etc. It's easy to see how rapidly the complexity quotient snowballs and how a standard amplification system might not be able to handle such an increased amount of information.

The first Roland synthesizer guitar, the GS-500, was introduced in 1977. It was with the Roland series of synth guitars, which continues to this day, that the hexaphonic pickup found its most widespread utilization. As a transducer for analog-to-digital sound, the hexaphonic pickup is the only way to go. For a guitar to control a synthesizer module (thereby allowing it to make sounds far beyond the normal guitar or guitar effects palette) each string's signal needs to be isolated. An analog-to-digital converter requires that sounds be isolated, from string to string, in terms of pitch, and note start and stop. Converted to a digital signal, the Roland systems take the information from the guitar and turn it into sounds such as percussion, keys, saxophone, etc. The latest synth pickups on the market, such as the Roland GK-3 and the Axon PU100 designed in conjunction with Seymour Duncan, represent the latest technology in the field.

Some purists may balk, but my opinion is that synth guitar and its tonal possibilities would have been very popular with the players of yesterday. Alvino Rey used to have a sight gag in his act where he would "deliver" a baby lapsteel. I'd bet he would have loved to hook his lapsteels up to a unit that could make each note sound exactly like a crying baby. Taking it in another direction, I think of hexaphonic pickups and I wonder what it would have been like if one of the gods of guitar like Hendrix had had a separate Marshall stack for each string of his Strat. Perhaps cacophony; maybe brilliance. Either way, hexaphonic pickups and their ultimate utilization may be one of the last frontiers in the quest for the ultimate electric guitar tone.

(snip)

I would point out to Mr Marx the 1970 era  Baldwin  / Innovex-Condor GMS which had a hex PU in a Ovation Tornado ( 335) over 41 years ago:
http://guitarz.blogspot.com/2011/01/innovex-condor-gms-vintage-guitar-synth.html




http://www.ovationtribute.com/Catalogues/Storm%20Sheets/Tornado%20Brochure.html

billbax

Message to Elantric,

What is the signal-to-noise ratio, separate string output, channel cross-talk, and maximum output level?  As you know cross-talk between strings is pretty much a string/hex pickup physical phenomena, not a typical -120dBm separation op-amp issue.  Also, do you have a THD figure on a separate string with the Septar kit?  Some serious published figures and findings might wet forum users' appetites.

regards,

Bill Baxendale

scratch17

#12
Celemony has demonstrated polyphonic audio to MIDI conversion.

There is a video on their website detailing the work that can be done with it's new technology dubbed ARA (their term for a new API that they developed). The video (named "Showcase") can be found here:

http://www.celemony.com/cms/index.php?id=ara

At about the 8 minute mark of the video, the demonstrator selects an audio clip from the Melodyne Editor plug-in, running under ARA in Presonus' DAW (Studio 1 Version 2), and pastes it to a new MIDI track. Yes, you read that right. He selects an audio clip and pastes it to a MIDI track. This is done in real time, as if you were doubling a guitar track. However, the MIDI track has a synth sound assigned.

Playback is flawless. The MIDI track plays the synth sound as if the track was originally played on a MIDI controller. There are no glitches at all.

From the videos, it appears that ARA facilitates a real-time integration of tasks between Melodyne Editor and the Presonus DAW. There is no pre-rendering necessary for audio to be available in Editor, as it is in DAWs that do not yet support ARA. Celemony says that they expect to have other DAW manufacturers implement the ARA architecture. For now, it is only implemented in Studio 1 V2.

It is not clear as to whether the stand-alone version of Melodyne Editor will include the polyphonic audio to MIDI conversion function in the upcoming version 2 (to be released by Celemony on November 1).

Note that as an API, ARA is neither DAW nor plug-in specific. It is simply an interface that allows a greater level of communication between these programs. ARA implementation also has no bearing on the plug-in container format (VST, AU, etc.). I can see how certain types of plug-ins might be able to use awareness of pitch, tempo, and time signature, could do more than simply play or alter a sound.

For guitarists (who have little or no keyboard chops), this is excellent news.

Steven Kastner
Hamer Duotone, Brian Moore i213, Taylor 710 BCE 

VG-99, FC-300, RMC Fanout
RJM Mastermind GT10
Kemper Profiling Amp
Radial JDV Mk3, X-Amp
Mesa Recto Pre + 20/20
68 Fender Bandmaster (AB763)
Marshal AS80R

UA Apollo X6, Twin X, Logic Pro, Luna, Melodyne Studio

Elantric

#13
Great technology and on my "must have" list.

Of course this does nothing for live performance use, but its a valuable studio tool! (Melodyne Audio to MIDI Conversion is NOT Real time! )

Melodyne's feature set makes me want to dig out my old 1970's 1980's band tapes and rework / remix them to modern standards.


scratch17

#14
After doing some more research on the Celemony web site, I found that this technology is indeed included with Melodyne Editor, in either plug-in or stand-alone modes. The ARA technology makes workflow totally real time. Without it, you need to import the audio from your DAW and have editor render it into a format that can be used by Editor to do its stuff.

I plan to buy Editor tomorrow, so I get the free upgrade to version  2, which is to be released on November 1st. Version 2 includes Rewire support, so I'll be able to use it with Reason 5.

I may buy the crossgrade to Presonus Studio One V2 if I find that working with it in Logic Pro 9 is too slow.

Steven
Hamer Duotone, Brian Moore i213, Taylor 710 BCE 

VG-99, FC-300, RMC Fanout
RJM Mastermind GT10
Kemper Profiling Amp
Radial JDV Mk3, X-Amp
Mesa Recto Pre + 20/20
68 Fender Bandmaster (AB763)
Marshal AS80R

UA Apollo X6, Twin X, Logic Pro, Luna, Melodyne Studio

Elantric

#15
Cool!


Please report your hands on results with Melodyne- either good or bad.

paults

#16
I regularly use the version bundled with Pro Tools - it works really well for pitch and time massaging (I'm not calling it correction, because it doesn't sound like an old Cher single).  The upgrade to the full version is on sale until v2.0 comes out, and includes getting v2.0 when it arrives - projected to be in mid-November.

scratch17

#17
Wow, this thing really works!

I finally got over my cold and went to Guitar Center today. I got Melodyne Editor 1.x for $199 + tax. That's about $35 less than online retailers are selling it.

After I started the installer, it took me to a Celemony web page that started a download to update the program to its latest version (1.3). Activation was easy after I read the FAQ and figured out where my serial number was, and how to get it into the online activation system. I filled in the info required to create a Celemony user account and activation succeeded.

I fired up the program in stand-alone mode, and watched the first steps video. I'd already seen many of the online videos, but the more you see the use of this complex program, the easier it gets. For some reason I did not get the demo files loaded into my drive, and my forum activation awaits a confirmation email.

I wanted to try the save audio file as a MIDI file option right away. So I grabbed a WAV file of one of the tracks from an archived "Best of the Guess Who" quad reel tape that I already had on my drive. It was, I believe, the right front track from "These Eyes".

When Editor first came out with Direct Note Access, Celemony recommended that you not use complex, multi-instrument audio tracks, like a track from a fully built album. They suggested that you limit the editing to tracks with a single polyphonic instrument. The program also didn't support audio to MIDI file conversion at that time, but it was promised in the future.

So for me to use a track like this as a first test was pushing the envelope quite a bit.

I opened the WAV file in Editor. It took my Macbook Pro (quad i7) about a minute to load and render the audio. When done, I saw all of the notes as blobs, as expected. Now for the acid test. I went to the file menu, selected "Save AS" and chose MIDI file in the dialog that opened. Almost instantly a MIDI file appeared on my desktop. I was surprised at how fast this was.

Now I fired up Logic Pro 9, and opened the MIDI file. It loaded with a Grand Piano instrument voice.

I hit play and was amazed. It worked perfectly.

I now have a Grand piano playing the two or three parts that were originally vocals, guitar and electric piano mixed together into an audio track.

Then I loaded a drum WAV file and followed the same procedure. The Grand Piano played the drum!

Of course this program has many other equally awesome features. But this forum is dedicated to V-Guitar and guitar synthesis, so I'll direct you to the Celemony web site if you're interested any any of these other capabilities.

I've attached the MIDI file of the track with the polyphonic audio to MIDI conversion so you can check out the results for yourselves.

Note that the new retail of Editor when version 2 is released some time this month will be $399 or 399 Euro. A free upgrade to version 2 will be available if you buy the program before V2 is released.

Version 2 has some really nice time-based editing upgrades, one of which is potentially a new sound design tool. You can edit starting transients, and even transients within a note.

Micro-tonal support is also available for International styles that do not employ only true-tempered scales.

Finally, they will add Rewire support, so I'll be able to use Editor with Reason 5.

Very cool.

Steven Kastner

Hamer Duotone, Brian Moore i213, Taylor 710 BCE 

VG-99, FC-300, RMC Fanout
RJM Mastermind GT10
Kemper Profiling Amp
Radial JDV Mk3, X-Amp
Mesa Recto Pre + 20/20
68 Fender Bandmaster (AB763)
Marshal AS80R

UA Apollo X6, Twin X, Logic Pro, Luna, Melodyne Studio

Elantric

#18
I really enjoyed reading your review!

Also Celemony Melodyne does many other things beside Wave to MIDI - I have seen a demo at NAMM where  an old 1950's jazz quartet track had its melodic key center changed on the fly from major to minor, and autotuned, etc, etc,


"A pioneering technology that makes the impossible possible: edit individual notes within chords and polyphonic audio material! Yes, audio, not MIDI. The makers of Melodyne did it again ... www.celemony.com

See also this interview with Peter Neubaecker:"




audiotrax

#19
I recently got this software, and it's truly like magic from a distant Star Trek future :D
It's totally changed what I can do with my recording workflow, because now every audio track in my DAW has completely independant timing and pitch that can be molded around like a MIDI file.  As long as you don't stretch things around to ridiculous amounts, it's totally transparent. That is mind boggling when you think about it!  I've experimented a bit using Melodyne as a polyphonic guitar to MIDI converter, and so far it's working pretty good.  Not ready to give up my Roland hardware yet, but this seems to have a lot of promise - and no hex pickup is required.
Owner of: VG-88, GI-10, Cubase 5, Kontakt, SampleTank, var VSTI's, Roland JV1080.  Strat with GK-2A, two Roland GR500 analog guitar synths

aliensporebomb

#20
What amazed me was there was a demo I saw where there was an old recording of a band and they were totally killing it and the energy level was high.  Then someone hit a clam and an otherwise great take was not really usable.  But with this software they were able to take the clam and move a couple of the notes in a four note chord to their proper places and the take was usable again.  It's amazing technologically! 

It makes me wonder if it wouldn't be really cool for bizarro world remixes of existing material.

A year or two ago Sarah Hicks who is one of the associate conductors in the Minnesota Orchestra took an existing piece (I forget which one) and got idea to take it  in a new context then brought all the parts into Finale and transposed it into a totally different key then printed all the parts out and had the orchestra play a full movement in this new key.  It was like taking a familiar piece in a totally diffferent context - very strange.

So you could do that here easily.
My music projects online at http://www.aliensporebomb.com/

GK Devices:  Roland VG-99, Boss GP-10, Boss SY-1000.

septarboards

Hi Guys,

For those interested in the Pd abstractions, I will make some available in the new year via my website.

@Elantric - Thanks for the kind words. Glad the kit worked out for you.

We are now completely out of Septar boards with the aim to produce a superior model in the new year.

Happy New Year!

Ricky

clearlight

How much are these Septar boards?
My Music
My Band Website
GUITARS: 2x RG1521, 3x RG321 w/gk, Rg721 Fretless Modified, AmStd FatStrat w/gk, various others....
XV5050,Triton etc..
KOMPLETE 7
VGUITAR Stuff: VG99, FC300, RC5-

septarboards

£40. Unfortunately we are now completely sold out. Keep an eye on the website for new models sometime in 2012.

audiotrax

I have Melodyne.  What you are talking about is a neat idea in theory, but in my opinion after all the money, time and effort you put into doing that, I really don't think your going to be happy with the results.  You also for all your trouble will not be able to play your MIDI instruments in real time.  You would be much better of using a dedicated pitch to midi system (or You Rock style controller), and using to daw to edit out any glitches afterward.  Melodyne works well on interpreting audio data, but it's just not accurate enough to convert intricate passages into clean MIDI.  I thought it would be a good replacement for a hex system, but I ended going back to my GI10.
Owner of: VG-88, GI-10, Cubase 5, Kontakt, SampleTank, var VSTI's, Roland JV1080.  Strat with GK-2A, two Roland GR500 analog guitar synths