Roland_Boss - MIDI over USB Sysex Reverse Engineering

Started by vtgearhead, December 12, 2016, 12:43:41 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

vtgearhead

This is really annoying.  I was able to use USBPCap to sniff communication between Fender FUSE and the Mustang Amp.  However, both Robert and I have both found that it cannot see traffic between BTS and the Katana amp.  I downloaded a time-limited commercial package that can do the job, but it's much clunkier to use than Wireshark.  In particular, there is little control over export of the capture and I'll need to write an application to post-process the data (pulling raw data out of a number of columns and removing USB protocol bytes inserted by the driver).

If there anyone on the forum who can explain why it's proving so difficult to capture this communication?  This isn't going to stop progress, but it certainly doesn't make it fun.

admin

Have you tried Bomes Sendsx app (MIDI MONITOR) and see if the traffic is MIDI SYSEX?

gumtown

Not sure on the USB protocols, but these units use midi over USB to communicate,
I am currently extracting data between the GT-001 and BTS by using "Bomes SendSX", setting the Midi input to the same as the BTS USB/midi setting.
Although it won't read direct from the Unit, when BTS is activated, it places the Unit into 'verbose mode' which makes it repeat all commands it was sent, back to BTS which the SendSX program will read as a SYSX stream.

You could get tricky and use MidiOX and MidiYoke, by rerouting midYoke as a go between pipe between BTS and the amp, and set BTS to read the midiYoke port as being the amp. You can intercept 2 way midi traffic that way.
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead

Quote from: admsustainiac on December 12, 2016, 12:53:35 PM
Have you tried Bomes Sendsx app (MIDI MONITOR) and see if the traffic is MIDI SYSEX?

I can see the data from USBLyzer and it's most definitely sysex.  Although, if USBPCap did it's job it really wouldn't matter what protocol was being used.  I'm just trying to capture the raw bytes.

vtgearhead

#4
Quote from: gumtown on December 12, 2016, 12:55:23 PM
Not sure on the USB protocols, but these units use midi over USB to communicate,
I am currently extracting data between the GT-001 and BTS by using "Bomes SendSX", setting the Midi input to the same as the BTS USB/midi setting.
Although it won't read direct from the Unit, when BTS is activated, it places the Unit into 'verbose mode' which makes it repeat all commands it was sent, back to BTS which the SendSX program will read as a SYSX stream.
When I use BTS to switch presets the amp responds with a long series of 340 byte packets, which I believe represent all the state changes.  The original command may be in there.  Need to look closer.
Quote
You could get tricky and use MidiOX and MidiYoke, by rerouting midYoke as a go between pipe between BTS and the amp, and set BTS to read the midiYoke port as being the amp. You can intercept 2 way midi traffic that way.
That sounds interesting.  Would you expect this to work even on non-class-compliant devices like the Katana?  I'll investigate this approach.

Update:  MidiYoke hasn't been updated in years and will not install on Windows 7.  As far as I'm aware, Katana BTS won't run on anything earlier than Win7, so looks like I'm out of luck. 

Update2:  MidiOx installs on Windows 7, but cannot see communication between BTS and the amp since it is not using a MIDI device interface.

Update3: Bome SendSx actually can read data from the amplifier.  It may be tricky to make sense of it, though.  When I issue a simple command like decrement volume by 1, I can see two messages.  One the face of it, that isn't making much sense.  I'll check using USBLyzer to see which (if either) actually represents the outgoing command.

Will get done with USBLyzer, but won't be a lot of fun.

CodeSmart

I used USBLyzer for capturing FX-1 Sysex of the GT-1.
Tedious but works. Effort stopped due to preparing NZ traveling.
As said USBPCap won't capture Boss. Good luck friends.
But I got more gear than I need...and I like it!

whippinpost91850


CodeSmart

Thanks.
Flying to Auckland.
30 flying hours and 12h time zones away, and from North Pole to South Pole.
Stopping in Frankfurt and Singapore for a beer ;D


But I got more gear than I need...and I like it!

gumtown

#8
QuoteUpdate2:  MidiOx installs on Windows 7, but cannot see communication between BTS and the amp since it is not using a MIDI device interface.
The Katana USB is 'the midi device' modiOX, just like SendSX, will read not from the USB, but from where the data leaves the USB/midi device and enters the program/app.
BTS will still see the Katana as 'midi device'.
When I use midi yoke (configured with graphical routing via midiOX) You will see a list of midi inputs one side including the Katana/Boss USB device and 8 midi yoke virtual ports, and midi output devices the other side (including those as described above).
You then drag a link from the Katana input to a midi Yoke output (yoke1), then drag another connection line to the midiOX monitor (sometime you might need to use 2 yoke ports and zigzag the connections in series).
Then in BTS select Yoke1 as the midi device instead of the Katana, the data will pass through Yoke 1 and thru to the Katana, and you can monitor the data from Yoke 1.

I hope that makes some sense....  ;)

That is how I have been reverse engineering all the Boss GT and Roland gear for the last 10 years.
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

whippinpost91850

Sounds like a wonderful time. Hope you get to spend a little time with Gumtown.. Cheers

gumtown

Quote from: CodeSmart on December 12, 2016, 02:30:35 PM
Thanks.
Flying to Auckland.
30 flying hours and 12h time zones away, and from North Pole to South Pole.
Stopping in Frankfurt and Singapore for a beer ;D
Last time I landed in Singapore, I thought the plane was going to land in the middle of a city street.
Back then the airport had a smokers room, didn't need to light up, just inhale.  :P
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead


vtgearhead

Quote from: gumtown on December 12, 2016, 02:33:17 PM
When I use midi yoke (configured with graphical routing via midiOX) You will see a list of midi inputs one side including the Katana/Boss USB device and 8 midi yoke virtual ports, and midi output devices the other side (including those as described above).
I hope that makes some sense....  ;)

That is how I have been reverse engineering all the Boss GT and Roland gear for the last 10 years.

That looks very useful, but as I posted earlier MidiYoke will not install on Windows 7.  The installer gets terminated with prejudice towards the end of the process.

USBLyzer looks like the way to go.  Here's the basic procedure:

1. Set it to capture only 'URB Bulk or Interrupt Transfer' and start recording. 

2. Go through a specific set of adjustments on BTS (e.g. Volume to 0, then 100, etc.), keeping notes on what's been done. 

3. When a batch is complete, stop recording and filter out everything but outgoing data. 

4. Export to CSV format.

5. Post-process with a Perl script to remove and interpret USB/Midi event packet headers, leaving only the actual sysex command.

6. Correlate each command with the actions done in step 2

7. Repeat, rinse.

Here's what I have for the last 15 minutes of work:

** Patch Change

Panel:
F0 41 00 00 00 00 33 12 00 01 00 00 00 00 7F F7

One:
F0 41 00 00 00 00 33 12 00 01 00 00 00 01 7E F7

Two:
F0 41 00 00 00 00 33 12 00 01 00 00 00 02 7D F7

Three:
F0 41 00 00 00 00 33 12 00 01 00 00 00 03 7C F7

Four:
F0 41 00 00 00 00 33 12 00 01 00 00 00 04 7B F7

--------------------
** Amp Type

Acoustic:
F0 41 00 00 00 00 33 12 00 00 04 20 00 5C F7

Clean:
F0 41 00 00 00 00 33 12 00 00 04 20 01 5B F7

Crunch:
F0 41 00 00 00 00 33 12 00 00 04 20 02 5A F7

Lead:
F0 41 00 00 00 00 33 12 00 00 04 20 03 59 F7

Brown:
F0 41 00 00 00 00 33 12 00 00 04 20 04 58 F7

-----------------
** Gain

0 =
F0 41 00 00 00 00 33 12 00 00 04 21 00 5B F7
100 =
F0 41 00 00 00 00 33 12 00 00 04 21 64 77 F7

----------------
** Volume

0 =
F0 41 00 00 00 00 33 12 00 00 04 22 00 5A F7
100 =
F0 41 00 00 00 00 33 12 00 00 04 22 64 76 F7

----------------
** Bass

0 =
F0 41 00 00 00 00 33 12 00 00 04 23 00 59 F7
100 =
F0 41 00 00 00 00 33 12 00 00 04 23 64 75 F7

---------------
** Mid

0 =
F0 41 00 00 00 00 33 12 00 00 04 24 00 58 F7
100 =
F0 41 00 00 00 00 33 12 00 00 04 24 64 74 F7

---------------
** Treble

0 =
F0 41 00 00 00 00 33 12 00 00 04 25 00 57 F7
100 =
F0 41 00 00 00 00 33 12 00 00 04 25 64 73 F7

---------------
** Presence

0 =
F0 41 00 00 00 00 33 12 00 00 04 26 00 56 F7
100 =
F0 41 00 00 00 00 33 12 00 00 04 26 64 72 F7

gumtown

Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead

Quote from: gumtown on December 12, 2016, 05:09:29 PM
I'm using midiYoke and midiOX on Windows 7...
I wish I knew the secret for installing midiYoke.  It chugs away for a bit on the installation, then pops up a dialog telling me that the installer was terminated prematurely.  Tried several times.  Would you mind sending the package you're using?  Perhaps I have a broken version.

midiOX didn't present any problems.

gumtown

Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead

#16
Ok, the trick is to execute the installer from a command prompt that was started with "Run as Administrator".  It seems to be showing up in MidiOx, but I haven't a clue how to use it.  I can see all the inputs, outputs and what I think are the MidiYoke channels.  Would you mind giving me a quick step-by-step on how to wire things up?  Sorry to be dense, but it isn't very intuitive.

Update:  In particular, there is nothing in the MidiOx device setup dialog that responds to drag and drop. 

Elantric

#17
Elantric said: ↑
QuoteFollow VTGEARHEAD (snhirsh) for latest progress.

It's going to require different tool than a Helix, owning to lack of USB Host to MIDI support and a MIDI Command Interpterter.
And the fact Katana is NOT USB CLASS COMPLIANT (prevents IPad based remote control

But we are working on a solution
It may first appear as a firmware update to one of PrimovaSounds MIDX-20 Custom USB Host to MIDI I/O boxes but I figure not until spring 2017 at earliest time

http://www.primovasound.com/
Click to expand... 

Snhirsch wrote>
QuoteBeyond the issue of class-compliance (or lack thereof) is the fact that all the interesting software controls require sysex messages. At the present time, the only off-the-shelf solution would be to put a MidiSolutions Event Processor+ in front of a MIDX-10 or 20. You'd then have to program the Event Processor to map PC and CC messages to the appropriate raw sysex. Since the Event Processor doesn't know anything in particular about the Katana, this would be a rather tedious exercise of stringing raw hex bytes together. I'm envisioning a small Linux application that would run on a Beaglebone or RaspberryPi computer (philosophically similar to what I've done for the Fender Mustang). The application would know about the Katana and have a means for the user to reference operations by name when putting together a "scene" for the amplifier. Each scene could then be mapped to PC or CC. Nothing firm in mind, just a direction I want to head.

Codesmart (developer of MIDX-10 and 20) is on vacation through the end of January, but when he returns we intend to coordinate our efforts. I'll be working on a piece of opensource software that will be free if you want to set things up yourself. He will be working on an extension to one of his commercially available products. If possible, the capabilities and API will be the same.

I'm off from work the next few weeks and hope to pull together at least a proof-of-concept by the end of December.
Menu Unlike+ QuoteReply

vtgearhead

#18
Well, this is interesting.  All of us were of the belief that Katana was not MIDI class-compliant.  Perhaps I don't fully understand the notion of "class compliance", but Linux certainly thinks it's a MIDI device:

hirsch@z87:~$ amidi -l
Dir Device    Name
IO  hw:1,0,0  KATANA MIDI 1
IO  hw:1,0,1  KATANA MIDI 2
IO  hw:1,0,2  KATANA MIDI 3

I believe those devices correspond to 'KATANA', 'KATANA DAW CONTROL' and 'KATANA CONTROL'.  If this is really true it makes life one heck of a lot simpler.  Stay tuned.

I'm back...  The amp is definitely a MIDI device as far as Linux is concerned.  I wrote a five liner Python program that uses 'mido' to contact the amp and send messages.  It responds properly to PC.  Sysex up next.


Elantric

#19
Linux sees Roland /Boss gear typically due to "Jack Audio" rolled into the distro.

I understand Years ago the consortium of developers for  "Jack" Audio for Linux implemented support for Edirol / Roland  / Boss gear.

"Jack Audio" on RPi explains why Raspberry Pi works with Roland /Boss gear with just a USB cable - add a $18 USB <> MIDI cable and you are done:

( same strategy as Boss GP-10 Raspberry Pi USB <> Adapter detailed here:
https://www.vguitarforums.com/smf/index.php?topic=11998.0

http://www.jackaudio.org

vtgearhead

I read up on MIDI descriptors and dumped the Roland interface with 'lsusb'.  It definitely does not id itself as a MIDI interface.  Everything is "255 Vendor Specific".  In contrast, my M-Audio MidiSport has "MIDI" all through the device dump.

So, the plot thickens.  Jack audio (a userspace application) is not installed on my machine.  What I'm seeing is happening at the kernel level.  The udev subsystem is responding to the amplifier plug event by handing it off to the kernel MIDI driver.  Without digging through the kernel driver source code, I can only assume that one of the maintainers baked this in as special behavior.  Whatever the root cause, this is great news.  It means that I can use librtmidi to communicate with the Katana and don't have to reinvent the wheel by rolling my own low level I/O routines.   Should have some code working within a day or so.

gumtown

The Linux Roland/Boss midi is built into the Linux Kernel for about the last 3 years now.
Jack is a means of interfacing hardware with the kernel,
ALSA is also used in Linux for midi/audio.
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead

I love open-source software :-).  It took about 20 minutes to write Python code to send lists of sysex messages to the amp.  I want to see what the latency is for a modest sized set of commands.

Isn't Jack a user-space tool for routing between applications and kernel device drivers?  I wasn't of the impression that it handled matters down at the hardware level.

Elantric

#23
Quotebelieve those devices correspond to 'KATANA', 'KATANA DAW CONTROL' and 'KATANA CONTROL'.  If this is really true it makes life one heck of a lot simpler.  Stay tuned.


Among the three Katana MIDI devices you list, Roland states in the Boss Tone Studio Manual to use the "Katana" device for connection to BTS

My thought was review and document the MIDI data traffic that occurs to the "KATANA" MIDI Device while moving GUI controls in BTS

vtgearhead

#24
Quote from: Elantric on December 13, 2016, 05:50:08 PM
Among the three Katana MIDI devices you list, Roland states in the Boss Tone Studio Manual to use the "Katana" device for connection to BTS
Yes, that's the control device
Quote
My thought was review and document the MIDI data traffic that occurs to the "KATANA" MIDI Device while moving GUI controls in BTS
I've spent the past day doing exactly that.  Almost finished.  I see where there may be some confusion as to what I've been up to.  All the reverse engineering is being done under Windows 7.  I'm monitoring communication between BTS and the amp using USBLyzer and Bome SendSX.  Only USBLyzer is capable of reading outgoing communication to the amp.  Both can see return data.  About 95% of the commands are echoed back by the amp, so watching return traffic is a reasonable way to grab them.  Thanks to Mssr. Gumtown for suggesting SendSX - never would have thought to try it. 

Just took a break to write some test code after realizing that Linux presents the amp as a MIDI device.  Nothing too fancy at this point - just reads a list of commands from a text file and sends one after the other.  It does look like the response to sysex is very fast. 

My eyes are crossing from reading hex dumps :-).  Will get back to it tomorrow and see how some real-life "scene" changes behave under playing conditions.  Still hopeful that "virtual presets" will be practical.