Roland_Boss - MIDI over USB Sysex Reverse Engineering

Started by vtgearhead, December 12, 2016, 12:43:41 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

gumtown

#25
QuoteMy eyes are crossing from reading hex dumps :-).

is it starting to look like this??



Are you sure your reality is what you think it is ??   ;D
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead

Quote from: gumtown on December 13, 2016, 06:21:17 PM
is it starting to look like this??


I haven't seen anything exactly like that since the '60s, but yeah, staring at code or data dumps for a long period of time can really wear on you :-)

CodeSmart

Great work Snhirsch, considering you were not supposed to do this until your vacation begun  ;D
It's a pity I didn't have time to complete the GT-1. It would have been interesting to try to interface these two new devices as similar as possible, but with this warp speed you're probably done-done when I get back  :)
But I got more gear than I need...and I like it!

jwhitcomb3

It's been a while since I've played with sysex. Which bytes does Roland include in the checksum calculation?

sixeight

Quote from: jwhitcomb3 on December 14, 2016, 03:51:09 AM
It's been a while since I've played with sysex. Which bytes does Roland include in the checksum calculation?

Only address and data. See http://www.2writers.com/eddie/tutsysex.htm for details.

QuoteI'm monitoring communication between BTS and the amp using USBLyzer and Bome SendSX.  Only USBLyzer is capable of reading outgoing communication to the amp.  Both can see return data.

On Mac I use MIDI monitor, which is capable of spying on outgoing MIDI traffic.

vtgearhead

#30
Quote from: sixeight on December 14, 2016, 04:07:50 AM
Only address and data. See http://www.2writers.com/eddie/tutsysex.htm for details.
The explanation on that page is correct, but the author is clearly not a software guy.  Let's say our message is "F0 41 00 00 00 00 33 12 60 00 12 14 01 79 F7" (command to set Katana Reverb type to 'red').  Here's how that final '0x79' is derived:

# Take the address bytes (4) and data byte:
vals = [0x60, 0x00, 0x12, 0x14, 0x01]

# Sum with 7-bit wraparound
accum = 0
for val in vals:
    accum = (accum + val) & 0x7F

# Lower 7 bits of difference w/ 128 is the "checksum"
cksum = (128 - accum) & 0x7F

Send the resulting byte as the last piece of data.

vtgearhead

Quote from: CodeSmart on December 13, 2016, 10:27:40 PM
Great work Snhirsch, considering you were not supposed to do this until your vacation begun  ;D
It began last Friday evening - I'm not one to waste time when on a mission :-)
Quote
It's a pity I didn't have time to complete the GT-1. It would have been interesting to try to interface these two new devices as similar as possible, but with this warp speed you're probably done-done when I get back  :)
For some definition of "done-done", perhaps.  This work is going on two paths, depending on how burned out I get on either one:

** Reverse Engineering + Documentation:

I'm almost there on capturing BTS commands (you know how tedious this work is).  Next step is to understand the opening dialog between BTS and the Katana to get a handle on best way to capture entire state.

** Virtual preset performance testing:

There's now no question that individual parameters can be controlled in realtime.  Next question is to see whether I can broadly reconfigure the amp in a reasonable amount of time with a sequence of sysex commands.

My goal is to have an ability to capture and store the entire amp state and recall it from a PC command.  It might work something like this: 

  - User selects 'Panel' and manually (possibly with help of BTS) tweaks a setting they like
  - Disconnect BTS and connect my program
  - Issue a command to capture the state and store under given PC #

These PC #s (> 5) would then be "virtual presets" to be recalled by sending the PC to my application during performance.

vtgearhead

Here's a question for my fellow propellor-heads:

When communicating with Roland gear using sysex over USB, is it necessary to explicitly read return data from the unit under control?  On 5-pin MIDI it's obvious that one can ignore this stream (since it's inherently an unreliable medium where sender can never know directly whether data made it to the target or not).  With "raw" USB (as used for communication with my distinctly non-MIDI Fender Mustang) it is essential for client code to continually read in return data.  If not done, the return path buffers eventually block and the amp stops responding.  The Katana is recognized as a class-compliant MIDI device by Linux, which introduces another level of abstraction between client and USB session layer.  So, my question is: What will happen if I never read return data from the logical MIDI port?  Does it simply start discarding incoming or will this back up down to the USB physical driver level and stuff up communications?

sixeight

I haven't really played with USB midi, apart from using the stock libraries for Arduino and Teensy. Maybe this library will give you a clue:

https://github.com/FortySevenEffects/arduino_midi_library/blob/master/src/midi_UsbTransport.hpp

The USB stuff is really beyond me...

CodeSmart

Quote from: snhirsch on December 14, 2016, 07:45:40 AM
So, my question is: What will happen if I never read return data from the logical MIDI port?  Does it simply start discarding incoming or will this back up down to the USB physical driver level and stuff up communications?

To my knowledge it will start to discard when buffer is full. In my world it will not nuke the Boss/Roland thing if not reading. The Mustang was a completely different beast.
But I got more gear than I need...and I like it!

gumtown

With the most recent Boss gear (GT-100/001 onwards), they seem to have implemented a 'verbose mode',
using a subset of mystery commands with the address starting in the "7F" range.

In verbose mode, the device will send back data without specific data request commands being sent to the device.

Send a patch change, the device will return the patch data.

Set it to use 'Wave pedal' for an assign, the device will stream back the data changes in real-time.

Send a "7F" addressed command to enable the tuner, the device will stream back the tuner increments (with data in the "7F" address) as a note is played.

The returned USB sysx data does not require any acknowledgement and can be ignored.

Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead

Quote from: gumtown on December 14, 2016, 11:03:00 AM
The returned USB sysx data does not require any acknowledgement and can be ignored.

I knew it didn't require any specific acknowledgement, but was trying to establish whether I even needed to go through the motions of receiving to avoid a full buffer somewhere.  The Mustang (handled as a USB HID device - not MIDI) will eventually choke if an application on the host does not actively receive incoming data.  In this case the device blocks on USB transmit, not on expectation of a reply (one could argue that it should be smarter about handling the situation, but there we are).  I guess I'll have to experiment to see if it's a problem with Linux + Boss Katana.


vtgearhead

I'm pleased to report that the mind-numbing phase is complete.  I have captured the sysex messages from every control on BTS and documented them as concisely as possible.  There are still some mysteries overall.

Example:

Moving the BTS Gain knob sends values to address 00 00 04 21, but moving the physical knob on the amp causes it to send the corresponding new value at 00 00 04 21 along with a similar (but not identical) value at address 60 00 00 52.  They always seem to come in pairs.  I don't quite understand this.

One interesting experiment to try tomorrow:  It's possible that the amp has memory slots beyond the four programs accessible from the panel.  I'm going to try saving a snapshot into one of them and see if it can be recalled under software control.

carlb

#38
There is some sort of memory up to "channel" number 10. When accessing them via the Midi-Mouse there is definitely some sort of recall of prior settings up there ...

Not sure how to write to them, however.

Six and seven seem to have low output, but that's likely just where they were originally set (?). Eight, nine, and ten have normal output level and retain their past settings: tweak the knobs, change patch up or down and then back again, and the original settings are still there.

Interesting...

Any patch change request from 11 up to 128 doesn't seem to do much ...
ES Les Paul, internal Roland GK
Boss SY-1000, Valeton Coral Amp pedal
Morningstar MC8 & MC6
QSC CP8 powered speaker

Elantric

#39
QuoteThere are still some mysteries overall.

Example:

Moving the BTS Gain knob sends values to address 00 00 04 21, but moving the physical knob on the amp causes it to send the corresponding new value at 00 00 04 21 along with a similar (but not identical) value at address 60 00 00 52.  They always seem to come in pairs.  I don't quite understand this.

This might be related to an observation by some on the TGP Katana thread

They report when moving the "Booster / MOD" knob on the Katana, they see not just Boost gain but other parameters change as well on Boss Tone Studio. They asked if this "coupled" behavior of moving one physical Katana FX knob that changes multiple internal FX targets  can be ussr modified or changed

jwhitcomb3

Quote from: Elantric on December 15, 2016, 07:30:44 AM
This might be related to an observation by some on tyne TGP Katana thread

They report when moving the "Booster / MOD" knob on the Katana, they see not just Boost gain but other parameters change as well on Boss Tone Studio. They asked if this "coupled" behavior of moving one physical Katana FX knob that changes multiple internal FX targets  can be ussr modified or changed
Interesting.

I noticed that, depending on the algorithm selected, moving the physical effect knob on the amp can change one, two, or three parameters in the Boss Tone Studio. Does the number of parameter change messages correspond to the number of parameters controlled by the knob?

vtgearhead

Quote from: carlb on December 15, 2016, 07:10:57 AM
There is some sort of memory up to "channel" number 10. When accessing them via the Midi-Mouse there is definitely some sort of recall of prior settings up there ...

Not sure how to write to them, however.

Six and seven seem to have low output, but that's likely just where they were originally set (?). Eight, nine, and ten have normal output level and retain their past settings: tweak the knobs, change patch up or down and then back again, and the original settings are still there.

Interesting...

Any patch change request from 11 up to 128 doesn't seem to do much ...

I'm not able to reproduce that here.  There is simply no response to PC# > 4.  Do you see any change to the front panel LEDs when you select, e.g. PC #6? 

carlb

Nothing seen on the panel, it's what's coming out of the amp when a channel (patch) change command is sent from the Midi-Mouse. From six on up to ten, there's some different set of effects and gain settings on each "channel."

Not sure how you'd write to those channels, but they definitely "remember" their settings when you come back to them.

Very odd ... heh!
ES Les Paul, internal Roland GK
Boss SY-1000, Valeton Coral Amp pedal
Morningstar MC8 & MC6
QSC CP8 powered speaker

gumtown

#43
Have you tried data requests to the Katana to see if you have found all the patch data?

If you use SendSX, set the midi in/out ports to the Katana,
and paste this in the midi out screen
F0 41 00 00 00 00 33 11 60 00 00 00 00 00 11 00 0F F7  (is the katana product id 00 00 33 ?)

The above is used in the GT-001 as a patch data request (address from 60 00 00 00), I have changed the id from 06 to 33, they seem to share the same temporary patch address (same as GT-100, GT-001, GT-1).
The Katana should send as much patch data as it has, and you can then see if something is missing from the list.

And the same for any data in the SYSTEM area (address from 00 00 00 00)
F0 41 00 00 00 00 33 11 00 00 00 00 00 08 7F 00 79 F7

You could also try your luck at extracting anything you can find ay different addresses, below will request the maximum data size.
F0 41 00 00 00 00 33 11 00 00 00 00 7F 00 00 00 79 F7

the checksum will be incorrect, but the Boss gear has never been fussy on checksum for data requests.

Try different address ranges like
10 00 00 00 usually User patch's address start
20 00 00 00 usually preset patch's address start
30 00 00 00 Boss User quick patches (QFX) usually reside at this address
40 00 00 00 Boss Preset quick patches (QFX) usually reside at this address
7F 00 00 00 those weird BTS commands in this range
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead

#44
I haven't tried to do an exhaustive dump. I have looked closely at the conversation between BTS and amp at startup, when the application does number of disjoint reads (some quite large).  What threw me initially is the fact that the amp skips blocks of addresses.  So, the 'size' passed to the dump command sets an upper bound rather than an expected count?  The device is free to skip over chunks of the address space that have no parms?  Do I understand this correctly?

Updated:

Spent some quality time reading the GP-10 MIDI spec and I don't think the Katana parameter map is organized quite the same way.

All front panel rotary controls are here :
00 00 04 20 - 00 00 04 29

BTS asks for blocks of data at:
00 00 00 xx ??
00 00 01 xx ??
00 00 02 xx ??
00 00 03 xx ??
00 00 04 xx (front panel)

Are the other blocks user presets?

00 02 xx xx ??

No idea what happens here.

Almost everything else starts at 60 00 00 00 (efx, button assign, etc, etc.)


CodeSmart

Kia Ora. Out of topic. Proud my butt is still alive after 30hrs flight chair. Just landed. I'm in Kiwi land now and need to learn the Haka before I can join in with some clever Sysex ideas. See ya later guys ;D
But I got more gear than I need...and I like it!

sixeight

Quote from: CodeSmart on December 16, 2016, 05:32:25 AM
Kia Ora. Out of topic. Proud my butt is still alive after 30hrs flight chair. Just landed. I'm in Kiwi land now and need to learn the Haka before I can join in with some clever Sysex ideas. See ya later guys ;D

Enjoy your time in New Zealand. Will you be visiting Colin? You may be the person who has met the most vguitarforums members on the most continents soon...

carlb

Yah, jealous here! Always wanted to travel New Zealand. Relax, sleep-in, enjoy, try new things to eat, and explore. :)
ES Les Paul, internal Roland GK
Boss SY-1000, Valeton Coral Amp pedal
Morningstar MC8 & MC6
QSC CP8 powered speaker

whippinpost91850

Congrats on the safe journey and learn what ever your wife wants you to learn :)

gumtown

#49
Have a look at the GT-100 midi implementation, the Katana is possibly closer to that than the GP-10.
https://static.roland.com/assets/media/pdf/GT-100_GT-001_MIDI_Imple_e01_W.pdf



Quote from: carlb on December 16, 2016, 06:55:35 AM
Yah, jealous here! Always wanted to travel New Zealand. Relax, sleep-in, enjoy, try new things to eat, and explore. :)
Yip, we have Mc D's, KFC, Burger King, Carls jr, and if you want to get a bit metro, Starbucks.

Traditional feast is "hangi", or you could try "Puha and Pakeha"
<iframe width="200" height="113" src="https://www.youtube.com/embed/dBej8cZj43Q?rel=0" frameborder="0" allowfullscreen></iframe>
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/