Roland_Boss - MIDI over USB Sysex Reverse Engineering

Started by vtgearhead, December 12, 2016, 12:43:41 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

vtgearhead

Quote from: CodeSmart on March 13, 2017, 02:52:09 PM
You tell missis you need to buy a new amp to sort this out...or don't tell and hide it in underneath your working bench  ;D

And, then, if she spots it later on you can say "..Oh, that's not new, I've had it for a while Dear".  Well, it works for me most of the time.

gumtown

I won't be buying any new gear for a little while,
the missus is still getting over me buying a motorbike recently (Triumph speedmaster).

I just realised that the Katana midi sysx headers are longer than usual, and I was creating the wrong checksum.
I think I have that sorted....
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/


alancarl

Thanks Snhirsch for taking the time.
There's a lot to be learned in your documentation.
Do you by any chance use a script or a spreadsheet to calculate the Roland Checksum?
Thanks again,
Al

sixeight

Roland checksum is the same for all Boss/Roland devices. It is described here:

http://www.2writers.com/eddie/tutsysex.htm

Elantric

#155
QuoteRoland checksum is the same for all Boss/Roland devices. It is described here:
http://www.2writers.com/eddie/tutsysex.htm


just in case that site vanishes - see copy below:

ROLAND MIDI SYSEX CHECKSUM
----

HomeMIDIMIDI Tutorials
MIDI SysEx Tutorial

In an effort to bring System Exclusive (SysEx) messages to the masses, I embarked on a mission to produce this tutorial which will help people plumb the depths of this wonderful and powerful area of MIDI.

To understand SysEx messages you will need to have an understanding of hexadecimal.

The first part deals with the actual contents of the SysEx message and their function in the scheme of things.

The second part deals with the infamous Roland checksum.

By the end of this you may be disappointed that it is all quite easy after all!

System Exclusive (SysEx) Tutorial

Part 1 : The Anatomy of a SysEx Message (or Do this please!)

This discussion on SysEx is aimed at people using Roland equipment, but will put you in good stead to apply the knowledge to other makes of MIDI equipment.

The idea of SysEx is to change settings in a synth that cannot be accessed by any other means. Usually, anything that can be changed in a synth can be done via SysEx, but because of its unwieldy structure and length, the more common parameters can be accessed via controllers (like volume et al) and special events (like patch change, pitch bend et al). SysEx is the only means of retrieving data from a synth.

The Roland SysEx message is made up of nine parts. All notation is in hex, but without the trailing "h" for simplicity's sake.


[1]  [2]  [3]  [4]  [5]  [6]      [7]  [8]  [9]
F0   41   10   42   12   40007F   00   41   F7
I will discuss each part briefly before going into detail on the usage of SysEx messages.

Parts [1], [2] and [9] are part of the MIDI specification and are required by all SysEx messages. What is in between is specific to the manufacturer, identified by part [2], which is 41h in Roland's case.

Part [3] is known as the Device ID. Most Roland MIDI devices use the default of 10h, but is provided for us to change as we see fit. The idea behind the Device ID is that if you have more than one MIDI device of the same type in a daisy chain (connected to one another) you can change the Device ID on each of them so that you can send SysEx messages that will be accepted by only one of them, not all.

Part [4] is the Model ID. GS synths will all respond to SysEx with a Model ID of 42h, however they generally have their own Model ID as well.

Part [5] specifies whether we are sending (12h) or requesting (11h) information. If a synth receives a SysEx message it recognizes, it will look this part to determine whether it needs to change an internal setting or reply (with its own SysEx message) with the value of a setting.

Part [6] is the start address on which the SysEx intends to act. It is at this address that you may wish to put a value (or values) or retrieve the current value(s). It always contains three bytes. Most synth manuals will provide you with a long "address map" table which explains what lives at each address. Although daunting on a first perusal, once you understand its function it becomes a wonderful resource.

Part [7] has two functions. If part [5] is 12h (sending data) then part [7] contains the data we are sending and can be one byte or many bytes in length. If it is 11h (requesting data) then it is the number of bytes we want the synth to send us. I will expand on this later with examples.

Part [8] is the infamous Roland checksum which gets a whole section to itself in this tutorial.

That, in a nutshell, is what a SysEx message is made up of. If you're still a little hazy then don't be concerned. By way of examples you will see SysEx in action and will soon realize that there is no mystery to SysEx after all.

Example 1 : The GS reset. (or Sending a single byte.)


[1]  [2]  [3]  [4]  [5]  [6]      [7]  [8]  [9]
F0   41   10   42   12   40007F   00   41   F7
Open your text books...er...manuals to the address map for your synth. If it is a GS synth, run your finger down the left most column (which usually contains the address location) and find 40 00 7F. Different manuals describe it differently (how consistent!) but in essence it is the famous GS Reset. (The SC-88 manual describes it as "MODE SET 00=GS Reset")

What this message is telling the synth is "put the value 00h at address 40007F". What the synth does in this particular instance is perform an initialization.

Example 2 : Patch Change. (or Sending more than one byte.)


[1]  [2]  [3]  [4]  [5]  [6]      [7]   [8]  [9]
F0   41   10   42   12   401100   0801  26   F7
We can send more than one byte of data in a single SysEx message. This is very useful when we want to set the value of several settings that follow each other in the address map. By sending more than one byte in part [7], which is the data part, the first byte will be put in the address specified and the following bytes in each successive address location. Each address location can only hold one byte of data.

In the example above we are putting the value 08h into address 401100h and the value 01h into address 401101h. Think about it this way, if we send more than one byte to an address then each successive byte will be put in the address below the previous one in the map in the manual.

For the above example, the SC-88 manual has the following in the address map:

Address(H)  Size (H)  Data (H)  Parameter     Description
----------  --------  --------  -----------   -----------------
40 1x 00    00 00 02  00 - 7F   Tone Number   CC#00 Value 0-127
40 1x 01#             00 - 7F                 P.C. Value 1-128
A quick time out here to explain the manual notation:

1. x in the address represents the synth part number, 1 though F.
      Thus x=1 for part 1,
      hence 40 11 00 is the start address for part 1.
2. # in the address means that we cannot send data to this address, we
      must send multiple bytes to a previous address that does not have a #.
3. CC#00 is the GS variation number.
4. P.C. is the regular patch change.
Now you can see how 08h goes to the address specified in the SysEx message, and 01h goes to the next address.

For those with an external Sound Canvas with a display, if you successfully send this example to your synth you will see the patch name "Piano 2w"!

If there is anything you do not understand, please e-mail me and try as best possible to explain what confuses you and what part (if not all) you still don't understand. I have discovered that the way things are explained can make perfect sense to one person, and absolute nonsense to another. It's all to do with the words used and the order of the explanation. (Thank goodness Yoda gave self-defense lessons and not English lessons!)

Example 3 : Receiving data. (or What are you thinking?)


[1]  [2]  [3]  [4]  [5]  [6]      [7]      [8]  [9]
F0   41   10   42   11   401100   000002   2D   F7
To find out what value(s) your synth has in an address (or range of addresses) we use the 11h value in part [5].

Part [6] identifies the start address that we're interested in.

Part [7] is now an indication of the number of bytes we want back, as opposed to being the data we sent to the synth when using 12h in part [5]. This part must always contain three bytes (six hex digits).

That's it. The synth will respond with a SysEx message containing the value(s) we asked for.

This example is the converse of example 2. Instead of setting the Varaition/Patch Change, we are asking the synth to tell us what the current setting is. The synth's response is in the format of a SysEx message. The message will be exactly what could have put those values in those addresses in the first place!
In other words, if you successfully send "example 2" to the synth and then send "example 3" to the synth, then the contents of the synth's response will be the bytes that make up "example 2"!!

There should be more examples in your manual which you can now try. Try and predict the result before you send the example. If the result confirms your prediction then your SysEx world has just become a universe!

System Exclusive (SysEx) Tutorial

Part 2 : The Roland Checksum (or Did I get it right?)

The idea behind the checksum is simple. When we send a SysEx message to a synth, we want it to do only what we ask and nothing else. Consider the real world where messages don't always reach their destination in their original form. The results could be disasterous.

If a SysEx message gets corrupted during transmission to the synth, it is entirely possible that it could become a totally different (but valid) message! So a reverb change could conceivably become a GS Reset!!
To avoid this, Roland uses a checksum that is calculated using the address and data parts of the SysEx message, thus ensuring that if the address or data is corrupted somehow, they will no longer match the checksum and the SysEx message will be discarded.
(For the cynics out there: Yes it is conceivable that the address/data and the checksum could be corrupted in such a way that a different, valid message is formed. However, the likelyhood is far smaller than with no checksum present.)

Calculating the checksum follows a simple formula which is easily learned and applied. Following this formula will gaurantee correct checksums:

1. Convert the hex values to decimal.
2. Add the values together, but if the answer to any sum exceeds 127 then subtract 128.
3. Subtract the final answer from 128.
4. Convert the resulting value back to hex and use as the checksum.

Here is a practical example:


[1]  [2]  [3]  [4]  [5]  [6]      [7]   [8]  [9]
F0   41   10   42   12   401100   4163  0B   F7
Remember the checksum is only calculated from the address [6] and the data [7], thus we need only look at 40 11 00 41 63 in this example.

1. Convert hex to decimal:
   40h = 64
   11h = 17
   00h = 0
   41h = 65
   63h = 99

2. Add values, keeping result under 128:
   64 + 17 = 81
   81 + 0 = 81
   81 + 65 = 146 ( 146 - 128 = 18 )
   18 + 99 = 117

3. Subtract answer from 128
   128 - 117 = 11

4. Covert to hex:
   11 = 0Bh
It's that simple! Part [8] is the checksum and is indeed 0Bh in this instance.

The manual shows a different method which involves slightly more complicated arithmatic. If you prefer this method then by all means use it.

In essence it involves adding all the values together, dividing by 128 and subtracting the remainder from 128. Thus:

1. Convert hex to decimal:
   40h = 64
   11h = 17
   00h = 0
   41h = 65
   63h = 99

2. Add values:
   64 + 17 + 0 + 65 + 99 = 245

3. Divide by 128
   245 / 128 = 1 remainder 117

4. Subtract remainder from 128
   128 - 117 = 11

5. Covert to hex:
   11 = 0Bh
If you know for a fact that I have made any mistakes above, please let me know.

Feed back on clarity and semantics will also be greatly appreciated.

Thanks, Eddie Lotter

alancarl

Thanks Steve and Six 8 for the info.I appreciate the careful explanations of the anatomy of SYSEX messages..I have learned a few things about SYSEX in the last 2 days that clear up a lot of confusion for me:-)

I do know the formulaae for calculating the Roland Checksum...but I'm just looking for an automatic way to calculate them..I don't use spreadsheets hardly ever but a few friends do so I am going to ask and see what I can learn.
I know it sounds a bit lazy but I'm just trying to find a way that requires the least amount of typing and mousing...I usually have the urge to delve into these sort of projects right after a big gig and hand and arm usage is really important to monitor!

Thanks again Steve and 6 Eight!
Al

Elantric

#157
Gumtown already explained a lot of the Roland MIDI Protocol in these threads here

https://www.vguitarforums.com/smf/index.php?topic=3205.25

https://www.vguitarforums.com/smf/index.php?topic=3348.msg24880#msg24880
-----
Here is the Boss GT-10 midi implimentation document, which describes the way things work
https://static.roland.com/assets/media/pdf/GT-10_MI.pdf


Most GR-55 paramaters are 1 byte in size, except for things like
Synth portamento = 2 bytes
patch master level = 2 bytes
BPM temp = 2 bytes
Assign Target & Min & Max values = 3 bytes each
plus the Assign min & max values are centered at a value of 1024, and - values are subtracted and + values are added to that figure.
Some Rate values = 3 bytes
Most Delay times = 5 bytes (one to select bpm/time, one for bpm values and 3 bytes for time value, each byte only uses LSB nibble and is worked mostly on 8 bit values as opposed to typical 7 bit midi spec values)
The GR-55 has a very complex midi specification.
https://www.vguitarforums.com/smf/index.php?action=dlattach;topic=3348.0;attach=10864
The midi spec on other devices could be very different, i have found many other Roland products do not allow realtime sysex data control, typically most pre-2005. The GR-20 is one example of that, no realtime control built into the midi spec.

vtgearhead

FWIW, from my reverse-engineered Katana API document:

https://github.com/snhirsch/katana-midi-bridge/blob/master/doc/katana_sysex.txt

[Checksum Algorithm]

Let's say our message is:

F0 41 00 00 00 00 33 12 60 00 12 14 01 79 F7

(command to set Katana Reverb type 'red').  Here's how that final
'0x79' was derived (in Python syntax):

# Take the address (4 bytes) and data (1 byte) portion:
vals = [0x60, 0x00, 0x12, 0x14, 0x01]

# Sum with 7-bit wraparound
accum = 0
for val in vals:
    accum = (accum + val) & 0x7F

# Checksum is lower 7 bits of the difference w/ 128
cksum = (128 - accum) & 0x7F

I have no idea how one would do this in spreadsheet macros - or even if it's possible.

Elantric

Snhirsch wrote>

As I and others continue our dive into the Katana sysex API, it's becoming more and more apparent that Katana borrows or outright reuses most of the GT100/GT001 specification.  Robin Van Boven has been doing some deep poking around and pointed out that the panel gain knob ends up tweaking an address mapped to preamp gain in the GT001 spec.  This implies strongly that the GT preamp modeling is included in Katana, but not directly exposed in BTS.  BTS abstracts it behind the front-panel knobs, but it does respond to direct setting.  There are very likely to be a bunch of undocumented effects and models in the Katana.  Should be fun exploring this.

Beanow

#160
Quote from: Elantric on April 06, 2017, 08:18:51 AM
Snhirsch wrote>

As I and others continue our dive into the Katana sysex API, it's becoming more and more apparent that Katana borrows or outright reuses most of the GT100/GT001 specification.  Robin Van Boven has been doing some deep poking around and pointed out that the panel gain knob ends up tweaking an address mapped to preamp gain in the GT001 spec.  This implies strongly that the GT preamp modeling is included in Katana, but not directly exposed in BTS.  BTS abstracts it behind the front-panel knobs, but it does respond to direct setting.  There are very likely to be a bunch of undocumented effects and models in the Katana.  Should be fun exploring this.


I have found the other sneaky amps indeed, had just written this post about it before reading this. https://www.vguitarforums.com/smf/index.php?topic=20609.0

gumtown

So I have just had an "Ah Ha !! " moment,
when I have just realised that I can run Katana FxFloorBoard sending on MidiYoke 2 and Katana Tone Studio receiving on MidiYoke 2,
and have Tone Studio recognise the parameters changes made in FxFloorBoard in real time.

Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead

I would really appreciate a step by step on how to set that up.

gumtown

Quote from: snhirsch on April 23, 2017, 06:02:07 PM
I would really appreciate a step by step on how to set that up.

Install midiyoke (I use win 7 64 and the Yoke Nt version)
http://www.midiox.com/index.htm?http://www.midiox.com/myoke.htm

When you use any midi software, it should now show 8 virtual midi Yoke ports (8in/8out).
The in's and out's are connected, so a midi software sending on Yoke 1 will allow another midi software to receive on Yoke 1.
There are other windows midi connection software, but I prefer midiyoke as it is system resident.

Open Katana Tone Studio and set to offline mode, then from the System > Device menu, select a midiyoke port to receive on.

Open another midi software (KatanaFxFloorBoard in this case in debug mode) and select the same midiyoke port to send on.

Changing the Katana FxFloorBoard parameters will now show the relevant changes in the Tone Studio. (Tone studio recognises sysx parameter data input).
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead

Thanks, Colin.  I'll try MidiYoke again.  Last time around I could not get it to do much of anything.

vit3k

#165
In katana midi bridge you use addresses starting from 96. In spec they're different. Where do they come from?

And I have one question: is it possible to switch boost and mod with sysex different way than setting the knob value?

vtgearhead

Quote from: vit3k on August 11, 2017, 02:52:14 PM
In katana midi bridge you use addresses starting from 96. In spec they're different. Where do they come from?

Can you rephrase that question?  I don't understand what you are asking.

Quote
And I have one question: is it possible to switch boost and mod with sysex different way than setting the knob value?

Yes.  If using sysex communication, do not target the knob.  Instead, control all (5) effects (boost,mod,delay,fx,reverb) individually and have any or all of them active at the same time.

gumtown

Address 96 is decimal for the hexadecimal address of 60 00 00 00, which is the Katana "temporary scratch pad memory", or the Katana's unsaved or current patch data (what you hear), any loaded patch/channel is copied to this address range for DSP processing.

Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vit3k

QuoteAddress 96 is decimal for the hexadecimal address of 60 00 00 00

Sorry for late response.
Wow, I feel stupid now ;D Thanks for clarification.

QuoteYes.  If using sysex communication, do not target the knob.  Instead, control all (5) effects (boost,mod,delay,fx,reverb) individually and have any or all of them active at the same time.

Thanks I will check this out.

gumtown

#169
At the moment analysing the Katana Firmware 2 chain,
I didn't notice before, the specific order which the Katana lays out the effects.
It still uses the GT-100 chain positions.

    // default chain = 11     0F     05    02   0D   0C    04    01    07    06   08    09    0E   12           00    03     0B   0A   13     10
     //fx name =      cn-s    od     fx1 ch_a  ns1   fv    eq     lp    dd    fx2   ce     rv   ns2  cn_m1_s2 cs   ch_b   pdl  fx3 cn_m usb

cn-s = chain split, channel A starts
cn_m1_s2 = chain merge 1/split2 (channel A finishes/channel B starts)
cn_m = chain merge (channel B finishes)
ch_a = preamp (channel A)
ce = Delay 2 (ex Chorus in GT-100)
od = Booster
fx1 = Mod effects
ns1 = noise suppressor
dd = delay 1
rv = reverb

and the unused GT-100 effects
ch_b = preamp channel B
cs = compressor
fx3 = Accel effects
pdl = pedal Fx


this part is the Katana (firmware 2) effects
0F     05    02   0D   0C    04    01    07    06   08    09
and the rest remains static around it.

Note how the NS2 noise suppressor seems to be included in the Katana channel A string,
question is, could NS2 be active?

In the previous Katana FxFloorBoard, the NS2 had been placed in the channel B string (just after cn_m1_S2).

This might explain some chain order inconsistencies.


Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead

Quote from: gumtown on October 07, 2017, 12:42:22 AM
At the moment analysing the Katana Firmware 2 chain,
I didn't notice before, the specific order which the Katana lays out the effects.
It still uses the GT-100 chain positions.

    // default chain = 11     0F     05    02   0D   0C    04    01    07    06   08    09    0E   12           00    03     0B   0A   13     10
     //fx name =      cn-s    od     fx1 ch_a  ns1   fv    eq     lp    dd    fx2   ce     rv   ns2  cn_m1_s2 cs   ch_b   pdl  fx3 cn_m usb

cn-s = chain split, channel A starts
cn_m1_s2 = chain merge 1/split2 (channel A finishes/channel B starts)
cn_m = chain merge (channel B finishes)
ch_a = preamp (channel A)
ce = Delay 2 (ex Chorus in GT-100)
od = Booster
fx1 = Mod effects
ns1 = noise suppressor
dd = delay 1
rv = reverb

and the unused GT-100 effects
ch_b = preamp channel B
cs = compressor
fx3 = Accel effects
pdl = pedal Fx


this part is the Katana (firmware 2) effects
0F     05    02   0D   0C    04    01    07    06   08    09
and the rest remains static around it.

Note how the NS2 noise suppressor seems to be included in the Katana channel A string,
question is, could NS2 be active?

In the previous Katana FxFloorBoard, the NS2 had been placed in the channel B string (just after cn_m1_S2).

This might explain some chain order inconsistencies.

If you can get chain ordering to persist in a stable manner it will be great.  I was never able to achieve consistent results using FxFloorboard. 

gumtown

#171
Attached is a Katana .tsl patch in a text format.
This is a Tone Studio Firmware 2 patch, which has the Delay 2, and new effects in it,
but...
the extra patch data with the cryptic pedal assigns is not contained in the patch data ??
Not sure what is going on with the Katana there, like there was a half finished plan to use extensive assigns,
but never made it to the Tone Studio patch format.
What does the extra 600 bytes of patch data do if it is not saved in a patch file. ?

Attached is also the address map with the above assigns
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

CodeSmart

Quote from: gumtown on October 10, 2017, 03:23:41 AM
Not sure what is going on with the Katana there, like there was a half finished plan to use extensive assigns,
but never made it to the Tone Studio patch format.
What does the extra 600 bytes of patch data do if it is not saved in a patch file. ?

Yes, it's weird...I also find this very strange.
But I got more gear than I need...and I like it!

vtgearhead

Quote from: CodeSmart on October 10, 2017, 03:06:27 PM
Yes, it's weird...I also find this very strange.

Perhaps it's considered to be part of the persistent amp state and not considered to be patch dependent? 

Elantric

Quote from: snhirsch on October 11, 2017, 06:30:31 AM
Perhaps it's considered to be part of the persistent amp state and not considered to be patch dependent?

Agreed!

Most functions that are  accessible via the "SYSTEM" link (on lower right in BTS) is a global  - unless otherwise stated as having a "Patch" option". )