Author Topic: Boss MS-3 - MIDI over USB Sysex Reverse Engineering  (Read 1022 times)

0 Members and 1 Guest are viewing this topic.

Offline gumtown

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #50 on: September 29, 2017, 03:28:33 AM »
Data retrieval is quite straight forward.

F0 41 00 00 00 00 3B 11 60 00 XX XX 00 00 ZZ ZZ CS F7"
where
F0 41 00 00 00 3B = usual boring sysx header stuff
11 = data request mode
60 00 XX XX = start address of temp data block
00 00 ZZ ZZ = data size required from start address
CS = checksum
F7 = end of message

You might find some useful Arduino code here
https://github.com/gumtown/V-tone

A 12 knob box with 3 lcd displays



I also started on a MKii version using rotary encoders, but couldn't get the Arduino to scan 12 encorders



 
« Last Edit: September 29, 2017, 03:35:23 AM by gumtown »
Free "GR-55 FloorBoard" editor software from http://sourceforge.net/projects/grfloorboard/

Offline MrHaroldA

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #51 on: September 29, 2017, 03:36:21 AM »
Tnx! But I suspect that to be asynchronous, and the Arduino library uses a poller in it's examples to capture SysEx data. I have to code in some basic functionality first, and then I'll try to query the MS-3 for the active patch number, FX state, and maybe even some settings like 'switch on push/release'.

Offline sixeight

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #52 on: September 29, 2017, 04:51:49 AM »
Tnx! But I suspect that to be asynchronous, and the Arduino library uses a poller in it's examples to capture SysEx data. I have to code in some basic functionality first, and then I'll try to query the MS-3 for the active patch number, FX state, and maybe even some settings like 'switch on push/release'.

That is correct. Most of the data has to be "pulled out" of the MS-3. I do the same with my VController project for patch names and effect states.

Anything you change on the MS-3 should be sent in sysex automatically once editor mode is on. This way the BTS for MS3 keeps track of changes on the unit itself.

A good tutorial for the Roland midi system is http://www.2writers.com/eddie/tutsysex.htm

Offline snhirsch

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #53 on: September 29, 2017, 06:05:44 AM »
00 00 ZZ ZZ = data size required from start address

One word of warning:  This number is actually the address offset from start to the last byte you want to receive.  Depending upon your choice of starting address and offset you may get fewer bytes back than a simple subtraction would suggest. 

Update: When reading known parameters from their proper address you can ignore this distinction.  However, keep it in mind when doing bulk reads or you'll experience modest grief.  The MIDI sysex address range has discontinuities after each 128-byte page due to the need for data bytes to have bit 7 clear,  but the missing region is still accounted for when determining where to cut off a read.

« Last Edit: September 29, 2017, 07:39:38 AM by snhirsch »

Offline MrHaroldA

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #54 on: September 29, 2017, 06:21:44 AM »
Are the query addresses the same as the set addresses? For example, if I set FX1 at '0x60, 0x00, 0x00, 0x30', can I read it at that address too?

Offline sixeight

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #55 on: September 29, 2017, 07:30:34 AM »
Are the query addresses the same as the set addresses? For example, if I set FX1 at '0x60, 0x00, 0x00, 0x30', can I read it at that address too?

Yes you can. You will have to specify the number of bytes you want to read, so '0x60, 0x00, 0x00, 0x30 0x00 0x00, 0x00, 0x01' will read one byte from your address.

For the GP10 I have used the following code for reading data from the device:
Code: [Select]
void request_GP10(uint32_t address, uint8_t no_of_bytes) {
  uint8_t *ad = (uint8_t*)&address; //Split the 32-bit address into four bytes: ad[3], ad[2], ad[1] and ad[0]
  uint8_t no1 = no_of_bytes / 128;
  uint8_t no2 = no_of_bytes % 128;
  uint8_t checksum = MIDI_calc_Roland_checksum(ad[3] + ad[2] + ad[1] + ad[0] +  no1 + no2); // Calculate the Roland checksum
  uint8_t sysexmessage[18] = {0xF0, 0x41, GP10_device_id, 0x00, 0x00, 0x00, 0x05, 0x11, ad[3], ad[2], ad[1], ad[0], 0x00, 0x00, no1, no2, checksum, 0xF7};
  MIDI.sendSysEx(17, sysexmessage);
}

// Calculate the Roland checksum
uint8_t MIDI_calc_Roland_checksum(uint16_t sum) {
  uint8_t checksum = 0x80 - (sum % 0x80);
  if (checksum == 0x80) checksum = 0;
  return checksum;
}

Offline MrHaroldA

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #56 on: October 01, 2017, 02:06:15 PM »
I've added support for loading, editing and storing "FX sets"! :D Next up is actually reading something from the MS3 ...



But first a beer and a bath.
« Last Edit: October 01, 2017, 02:17:01 PM by MrHaroldA »

Offline MrHaroldA

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #57 on: October 05, 2017, 01:48:48 AM »
Next up is actually reading something from the MS3 ...

Yeah ... this is the tricky part.

If I query the MS-3, I get an answer. But if I don't query it, I can't get the Arduino to receive anything. For example: patch changes are sent by the MS-3 without having to query it, but I can only receive it by querying it with some random query.

Running my poller outside of USB_STATE_RUNNING makes no difference.

Code: [Select]
if (Usb.getUsbTaskState() == USB_STATE_RUNNING)

I'll try to find some examples of other implementations ... or do you guys have some ideas to try?


EDIT: I also get bundled replies when querying while switching patches:

Code: [Select]
Query: F0, 41, 0, 0, 0, 0, 3B, 11, 60, 0, 0, 30, 0, 0, 0, 1, 6F, F7: 0
Receiving: 20:  04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 00 30 07 00 70 F7
Query: F0, 41, 0, 0, 0, 0, 3B, 11, 60, 0, 0, 30, 0, 0, 0, 1, 6F, F7: 0
Receiving: 64:  04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 00 30 07 00 70 F7 04 F0 41 00 04 00 00 00 04 3B
12 00 04 01 00 00 04 00 00 7F 05 F7 00 00 04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 06 53 07 51 76 F7
Query: F0, 41, 0, 0, 0, 0, 3B, 11, 60, 0, 0, 30, 0, 0, 0, 1, 6F, F7: 0
Receiving: 20:  04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 00 30 07 00 70 F7
« Last Edit: October 05, 2017, 02:04:47 AM by MrHaroldA »

Offline sixeight

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #58 on: October 05, 2017, 02:22:41 AM »
If I query the MS-3, I get an answer. But if I don't query it, I can't get the Arduino to receive anything. For example: patch changes are sent by the MS-3 without having to query it, but I can only receive it by querying it with some random query.

Sorry, you lost me here.
Does the MS-3 not send any data when you change a parameter on the unit? That would mean it is not properly in editor mode.
Or is the problem with receiving MIDI data in general. It is normal that you have to check the USB handler for new data.

Offline MrHaroldA

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #59 on: October 05, 2017, 04:37:54 AM »
Sorry, you lost me here.
Does the MS-3 not send any data when you change a parameter on the unit? That would mean it is not properly in editor mode.
Or is the problem with receiving MIDI data in general. It is normal that you have to check the USB handler for new data.

If I sniff the Midi with my PC, the MS-3 transmits the patch changes correctly; so the editor mode must be set correctly too.

But if I poll the USB from my Arduino, it never registers that patch change; unless I'm querying it at the same time. The patch change joins the queried data in one big message.

In my current project, all I need is the patch change to load a different set of FX combinations. Querying the effect state would be a bonus to select which FX set matches, and should be activated once a patch is loaded.

Offline gumtown

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #60 on: October 05, 2017, 04:53:32 AM »
Arduino interrupt on midi input?
Free "GR-55 FloorBoard" editor software from http://sourceforge.net/projects/grfloorboard/

Offline sixeight

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #61 on: October 05, 2017, 06:19:27 AM »
Very hard to help you without seeing your code. Have you seen the code of vit3k:

https://github.com/vit3k/katana_fx_controller

The bit about Katana MIDI:
https://github.com/vit3k/katana_fx_controller/blob/master/src/katana/midi.cpp

Offline philjynx

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #62 on: October 05, 2017, 08:01:06 AM »
Does your code check for incoming USB data continuously? If it doesn't and only checks after each of your sends, that would explain why you're not getting the info you want.
I am playing all the right notes. But not necessarily in the right order.

Offline MrHaroldA

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #63 on: October 05, 2017, 01:03:34 PM »
Very hard to help you without seeing your code. Have you seen the code of vit3k:

https://github.com/vit3k/katana_fx_controller

That is in fact very helpful!!! I stripped away all Katana-specific stuff and ended up with basically main.cpp and the katana class. I'll have to study the midi class some more, but I already spotted some things I didn't know before. Also the queue seems like a very good idea!

But ... it seems to have a memory leak? Every 'katana.set' I do, I lose around 17 bytes of RAM, until it runs out.


Does your code check for incoming USB data continuously? If it doesn't and only checks after each of your sends, that would explain why you're not getting the info you want.

Yes, I check constantly ... I have to run my code against that of the Katana FX Controller to see what I'm doing wrong.

Offline philjynx

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #64 on: October 05, 2017, 01:23:50 PM »
That is in fact very helpful!!! I stripped away all Katana-specific stuff and ended up with basically main.cpp and the katana class. I'll have to study the midi class some more, but I already spotted some things I didn't know before. Also the queue seems like a very good idea!

But ... it seems to have a memory leak? Every 'katana.set' I do, I lose around 17 bytes of RAM, until it runs out.


Yes, I check constantly ... I have to run my code against that of the Katana FX Controller to see what I'm doing wrong.

Without looking at the code, my guess would be that 'katana.set' assigns some memory that is not subsequently released.
I am playing all the right notes. But not necessarily in the right order.

Offline MrHaroldA

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #65 on: October 05, 2017, 01:58:20 PM »
Without looking at the code, my guess would be that 'katana.set' assigns some memory that is not subsequently released.

Nope: it's in the receive part ;)

Code: [Select]
Initialized
Free RAM: 1056
Sending F0 41 00 00 00 00 3B 12 7F 00 00 01 01 7F F7
Sending F0 41 00 00 00 00 3B 12 60 00 00 30 00 70 F7
Received: F0 41 00 00 00 00 3B 12 00 01 00 00 00 01 7E F7 00
*** Memory usage changed: 933
Received: F0 41 00 00 00 00 3B 12 60 00 06 53 64 63 F7
Received: F0 41 00 00 00 00 3B 12 00 01 00 00 00 00 7F F7 00
*** Memory usage changed: 895
Received: F0 41 00 00 00 00 3B 12 60 00 06 53 64 63 F7
...

I'm beginning to suspect timing issues with my MS3. It fails to boot into editor mode occasionally, misses FX states sometimes when sending bulk messages ... I'll check the SysEx trace for the timings the Editor uses.


Edit: note the extra 00 suffix on the patch changes received from the MS3 ...
« Last Edit: October 05, 2017, 02:17:58 PM by MrHaroldA »

Offline gumtown

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #66 on: October 05, 2017, 03:46:48 PM »
You will need at least 20ms wait between sending sysx data,
that is the minimum required by midi spec for a receiving device to digest data,
and also determine the gap between the last message and the next.

If you are expecting a reply, only send one message at one time.
Free "GR-55 FloorBoard" editor software from http://sourceforge.net/projects/grfloorboard/

Offline snhirsch

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #67 on: October 11, 2017, 07:26:28 AM »
You will need at least 20ms wait between sending sysx data,
that is the minimum required by midi spec for a receiving device to digest data,
and also determine the gap between the last message and the next.

If you are expecting a reply, only send one message at one time.

I was not aware of the 20ms requirement for message separation.  Explains a lot of weirdness I experienced when working on my Katana bridge.

Offline MrHaroldA

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #68 on: October 11, 2017, 07:40:59 AM »
I measured around 4ms delay between events in the Editor software... And 60ms after setting up the Editor mode. This sure made things way more reliable!

Offline vit3k

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #69 on: October 20, 2017, 02:01:22 PM »
That is in fact very helpful!!! I stripped away all Katana-specific stuff and ended up with basically main.cpp and the katana class. I'll have to study the midi class some more, but I already spotted some things I didn't know before. Also the queue seems like a very good idea!

But ... it seems to have a memory leak? Every 'katana.set' I do, I lose around 17 bytes of RAM, until it runs out.


Yes, I check constantly ... I have to run my code against that of the Katana FX Controller to see what I'm doing wrong.

It definitely has a memory leak somewhere because it restarts itself after a while. I don't have time to work on this project right now but if you found out where it is I would really like to know.

« Last Edit: October 20, 2017, 02:15:45 PM by vit3k »

Offline philjynx

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #70 on: October 21, 2017, 12:02:25 PM »
You will need at least 20ms wait between sending sysx data,
that is the minimum required by midi spec for a receiving device to digest data,
and also determine the gap between the last message and the next.

If you are expecting a reply, only send one message at one time.

Purely hypothetical question for you.  If our gizmo was sending midi to some other gizmo and our gizmo had a partial brain fade resulting in a gap of more than 20 ms in its message, that would be seen as the end of the message by the receiving gizmo?
I am playing all the right notes. But not necessarily in the right order.

Offline sixeight

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #71 on: October 21, 2017, 12:20:06 PM »
Purely hypothetical question for you.  If our gizmo was sending midi to some other gizmo and our gizmo had a partial brain fade resulting in a gap of more than 20 ms in its message, that would be seen as the end of the message by the receiving gizmo?

The same thing may happen that also happens with the Arduino midi library. There is only one buffer used for midi in messages. The buffer is filled through an interrupt routine triggered by receiving midi data. So it could be that the code is reading a message, but while it is processing the data it is written over by the next message through the interrupt routine.

I have seen this happen with the VController when it is reading Zoom devices. I am considering changing the midi library to support multiple midi buffers. But I am still investigating the issue.

Offline MrHaroldA

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #72 on: October 29, 2017, 09:47:16 AM »
But I am still investigating the issue.

It seems a lot of messages are lost if I send a queue item (TX) and wait for the response (RX). I have the response time-out set to 100ms and I randomly (yet predictable) get an answer.

Code: [Select]
Single press: Load patch 0 set 0
Add to queue: 0
Add to queue: 1
Add to queue: 2
Add to queue: 3
Add to queue: 4
Add to queue: 5
Add to queue: 6
Add to queue: 7
Add to queue: 8
Add to queue: 9
Add to queue: 10
Add to queue: 11
Add to queue: 12
Handle queue: 0
TX: F0 41 00 00 00 00 3B 12 60 00 00 30 00 70 F7 (15)
RX: F0 41 00 00 00 00 3B 12 60 00 00 30 00 70 F7 (15)
Unhandled parameter: 0x60000030
Handle queue: 1
TX: F0 41 00 00 00 00 3B 12 60 00 04 30 00 6C F7 (15)
RX: F0 41 00 00 00 00 3B 12 60 00 04 30 00 6C F7 (15)
Unhandled parameter: 0x60000430
Handle queue: 2
TX: F0 41 00 00 00 00 3B 12 60 00 00 20 01 7F F7 (15)
*** Time-out reached.
Handle queue: 3
TX: F0 41 00 00 00 00 3B 12 60 00 00 21 00 7F F7 (15)
RX: F0 41 00 00 00 00 3B 12 60 00 00 21 00 7F F7 (15)
Unhandled parameter: 0x60000021
Handle queue: 4
TX: F0 41 00 00 00 00 3B 12 60 00 00 22 01 7D F7 (15)
*** Time-out reached.
Handle queue: 5
TX: F0 41 00 00 00 00 3B 12 60 00 02 29 01 74 F7 (15)
*** Time-out reached.
Handle queue: 6
TX: F0 41 00 00 00 00 3B 12 60 00 05 1E 00 7D F7 (15)
*** Time-out reached.
Handle queue: 7
TX: F0 41 00 00 00 00 3B 12 60 00 06 10 00 0A F7 (15)
*** Time-out reached.
Handle queue: 8
TX: F0 41 00 00 00 00 3B 12 60 00 06 30 00 6A F7 (15)
*** Time-out reached.
Handle queue: 9
TX: F0 41 00 00 00 00 3B 12 60 00 06 56 00 44 F7 (15)
*** Time-out reached.
Handle queue: 10
TX: F0 41 00 00 00 00 3B 12 60 00 0B 02 00 13 F7 (15)
*** Time-out reached.
Handle queue: 11
TX: F0 41 00 00 00 00 3B 12 60 00 0B 05 00 10 F7 (15)
*** Time-out reached.
Handle queue: 12
TX: F0 41 00 00 00 00 3B 12 60 00 0B 06 00 0F F7 (15)
*** Time-out reached.

Since I do get some answers, I guess my implementation is correct, but the Midi library fails to capture all received data.
« Last Edit: October 29, 2017, 09:56:56 AM by MrHaroldA »

Offline sixeight

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #73 on: October 29, 2017, 10:22:46 AM »
Since I do get some answers, I guess my implementation is correct, but the Midi library fails to capture all received data.

What happens if you change the order of the data you are requesting?
Is it certain addresses that do not respond? If you request data that does not exist, you will not get an answer.

Any chance you could give us a look at your code?

Offline gumtown

Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
« Reply #74 on: October 29, 2017, 11:46:35 AM »
If you want an answer to a sysx data request, then use "11" instead of "12" in your messages.
You are sending data change, not requesting a reply.
The MS-3 is possibly only sending back the same value you are changing.
Free "GR-55 FloorBoard" editor software from http://sourceforge.net/projects/grfloorboard/

 

Blog