Boss MS-3 - MIDI over USB Sysex Reverse Engineering

Started by MrHaroldA, August 29, 2017, 12:13:37 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

MrHaroldA

Tnx! But I suspect that to be asynchronous, and the Arduino library uses a poller in it's examples to capture SysEx data. I have to code in some basic functionality first, and then I'll try to query the MS-3 for the active patch number, FX state, and maybe even some settings like 'switch on push/release'.
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

sixeight

Quote from: MrHaroldA on September 29, 2017, 02:36:21 AM
Tnx! But I suspect that to be asynchronous, and the Arduino library uses a poller in it's examples to capture SysEx data. I have to code in some basic functionality first, and then I'll try to query the MS-3 for the active patch number, FX state, and maybe even some settings like 'switch on push/release'.

That is correct. Most of the data has to be "pulled out" of the MS-3. I do the same with my VController project for patch names and effect states.

Anything you change on the MS-3 should be sent in sysex automatically once editor mode is on. This way the BTS for MS3 keeps track of changes on the unit itself.

A good tutorial for the Roland midi system is http://www.2writers.com/eddie/tutsysex.htm

vtgearhead

#52
Quote from: gumtown on September 29, 2017, 02:28:33 AM
00 00 ZZ ZZ = data size required from start address

One word of warning:  This number is actually the address offset from start to the last byte you want to receive.  Depending upon your choice of starting address and offset you may get fewer bytes back than a simple subtraction would suggest. 

Update: When reading known parameters from their proper address you can ignore this distinction.  However, keep it in mind when doing bulk reads or you'll experience modest grief.  The MIDI sysex address range has discontinuities after each 128-byte page due to the need for data bytes to have bit 7 clear,  but the missing region is still accounted for when determining where to cut off a read.


MrHaroldA

Are the query addresses the same as the set addresses? For example, if I set FX1 at '0x60, 0x00, 0x00, 0x30', can I read it at that address too?
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

sixeight

Quote from: MrHaroldA on September 29, 2017, 05:21:44 AM
Are the query addresses the same as the set addresses? For example, if I set FX1 at '0x60, 0x00, 0x00, 0x30', can I read it at that address too?

Yes you can. You will have to specify the number of bytes you want to read, so '0x60, 0x00, 0x00, 0x30 0x00 0x00, 0x00, 0x01' will read one byte from your address.

For the GP10 I have used the following code for reading data from the device:
void request_GP10(uint32_t address, uint8_t no_of_bytes) {
  uint8_t *ad = (uint8_t*)&address; //Split the 32-bit address into four bytes: ad[3], ad[2], ad[1] and ad[0]
  uint8_t no1 = no_of_bytes / 128;
  uint8_t no2 = no_of_bytes % 128;
  uint8_t checksum = MIDI_calc_Roland_checksum(ad[3] + ad[2] + ad[1] + ad[0] +  no1 + no2); // Calculate the Roland checksum
  uint8_t sysexmessage[18] = {0xF0, 0x41, GP10_device_id, 0x00, 0x00, 0x00, 0x05, 0x11, ad[3], ad[2], ad[1], ad[0], 0x00, 0x00, no1, no2, checksum, 0xF7};
  MIDI.sendSysEx(17, sysexmessage);
}

// Calculate the Roland checksum
uint8_t MIDI_calc_Roland_checksum(uint16_t sum) {
  uint8_t checksum = 0x80 - (sum % 0x80);
  if (checksum == 0x80) checksum = 0;
  return checksum;
}

MrHaroldA

#55
I've added support for loading, editing and storing "FX sets"! :D Next up is actually reading something from the MS3 ...



But first a beer and a bath.
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

MrHaroldA

#56
Quote from: MrHaroldA on October 01, 2017, 01:06:15 PMNext up is actually reading something from the MS3 ...

Yeah ... this is the tricky part.

If I query the MS-3, I get an answer. But if I don't query it, I can't get the Arduino to receive anything. For example: patch changes are sent by the MS-3 without having to query it, but I can only receive it by querying it with some random query.

Running my poller outside of USB_STATE_RUNNING makes no difference.

if (Usb.getUsbTaskState() == USB_STATE_RUNNING)

I'll try to find some examples of other implementations ... or do you guys have some ideas to try?


EDIT: I also get bundled replies when querying while switching patches:


Query: F0, 41, 0, 0, 0, 0, 3B, 11, 60, 0, 0, 30, 0, 0, 0, 1, 6F, F7: 0
Receiving: 20:  04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 00 30 07 00 70 F7
Query: F0, 41, 0, 0, 0, 0, 3B, 11, 60, 0, 0, 30, 0, 0, 0, 1, 6F, F7: 0
Receiving: 64:  04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 00 30 07 00 70 F7 04 F0 41 00 04 00 00 00 04 3B
12 00 04 01 00 00 04 00 00 7F 05 F7 00 00 04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 06 53 07 51 76 F7
Query: F0, 41, 0, 0, 0, 0, 3B, 11, 60, 0, 0, 30, 0, 0, 0, 1, 6F, F7: 0
Receiving: 20:  04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 00 30 07 00 70 F7
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

sixeight

Quote from: MrHaroldA on October 05, 2017, 12:48:48 AM
If I query the MS-3, I get an answer. But if I don't query it, I can't get the Arduino to receive anything. For example: patch changes are sent by the MS-3 without having to query it, but I can only receive it by querying it with some random query.

Sorry, you lost me here.
Does the MS-3 not send any data when you change a parameter on the unit? That would mean it is not properly in editor mode.
Or is the problem with receiving MIDI data in general. It is normal that you have to check the USB handler for new data.

MrHaroldA

Quote from: sixeight on October 05, 2017, 01:22:41 AM
Sorry, you lost me here.
Does the MS-3 not send any data when you change a parameter on the unit? That would mean it is not properly in editor mode.
Or is the problem with receiving MIDI data in general. It is normal that you have to check the USB handler for new data.

If I sniff the Midi with my PC, the MS-3 transmits the patch changes correctly; so the editor mode must be set correctly too.

But if I poll the USB from my Arduino, it never registers that patch change; unless I'm querying it at the same time. The patch change joins the queried data in one big message.

In my current project, all I need is the patch change to load a different set of FX combinations. Querying the effect state would be a bonus to select which FX set matches, and should be activated once a patch is loaded.
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

gumtown

Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

sixeight


MrHaroldA

Quote from: sixeight on October 05, 2017, 05:19:27 AM
Very hard to help you without seeing your code. Have you seen the code of vit3k:

https://github.com/vit3k/katana_fx_controller

That is in fact very helpful!!! I stripped away all Katana-specific stuff and ended up with basically main.cpp and the katana class. I'll have to study the midi class some more, but I already spotted some things I didn't know before. Also the queue seems like a very good idea!

But ... it seems to have a memory leak? Every 'katana.set' I do, I lose around 17 bytes of RAM, until it runs out.


Quote from:  philjynx on October 05, 2017, 07:01:06 AM
Does your code check for incoming USB data continuously? If it doesn't and only checks after each of your sends, that would explain why you're not getting the info you want.

Yes, I check constantly ... I have to run my code against that of the Katana FX Controller to see what I'm doing wrong.
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

MrHaroldA

#62
Quote from:  philjynx on October 05, 2017, 12:23:50 PM
Without looking at the code, my guess would be that 'katana.set' assigns some memory that is not subsequently released.

Nope: it's in the receive part ;)


Initialized
Free RAM: 1056
Sending F0 41 00 00 00 00 3B 12 7F 00 00 01 01 7F F7
Sending F0 41 00 00 00 00 3B 12 60 00 00 30 00 70 F7
Received: F0 41 00 00 00 00 3B 12 00 01 00 00 00 01 7E F7 00
*** Memory usage changed: 933
Received: F0 41 00 00 00 00 3B 12 60 00 06 53 64 63 F7
Received: F0 41 00 00 00 00 3B 12 00 01 00 00 00 00 7F F7 00
*** Memory usage changed: 895
Received: F0 41 00 00 00 00 3B 12 60 00 06 53 64 63 F7
...


I'm beginning to suspect timing issues with my MS3. It fails to boot into editor mode occasionally, misses FX states sometimes when sending bulk messages ... I'll check the SysEx trace for the timings the Editor uses.


Edit: note the extra 00 suffix on the patch changes received from the MS3 ...
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

gumtown

You will need at least 20ms wait between sending sysx data,
that is the minimum required by midi spec for a receiving device to digest data,
and also determine the gap between the last message and the next.

If you are expecting a reply, only send one message at one time.
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vtgearhead

Quote from: gumtown on October 05, 2017, 02:46:48 PM
You will need at least 20ms wait between sending sysx data,
that is the minimum required by midi spec for a receiving device to digest data,
and also determine the gap between the last message and the next.

If you are expecting a reply, only send one message at one time.

I was not aware of the 20ms requirement for message separation.  Explains a lot of weirdness I experienced when working on my Katana bridge.

MrHaroldA

I measured around 4ms delay between events in the Editor software... And 60ms after setting up the Editor mode. This sure made things way more reliable!
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

vit3k

#66
Quote from: MrHaroldA on October 05, 2017, 12:03:34 PM
That is in fact very helpful!!! I stripped away all Katana-specific stuff and ended up with basically main.cpp and the katana class. I'll have to study the midi class some more, but I already spotted some things I didn't know before. Also the queue seems like a very good idea!

But ... it seems to have a memory leak? Every 'katana.set' I do, I lose around 17 bytes of RAM, until it runs out.


Yes, I check constantly ... I have to run my code against that of the Katana FX Controller to see what I'm doing wrong.

It definitely has a memory leak somewhere because it restarts itself after a while. I don't have time to work on this project right now but if you found out where it is I would really like to know.


sixeight

Quote from:  philjynx on October 21, 2017, 11:02:25 AM
Purely hypothetical question for you.  If our gizmo was sending midi to some other gizmo and our gizmo had a partial brain fade resulting in a gap of more than 20 ms in its message, that would be seen as the end of the message by the receiving gizmo?

The same thing may happen that also happens with the Arduino midi library. There is only one buffer used for midi in messages. The buffer is filled through an interrupt routine triggered by receiving midi data. So it could be that the code is reading a message, but while it is processing the data it is written over by the next message through the interrupt routine.

I have seen this happen with the VController when it is reading Zoom devices. I am considering changing the midi library to support multiple midi buffers. But I am still investigating the issue.

MrHaroldA

#68
Quote from: sixeight on October 21, 2017, 11:20:06 AM
But I am still investigating the issue.

It seems a lot of messages are lost if I send a queue item (TX) and wait for the response (RX). I have the response time-out set to 100ms and I randomly (yet predictable) get an answer.


Single press: Load patch 0 set 0
Add to queue: 0
Add to queue: 1
Add to queue: 2
Add to queue: 3
Add to queue: 4
Add to queue: 5
Add to queue: 6
Add to queue: 7
Add to queue: 8
Add to queue: 9
Add to queue: 10
Add to queue: 11
Add to queue: 12
Handle queue: 0
TX: F0 41 00 00 00 00 3B 12 60 00 00 30 00 70 F7 (15)
RX: F0 41 00 00 00 00 3B 12 60 00 00 30 00 70 F7 (15)
Unhandled parameter: 0x60000030
Handle queue: 1
TX: F0 41 00 00 00 00 3B 12 60 00 04 30 00 6C F7 (15)
RX: F0 41 00 00 00 00 3B 12 60 00 04 30 00 6C F7 (15)
Unhandled parameter: 0x60000430
Handle queue: 2
TX: F0 41 00 00 00 00 3B 12 60 00 00 20 01 7F F7 (15)
*** Time-out reached.
Handle queue: 3
TX: F0 41 00 00 00 00 3B 12 60 00 00 21 00 7F F7 (15)
RX: F0 41 00 00 00 00 3B 12 60 00 00 21 00 7F F7 (15)
Unhandled parameter: 0x60000021
Handle queue: 4
TX: F0 41 00 00 00 00 3B 12 60 00 00 22 01 7D F7 (15)
*** Time-out reached.
Handle queue: 5
TX: F0 41 00 00 00 00 3B 12 60 00 02 29 01 74 F7 (15)
*** Time-out reached.
Handle queue: 6
TX: F0 41 00 00 00 00 3B 12 60 00 05 1E 00 7D F7 (15)
*** Time-out reached.
Handle queue: 7
TX: F0 41 00 00 00 00 3B 12 60 00 06 10 00 0A F7 (15)
*** Time-out reached.
Handle queue: 8
TX: F0 41 00 00 00 00 3B 12 60 00 06 30 00 6A F7 (15)
*** Time-out reached.
Handle queue: 9
TX: F0 41 00 00 00 00 3B 12 60 00 06 56 00 44 F7 (15)
*** Time-out reached.
Handle queue: 10
TX: F0 41 00 00 00 00 3B 12 60 00 0B 02 00 13 F7 (15)
*** Time-out reached.
Handle queue: 11
TX: F0 41 00 00 00 00 3B 12 60 00 0B 05 00 10 F7 (15)
*** Time-out reached.
Handle queue: 12
TX: F0 41 00 00 00 00 3B 12 60 00 0B 06 00 0F F7 (15)
*** Time-out reached.


Since I do get some answers, I guess my implementation is correct, but the Midi library fails to capture all received data.
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

sixeight

Quote from: MrHaroldA on October 29, 2017, 08:47:16 AM
Since I do get some answers, I guess my implementation is correct, but the Midi library fails to capture all received data.

What happens if you change the order of the data you are requesting?
Is it certain addresses that do not respond? If you request data that does not exist, you will not get an answer.

Any chance you could give us a look at your code?

gumtown

If you want an answer to a sysx data request, then use "11" instead of "12" in your messages.
You are sending data change, not requesting a reply.
The MS-3 is possibly only sending back the same value you are changing.
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

MrHaroldA

Quote from: gumtown on October 29, 2017, 10:46:35 AM
The MS-3 is possibly only sending back the same value you are changing.

I know, but the MS-3 should return everything I set. I do have to check if the MS-3 also returns something if there was no change in state, for example by enabling the same effect twice.

But I'm watching Hello Kitty with my daughter right now ;)

I plan to release my MS-3 library soon, so I'll look into publishing it later this evening...
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

MrHaroldA

Quote from: MrHaroldA on October 29, 2017, 10:55:42 AM
I know, but the MS-3 should return everything I set. I do have to check if the MS-3 also returns something if there was no change in state, for example by enabling the same effect twice.

Hello Kitty is finished, daughter is sleeping, and watched a Stranger Things with the wife ... 8)

I've checked the replies, and while they are consistent they seem random over different effect sets.

I'll now clean up my library a little and write an example implementation.
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

MrHaroldA

Here's my MS-3 library: https://github.com/MrHaroldA/MS3

I've added a really simple example to show how to use the library.

I would really love your remarks on this! Note that I don't really know what I'm doing, I'm just a simple php scripter by daytime ;)
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

MrHaroldA

Quote from: sixeight on October 29, 2017, 09:22:46 AM
What happens if you change the order of the data you are requesting?
Is it certain addresses that do not respond? If you request data that does not exist, you will not get an answer.

Any chance you could give us a look at your code?

I've just added an effect_state example, which also misses out on a lot of effect states ... unless you enable debug/verbose mode!  ???


// Uncomment this to enable verbose debug messages.
#define MS3_DEBUG_MODE // *** Enable this!


It seems I have severe timing issues?
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3