VGuitar Forums

Boss MS-3 => Boss MS-3 - Top Things to Know => Topic started by: MrHaroldA on August 29, 2017, 12:13:37 AM

Title: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on August 29, 2017, 12:13:37 AM
Hey guys, I'm Harold and I'm new here ;)

I'm the owner of the new Boss MS-3, and after a quick inspection with MidiOX, it receives SysEX over the emulated USB/Midi port, just like the Katana (and others) do!

The MS-3 lacks a Midi IN port, so I'm now trying to build a compatible Midi controller for it, that will connect through the USB connector instead. At the heart of it will be an Arduino; probably a Nano (or mini) with an USB shield, as the Arduino itself isn't capable of becoming a USB host.

I also have an Arduino Due here, which might be able to act as host ... but is pretty large ;)


I flipped through this thread, and didn't find anyone using an Arduino to control their Boss/Roland hardware, is that correct?
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on August 29, 2017, 01:44:34 AM
Boss USB is not 'class compliant' so a host shield won't work out of the box.
You will have to find the device specific USB end point enumerators.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on August 29, 2017, 02:05:01 AM
Quote from:  gumtown on August 29, 2017, 01:44:34 AM
Boss USB is not 'class compliant' so a host shield won't work out of the box.
You will have to find the device specific USB end point enumerators.
Hmmm... I hoped it to be class compliant, since my Linux machine recognized it (and even tried to use it as primary soundcard ;) )

I'll look into this, but I guess this will turn out to be pretty difficult for me, as I'm not familiar with this and the Boss of course won't display any debug information; it either works, or it doesn't.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: admin on August 29, 2017, 07:09:13 AM
(https://static.roland.com/assets/images/products/gallery/ms-3_R_gal.jpg)
The MS-3 with missing MIDI Input jack is yet another Roland Boss product that is crippled from the start
with 80% of what you need



If you  are using a raspberry pi ,or Beaglebone black - thanks to Linux Jack audio the MS-3  can be seen as a MIDI device in that environment.

same situation as the Boss GP-10)

Details
https://www.vguitarforums.com/smf/index.php?topic=11998
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vit3k on September 01, 2017, 02:53:19 PM
Quote from:  gumtown on August 29, 2017, 01:44:34 AM
Boss USB is not 'class compliant' so a host shield won't work out of the box.
You will have to find the device specific USB end point enumerators.

USB host shield for Arduino works just fine with katana. At least program change messages.

I have some code here: https://github.com/vit3k/arduino_midi
It is work in progress and may not work with current commit. Also I was using hub to use multiple devices so code is more complex because of it. I didn't check sysex though but it will probably work just fine.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: admin on September 01, 2017, 04:13:37 PM
Quote from:  vit3k on September 01, 2017, 02:53:19 PM
USB host shield for Arduino works just fine with katana. At least program change messages.

I have some code here: https://github.com/vit3k/arduino_midi
It is work in progress and may not work with current commit. Also I was using hub to use multiple devices so code is more complex because of it. I didn't check sysex though but it will probably work just fine.

Thanks for the news regarding USB host shield for Arduino works just fine with Katana!

(perhaps there is hope yet !)
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: Beanow on September 03, 2017, 04:34:32 AM
Quote from:  MrHaroldA on August 29, 2017, 12:13:37 AM
I flipped through this thread, and didn't find anyone using an Arduino to control their Boss/Roland hardware, is that correct?

While not an arduino, the MIDX-20 is built with a PIC micro controller.
https://www.vguitarforums.com/smf/index.php?topic=21261.0

Quote from:  MrHaroldA on August 29, 2017, 02:05:01 AM
Hmmm... I hoped it to be class compliant, since my Linux machine recognized it (and even tried to use it as primary soundcard ;) )
The Linux kernel ships with a large amount of drivers (kernel modules) out of the box, including "snd_usb_audio" maintained by the Alsa folks. This module seems generic based on it's name, but actually contains many vendor-specific specifications and workarounds. Including the Boss/Roland non-standard audio and midi interfaces.
Some behind the scenes look would be the quirks table in the kernel source. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/sound/usb/quirks-table.h?h=v4.13-rc7#n1780

What I would recommend for testing if Sysex is received by the Katana or not, is to send a "Universal non-realtime identity request". It's a standard probe to find out more about the devices responding to Sysex requests. I would be formatted as hexadecimal raw midi:
F0 7E 7F 06 01 F7
This will request devices with any device ID to respond.
You should get a reply that looks something like this:
F0 7E 00 06 02 41 33 03 00 00 01 00 00 00 F7
Interpreted this reveals: 41 (Boss/Roland) 33 03 00 00 (Katana) 01 00 00 00 (Firmware version).
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vit3k on September 04, 2017, 02:20:19 PM
Quote from:  gumtown on August 29, 2017, 01:44:34 AM
Boss USB is not 'class compliant' so a host shield won't work out of the box.
You will have to find the device specific USB end point enumerators.

Today I've tried to use sysex messages with Arduino and Katana. Unfortunately it doesn't work. I can send messages but don't receive any. Looks like input endpoint is wrong.

Looks like gumtown is right and some changes to library will be needed. I will get back to this when I have some time. Gumtown - do you have some more information on this subject?

Or there is a bug in my code  ???

https://github.com/vit3k/katana_fx_controller

Edit:
I found a solution. I think this could be a bug in usb host library

usbh_midi.cpp flie, line 297:
if ((epDesc->bmAttributes & 0x02) == 2) {//bulk
should be changed to:
if ((epDesc->bmAttributes & 0x03) == 2) {//bulk

and that's because:
Bits 0..1 Transfer Type
00 = Control
01 = Isochronous
10 = Bulk
11 = Interrupt

With this change I receive all sysex messages. Tried with put Katana in BTS mode. It sends all changes to arduino :)
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: Beanow on September 07, 2017, 03:11:42 AM
Quote from:  vit3k on September 04, 2017, 02:20:19 PM
Edit:
I found a solution. I think this could be a bug in usb host library

usbh_midi.cpp flie, line 297:
if ((epDesc->bmAttributes & 0x02) == 2) {//bulk
should be changed to:
if ((epDesc->bmAttributes & 0x03) == 2) {//bulk

and that's because:
Bits 0..1 Transfer Type
00 = Control
01 = Isochronous
10 = Bulk
11 = Interrupt

Edit:
Depending on the intentions of that line, that smells like a bug indeed.
For example, if epDesc->bmAttributes was 0b11 (Interrupt) it would evaluate as true, while commented as bulk and seems to check for bulk (== 2).
The pattern this check uses is of a "is this bit set?" suitable for flags. But it looks like the field is actually a 2-bit uint not a set of binary flags.

They are important properties to look at. On linux lsusb reports:


Interface #1
EP 13 OUT
  Transfer Type            Isochronous
  Synch Type               Asynchronous
  Usage Type               Data

Interface #2
EP 14 IN
  Transfer Type            Isochronous
  Synch Type               Asynchronous
  Usage Type               Implicit feedback Data

Interface #3, alternate 0
EP 3 OUT
  Transfer Type            Bulk
  Synch Type               None
  Usage Type               Data

EP 4 IN
  Transfer Type            Bulk
  Synch Type               None
  Usage Type               Data

Interface #3, alternate 1
EP 3 OUT
  Transfer Type            Interrupt
  Synch Type               None
  Usage Type               Data

EP 5 IN
  Transfer Type            Interrupt
  Synch Type               None
  Usage Type               Data


From my understanding interface 1 and 2 are your audio in and out.
Whereas interface 3 is for midi and it can be configured to run in interrupt or bulk mode.
This is about where my knowledge of the USB and MIDI spec ends through, but I do know that if these settings and endpoints are mismatched it won't work.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vtgearhead on September 07, 2017, 05:13:42 AM
Quote from:  vit3k on September 04, 2017, 02:20:19 PM
I found a solution. I think this could be a bug in usb host library

usbh_midi.cpp flie, line 297:
if ((epDesc->bmAttributes & 0x02) == 2) {//bulk
should be changed to:
if ((epDesc->bmAttributes & 0x03) == 2) {//bulk

and that's because:
Bits 0..1 Transfer Type
00 = Control
01 = Isochronous
10 = Bulk
11 = Interrupt

With this change I receive all sysex messages. Tried with put Katana in BTS mode. It sends all changes to arduino :)

If you need to consider bits 0 and 1 for the comparison then the original code is wrong.  Nice catch!

Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vit3k on September 07, 2017, 12:26:45 PM
Yes, it's a bug. I've post it on library's github and it's already fixed and merged.

https://github.com/felis/USB_Host_Shield_2.0/issues/313

And currently I'm trying to move to Teensy 3.2 as arduino lacks flash and ram and of course it doesn't work anymore.

Does anyone have any experience in running usb host shield (full sized or mini) with teensy? Is it possible that because of fast SPI it's not working on breadboard or with 20cm cables?
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 19, 2017, 01:49:03 PM
Hey guys; looks like you all did a fine job while I was away on vacation!  8)

vit3k's fix is already released in v1.3.1 of the host library, so I hacked up this tiny script to test if I could send SysEx to my MS-3:

#include <usbh_midi.h>

USB  Usb;
USBH_MIDI  Midi(&Usb);

// SysEx data for loading Patch 0 & 1.
uint8_t data1[] = {0xF0, 0x41, 0x00, 0x00, 0x00, 0x00, 0x3B, 0x12, 0x00, 0x01, 0x00, 0x00, 0x00, 0x7F, 0xF7};
uint8_t data2[] = {0xF0, 0x41, 0x00, 0x00, 0x00, 0x00, 0x3B, 0x12, 0x00, 0x01, 0x00, 0x00, 0x01, 0x7E, 0xF7};

void setup() {
    Serial.begin(38400);
    while (!Serial) {}

    // Halt if USB won't init.
    if (Usb.Init() == -1) {
        Serial.println(F("USB Init error"));
        while (1);
    }

    delay(200); // @TODO: Why?
    Serial.println(F("Ready!"));
}

void loop() {
    Usb.Task();
    if (Usb.getUsbTaskState() == USB_STATE_RUNNING) {
        // @TODO: Handle incoming USB data.
    }

    Serial.print(F("Load patch 0: "));
    Serial.println(Midi.SendSysEx(data1, sizeof(data1))); // sizeof(data1) == 15;
    delay(1000);

    Serial.print(F("Load patch 1: "));
    Serial.println(Midi.SendSysEx(data2, sizeof(data2)));
    delay(1000);
}


Resulting in:

Ready!
Load patch 0: 13
Load patch 1: 13
Load patch 0: 14
Load patch 1: 13
Load patch 0: 13
Load patch 1: 13
...


(the second time patch 0 is loaded it always returns 14, the rest returns 13)


It would have been very nice if this would work, but I'd also be surprised if it did ;)

I'm off to polish my USB/Midi/Arduino/C/debug knowledge now ...
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vit3k on September 21, 2017, 06:17:09 AM
I think sizeof will return the size of pointer. Pass the size of array or use SendData method that will calculate size of sysex message for you.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 22, 2017, 02:20:43 AM
Quote from:  vit3k on September 21, 2017, 06:17:09 AM
I think sizeof will return the size of pointer. Pass the size of array or use SendData method that will calculate size of sysex message for you.

'sizeof(data1)' returns 15 in this case, which is the number of bytes in the array ...

My best guess is now that I missed a byte, as 'F0 41 00 00 00 00 3B 12 00 01 00 00 00 00 7F F7' is 16 bytes ;)

I hope to report back this weekend!
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 22, 2017, 02:13:05 PM
Moving Midi.SendSysEx() into the 'if (Usb.getUsbTaskState() == USB_STATE_RUNNING) {' block made the MS-3 react to my messages! :D

It now flashes "BULK DATA RECEIVING..." it it's display ... instead of changing patches.

If I manually send the SysEx sequence above with MIDI OX, the patch is loaded correctly, so now I have to figure out why the MS-3 does not consider this to be a SysEx command if it's coming from my Arduino.

The SendData method acts the same, btw. Both methods return 0, which translates to "success".
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 22, 2017, 03:41:44 PM
This is weird. My SysEx (F0 41 00 00 00 00 3B 12 00 01 00 00 00 00 7F F7) for loading patch 1 works when the MS-3 Editor/Librarian is open, but doesn't when it is not running. So, I sniffed the FX1 toggle and used that instead; works like a charm, even from the Arduino!!!  8)

But here's another strange thing: the display flashes "BULK DATA RECEIVING..." when the Editor/Librarian isn't running, but not if it is. So, I thought it might issue a SysEx command to silence the display, but it should send one to re-activate it too when exiting the Editor/Librarian; but it doesn't.

Using amidi on Linux also flashes the "BULK DATA" message ...


Did they hide something in the Windows driver?  ???
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on September 22, 2017, 04:36:54 PM
You need to send a sysx command to put the MS-3 into "verbose mode".

"name": "editor_comunication_level",
      "address": "7F,00,00,00",
      "size": "00,00,00,01"
    },
    {
      "name": "editor_comunication_mode",
      "address": "7F,00,00,01",
      "size": "00,00,00,01"
    },
    {
      "name": "running_mode",
      "address": "7F,00,00,02",
      "size": "00,00,00,01"
    },
    {
      "name": "patch_copy",
      "address": "7F,00,01,00",
      "size": "00,00,00,04"
    },
    {
      "name": "patch_write",
      "address": "7F,00,01,04",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_1_pitch",
      "address": "7F,00,02,00",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_2_pitch",
      "address": "7F,00,02,02",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_3_pitch",
      "address": "7F,00,02,04",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_4_pitch",
      "address": "7F,00,02,06",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_5_pitch",
      "address": "7F,00,02,08",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_6_pitch",
      "address": "7F,00,02,0A",
      "size": "00,00,00,02"
    },
    {
      "name": "patch_clear",
      "address": "7F,01,00,03",
      "size": "00,00,00,01"
    },
    {
      "name": "patch_initialize",
      "address": "7F,01,00,04",
      "size": "00,00,00,01"
    },
    {
      "name": "tuner_single_mode_pitch",
      "address": "7F,00,05,00",
      "size": "00,00,00,02"
    },
    {
      "name": "usb_direct_mon",
      "address": "7F,00,06,00",
      "size": "00,00,00,01"
    },
    {
      "name": "usb_in_out_mode",
      "address": "7F,00,06,01",
      "size": "00,00,00,01"
    },
    {
      "name": "patch_map",
      "address": "7F,70,00,00",
      "size": "00,00,00,01"
    },
    {
      "name": "update_patch_map",
      "address": "7F,71,00,00",
      "size": "00,00,00,01"
    },
    {
      "name": "manual_sw",
      "address": "7F,01,00,07",
      "size": "00,00,00,01"
    },
     {
      "name": "quick_setting_wirte",
      "address": "7F,01,00,00",
      "size": "00,00,00,02"
    },
    {
      "name": "ab_utilty",
      "address": "7F,01,00,02",
      "size": "00,00,00,01"
    },
    {
      "name": "ctl_midi_send",
      "address": "7F,00,07,00",
      "size": "00,00,00,01"
    }
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 23, 2017, 08:47:08 AM
Quote from:  gumtown on September 22, 2017, 04:36:54 PM
You need to send a sysx command to put the MS-3 into "verbose mode".

Tnx! I'll crosscheck the SysEx commands the Editor sends with your list; or is it the "running_mode" you are referring to?
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on September 23, 2017, 01:34:15 PM
I am not sure which one, somewhere in the first 3 addresses. I suspect 7F 00 00 01
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vit3k on September 23, 2017, 11:51:05 PM
Quote from:  snhirsch
If it behaves like the Katana, then:

7F,00,00,00,01   Enters editor mode
7F,00,00,00,00   Exists editor mode

I think 7F,00,00,01 set to 01 enters and 00 leaves.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 24, 2017, 01:43:21 AM
Anybody got the checksum for that command at hand? I still have to write the logic to create them myself.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on September 24, 2017, 02:21:40 AM
Those commands are send 'as is' with no checksum and no sysx header/footer.

Arduino checksum calc:

int sum = 128 - (data_size % 128);
if (sum == 128) { sum = 0; };

String checksum = String(sum, 16);
if (sum < 16) { checksum = String(("0") + checksum); };
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 24, 2017, 03:03:35 AM
I've tried these combinations with the SendData() method:


0x7F, 0x00, 0x00, 0x01
0x7F, 0x00, 0x00, 0x00, 0x01
0x7F, 0x00, 0x00, 0x01, 0x01
0x7F, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01


But the display still flashes the BULK DATA message when receiving SysEx.

Here's my current working (except for the flashing display) code, including the logic for calculating the checkum:

#include <usbh_midi.h>

USB Usb;
USBH_MIDI Midi(&Usb);

const uint8_t SYSEX_START = 0xF0;
const uint8_t SYSEX_END = 0xF7;
const uint8_t MANUFACTURER_ID = 0x41;
const uint8_t DEVICE_ID = 0x00;
const uint8_t MODEL_ID[] = {0x00, 0x00, 0x00, 0x3b};
const uint8_t QUERY = 0x11;
const uint8_t SET = 0x12;

// SYSEX_START + MANUFACTURER_ID + DEVICE_ID + MODEL_ID + QUERY/SET + PARAMETER + <variable dataSize> + CHECKSUM + SYSEX_END.
const uint8_t FIXED_MESSAGE_LENGTH = 14;

// Special instructions.
const uint8_t EDITOR_MODE[] = {0x7F, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01};

// Toggles.
const uint8_t TOGGLE_ON[] = {0x01};
const uint8_t TOGGLE_OFF[] = {0x00};
const uint8_t TOGGLE_FX1[] = {0x60, 0x00, 0x00, 0x30};

// Global variables.
bool ready = false;

void setParameter(uint8_t *parameter, uint8_t *data, uint8_t dataSize) {
    uint8_t message[FIXED_MESSAGE_LENGTH + dataSize], i;

    // Construct the full message.
    message[0] = SYSEX_START;
    message[1] = MANUFACTURER_ID;
    message[2] = DEVICE_ID;
    for (i = 0; i <= 3; i++) {
        message[i + 3] = MODEL_ID[i];
    }
    message[7] = SET;
    for (i = 0; i <= 3; i++) {
        message[i + 8] = parameter[i];
    }
    for (i = 0; i < dataSize; i++) {
        message[i + 12] = data[i];
    }
    message[sizeof(message) - 2] = checksum(parameter, data, dataSize);
    message[sizeof(message) - 1] = SYSEX_END;

    // Debug printing.
    Serial.print(message[0], HEX);
    for (i = 1; i < sizeof(message); i++) {
        Serial.print(", ");
        Serial.print(message[i], HEX);
    }
    Serial.print(": ");
    Serial.println(Midi.SendSysEx(message, sizeof(message)));
}

uint8_t checksum(uint8_t *parameter, uint8_t *data, uint8_t dataSize) {
    uint8_t sum, i;

    for (i = 0; i <= 3; i++) {
        sum = (sum + parameter[i]) & 0x7F;
    }
    for (i = 0; i < dataSize; i++) {
        sum = (sum + data[i]) & 0x7F;
    }

    return (128 - sum) & 0x7F;
}

void setup() {
    Serial.begin(38400);
    while (!Serial) {}

    // Halt if USB won't init.
    if (Usb.Init() == -1) {
        Serial.println(F("USB Init error"));
        while (1);
    }

    delay(200); // @TODO: Why?
    Serial.println(F("Ready!"));
}

void loop() {
    Usb.Task();
    if (Usb.getUsbTaskState() == USB_STATE_RUNNING) {
        if (!ready) {
            Serial.print(F("Set editor mode: "));
            Serial.println(Midi.SendData(EDITOR_MODE));
            ready = true;
        }

        Serial.print(F("Set FX1 to 1: "));
        setParameter(TOGGLE_FX1, TOGGLE_ON, sizeof(TOGGLE_ON));
        delay(1000);

        Serial.print(F("Set FX1 to 0: "));
        setParameter(TOGGLE_FX1, TOGGLE_OFF, sizeof(TOGGLE_ON));
        delay(1000);
    }
}
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: sixeight on September 24, 2017, 03:30:29 AM
QuoteI've tried these combinations with the SendData() method:

You will have to send 0xF0 before and 0xF7 at the end to make it a proper system message...
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 24, 2017, 03:32:33 AM
Quote from:  sixeight on September 24, 2017, 03:30:29 AM
You will have to send 0xF0 before and 0xF7 at the end to make it a proper system message...

But ...

Quote from:  gumtown on September 24, 2017, 02:21:40 AM
Those commands are send 'as is' with no checksum and no sysx header/footer.

I'll try anyway; one moment please!
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 24, 2017, 03:35:55 AM
Quote from:  MrHaroldA on September 24, 2017, 03:32:33 AM
I'll try anyway; one moment please!

Nope: still flashes ...
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vtgearhead on September 24, 2017, 06:10:25 AM
Quote from:  MrHaroldA on September 24, 2017, 03:03:35 AM

uint8_t checksum(uint8_t *parameter, uint8_t *data, uint8_t dataSize) {
    uint8_t sum, i;

    for (i = 0; i <= 3; i++) {
        sum = (sum + parameter[i]) & 0x7F;
                   ------  Bzzzt!
    }
    for (i = 0; i < dataSize; i++) {
        sum = (sum + data[i]) & 0x7F;
    }

    return (128 - sum) & 0x7F;
}



Um, unless I'm missing something 'sum' is uninitialized at the first point it's used.  See buzzer above.

Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 24, 2017, 06:21:22 AM
Quote from:  snhirsch
You are correct - I just fixed my posting above. 

To clarify:


  • These commands DO NOT require the usual sysex prefix / suffix and checksum
  • The command to enter edit = 7F, 00, 00, 01, 01
  • The command to exit edit = 7F, 00, 00, 01, 00


That still doesn't work...

Sniffing the Editor startup revails these messages that are sent first:


SYSX: F0 7E 00 06 02 41 3B 03 00 00 00 00 00 00 F7
SYSX: F0 7E 00 06 02 41 3B 03 00 00 00 00 00 00 F7
SYSX: F0 41 00 00 00 00 3B 12 7F 00 00 00 00 01 F7


The last one mildly resembles the editor_mode code posted above.

After that, a repetitive number of 30 byte SysEx messages appear.

Edit: full log in the attachment!
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: sixeight on September 24, 2017, 08:24:59 AM
QuoteF0 41 00 00 00 00 3B 12 7F 00 00 00 00 01 F7

41 00 00 00 00 3B is the identifyer for Boss MS3
12 means write data (11 is data request)
7F 00 00 00 is the address
00 is the data
01 the checksum (80 - sum of data and address: 7F + 00 +00 +00 +00 = 7F, 80 - 7F = 01

Try sending F0 41 00 00 00 00 3B 12 7F 00 00 00 01 00 F7
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vit3k on September 24, 2017, 10:05:38 AM
For Katana I'm sending whole sysex message with checksum to enable editor mode as for any other command. Here is my method for calculating checksum working on Arduino. You can use it if you want.



byte Katana::calculateChecksum(byte* vals, byte valsSize)
{
    byte acc = 0;
    for(byte i = 0; i < valsSize; i++)
    {
        acc = (acc + vals[i]) & 0x7F;
    }

    return (128 - acc) & 0x7F;
}


Vals has to point to the beginning of address with correct size (4 for address + some for data). I don't know if the same works on MS-3.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vtgearhead on September 24, 2017, 11:38:29 AM
Quote from:  sixeight on September 24, 2017, 08:24:59 AM
41 00 00 00 00 3B is the identifyer for Boss MS3
12 means write data (11 is data request)
7F 00 00 00 is the address
00 is the data
01 the checksum (80 - sum of data and address: 7F + 00 +00 +00 +00 = 7F, 80 - 7F = 01

Try sending F0 41 00 00 00 00 3B 12 7F 00 00 00 01 00 F7

And you are of course correct - I'll delete my earlier, misleading posts.  Sorry about that!
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 24, 2017, 01:25:42 PM
Quote from:  sixeight on September 24, 2017, 08:24:59 AM
Try sending F0 41 00 00 00 00 3B 12 7F 00 00 00 01 00 F7

Is that 00 in the right position?

Sending "F0, 41, 0, 0, 0, 0, 3B, 12, 7F, 0, 0, 0, 0, 1, F7" didn't do the trick, neither did "F0, 41, 0, 0, 0, 0, 3B, 12, 7F, 0, 0, 0, 1, 0, F7"

I'm off to bed now; the hunt continues tomorrow!
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vtgearhead on September 24, 2017, 02:53:27 PM
Now that I've re-read the sources, this is what I used to put a Katana into edit mode:

0xF0, 0x41, 0x00, 0x00, 0x00, 0x00, 0x33, 0x12, 0x7f, 0x00, 0x00, 0x01, 0x01, 0x7F, 0xF7

And this should cancel edit mode:

0xF0, 0x41, 0x00, 0x00, 0x00, 0x00, 0x33, 0x12, 0x7f, 0x00, 0x00, 0x01, 0x00, 0x00, 0xF7

Given how many times I stuck my foot in my mouth during my participation in this thread I'd appreciate someone else reviewing this post for correctness.

UPDATE:  All I know is that the strings above are (a) what is calculated by the checksum code in my program and (b) work properly.  I was looking at a post that showed different checksum bytes.

Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vtgearhead on September 24, 2017, 02:57:27 PM
Quote from:  sixeight on September 24, 2017, 08:24:59 AM
41 00 00 00 00 3B is the identifyer for Boss MS3
12 means write data (11 is data request)
7F 00 00 00 is the address
00 is the data
01 the checksum (80 - sum of data and address: 7F + 00 +00 +00 +00 = 7F, 80 - 7F = 01

Try sending F0 41 00 00 00 00 3B 12 7F 00 00 00 01 00 F7

Six-Eight:  Are you sure about that algorithm?  I've always used this:

# Example: Take the address (4 bytes) and data (1 byte) in this case
vals = [0x60, 0x00, 0x12, 0x14, 0x01]

# Sum with 7-bit wraparound
accum = 0
for val in vals:
    accum = (accum + val) & 0x7F

# Checksum is lower 7 bits of the difference w/ 128
cksum = (128 - accum) & 0x7F

Your post is not doing a difference with 128.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: sixeight on September 25, 2017, 12:51:12 AM
QuoteIs that 00 in the right position?

Yes. It is the checksum (7F + 00 + 00 + 00 + 01 = 0x80; ox80 - 0x80 = 00)

Maybe these will work:
F0 41 00 00 00 00 3B 12 7F 00 00 01 01 7F F7
F0 41 00 00 00 00 3B 12 7F 00 00 02 01 7E F7
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 25, 2017, 04:31:48 AM
None of these options work, so I replicated my Arduino program in a Linux bash script:


#!/bin/bash
MIDI='hw:2,0,0'

echo "Setting editor mode" # first three messages from the editor start.
amidi -p ${MIDI} -S 'F0 7E 00 06 02 41 3B 03 00 00 00 00 00 00 F7'
amidi -p ${MIDI} -S 'F0 7E 00 06 02 41 3B 03 00 00 00 00 00 00 F7'
amidi -p ${MIDI} -S 'F0 41 00 00 00 00 3B 12 7F 00 00 00 00 01 F7' # This already triggers BULK DATA

while true; do
  echo "Setting FX1 ON"
  amidi -p ${MIDI} -S 'F0 41 0 0 0 0 3B 12 60 0 0 30 1 6F F7'
  sleep 1

  echo "Setting FX1 OFF"
  amidi -p ${MIDI} -S 'F0 41 0 0 0 0 3B 12 60 0 0 30 0 70 F7'
  sleep 1
done


This doesn't need compilation, upload, verify, reboot, serial terminal, etc, so it's way faster to try stuff.

Anyway; this doesn't work either. FX1 toggles nicely, but with the "BULK DATA RECEIVING..." overlay in the display even by setting the editor mode exactly like the editor does.

I'm off again, have to sell my condo today ;) See you guys later, and thank you for all your suggestions!!!
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 26, 2017, 01:05:18 PM
I can't find any other commands the Editor/Librarian sends than those three posted above; and even mimicking the timing does not prevent the display from displaying "BULK DATA RECEIVING..." upon the third SysEx command.

This is becoming very annoying.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vtgearhead on September 26, 2017, 02:49:51 PM
There must be other traffic.  How are you intercepting communication?   At Codesmart's suggestion I've used a low-level MIDI analyzer package that shows absolutely everything in both directions.  Since you'll be seeing raw USB it does require some interpretation to get rid of the data length bytes.  When diving into the Katana last year, I captured bulk data and processed out the length bytes with a Perl script.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on September 26, 2017, 09:20:11 PM
This one definitely works for me

F0 41 00 00 00 00 3B 12 7F 00 00 01 01 7F F7

Puts the MS-3 into "verbose mode" and stops the "Bulk Data" messages, and it also sends and knob turn/parameter changes back out the Midi/USB.

I am using Bome's SendSX to read and send midi, and a partially constructed MS-3_FxFloorBoard editor.

The MS-3 internal structure is definitely not based on the GT-100, like the GT-1 and Katana are.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumbo on September 26, 2017, 11:30:04 PM
...I'm convinced that all you guys are just really TALKING DIRTY to each other in some kind of encrypted code....   >:(
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 26, 2017, 11:50:21 PM
Quote from:  snhirsch on September 26, 2017, 02:49:51 PMHow are you intercepting communication?   At Codesmart's suggestion I've used a low-level MIDI analyzer package that shows absolutely everything in both directions.

I'm using MIDI-OX right now, which may be too high level; I don't really know these tools. Any suggestions? Linux or Windows only.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on September 27, 2017, 12:51:00 AM
Quote from:  gumbo on September 26, 2017, 11:30:04 PM
...I'm convinced that all you guys are just really TALKING DIRTY to each other in some kind of encrypted code....   >:(
Figure this out  :-*
59 65 61 68 20 42 61 62 79 20 21 20 54 61 6C 6B
20 64 69 72 74 79 20 74 6F 20 6D 65 20 21 21
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 27, 2017, 12:54:23 AM
Hint: 546865726520617265206f6e6c696e6520636f6e7665727465727320617661696c61626c6520746f20636f6e766572742068657820746f2063686172732e
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on September 27, 2017, 01:04:34 AM
It might seem odd, but my go to hex/code editor tool is PSPad.

http://www.pspad.com/en/
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 27, 2017, 01:25:16 AM
Quote from:  gumtown on September 27, 2017, 01:04:34 AM
It might seem odd, but my go to hex/code editor tool is PSPad.

I'm using Atom IDE, but that can't sniff Midi on a low level ...
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: sixeight on September 27, 2017, 05:17:30 AM
Quote from:  gumtown on September 27, 2017, 12:51:00 AM
Figure this out  :-*
59 65 61 68 20 42 61 62 79 20 21 20 54 61 6C 6B
20 64 69 72 74 79 20 74 6F 20 6D 65 20 21 21

Don't let Elantric read that one. He will ban you from this forum. ;D
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 27, 2017, 01:11:52 PM
Quote from:  gumtown on September 26, 2017, 09:20:11 PM
This one definitely works for me

F0 41 00 00 00 00 3B 12 7F 00 00 01 01 7F F7

I sure hope you edited that HEX code, or else I can't copy/paste like sh*t. :D

After decoding the sniffed USB Analyzer data, I reconstructed "F0 41 00 00 00 00 3B 12 7F 00 00 01 01 7F F7" as one of the first things the editor sends. This is different than what MIDI-OX logs!

(but it's exactly what you posted)

And ... IT WORKS!!!  ;D
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on September 27, 2017, 01:16:45 PM
Which is why ii don't use midiOX that much,
I still have 8 ports of MidiYoke installed, but prefer to use
Bomes SendSX
https://www.bome.com/products/sendsx
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 29, 2017, 12:05:06 AM
Ok, setting data works fine and reliable. Now I have to figure out how to receive data from the MS-3 ... I'll probably first sniff what I can expect, and then look into the Arduino Midi Host Shield software on how to actually receive the variable length SysEx data.

I also have to start on the hardware prototype, it will make it all way more "touchable".
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on September 29, 2017, 02:28:33 AM
Data retrieval is quite straight forward.

F0 41 00 00 00 00 3B 11 60 00 XX XX 00 00 ZZ ZZ CS F7"
where
F0 41 00 00 00 3B = usual boring sysx header stuff
11 = data request mode
60 00 XX XX = start address of temp data block
00 00 ZZ ZZ = data size required from start address
CS = checksum
F7 = end of message

You might find some useful Arduino code here
https://github.com/gumtown/V-tone

A 12 knob box with 3 lcd displays

https://www.youtube.com/watch?v=0RUX1Rtfcto&t=20s

I also started on a MKii version using rotary encoders, but couldn't get the Arduino to scan 12 encorders

https://www.youtube.com/watch?v=QQeukXJMfy4

Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 29, 2017, 02:36:21 AM
Tnx! But I suspect that to be asynchronous, and the Arduino library uses a poller in it's examples to capture SysEx data. I have to code in some basic functionality first, and then I'll try to query the MS-3 for the active patch number, FX state, and maybe even some settings like 'switch on push/release'.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: sixeight on September 29, 2017, 03:51:49 AM
Quote from: MrHaroldA on September 29, 2017, 02:36:21 AM
Tnx! But I suspect that to be asynchronous, and the Arduino library uses a poller in it's examples to capture SysEx data. I have to code in some basic functionality first, and then I'll try to query the MS-3 for the active patch number, FX state, and maybe even some settings like 'switch on push/release'.

That is correct. Most of the data has to be "pulled out" of the MS-3. I do the same with my VController project for patch names and effect states.

Anything you change on the MS-3 should be sent in sysex automatically once editor mode is on. This way the BTS for MS3 keeps track of changes on the unit itself.

A good tutorial for the Roland midi system is http://www.2writers.com/eddie/tutsysex.htm (http://www.2writers.com/eddie/tutsysex.htm)
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vtgearhead on September 29, 2017, 05:05:44 AM
Quote from: gumtown on September 29, 2017, 02:28:33 AM
00 00 ZZ ZZ = data size required from start address

One word of warning:  This number is actually the address offset from start to the last byte you want to receive.  Depending upon your choice of starting address and offset you may get fewer bytes back than a simple subtraction would suggest. 

Update: When reading known parameters from their proper address you can ignore this distinction.  However, keep it in mind when doing bulk reads or you'll experience modest grief.  The MIDI sysex address range has discontinuities after each 128-byte page due to the need for data bytes to have bit 7 clear,  but the missing region is still accounted for when determining where to cut off a read.

Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on September 29, 2017, 05:21:44 AM
Are the query addresses the same as the set addresses? For example, if I set FX1 at '0x60, 0x00, 0x00, 0x30', can I read it at that address too?
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: sixeight on September 29, 2017, 06:30:34 AM
Quote from: MrHaroldA on September 29, 2017, 05:21:44 AM
Are the query addresses the same as the set addresses? For example, if I set FX1 at '0x60, 0x00, 0x00, 0x30', can I read it at that address too?

Yes you can. You will have to specify the number of bytes you want to read, so '0x60, 0x00, 0x00, 0x30 0x00 0x00, 0x00, 0x01' will read one byte from your address.

For the GP10 I have used the following code for reading data from the device:
void request_GP10(uint32_t address, uint8_t no_of_bytes) {
  uint8_t *ad = (uint8_t*)&address; //Split the 32-bit address into four bytes: ad[3], ad[2], ad[1] and ad[0]
  uint8_t no1 = no_of_bytes / 128;
  uint8_t no2 = no_of_bytes % 128;
  uint8_t checksum = MIDI_calc_Roland_checksum(ad[3] + ad[2] + ad[1] + ad[0] +  no1 + no2); // Calculate the Roland checksum
  uint8_t sysexmessage[18] = {0xF0, 0x41, GP10_device_id, 0x00, 0x00, 0x00, 0x05, 0x11, ad[3], ad[2], ad[1], ad[0], 0x00, 0x00, no1, no2, checksum, 0xF7};
  MIDI.sendSysEx(17, sysexmessage);
}

// Calculate the Roland checksum
uint8_t MIDI_calc_Roland_checksum(uint16_t sum) {
  uint8_t checksum = 0x80 - (sum % 0x80);
  if (checksum == 0x80) checksum = 0;
  return checksum;
}
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on October 01, 2017, 01:06:15 PM
I've added support for loading, editing and storing "FX sets"! :D Next up is actually reading something from the MS3 ...

https://www.youtube.com/watch?v=9xDCUllpelA

But first a beer and a bath.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on October 05, 2017, 12:48:48 AM
Quote from: MrHaroldA on October 01, 2017, 01:06:15 PMNext up is actually reading something from the MS3 ...

Yeah ... this is the tricky part.

If I query the MS-3, I get an answer. But if I don't query it, I can't get the Arduino to receive anything. For example: patch changes are sent by the MS-3 without having to query it, but I can only receive it by querying it with some random query.

Running my poller outside of USB_STATE_RUNNING makes no difference.

if (Usb.getUsbTaskState() == USB_STATE_RUNNING)

I'll try to find some examples of other implementations ... or do you guys have some ideas to try?


EDIT: I also get bundled replies when querying while switching patches:


Query: F0, 41, 0, 0, 0, 0, 3B, 11, 60, 0, 0, 30, 0, 0, 0, 1, 6F, F7: 0
Receiving: 20:  04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 00 30 07 00 70 F7
Query: F0, 41, 0, 0, 0, 0, 3B, 11, 60, 0, 0, 30, 0, 0, 0, 1, 6F, F7: 0
Receiving: 64:  04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 00 30 07 00 70 F7 04 F0 41 00 04 00 00 00 04 3B
12 00 04 01 00 00 04 00 00 7F 05 F7 00 00 04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 06 53 07 51 76 F7
Query: F0, 41, 0, 0, 0, 0, 3B, 11, 60, 0, 0, 30, 0, 0, 0, 1, 6F, F7: 0
Receiving: 20:  04 F0 41 00 04 00 00 00 04 3B 12 60 04 00 00 30 07 00 70 F7
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: sixeight on October 05, 2017, 01:22:41 AM
Quote from: MrHaroldA on October 05, 2017, 12:48:48 AM
If I query the MS-3, I get an answer. But if I don't query it, I can't get the Arduino to receive anything. For example: patch changes are sent by the MS-3 without having to query it, but I can only receive it by querying it with some random query.

Sorry, you lost me here.
Does the MS-3 not send any data when you change a parameter on the unit? That would mean it is not properly in editor mode.
Or is the problem with receiving MIDI data in general. It is normal that you have to check the USB handler for new data.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on October 05, 2017, 03:37:54 AM
Quote from: sixeight on October 05, 2017, 01:22:41 AM
Sorry, you lost me here.
Does the MS-3 not send any data when you change a parameter on the unit? That would mean it is not properly in editor mode.
Or is the problem with receiving MIDI data in general. It is normal that you have to check the USB handler for new data.

If I sniff the Midi with my PC, the MS-3 transmits the patch changes correctly; so the editor mode must be set correctly too.

But if I poll the USB from my Arduino, it never registers that patch change; unless I'm querying it at the same time. The patch change joins the queried data in one big message.

In my current project, all I need is the patch change to load a different set of FX combinations. Querying the effect state would be a bonus to select which FX set matches, and should be activated once a patch is loaded.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on October 05, 2017, 03:53:32 AM
Arduino interrupt on midi input?
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: sixeight on October 05, 2017, 05:19:27 AM
Very hard to help you without seeing your code. Have you seen the code of vit3k:

https://github.com/vit3k/katana_fx_controller (https://github.com/vit3k/katana_fx_controller)

The bit about Katana MIDI:
https://github.com/vit3k/katana_fx_controller/blob/master/src/katana/midi.cpp (https://github.com/vit3k/katana_fx_controller/blob/master/src/katana/midi.cpp)
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on October 05, 2017, 12:03:34 PM
Quote from: sixeight on October 05, 2017, 05:19:27 AM
Very hard to help you without seeing your code. Have you seen the code of vit3k:

https://github.com/vit3k/katana_fx_controller (https://github.com/vit3k/katana_fx_controller)

That is in fact very helpful!!! I stripped away all Katana-specific stuff and ended up with basically main.cpp and the katana class. I'll have to study the midi class some more, but I already spotted some things I didn't know before. Also the queue seems like a very good idea!

But ... it seems to have a memory leak? Every 'katana.set' I do, I lose around 17 bytes of RAM, until it runs out.


Quote from:  philjynx on October 05, 2017, 07:01:06 AM
Does your code check for incoming USB data continuously? If it doesn't and only checks after each of your sends, that would explain why you're not getting the info you want.

Yes, I check constantly ... I have to run my code against that of the Katana FX Controller to see what I'm doing wrong.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on October 05, 2017, 12:58:20 PM
Quote from:  philjynx on October 05, 2017, 12:23:50 PM
Without looking at the code, my guess would be that 'katana.set' assigns some memory that is not subsequently released.

Nope: it's in the receive part ;)


Initialized
Free RAM: 1056
Sending F0 41 00 00 00 00 3B 12 7F 00 00 01 01 7F F7
Sending F0 41 00 00 00 00 3B 12 60 00 00 30 00 70 F7
Received: F0 41 00 00 00 00 3B 12 00 01 00 00 00 01 7E F7 00
*** Memory usage changed: 933
Received: F0 41 00 00 00 00 3B 12 60 00 06 53 64 63 F7
Received: F0 41 00 00 00 00 3B 12 00 01 00 00 00 00 7F F7 00
*** Memory usage changed: 895
Received: F0 41 00 00 00 00 3B 12 60 00 06 53 64 63 F7
...


I'm beginning to suspect timing issues with my MS3. It fails to boot into editor mode occasionally, misses FX states sometimes when sending bulk messages ... I'll check the SysEx trace for the timings the Editor uses.


Edit: note the extra 00 suffix on the patch changes received from the MS3 ...
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on October 05, 2017, 02:46:48 PM
You will need at least 20ms wait between sending sysx data,
that is the minimum required by midi spec for a receiving device to digest data,
and also determine the gap between the last message and the next.

If you are expecting a reply, only send one message at one time.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vtgearhead on October 11, 2017, 06:26:28 AM
Quote from: gumtown on October 05, 2017, 02:46:48 PM
You will need at least 20ms wait between sending sysx data,
that is the minimum required by midi spec for a receiving device to digest data,
and also determine the gap between the last message and the next.

If you are expecting a reply, only send one message at one time.

I was not aware of the 20ms requirement for message separation.  Explains a lot of weirdness I experienced when working on my Katana bridge.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on October 11, 2017, 06:40:59 AM
I measured around 4ms delay between events in the Editor software... And 60ms after setting up the Editor mode. This sure made things way more reliable!
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: vit3k on October 20, 2017, 01:01:22 PM
Quote from: MrHaroldA on October 05, 2017, 12:03:34 PM
That is in fact very helpful!!! I stripped away all Katana-specific stuff and ended up with basically main.cpp and the katana class. I'll have to study the midi class some more, but I already spotted some things I didn't know before. Also the queue seems like a very good idea!

But ... it seems to have a memory leak? Every 'katana.set' I do, I lose around 17 bytes of RAM, until it runs out.


Yes, I check constantly ... I have to run my code against that of the Katana FX Controller to see what I'm doing wrong.

It definitely has a memory leak somewhere because it restarts itself after a while. I don't have time to work on this project right now but if you found out where it is I would really like to know.

Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: sixeight on October 21, 2017, 11:20:06 AM
Quote from:  philjynx on October 21, 2017, 11:02:25 AM
Purely hypothetical question for you.  If our gizmo was sending midi to some other gizmo and our gizmo had a partial brain fade resulting in a gap of more than 20 ms in its message, that would be seen as the end of the message by the receiving gizmo?

The same thing may happen that also happens with the Arduino midi library. There is only one buffer used for midi in messages. The buffer is filled through an interrupt routine triggered by receiving midi data. So it could be that the code is reading a message, but while it is processing the data it is written over by the next message through the interrupt routine.

I have seen this happen with the VController when it is reading Zoom devices. I am considering changing the midi library to support multiple midi buffers. But I am still investigating the issue.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on October 29, 2017, 08:47:16 AM
Quote from: sixeight on October 21, 2017, 11:20:06 AM
But I am still investigating the issue.

It seems a lot of messages are lost if I send a queue item (TX) and wait for the response (RX). I have the response time-out set to 100ms and I randomly (yet predictable) get an answer.


Single press: Load patch 0 set 0
Add to queue: 0
Add to queue: 1
Add to queue: 2
Add to queue: 3
Add to queue: 4
Add to queue: 5
Add to queue: 6
Add to queue: 7
Add to queue: 8
Add to queue: 9
Add to queue: 10
Add to queue: 11
Add to queue: 12
Handle queue: 0
TX: F0 41 00 00 00 00 3B 12 60 00 00 30 00 70 F7 (15)
RX: F0 41 00 00 00 00 3B 12 60 00 00 30 00 70 F7 (15)
Unhandled parameter: 0x60000030
Handle queue: 1
TX: F0 41 00 00 00 00 3B 12 60 00 04 30 00 6C F7 (15)
RX: F0 41 00 00 00 00 3B 12 60 00 04 30 00 6C F7 (15)
Unhandled parameter: 0x60000430
Handle queue: 2
TX: F0 41 00 00 00 00 3B 12 60 00 00 20 01 7F F7 (15)
*** Time-out reached.
Handle queue: 3
TX: F0 41 00 00 00 00 3B 12 60 00 00 21 00 7F F7 (15)
RX: F0 41 00 00 00 00 3B 12 60 00 00 21 00 7F F7 (15)
Unhandled parameter: 0x60000021
Handle queue: 4
TX: F0 41 00 00 00 00 3B 12 60 00 00 22 01 7D F7 (15)
*** Time-out reached.
Handle queue: 5
TX: F0 41 00 00 00 00 3B 12 60 00 02 29 01 74 F7 (15)
*** Time-out reached.
Handle queue: 6
TX: F0 41 00 00 00 00 3B 12 60 00 05 1E 00 7D F7 (15)
*** Time-out reached.
Handle queue: 7
TX: F0 41 00 00 00 00 3B 12 60 00 06 10 00 0A F7 (15)
*** Time-out reached.
Handle queue: 8
TX: F0 41 00 00 00 00 3B 12 60 00 06 30 00 6A F7 (15)
*** Time-out reached.
Handle queue: 9
TX: F0 41 00 00 00 00 3B 12 60 00 06 56 00 44 F7 (15)
*** Time-out reached.
Handle queue: 10
TX: F0 41 00 00 00 00 3B 12 60 00 0B 02 00 13 F7 (15)
*** Time-out reached.
Handle queue: 11
TX: F0 41 00 00 00 00 3B 12 60 00 0B 05 00 10 F7 (15)
*** Time-out reached.
Handle queue: 12
TX: F0 41 00 00 00 00 3B 12 60 00 0B 06 00 0F F7 (15)
*** Time-out reached.


Since I do get some answers, I guess my implementation is correct, but the Midi library fails to capture all received data.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: sixeight on October 29, 2017, 09:22:46 AM
Quote from: MrHaroldA on October 29, 2017, 08:47:16 AM
Since I do get some answers, I guess my implementation is correct, but the Midi library fails to capture all received data.

What happens if you change the order of the data you are requesting?
Is it certain addresses that do not respond? If you request data that does not exist, you will not get an answer.

Any chance you could give us a look at your code?
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on October 29, 2017, 10:46:35 AM
If you want an answer to a sysx data request, then use "11" instead of "12" in your messages.
You are sending data change, not requesting a reply.
The MS-3 is possibly only sending back the same value you are changing.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on October 29, 2017, 10:55:42 AM
Quote from: gumtown on October 29, 2017, 10:46:35 AM
The MS-3 is possibly only sending back the same value you are changing.

I know, but the MS-3 should return everything I set. I do have to check if the MS-3 also returns something if there was no change in state, for example by enabling the same effect twice.

But I'm watching Hello Kitty with my daughter right now ;)

I plan to release my MS-3 library soon, so I'll look into publishing it later this evening...
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on October 29, 2017, 01:30:34 PM
Quote from: MrHaroldA on October 29, 2017, 10:55:42 AM
I know, but the MS-3 should return everything I set. I do have to check if the MS-3 also returns something if there was no change in state, for example by enabling the same effect twice.

Hello Kitty is finished, daughter is sleeping, and watched a Stranger Things with the wife ... 8)

I've checked the replies, and while they are consistent they seem random over different effect sets.

I'll now clean up my library a little and write an example implementation.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on October 29, 2017, 02:46:08 PM
Here's my MS-3 library: https://github.com/MrHaroldA/MS3

I've added a really simple example to show how to use the library.

I would really love your remarks on this! Note that I don't really know what I'm doing, I'm just a simple php scripter by daytime ;)
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on October 30, 2017, 02:55:30 PM
Quote from: sixeight on October 29, 2017, 09:22:46 AM
What happens if you change the order of the data you are requesting?
Is it certain addresses that do not respond? If you request data that does not exist, you will not get an answer.

Any chance you could give us a look at your code?

I've just added an effect_state example (https://github.com/MrHaroldA/MS3/blob/master/examples/effect_state/effect_state.ino), which also misses out on a lot of effect states ... unless you enable debug/verbose mode!  ???


// Uncomment this to enable verbose debug messages.
#define MS3_DEBUG_MODE // *** Enable this!


It seems I have severe timing issues?
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on November 01, 2017, 06:39:28 AM
I've updated  effect_state example to keep trying to gather all data. The initial check is quite high, but I guess my queue overloads severely ;) Fixed in the latest revision.

All checks after that average around 150 tries to get 13 effect states ...

Ready!


Found all 13 effect states in 19560 tries.
FX1:  OFF
MOD1: OFF
L1:   OFF
L2:   OFF
L3:   OFF
FX2:  ON
MOD2: OFF
DLY:  OFF
REV:  OFF
NS:   ON
SOLO: OFF
CTL1: OFF
CTL2: OFF

Loaded patch 1. Checking all effect states.

Found all 13 effect states in 143 tries.
FX1:  ON
MOD1: ON
L1:   ON
L2:   ON
L3:   ON
FX2:  ON
MOD2: ON
DLY:  ON
REV:  ON
NS:   ON
SOLO: OFF
CTL1: OFF
CTL2: OFF

Loaded patch 2. Checking all effect states.

Found all 13 effect states in 143 tries.
FX1:  OFF
MOD1: OFF
L1:   OFF
L2:   OFF
L3:   OFF
FX2:  OFF
MOD2: OFF
DLY:  OFF
REV:  OFF
NS:   OFF
SOLO: OFF
CTL1: OFF
CTL2: OFF

Loaded patch 3. Checking all effect states.

Found all 13 effect states in 189 tries.
FX1:  OFF
MOD1: OFF
L1:   OFF
L2:   OFF
L3:   OFF
FX2:  OFF
MOD2: OFF
DLY:  OFF
REV:  OFF
NS:   OFF
SOLO: OFF
CTL1: OFF
CTL2: OFF


This now is quite a good test for measuring how much incoming data my implementation, or even the midi host library loses.
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on November 02, 2017, 02:26:05 PM
Found it! I was adding double items to the queue before the queue was empty, and I was picking up items from the queue before the timeout was over ...  ::)


Ready!

Loaded patch 0. Found all 13 effect states in 13 tries.
FX1:  OFF
MOD1: OFF
L1:   OFF
L2:   OFF
L3:   OFF
FX2:  ON
MOD2: OFF
DLY:  OFF
REV:  OFF
NS:   ON
SOLO: OFF
CTL1: OFF
CTL2: OFF

Loaded patch 1. Found all 13 effect states in 13 tries.
FX1:  ON
MOD1: ON
L1:   ON
L2:   ON
L3:   ON
FX2:  ON
MOD2: ON
DLY:  ON
REV:  ON
NS:   ON
SOLO: OFF
CTL1: OFF
CTL2: OFF

Loaded patch 2. Found all 13 effect states in 13 tries.
FX1:  OFF
MOD1: OFF
L1:   OFF
L2:   OFF
L3:   OFF
FX2:  OFF
MOD2: OFF
DLY:  OFF
REV:  OFF
NS:   OFF
SOLO: OFF
CTL1: OFF
CTL2: OFF

Loaded patch 3. Found all 13 effect states in 13 tries.
FX1:  OFF
MOD1: OFF
L1:   OFF
L2:   OFF
L3:   OFF
FX2:  OFF
MOD2: OFF
DLY:  OFF
REV:  OFF
NS:   OFF
SOLO: OFF
CTL1: OFF
CTL2: OFF


Is anybody reading along my monolog? You all seem awfully quiet ...
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: Elantric on November 02, 2017, 02:42:19 PM
I read each and every post

( click here:  and bookmark)

Show All Recent Posts
https://www.vguitarforums.com/smf/index.php?action=recent


glad to hear you located your problem
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: gumtown on November 02, 2017, 03:09:31 PM
Me too,  ;)
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on November 09, 2017, 01:40:38 PM
Well, I think i'm kinda done with the library ;) I even made some sort of Wiki (https://github.com/MrHaroldA/MS3/wiki) for it!

Receiving bulk data is still an issue, but debugging and improving the USB Host library is out-of-scope for now.

I'll try to focus on implementing the library and actually using my MS-3 now  ;)


Anybody looked at, or tried my library yet?
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: MrHaroldA on November 10, 2017, 01:15:19 AM
Quote from: sixeight on October 21, 2017, 11:20:06 AM
The same thing may happen that also happens with the Arduino midi library. There is only one buffer used for midi in messages. The buffer is filled through an interrupt routine triggered by receiving midi data. So it could be that the code is reading a message, but while it is processing the data it is written over by the next message through the interrupt routine.

I have seen this happen with the VController when it is reading Zoom devices. I am considering changing the midi library to support multiple midi buffers. But I am still investigating the issue.

@sixeight, can you please log an issue on the USB_Host_Shield_2.0 issue queue (https://github.com/felis/USB_Host_Shield_2.0/issues) for this? I think I run into this regularly, but I have no idea how to fix this nicely.


And @vit3k, your input is requested here: https://github.com/felis/USB_Host_Shield_2.0/issues/313#issuecomment-340712691
Title: Re: Boss MS-3 - MIDI over USB Sysex Reverse Engineering
Post by: sixeight on November 10, 2017, 06:52:00 AM
Quote from: MrHaroldA on November 10, 2017, 01:15:19 AM
@sixeight, can you please log an issue on the USB_Host_Shield_2.0 issue queue (https://github.com/felis/USB_Host_Shield_2.0/issues) for this? I think I run into this regularly, but I have no idea how to fix this nicely.

I don't have a USB host shield, so it is not a good idea to post an issue or fix if I am unable to test any of it.

It may be a better idea to post your problem and your code on the Arduino or Teensy forum.

I have not changed the serial midi sysex queue in the midi library after all. I was able to solve my problems by improving the loop time and do a check on expected length of sysex answers. In some cases the VController will ask for the same data twice now on serial3, but at least things are running ok again.