Boss MS-3 - MIDI over USB Sysex Reverse Engineering

Started by MrHaroldA, August 29, 2017, 12:13:37 AM

Previous topic - Next topic

0 Members and 2 Guests are viewing this topic.

MrHaroldA

Hey guys, I'm Harold and I'm new here ;)

I'm the owner of the new Boss MS-3, and after a quick inspection with MidiOX, it receives SysEX over the emulated USB/Midi port, just like the Katana (and others) do!

The MS-3 lacks a Midi IN port, so I'm now trying to build a compatible Midi controller for it, that will connect through the USB connector instead. At the heart of it will be an Arduino; probably a Nano (or mini) with an USB shield, as the Arduino itself isn't capable of becoming a USB host.

I also have an Arduino Due here, which might be able to act as host ... but is pretty large ;)


I flipped through this thread, and didn't find anyone using an Arduino to control their Boss/Roland hardware, is that correct?
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

gumtown

Boss USB is not 'class compliant' so a host shield won't work out of the box.
You will have to find the device specific USB end point enumerators.
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

MrHaroldA

Quote from:  gumtown on August 29, 2017, 01:44:34 AM
Boss USB is not 'class compliant' so a host shield won't work out of the box.
You will have to find the device specific USB end point enumerators.
Hmmm... I hoped it to be class compliant, since my Linux machine recognized it (and even tried to use it as primary soundcard ;) )

I'll look into this, but I guess this will turn out to be pretty difficult for me, as I'm not familiar with this and the Boss of course won't display any debug information; it either works, or it doesn't.
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

admin

#3

The MS-3 with missing MIDI Input jack is yet another Roland Boss product that is crippled from the start
with 80% of what you need



If you  are using a raspberry pi ,or Beaglebone black - thanks to Linux Jack audio the MS-3  can be seen as a MIDI device in that environment.

same situation as the Boss GP-10)

Details
https://www.vguitarforums.com/smf/index.php?topic=11998

vit3k

#4
Quote from:  gumtown on August 29, 2017, 01:44:34 AM
Boss USB is not 'class compliant' so a host shield won't work out of the box.
You will have to find the device specific USB end point enumerators.

USB host shield for Arduino works just fine with katana. At least program change messages.

I have some code here: https://github.com/vit3k/arduino_midi
It is work in progress and may not work with current commit. Also I was using hub to use multiple devices so code is more complex because of it. I didn't check sysex though but it will probably work just fine.

admin

Quote from:  vit3k on September 01, 2017, 02:53:19 PM
USB host shield for Arduino works just fine with katana. At least program change messages.

I have some code here: https://github.com/vit3k/arduino_midi
It is work in progress and may not work with current commit. Also I was using hub to use multiple devices so code is more complex because of it. I didn't check sysex though but it will probably work just fine.

Thanks for the news regarding USB host shield for Arduino works just fine with Katana!

(perhaps there is hope yet !)

Beanow

Quote from:  MrHaroldA on August 29, 2017, 12:13:37 AM
I flipped through this thread, and didn't find anyone using an Arduino to control their Boss/Roland hardware, is that correct?

While not an arduino, the MIDX-20 is built with a PIC micro controller.
https://www.vguitarforums.com/smf/index.php?topic=21261.0

Quote from:  MrHaroldA on August 29, 2017, 02:05:01 AM
Hmmm... I hoped it to be class compliant, since my Linux machine recognized it (and even tried to use it as primary soundcard ;) )
The Linux kernel ships with a large amount of drivers (kernel modules) out of the box, including "snd_usb_audio" maintained by the Alsa folks. This module seems generic based on it's name, but actually contains many vendor-specific specifications and workarounds. Including the Boss/Roland non-standard audio and midi interfaces.
Some behind the scenes look would be the quirks table in the kernel source. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/sound/usb/quirks-table.h?h=v4.13-rc7#n1780

What I would recommend for testing if Sysex is received by the Katana or not, is to send a "Universal non-realtime identity request". It's a standard probe to find out more about the devices responding to Sysex requests. I would be formatted as hexadecimal raw midi:
F0 7E 7F 06 01 F7
This will request devices with any device ID to respond.
You should get a reply that looks something like this:
F0 7E 00 06 02 41 33 03 00 00 01 00 00 00 F7
Interpreted this reveals: 41 (Boss/Roland) 33 03 00 00 (Katana) 01 00 00 00 (Firmware version).

vit3k

#7
Quote from:  gumtown on August 29, 2017, 01:44:34 AM
Boss USB is not 'class compliant' so a host shield won't work out of the box.
You will have to find the device specific USB end point enumerators.

Today I've tried to use sysex messages with Arduino and Katana. Unfortunately it doesn't work. I can send messages but don't receive any. Looks like input endpoint is wrong.

Looks like gumtown is right and some changes to library will be needed. I will get back to this when I have some time. Gumtown - do you have some more information on this subject?

Or there is a bug in my code  ???

https://github.com/vit3k/katana_fx_controller

Edit:
I found a solution. I think this could be a bug in usb host library

usbh_midi.cpp flie, line 297:
if ((epDesc->bmAttributes & 0x02) == 2) {//bulk
should be changed to:
if ((epDesc->bmAttributes & 0x03) == 2) {//bulk

and that's because:
Bits 0..1 Transfer Type
00 = Control
01 = Isochronous
10 = Bulk
11 = Interrupt

With this change I receive all sysex messages. Tried with put Katana in BTS mode. It sends all changes to arduino :)

Beanow

#8
Quote from:  vit3k on September 04, 2017, 02:20:19 PM
Edit:
I found a solution. I think this could be a bug in usb host library

usbh_midi.cpp flie, line 297:
if ((epDesc->bmAttributes & 0x02) == 2) {//bulk
should be changed to:
if ((epDesc->bmAttributes & 0x03) == 2) {//bulk

and that's because:
Bits 0..1 Transfer Type
00 = Control
01 = Isochronous
10 = Bulk
11 = Interrupt

Edit:
Depending on the intentions of that line, that smells like a bug indeed.
For example, if epDesc->bmAttributes was 0b11 (Interrupt) it would evaluate as true, while commented as bulk and seems to check for bulk (== 2).
The pattern this check uses is of a "is this bit set?" suitable for flags. But it looks like the field is actually a 2-bit uint not a set of binary flags.

They are important properties to look at. On linux lsusb reports:


Interface #1
EP 13 OUT
  Transfer Type            Isochronous
  Synch Type               Asynchronous
  Usage Type               Data

Interface #2
EP 14 IN
  Transfer Type            Isochronous
  Synch Type               Asynchronous
  Usage Type               Implicit feedback Data

Interface #3, alternate 0
EP 3 OUT
  Transfer Type            Bulk
  Synch Type               None
  Usage Type               Data

EP 4 IN
  Transfer Type            Bulk
  Synch Type               None
  Usage Type               Data

Interface #3, alternate 1
EP 3 OUT
  Transfer Type            Interrupt
  Synch Type               None
  Usage Type               Data

EP 5 IN
  Transfer Type            Interrupt
  Synch Type               None
  Usage Type               Data


From my understanding interface 1 and 2 are your audio in and out.
Whereas interface 3 is for midi and it can be configured to run in interrupt or bulk mode.
This is about where my knowledge of the USB and MIDI spec ends through, but I do know that if these settings and endpoints are mismatched it won't work.

vtgearhead

Quote from:  vit3k on September 04, 2017, 02:20:19 PM
I found a solution. I think this could be a bug in usb host library

usbh_midi.cpp flie, line 297:
if ((epDesc->bmAttributes & 0x02) == 2) {//bulk
should be changed to:
if ((epDesc->bmAttributes & 0x03) == 2) {//bulk

and that's because:
Bits 0..1 Transfer Type
00 = Control
01 = Isochronous
10 = Bulk
11 = Interrupt

With this change I receive all sysex messages. Tried with put Katana in BTS mode. It sends all changes to arduino :)

If you need to consider bits 0 and 1 for the comparison then the original code is wrong.  Nice catch!


vit3k

Yes, it's a bug. I've post it on library's github and it's already fixed and merged.

https://github.com/felis/USB_Host_Shield_2.0/issues/313

And currently I'm trying to move to Teensy 3.2 as arduino lacks flash and ram and of course it doesn't work anymore.

Does anyone have any experience in running usb host shield (full sized or mini) with teensy? Is it possible that because of fast SPI it's not working on breadboard or with 20cm cables?

MrHaroldA

#11
Hey guys; looks like you all did a fine job while I was away on vacation!  8)

vit3k's fix is already released in v1.3.1 of the host library, so I hacked up this tiny script to test if I could send SysEx to my MS-3:

#include <usbh_midi.h>

USB  Usb;
USBH_MIDI  Midi(&Usb);

// SysEx data for loading Patch 0 & 1.
uint8_t data1[] = {0xF0, 0x41, 0x00, 0x00, 0x00, 0x00, 0x3B, 0x12, 0x00, 0x01, 0x00, 0x00, 0x00, 0x7F, 0xF7};
uint8_t data2[] = {0xF0, 0x41, 0x00, 0x00, 0x00, 0x00, 0x3B, 0x12, 0x00, 0x01, 0x00, 0x00, 0x01, 0x7E, 0xF7};

void setup() {
    Serial.begin(38400);
    while (!Serial) {}

    // Halt if USB won't init.
    if (Usb.Init() == -1) {
        Serial.println(F("USB Init error"));
        while (1);
    }

    delay(200); // @TODO: Why?
    Serial.println(F("Ready!"));
}

void loop() {
    Usb.Task();
    if (Usb.getUsbTaskState() == USB_STATE_RUNNING) {
        // @TODO: Handle incoming USB data.
    }

    Serial.print(F("Load patch 0: "));
    Serial.println(Midi.SendSysEx(data1, sizeof(data1))); // sizeof(data1) == 15;
    delay(1000);

    Serial.print(F("Load patch 1: "));
    Serial.println(Midi.SendSysEx(data2, sizeof(data2)));
    delay(1000);
}


Resulting in:

Ready!
Load patch 0: 13
Load patch 1: 13
Load patch 0: 14
Load patch 1: 13
Load patch 0: 13
Load patch 1: 13
...


(the second time patch 0 is loaded it always returns 14, the rest returns 13)


It would have been very nice if this would work, but I'd also be surprised if it did ;)

I'm off to polish my USB/Midi/Arduino/C/debug knowledge now ...
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

vit3k

I think sizeof will return the size of pointer. Pass the size of array or use SendData method that will calculate size of sysex message for you.

MrHaroldA

Quote from:  vit3k on September 21, 2017, 06:17:09 AM
I think sizeof will return the size of pointer. Pass the size of array or use SendData method that will calculate size of sysex message for you.

'sizeof(data1)' returns 15 in this case, which is the number of bytes in the array ...

My best guess is now that I missed a byte, as 'F0 41 00 00 00 00 3B 12 00 01 00 00 00 00 7F F7' is 16 bytes ;)

I hope to report back this weekend!
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

MrHaroldA

Moving Midi.SendSysEx() into the 'if (Usb.getUsbTaskState() == USB_STATE_RUNNING) {' block made the MS-3 react to my messages! :D

It now flashes "BULK DATA RECEIVING..." it it's display ... instead of changing patches.

If I manually send the SysEx sequence above with MIDI OX, the patch is loaded correctly, so now I have to figure out why the MS-3 does not consider this to be a SysEx command if it's coming from my Arduino.

The SendData method acts the same, btw. Both methods return 0, which translates to "success".
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

MrHaroldA

This is weird. My SysEx (F0 41 00 00 00 00 3B 12 00 01 00 00 00 00 7F F7) for loading patch 1 works when the MS-3 Editor/Librarian is open, but doesn't when it is not running. So, I sniffed the FX1 toggle and used that instead; works like a charm, even from the Arduino!!!  8)

But here's another strange thing: the display flashes "BULK DATA RECEIVING..." when the Editor/Librarian isn't running, but not if it is. So, I thought it might issue a SysEx command to silence the display, but it should send one to re-activate it too when exiting the Editor/Librarian; but it doesn't.

Using amidi on Linux also flashes the "BULK DATA" message ...


Did they hide something in the Windows driver?  ???
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

gumtown

You need to send a sysx command to put the MS-3 into "verbose mode".

"name": "editor_comunication_level",
      "address": "7F,00,00,00",
      "size": "00,00,00,01"
    },
    {
      "name": "editor_comunication_mode",
      "address": "7F,00,00,01",
      "size": "00,00,00,01"
    },
    {
      "name": "running_mode",
      "address": "7F,00,00,02",
      "size": "00,00,00,01"
    },
    {
      "name": "patch_copy",
      "address": "7F,00,01,00",
      "size": "00,00,00,04"
    },
    {
      "name": "patch_write",
      "address": "7F,00,01,04",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_1_pitch",
      "address": "7F,00,02,00",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_2_pitch",
      "address": "7F,00,02,02",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_3_pitch",
      "address": "7F,00,02,04",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_4_pitch",
      "address": "7F,00,02,06",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_5_pitch",
      "address": "7F,00,02,08",
      "size": "00,00,00,02"
    },
    {
      "name": "tuner_multi_mode_str_6_pitch",
      "address": "7F,00,02,0A",
      "size": "00,00,00,02"
    },
    {
      "name": "patch_clear",
      "address": "7F,01,00,03",
      "size": "00,00,00,01"
    },
    {
      "name": "patch_initialize",
      "address": "7F,01,00,04",
      "size": "00,00,00,01"
    },
    {
      "name": "tuner_single_mode_pitch",
      "address": "7F,00,05,00",
      "size": "00,00,00,02"
    },
    {
      "name": "usb_direct_mon",
      "address": "7F,00,06,00",
      "size": "00,00,00,01"
    },
    {
      "name": "usb_in_out_mode",
      "address": "7F,00,06,01",
      "size": "00,00,00,01"
    },
    {
      "name": "patch_map",
      "address": "7F,70,00,00",
      "size": "00,00,00,01"
    },
    {
      "name": "update_patch_map",
      "address": "7F,71,00,00",
      "size": "00,00,00,01"
    },
    {
      "name": "manual_sw",
      "address": "7F,01,00,07",
      "size": "00,00,00,01"
    },
     {
      "name": "quick_setting_wirte",
      "address": "7F,01,00,00",
      "size": "00,00,00,02"
    },
    {
      "name": "ab_utilty",
      "address": "7F,01,00,02",
      "size": "00,00,00,01"
    },
    {
      "name": "ctl_midi_send",
      "address": "7F,00,07,00",
      "size": "00,00,00,01"
    }
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

MrHaroldA

Quote from:  gumtown on September 22, 2017, 04:36:54 PM
You need to send a sysx command to put the MS-3 into "verbose mode".

Tnx! I'll crosscheck the SysEx commands the Editor sends with your list; or is it the "running_mode" you are referring to?
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

gumtown

I am not sure which one, somewhere in the first 3 addresses. I suspect 7F 00 00 01
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

vit3k

Quote from:  snhirsch
If it behaves like the Katana, then:

7F,00,00,00,01   Enters editor mode
7F,00,00,00,00   Exists editor mode

I think 7F,00,00,01 set to 01 enters and 00 leaves.

MrHaroldA

Anybody got the checksum for that command at hand? I still have to write the logic to create them myself.
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

gumtown

Those commands are send 'as is' with no checksum and no sysx header/footer.

Arduino checksum calc:

int sum = 128 - (data_size % 128);
if (sum == 128) { sum = 0; };

String checksum = String(sum, 16);
if (sum < 16) { checksum = String(("0") + checksum); };
Free "GR-55 FloorBoard" editor software from https://sourceforge.net/projects/grfloorboard/

MrHaroldA

#22
I've tried these combinations with the SendData() method:


0x7F, 0x00, 0x00, 0x01
0x7F, 0x00, 0x00, 0x00, 0x01
0x7F, 0x00, 0x00, 0x01, 0x01
0x7F, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01


But the display still flashes the BULK DATA message when receiving SysEx.

Here's my current working (except for the flashing display) code, including the logic for calculating the checkum:

#include <usbh_midi.h>

USB Usb;
USBH_MIDI Midi(&Usb);

const uint8_t SYSEX_START = 0xF0;
const uint8_t SYSEX_END = 0xF7;
const uint8_t MANUFACTURER_ID = 0x41;
const uint8_t DEVICE_ID = 0x00;
const uint8_t MODEL_ID[] = {0x00, 0x00, 0x00, 0x3b};
const uint8_t QUERY = 0x11;
const uint8_t SET = 0x12;

// SYSEX_START + MANUFACTURER_ID + DEVICE_ID + MODEL_ID + QUERY/SET + PARAMETER + <variable dataSize> + CHECKSUM + SYSEX_END.
const uint8_t FIXED_MESSAGE_LENGTH = 14;

// Special instructions.
const uint8_t EDITOR_MODE[] = {0x7F, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01};

// Toggles.
const uint8_t TOGGLE_ON[] = {0x01};
const uint8_t TOGGLE_OFF[] = {0x00};
const uint8_t TOGGLE_FX1[] = {0x60, 0x00, 0x00, 0x30};

// Global variables.
bool ready = false;

void setParameter(uint8_t *parameter, uint8_t *data, uint8_t dataSize) {
    uint8_t message[FIXED_MESSAGE_LENGTH + dataSize], i;

    // Construct the full message.
    message[0] = SYSEX_START;
    message[1] = MANUFACTURER_ID;
    message[2] = DEVICE_ID;
    for (i = 0; i <= 3; i++) {
        message[i + 3] = MODEL_ID[i];
    }
    message[7] = SET;
    for (i = 0; i <= 3; i++) {
        message[i + 8] = parameter[i];
    }
    for (i = 0; i < dataSize; i++) {
        message[i + 12] = data[i];
    }
    message[sizeof(message) - 2] = checksum(parameter, data, dataSize);
    message[sizeof(message) - 1] = SYSEX_END;

    // Debug printing.
    Serial.print(message[0], HEX);
    for (i = 1; i < sizeof(message); i++) {
        Serial.print(", ");
        Serial.print(message[i], HEX);
    }
    Serial.print(": ");
    Serial.println(Midi.SendSysEx(message, sizeof(message)));
}

uint8_t checksum(uint8_t *parameter, uint8_t *data, uint8_t dataSize) {
    uint8_t sum, i;

    for (i = 0; i <= 3; i++) {
        sum = (sum + parameter[i]) & 0x7F;
    }
    for (i = 0; i < dataSize; i++) {
        sum = (sum + data[i]) & 0x7F;
    }

    return (128 - sum) & 0x7F;
}

void setup() {
    Serial.begin(38400);
    while (!Serial) {}

    // Halt if USB won't init.
    if (Usb.Init() == -1) {
        Serial.println(F("USB Init error"));
        while (1);
    }

    delay(200); // @TODO: Why?
    Serial.println(F("Ready!"));
}

void loop() {
    Usb.Task();
    if (Usb.getUsbTaskState() == USB_STATE_RUNNING) {
        if (!ready) {
            Serial.print(F("Set editor mode: "));
            Serial.println(Midi.SendData(EDITOR_MODE));
            ready = true;
        }

        Serial.print(F("Set FX1 to 1: "));
        setParameter(TOGGLE_FX1, TOGGLE_ON, sizeof(TOGGLE_ON));
        delay(1000);

        Serial.print(F("Set FX1 to 0: "));
        setParameter(TOGGLE_FX1, TOGGLE_OFF, sizeof(TOGGLE_ON));
        delay(1000);
    }
}
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3

sixeight

QuoteI've tried these combinations with the SendData() method:

You will have to send 0xF0 before and 0xF7 at the end to make it a proper system message...

MrHaroldA

Quote from:  sixeight on September 24, 2017, 03:30:29 AM
You will have to send 0xF0 before and 0xF7 at the end to make it a proper system message...

But ...

Quote from:  gumtown on September 24, 2017, 02:21:40 AM
Those commands are send 'as is' with no checksum and no sysx header/footer.

I'll try anyway; one moment please!
Boss MS-3 and Katana library for Arduino: https://github.com/MrHaroldA/MS3