Well, then I tried it out with an AV receiver, which even has a convenient USB port for flash drives already build in. However, the bluetooth adapter seemed like it didn't power up. The same happened with another hi-fi system's USB port. However, when powered with a phone charger or by directly applying 5V to the GND and +5V lines of the USB plug, it worked. My first guess was, that those USB ports don't output enough power for the adapter. This was easily disproved by being able to charge my phone over this port. I measured the voltage on the two power lines: 5V. So this should be alright.
Apparently it had something to do with the data lines. I opened the adapter to have a look at those and indeed, they where connected to the bluetooth chip. Maybe for configuration/programming in the factory. It seemed like this is is what pees on the parade. The AV receiver tries to communicate with the adapter and thereby freezes it.
Unsoldering the plug and bending over the data lines to disconnect them is what fixes the problem.