Since the two storage systems required different table layouts, they are not compatible. However, in most scenarios the two systems can coexist without problems, allowing you to migrate your robots over time. You should not mix the usage of the new and the old system within a single robot.
In most cases you can execute robots using both storage systems on the same RoboServer, as long as you don't provide a DatabaseRobotInfo Environment to robots which use the new storage system. This means that if you use a deployment descriptor or API code to configure the environments, you will have to make a distinction in your code between robots using the new and the old system. Alternatively you can temporarily give new robots a RobotId allowing them to be run with the RobotInfo/Message environments, and then remove the environments once all robots have been migrated.
If a new robot is executed with a Database Storage Environment, it will be ignored, as long as the robot doesn't contain any old Database Output Objects or Return Object steps.
If you execute a robot with both a Database message environment and database logging enabled for RoboServer, you will get two different logs: One you access through RoboManager, and another one you access via the Log View in the Management Console.