Edit on GitHub

Migrations

A common tasks in evolving application is the need to migrate data and data structures to a specific format. Common examples are adding a new column to a database table or importing or altering a Pimcore class definition.

To be able to execute migration changes across environments, Pimcore integrates the Doctrine Migrations library which provides a powerful migration framework. Building on Doctrine Migrations, Pimcore adds the following features:

  • Migrations can be split up into multiple migration sets which are independent from each other. By default, there is one global app migration set and one set for every bundle which implements migrations in its installer, but you can configure Pimcore to handle additional migration sets.
  • As Doctrine Migrations is targeted to DB migrations only, Pimcore adds a couple of features to use migrations also for other changes such as changing class definitions. For example, a migration in Pimcore is able to determine if it is in --dry-run state (simulation) in order to decide if it changes things or not. For pure DB changes this is easy, as collected SQL statements are simply not executed, but when changing a class definition much more is changed in the background. Therefore the migration itself needs to decide if it executes its change or not.

Besides of those details, please refer to the Doctrine Migrations documentation for further information on migrations. You can also take a look at the DoctrineMigrationsBundle documentation, but Pimcore uses its commands internally and does not load the bundle itself. Therefore, configurations under the doctrine_migrations key will not be available (this is handled by Pimcore as we support multiple migration sets).

In general, the migration commands provided by Pimcore are the same as provided by the DoctrineMigrationsBundle, but start with pimcore:migrations, e.g. pimcore:migrations:migrate.

Migrations can be either used as global migration set or as a bundle specific one. For bundles, a dedicated MigrationInstaller takes care of defining a migration set and of interacting with migrations. There is a dedicated documentation page on installers which describes the interaction between installers and migration in detail. This pages describes the basic functionality which apply to all migrations.

Using migrations

Let's use a simple example to demonstrate how to use migrations inside Pimcore. Assume your project needs a DB table which is not handled via Pimcore's classes but is just a plain DB table you'll use in your code. To create this table, we'll use a first migration which defines the basic table structure.

Start off by looking at the basic migration configuration. If you don't pass a migration set name via --set, it will default to the app migration set which creates migration classes in app/Resources/migrations.

$ bin/console pimcore:migrations:status

 == Configuration

    >> Name:                                               Migrations
    >> Database Driver:                                    pdo_mysql
    >> Database Name:                                      pimcore5
    >> Configuration Source:                               manually configured
    >> Version Table Name:                                 pimcore_migrations
    >> Version Column Name:                                version
    >> Migrations Namespace:                               App\Migrations
    >> Migrations Directory:                               app/Resources/migrations
    >> Previous Version:                                   Already at first version
    >> Current Version:                                    0
    >> Next Version:                                       Already at latest version
    >> Latest Version:                                     0
    >> Executed Migrations:                                0
    >> Executed Unavailable Migrations:                    0
    >> Available Migrations:                               0
    >> New Migrations:                                     0

Creating a first migration

Migrations can be generated with the pimcore:migrations:generate command:

$ bin/console pimcore:migrations:generate
Generated new migration class to "app/Resources/migrations/Version20171005123020.php"

As you can see, the migration class defines an up and a down method which will be executed when migrating in one of those migrations. In general, the down method should reverse the changes done in up. Each method receives a Doctrine\DBAL\Schema\Schema object containing an abstract database schema representation which you can use to alter the database structure. If this is not enough, you can add raw SQL queries via $this->addSql(). Please refer to the Doctrine Schema-Representation documentation for details on the schema object.

<?php

namespace App\Migrations;

use Doctrine\DBAL\Schema\Schema;
use Pimcore\Migrations\Migration\AbstractPimcoreMigration;

/**
 * Auto-generated Migration: Please modify to your needs!
 */
class Version20171005123020 extends AbstractPimcoreMigration
{
    /**
     * @param Schema $schema
     */
    public function up(Schema $schema)
    {
        // this up() migration is auto-generated, please modify it to your needs

    }

    /**
     * @param Schema $schema
     */
    public function down(Schema $schema)
    {
        // this down() migration is auto-generated, please modify it to your needs

    }
}

Let's update our migration to create a table.

<?php

namespace App\Migrations;

use Doctrine\DBAL\Schema\Schema;
use Pimcore\Migrations\Migration\AbstractPimcoreMigration;

class Version20171005123020 extends AbstractPimcoreMigration
{
    public function up(Schema $schema)
    {
        $table = $schema->createTable('foo');
        $table->addColumn('title', 'string');
        $table->addColumn('description', 'string');
    }

    public function down(Schema $schema)
    {
        $schema->dropTable('foo');
    }
}

Executing the pimcore:migrations:status command now will show us one migration to execute:

$ bin/console pimcore:migrations:status

 == Configuration

    >> Name:                                               Migrations
    >> Database Driver:                                    pdo_mysql
    >> Database Name:                                      pimcore5
    >> Configuration Source:                               manually configured
    >> Version Table Name:                                 pimcore_migrations
    >> Version Column Name:                                version
    >> Migrations Namespace:                               App\Migrations
    >> Migrations Directory:                               app/Resources/migrations
    >> Previous Version:                                   Already at first version
    >> Current Version:                                    0
    >> Next Version:                                       2017-10-05 12:30:20 (20171005123020)
    >> Latest Version:                                     2017-10-05 12:30:20 (20171005123020)
    >> Executed Migrations:                                0
    >> Executed Unavailable Migrations:                    0
    >> Available Migrations:                               1
    >> New Migrations:                                     1

To actually execute the migration, you can use the pimcore:migrations:migrate command which will migrate to the latest known version. Executing this command will apply every defined migration which wasn't executed yet.

$ bin/console pimcore:migrations:migrate

                    Migrations


WARNING! You are about to execute a database migration that could result in schema changes and data lost. Are you sure you wish to continue? (y/n)y
Migrating up to 20171005123020 from 0

  ++ migrating 20171005123020

     -> CREATE TABLE foo (id INT UNSIGNED AUTO_INCREMENT NOT NULL, title VARCHAR(255) NOT NULL, PRIMARY KEY(id)) DEFAULT CHARACTER SET UTF8MB4 COLLATE utf8mb4_unicode_ci ENGINE = InnoDB

  ++ migrated (0.72s)

  ------------------------

  ++ finished in 0.72s
  ++ 1 migrations executed
  ++ 1 sql queries

Updates to the initial schema

This works as expected. Our application now can use the created database table and re-create the table in every environment by simply migrating to the latest version. Now assume after some time you decide to add a new column to your table. To do so just generate a new migration and alter its code:

<?php

namespace App\Migrations;

use Doctrine\DBAL\Schema\Schema;
use Pimcore\Migrations\Migration\AbstractPimcoreMigration;

class Version20171005124853 extends AbstractPimcoreMigration
{
    public function up(Schema $schema)
    {
        $table = $schema->getTable('foo');
        $table->addColumn('description', 'text');
    }

    public function down(Schema $schema)
    {
        $table = $schema->getTable('foo');
        $table->dropColumn('description');
    }
}

A pimcore:migrations:status will now show a new migration which is ready to be applied:

$ bin/console pimcore:migrations:status
 
  == Configuration
 
     >> Name:                                               Migrations
     >> Database Driver:                                    pdo_mysql
     >> Database Name:                                      pimcore5
     >> Configuration Source:                               manually configured
     >> Version Table Name:                                 pimcore_migrations
     >> Version Column Name:                                version
     >> Migrations Namespace:                               App\Migrations
     >> Migrations Directory:                               app/Resources/migrations
     >> Previous Version:                                   0
     >> Current Version:                                    2017-10-05 12:30:20 (20171005123020)
     >> Next Version:                                       2017-10-05 12:48:53 (20171005124853)
     >> Latest Version:                                     2017-10-05 12:48:53 (20171005124853)
     >> Executed Migrations:                                1
     >> Executed Unavailable Migrations:                    0
     >> Available Migrations:                               2
     >> New Migrations:                                     1

Changes can again be applied via pimcore:migrations:migrate:

$ bin/console pimcore:migrations:migrate
  
                      Migrations
  
  
  WARNING! You are about to execute a database migration that could result in schema changes and data lost. Are you sure you wish to continue? (y/n)y
  Migrating up to 20171005124853 from 20171005123020
  
    ++ migrating 20171005124853
  
       -> ALTER TABLE foo ADD description LONGTEXT NOT NULL
  
    ++ migrated (0.92s)
  
    ------------------------
  
    ++ finished in 0.92s
    ++ 1 migrations executed
    ++ 1 sql queries

Migration Sets

As mentioned above, Pimcore's migrations support multiple migration sets which are completely independent from each other. Each migration set defines an own migration namespace and an own directory where migration classes will be located. When executing migrations for a specific migration set it has no effect on other migration set. E.g. a bundle can handle its own DB schema updates in its own migration set while being fully independent from app migrations which update class definitions which are valid for the whole application.

By default Pimcore defines the following migration sets:

  • A global app migration set which looks for migrations in app/Resources/migrations. This is the default migration set for all migrate commands if no specific set name was passed as option. All examples on this page refer to the app migration set.
  • One migration set for every bundle which implements a MigrationInstaller.

When interacting with migrations, you can use 2 command line options to choose the right migration set:

  • --bundle/-b with a bundle name: $ bin/console pimcore:migrations:status -b AppBundle
  • --set/-s with a set name: $ bin/console pimcore:migrations:status -s app

Defining custom migration sets

Additional migration sets can be added via configuration. To add a new set, add an entry to the pimcore.migrations.sets config entry:

pimcore:
    migrations:
        sets:
            custom_migrations:
                name: My Custom Migrations
                namespace: CustomMigrations
                directory: "%kernel.project_dir%/src/CustomMigrations"

After adding the entry, you can start using your migration set:

$ bin/console pimcore:migrations:status -s custom_migrations
  
   == Configuration
  
      >> Name:                                               My Custom Migrations
      >> Database Driver:                                    pdo_mysql
      >> Database Name:                                      pimcore5
      >> Configuration Source:                               manually configured
      >> Version Table Name:                                 pimcore_migrations
      >> Version Column Name:                                version
      >> Migrations Namespace:                               CustomMigrations
      >> Migrations Directory:                               src/CustomMigrations
      >> Previous Version:                                   Already at first version
      >> Current Version:                                    2017-10-05 12:54:29 (20171005125429)
      >> Next Version:                                       Already at latest version
      >> Latest Version:                                     0
      >> Executed Migrations:                                0
      >> Executed Unavailable Migrations:                    0
      >> Available Migrations:                               0
      >> New Migrations:                                     0

Using a predefined database connection in a migration set

By default, every migration set will operate on the default DBAL connection (the one used by Pimcore). If you want your set to always use another connection, you can add a connection entry to the set configuration.

# assuming there is an additional DBAL connection configured
# please refer to the doctrine bundle documentation regarding DBAL configuration
# https://symfony.com/doc/master/bundles/DoctrineBundle/configuration.html
doctrine:
    dbal:
        connections:
            custom_connection:
                # ...


pimcore:
    migrations:
        sets:
            custom_migrations:
                name: My Custom Migrations
                namespace: CustomMigrations
                directory: "%kernel.project_dir%/src/CustomMigrations"
                connection: custom_connection

Alternatively you can choose the connection to use by passing the --db option to the migration command.

If a connection config entry is passed, it will override the connection selected via --db.

Using the DI container inside migrations

As described on the DoctrineMigrationsBundle documentation page, migrations implementing ContainerAwareInterface will have direct access to the service container:

<?php

namespace App\Migrations;

use Doctrine\DBAL\Schema\Schema;
use Pimcore\Migrations\Migration\AbstractPimcoreMigration;
use Symfony\Component\DependencyInjection\ContainerAwareInterface;
use Symfony\Component\DependencyInjection\ContainerAwareTrait;

class Version20171005125429 extends AbstractPimcoreMigration implements ContainerAwareInterface
{
    use ContainerAwareTrait;

    public function up(Schema $schema)
    {
        $service = $this->container->get('service-id');
    }

    public function down(Schema $schema)
    {
    }
}

Non-DB Changes

Despite being a DB-oriented approach, migrations can be used to alter other structural elements as class definitions. Inside migrations, you can make full use of Pimcore's APIs, for example to alter or import a class definition. As an example, we'll alter the description field in a blogArticle class:

<?php

namespace App\Migrations;

use Doctrine\DBAL\Schema\Schema;
use Pimcore\Migrations\Migration\AbstractPimcoreMigration;
use Pimcore\Model\DataObject\ClassDefinition;

class Version20171005125429 extends AbstractPimcoreMigration
{
    public function up(Schema $schema)
    {
        // writeMessage writes to output and formats your message as doctrine would do for SQL queries
        $this->writeMessage('Adding description to blogArticle class');
        
        /** @var ClassDefinition $classDefinition */
        $classDefinition = ClassDefinition::getByName('blogArticle');
        $classDefinition->setDescription('[MIGRATIONS] ' . $classDefinition->getDescription() ?? '');
        $classDefinition->save();
    }

    public function down(Schema $schema)
    {
        $this->writeMessage('Removing description from blogArticle class');
        
        /** @var ClassDefinition $classDefinition */
        $classDefinition = ClassDefinition::getByName('blogArticle');
        $classDefinition->setDescription(preg_replace('/^\[MIGRATIONS\] /', '', $classDefinition->getDescription()));
        $classDefinition->save();
    }
}

This migration can be executed as every other migration by excuting update:

$ bin/console pimcore:migrations:migrate
  
                      Migrations
  
  
  WARNING! You are about to execute a database migration that could result in schema changes and data lost. Are you sure you wish to continue? (y/n)y
  Migrating up to 20171005125429 from 0
  
    ++ migrating 20171005125429
  
       -> Adding description to blogArticle class
  Migration 20171005125429 was executed but did not result in any SQL statements.
  
    ++ migrated (0.76s)
  
    ------------------------
  
    ++ finished in 0.76s
    ++ 1 migrations executed
    ++ 0 sql queries

As you can see, the change was successfully applied, but yielded a warning about SQL statements not being applied. As our changes are not executed through the schema object or the addSql method, the migrations library does not know what has been applied triggers the warning. To avoid this, the AbstractPimcoreMigration defines the following method, which can be used to suppress the warning if it returns false:

<?php

namespace AppBundle\Migrations;

// [...]

class Version20171005125429 extends AbstractPimcoreMigration
{
    public function doesSqlMigrations(): bool
    {
        return false;
    }
    
    // [...]
}

As a result, the warning is now gone

$ bin/console pimcore:migrations:migrate

                    Migrations


WARNING! You are about to execute a database migration that could result in schema changes and data lost. Are you sure you wish to continue? (y/n)y
Migrating up to 20171005125429 from 0

  ++ migrating 20171005125429

     -> Adding description to blogArticle class

  ++ migrated (0.34s)

  ------------------------

  ++ finished in 0.34s
  ++ 1 migrations executed
  ++ 0 sql queries

Handling --dry-run in non-DB migrations

Doctrine Migrations can be executed in a dry-run mode which does not actually change data. For plain SQL migrations this is quite easy as all SQL queries are collected before being executed. In dry-run mode those collected queries are simply not executed. For migrations not only handling DB changes, this is more complicated as for example a class definition change does not only change SQL structure but also writes class files, updates references etc. By default, migrations do not know about the dry-run state, but the AbstractPimcoreMigration implements the DryRunMigrationInterface which adds the dry-run context on the migration itself. If you use migrations to apply changes which not only affect the database, you can use this information to decide if you really want to execute the changes.

<?php

namespace App\Migrations;

use Doctrine\DBAL\Schema\Schema;
use Pimcore\Migrations\Migration\AbstractPimcoreMigration;
use Pimcore\Model\DataObject\ClassDefinition;

class Version20171005125429 extends AbstractPimcoreMigration
{
    public function doesSqlMigrations(): bool
    {
        return false;
    }

    public function up(Schema $schema)
    {
        // dryRunMessage will prefix the message with "DRY-RUN:" in dry run mode
        $this->writeMessage($this->dryRunMessage('Adding description to blogArticle class'));

        if ($this->isDryRun()) {
            // nothing to do
            return;
        }

        /** @var ClassDefinition $classDefinition */
        $classDefinition = ClassDefinition::getByName('blogArticle');
        $classDefinition->setDescription('[MIGRATIONS] ' . $classDefinition->getDescription() ?? '');
        $classDefinition->save();
    }

    public function down(Schema $schema)
    {
        $this->writeMessage($this->dryRunMessage('Removing description from blogArticle class'));

        if ($this->isDryRun()) {
            return;
        }

        /** @var ClassDefinition $classDefinition */
        $classDefinition = ClassDefinition::getByName('blogArticle');
        $classDefinition->setDescription(preg_replace('/^\[MIGRATIONS\] /', '', $classDefinition->getDescription()));
        $classDefinition->save();
    }
}

If you execute this migration with the --dry-run option, it will output its message (prefixed with DRY-RUN) but not actually change data. Also, the migration is not marked as executed and you can execute the migration by omitting the -dry-run option.

# simulate first
$ bin/console pimcore:migrations:migrate --dry-run

                    Migrations


Executing dry run of migration up to 20171005125429 from 0

  ++ migrating 20171005125429

     -> DRY-RUN: Adding description to blogArticle class

  ++ migrated (0.14s)

  ------------------------

  ++ finished in 0.14s
  ++ 1 migrations executed
  ++ 0 sql queries
  

# apply the change
$ bin/console pimcore:migrations:migrate

                    Migrations


WARNING! You are about to execute a database migration that could result in schema changes and data lost. Are you sure you wish to continue? (y/n)y
Migrating up to 20171005125429 from 0

  ++ migrating 20171005125429

     -> Adding description to blogArticle class

  ++ migrated (0.32s)

  ------------------------

  ++ finished in 0.32s
  ++ 1 migrations executed
  ++ 0 sql queries

For further details please see