The Data Director Bundle: The Interface Between Pimcore and Your Data Sources

The Data Director: One solution for everything

With its large feature set, the Data Director replaces many small-part packages for different requirements, reducing the initial effort to a one-time implementation. The Data Director is the bundle of choice for:

• Importing data into Pimcore elements (data objects, assets, documents)
• Checking and updating data as it is stored
• Exports of data in any format (e.g. CSV, XML, JSON, Excel) and with any follow-up actions (e.g. FTP upload, email sending, transfer to an API)
• REST API creation

Because the Data Director Bundle is highly flexible and customizable, it offers companies real added value in terms of efficiency and compatibility:

• Easy integration
  The great flexibility and possibilities for customization guarantee direct connection to existing systems.
• Maximum efficiency
  The solution is powerful and easy to use. This means that data imports and exports can be created quickly and in a target-oriented manner.
• Flexible and customizable
  Manual adjustments can be made to the target data model at any point during data imports and exports, without affecting automated processes.
• Sustainable
  The solution is open and can be freely developed and adapted to the specific needs of existing IT infrastructures.
• Transparency
  The status of imports and exports can be viewed at any time, and changes in certain fields can be subsequently tracked using searchable logs.

How the bundle works

Blackbit's Data Director allows external systems to be connected to Pimcore. The numerous functions and ready-to-use templates facilitate data maintenance, while intelligent features further accelerate the import and export of data.

The Data Director Bundle imports data from external XML, JSON, CSV, or Excel data sources (such as ERP systems), caches it in tabular form, and then generates or updates data objects, assets, and documents from the raw data.

The Pimcore backend graphical user interface can be used to define exactly which data should be pulled from the import source and to which object fields in the Pimcore objects they should be assigned. The feature supports all Pimcore data types, even importing assets and documents (including attribution of metadata, properties, tags, etc.) is possible.

While importing, data can be modified and adapted to your own data model. The bundle offers extensive functionality, a comfortable user interface, and is a battle-proven solution for importing structured data into Pimcore objects, assets, and documents. It reduces the effort for programming individual interfaces and shortens the time to go live with your Pimcore project.

Likewise, data of desired fields can be exported - including data of relations, images and thumbnails, etc. Thanks to the intelligent caching system, export processes run at high speeds, as export data is updated when an object gets saved, instead of in the moment the export is started. This background operation does not increase the time to save an object. For certain exports the bundle provides many ready-to-use templates:

• CSV exports
• CSV as zip file including assets
• JSON exports
• JSON as zip file including assets
• XML exports
• XML exports as zip file including assets
• Additional export templates can be added via separate add-on bundles, e.g. Google Shopping Feed, BMEcat export or interfaces to e-commerce APIs.

In addition, the bundle automatically creates REST API endpoints. For imports, this allows live interfaces to be implemented without having to load files from the source system into Pimcore and without cronjobs. Exports can be retrieved via URL using the REST API. The configuration is also done via the Pimcore backend.

Further possibilities are opened up by the option of not only creating export documents, but also specifying what is to be done with them. For this functionality, the Data Director Bundle is delivered with the following ready-to-use templates:

• Send export document via e-mail
• Upload the export document via (S)FTP, AWS S3, and other cloud storage providers.

With the automatic mapping functions and the templates provided, setting up imports and exports is done in minutes - in most cases without any programming. However, if necessary, import or export data can be customized using callback functions.

For a feature overview and detailed instructions on setting up imports and exports, see our Data Director video tutorials.

The workflow

• Step 1: Import CSV, XML, JSON or Excel into Pimcore
   

The import always takes place in two steps: The first step uses an intuitive user interface to specify which columns of a CSV file or which attributes of an XML or JSON document should be imported into Pimcore.

A preview directly shows whether the raw data fields are correctly structured. The import reads the data into an "intermediate table" so that the user can see which data has been imported from the source system before it is further processed in the next step.

• Step 2: Mapping the attributes via drag and drop

In the second step, the information from the intermediate table is mapped to the attributes of the classes. These mappings are individually defined in the Pimcore backend UI via drag-and-drop.

One or more key attributes are set there so that the import process can be repeated at any time and existing objects are updated.

Attribute mapping can also be used to modify source data. For each field mapping, a callback function can be specified, which performs e.g. date format operations or other calculations based on one or more imported or existing attributes of the current object.

Advantages of Data Director in comparison to other Pimcore import plugins

• Everything is configurable in Pimcore backend user interface - no creation of PHP files or anything similar necessary, even for complex imports.
• Better import performance
  • streaming input files allows to also import huge import files
  • a lot of optimizations to accelerate importing data to Pimcore elements - or to check previously if data actually changed (and if not skip import or saving to save time)
• Auto-complete functions to set up imports very fast and to suggest how to proceed even with complex imports
• Flexible import resource, you can import data from:
  • single files
  • folders
  • URLs
  • PHP script for complex requirements like importing a CSV file only if a PDF file with the same file name (but different file extension) exists in the import folder
  • cURL requests to import data from resources that need authentication (but of course you can also use a PHP script in this case which authenticates and then provides the data to be imported)
• Supported import formats:
  • CSV
  • XML
  • JSON
  • Excel
  • Pimcore elements (data objects, assets, documents)
  • Pimcore reports (e.g. to import data from Google Analytics, external databases, or other Pimcore database tables)
  • File system (e.g. to import assets)
• Full flexibility: for common use cases of data transformation the bundle provides ready-to-be used templates. But you can also edit those and even write custom (PHP or JavaScript) functions to set up a transformation pipeline, conditions, etc. for certain fields
• Import object hierarchy (set parent element of imported elements)
• Option to write-protect fields: only import data to the certain field if the field is empty
• Generate response documents, for example, to send success status back to the source system
• Automatically start imports, for example to:
  • automatically assign assets to data objects after uploading
  • automatically start import for uploaded CSV, JSON, XML, Excel file
• Trigger imports via push from external systems (via REST API) for live interfaces without exchanging files.
• Traceability: Import archive (for import files) and searchable import log history to always know when and why a certain field got a certain value.
• Revert imports: If data loss happened due to a mistake, only revert imported fields to a certain date - much better than having to restore a database dump (because data of non-imported fields is kept).
• Dataports are also stored as JSON files.
  Allows tracking of changes via VCS (Git) and easy deployment. Dataport configuration download and import are also supported.
• Optimize inheritance feature
• Permission system: for each dataport you can configure who is allowed to configure and/or execute the dataport.
• Import different types of elements:
  • Data objects
  • Assets
  • Documents
  • Quantity value units (incl. conversion factors)
• Built-in interface to translation providers (DeepL, AWS Translate) to set up automatic translation with just one click (incl. caching of translations to save costs for recurring translation strings).
• Artificial intelligence features, e.g. to automatically assign categories based on product name and description.

Advantages compared to other Pimcore export plugins

• Access any data which is connected to an exported object, for example when exporting products you can easily access assigned categories, images of the assigned categories, and even meta data of the images of the assigned categories - you can chain that as long as you want.
• Full flexibility in setting up a transformation pipeline to change values to the desired format (e.g. date format conversion, convert quantity values to certain units, etc.)
• Full flexibility for format/document structure of the export document.
• Predefined export templates to create CSV, XML, JSON exports with or without referenced asset files (e.g. CSV file plus assets (or thumbnails) packed in a zip archive).
• Automatically execute exports whenever an object gets saved whose data gets exported to:
  • Prepare export as soon as data changes, so that data does not have to be loaded at the moment an export is requested. This leads to very fast exports since the export document is already available at the moment of the request.
  • Upload exports automatically to a target system to always have up-to-date data there.
  • Automatically send data to the other APIs, e.g. online shops, marketplace distributors.
• Intelligent checks whether anything changed since the last export. If nothing changed, the result document gets delivered from the cache.
• Access exports via URL, for example, to pull data feed in an external system.

Advantages compared to own import implementation

• Minimization of programming effort: Only data modifications from the data source require minimal programming
• Performance
  • Data is only imported if nothing has changed since the last import
  • If an object has not changed during import, it does not need to be saved
  • Export data gets updated when objects change, so it is already prepared at the moment an export gets requested
• Flexibility
  • Imports and exports are fully customizable to the data source and your Pimcore data model
  • Supports all Pimcore elements including data objects, object bricks, field collections, assets, and documents
  • Supports importing tags and properties
• Transparency
  • with import archive and searchable import log history, you can always trace why a certain field got a certain value and which import file was responsible for this
  • always see current import/export status (progress of current job, number of queued items)
• Comfort functions
  • Automatically extract raw data fields from import resources
  • Automatically map raw data fields to Pimcore class fields
  • Optimize inheritance feature
  • Replace placeholders (for example to generate texts automatically)
  • Automatic translation (with DeepL API)
  • Error monitoring/notification of imports
  • Revert imports

The highlights at a glance

•  File-based imports: Monitoring of a directory and automatic import for new files, e.g. for
  • automatic asset imports into Pimcore (e.g. from a network drive),
  • automatic assignment of assets to Pimcore objects based on the file name.
• Import multiple files (CSV, XML, JSON, Excel) from a source directory with automatic record deduplication.
• Import of URLs as a data source
• Importing data from Pimcore objects:
  • Migrate data from one field type to another without data loss.
  • Dynamic bulk data manipulation (currently not possible in Pimcore grid operation such as increasing prices by 10%).
• Import data from Pimcore reports, e.g. to:
  • send reports by mail (e.g. for notifying responsible users when an action is required for report objects).
  • or to import data from Google Analytics into Pimcore data objects and further use this data.
• Importing documents (populating editables).
• Generating response documents:
  • Generate a response document with all successfully imported objects and send it back to the source system.
  • Track import errors.
  • Invoke another import that depends on the current one.
  • Generate export documents (JSON/CSV/XML etc.) that other systems can use as an import source.
  • Create response documents for single-page application/PWA front end requests.
• Perform imports and exports using the built-in REST API.
• Skip records from import source, e.g. if data is missing or its quality is insufficient.
• Easy import of relations.
• Automatic assignment of elements to relations via artificial intelligence/machine learning.
• Import of object hierarchies/object trees
  • Specification of the parent element possible
  • Option to optimize inheritance (data is entered as high as possible in the object hierarchy).
• Automatic translation of texts via translation providers (via DeepL or AWS API).
• Support for all Pimcore data types including relations with metadata, object modules, field collections, etc.
• Optimized performance:
  • If the source data for a certain system has not changed, the record import can be skipped.
  • If the data of an object has not been changed during import, it does not need to be saved.
• Generation of barcode and QR code images.
• Ability to undo imports:
  • If an error has crept in during attribute assignment and several thousand objects have been filled with incorrect data, the change can be undone. In this case, only the fields mapped in the import are reset, or only for the objects changed in the import - a great advantage over a complete backup restore.

Documentation

To learn more about all options and their configuration, please read the full documentation or watch the tutorial videos.

Get the Data Director Bundle here

You can easily order the bundle directly from the Blackbit Shop or send an email to info@blackbit.de.