Easily migrate from one DAM to another, either one-time or continuously (if the DAM generates events).
The underlying technology is Apache Camel, a fantastic integration framework with all sorts of EIP.
Every supported DAM is implemented as a Camel-Endpoint, providing a consumer (fetch asset from a DAM) and/or a producer (upsert assets to a DAM).
This is developer-ware, as every migration will require some custom mapping code.
A basic camel route will look like this:
public class MyRoutes extends RouteBuilder {
@Override
public void configure() {
from("source-dam://old.place.com?token=xxxxx&query=some-condition")
.transform().body(CommonAsset.class, asset -> {/* custom mapping code*/})
.to("target-dam://new.place.com?token=xxxxx&previous-ID=some-field");
}
}In practice however we would advise to use a queue for buffering and rate-limiting:
from("source-dam://old.place.com?token=xxxxx&query=some-condition")
.transform().body(CommonAsset.class, asset -> {/* custom mapping code*/})
.to("vm:my-queue?blockWhenFull=true&size=50");
from("vm:my-queue?concurrentConsumers=5")
.throttle(250).timePeriodMillis(60000) // in case you are rate-limited
.to("target-dam://new.place.com?token=xxxxx&previous-ID=some-field");This is where all the pre-built camel stuff comes in handy!
As we're trying to map from one DAM to another, and every DAM has different ideas when it comes to metadata, folders, relations, etc., we map everything to and from dam-common. Overview:
... note that metadata and relations themselves use Identifiable/CommonEntity themselves, allowing relations between arbitrary DAM objects (e.g. the folder structure an asset is assigned to is simply a relation to a CommonNode).
Usage: bynder://your-instance.bynder.com?token=xxxxx, based on their API.
type: String, required: yes, default: -
The API token used to authenticate against Bynder's API. To create one, head over to "Advanced Settings > Portal Settings > Permanent Tokens"
type: String, required: no (but recommended), default: -
The metaproperty text-field (name or UUID) where the asset ID of the previous DAM is stored. This enables deduplication and later updates to the asset.
type: locale, required: no, default: English
When encountering a Nameable that needs to be resolved to a string, which locale to prefer. This should probably be set to the default locale of the source.
type: boolean, required: no, default: false
Handy for development, does the mapping, but without the final call to the API.
type: boolean, required: no, default: false
Whether to update assets found via the previousIdField, even though they've already been uploaded before.
type: String, required: no, default: -
When in consumer mode, what asset query to run against Bynder ("and"-joined), e.g.
property_foo = bar and dateCreatedFrom = 2020-01-01T00:00:00 and tag = baz
type: int, required: no, default: 50
How many assets to fetch per page, max is 1000
type: String, required: no, default: -
When looking for a previousIdField, only look in this collection (yeah, special case)
Usage: celum-remote://some.place.com/remoteapi?clientId=xxxx&clientSecret=xxxx&serverSecret=xxxxx. Requires their Remote-SDK (which only works up to version 6.20).
Note that only the consumer side has been implemented - use CELUM REST if you need a producer.
type: String, required: yes, default: -
The client ID for the Remote-SDK (see appserver/conf/remote.yml)
type: String, required: yes, default: -
The client secret for the Remote-SDK (see appserver/conf/remote.yml)
type: String, required: yes, default: -
The server secret for the Remote-SDK (see appserver/conf/remote.yml)
type: String, required: yes (until listeners are implemented), default: -
A SearchUtil2 query that define the scope of the assets to be transferred, e.g. assetType=123 and info_456=true.
Note that due to the poor quality of the Remote-SDK, not every query parameter can be used (most notably everything involving a NOT, except emptiness-checks).
type: Long, required: no, default: 1
The download format ID that should be used, defaults to OriginalFormat (1 on most servers, 6 on older ones - see CMA).
type: Long, required: no, default: -
Under certain circumstances you may prefer an alternative download format for certain assets.
When enabled, the endpoint will check if the preferred format is available for a given asset - otherwise downloadFormat will be used.
type: path, required: no, default: -
An alternative to the assetQuery, expects one asset ID per line, which is then fetched as if found via a query. Useful for migrating a known set of assets.
Usage: celum-rest://instance.celum.hosting?clientId=foo&apiKey=bar. Requires the new REST API >= 6.20.
type: String, required: yes, default: -
The REST API client ID to use, the middle bit of restClient.<ClientIdentifier>.enabled, see their docs. E.g. mobileapp.
type: String, required: yes, default: -
The REST API key to use, as set in CMA > Authentication > Profiles > REST
type: String, required: no, default: -
When in consumer mode, what asset query to run against CELUM. The syntax is like SearchUtil2, e.g. nodeRecursive=123 & modified >= "2020-01-01"
type: int, required: no, default: 1
When in consumer mode, what download format to request. Usually the original file format.
type: long, required: no (but recommended), default: -
The information field where the asset ID of the previous DAM is stored. This enables deduplication and later updates to already-uploaded asset.
type: boolean, required: no, default: false
Whether to update assets found via the previousIdField, even though they've already been uploaded before.
type: long, required: no, default: -
When in producer mode, and you have not provided a relation to indicate which node the asset should be uploaded to, fall back to this node. This probably indicates a problem in your mapping code, but at least it got uploaded (and could still be moved later on).
type: String, required: no, default: fetched from the instance
The default locale of this CELUM instance. When not provided, it will be queried via the API.
type: String, required: no, default: tollerant
When creating new folders, which validation mode to create them with (strict or tollerant).
type: String, required: no, default: -
When in producer mode, which permission defining node types to copy, when an asset it assigned to it. E.g. folder:101,public:105
type: String, required: no, default: -
When in producer mode, which non-permission defining node types to copy, when an asset it assigned to it. E.g. collection:103
type: long, required: no, default: -
Node-ID of the trashbin -> ignore assets that are assigned to it.
type: bean name, required: no, default: NoopDuplicateStrategy
When uploading new assets, which strategy to use to detect duplicates. Existing strageties: FilenameDuplicateStrategy, ChecksumDuplicateStrategy.
type: boolean, required: no, default: false
Search for duplicates globally across the entire system instead of only within the target node.
type: boolean, required: no, default: false
Links globally found duplicates to the target node (only works with searchForDuplicatesGlobally).
type: boolean, required: no, default: false
When in producer mode, fetch all node referencing keywords ahead of time, which reduces search calls at runtime.
type: boolean, required: no, default: false
Whether to skip metadata updates when encountering an asset matching through previousIdField.
type: boolean, required: no, default: false
Whether to skip node assignment updates when encountering an asset matching through previousIdField
Usage: cloudinary://cloud-name?apiKey=x&apiSecret=y. Cloud name refers to the name of the instance on Cloudinary, e.g. test01-some-customer.
Note that only the producer side has been implemented so far.
type: String, required: yes, default: -
The API key as delivered by Cloudinary.
type: String, required: yes, default: -
The API key as delivered by Cloudinary.
type: String, required: no, default: -
The metadata field where the asset ID of the previous DAM is stored. This enables deduplication and later updates to already-uploaded asset.
type: boolean, required: no, default: false
Instead of using a previousId, put the previous DAM ID into their public_id.
type: boolean, required: no, default: false
Whether to skip existing files (requires previousId or previousIdAsPublicId to be set)
type: path, required: no, default: -
When defined, log Cloudinary upload results in a JSONL file.
type: String, required: no, default: EN
The default locale used for paths and multilingual fields
type: String, required: no, default: ""
The fallback asset_path that an asset is uploaded to, when none has been sent. Default is empty string -> root level
type: , required: no, default: false
Whether to overwrite existing files (requires previousId or previousIdAsPublicId to be set). Only use this if you're sending pre-filtered assets deemed relevant to overwrite.
Usage: pixxio://customer.px.media?apiKey=x.
Note that only the producer side has been implemented so far.
type: String, required: yes, default: -
The API key as set in pixx.io (Settings > API > API-Key).
type: long, required: no (but recommended), default: -
The meta-property field where the asset ID of the previous DAM is stored. This enables deduplication and later updates to already-uploaded asset.
type: boolean, required: no, default: false
Whether to skip existing files (requires previousIdField to be set).
type: boolean, required: no, default: false
When uploading a local file, delete it after a sucessfull upload (files downloaded from a URL will always be deleted again). Relevance depends on the source.
type: long, required: no, default: 10k
Connect timeout in milliseconds when downloading assets.
type: long, required: no, default: 300k
Read timeout in milliseconds when downloading assets.
type: long, required: no, default: 300k
Timeout in milliseconds when uploading assets.
Usage: sharedien://api.demo.sharedien.com?entityType=Asset&token=x
Note that only the producer side has been implemented so far.
type: String, required: yes, default: -
The Sharedien API token to use (Admin > Security > Access Token).
type: String, required: no, default: Asset
The entity type to transfer, possible values: 'Asset', 'Category', any custom entity type identifier.
type: boolean, required: no, default: false
Whether to completely skip creation of entities which doesn't exist yet.
type: boolean, required: no, default: false
Whether to completely skip existing entities instead of trying to update them.
type: boolean, required: no, default: false
Whether to skip metadata updates on existing entities.
type: boolean, required: no, default: false
Whether to skip relation updates on existing entities.
type: boolean, required: no, default: false
Whether to throw an Exception after failing to process the message
type: String, required: no, default: {}
The full search request JSON. Refer to api.{customer}.sharedien.com and look at /assets/search for details.
type: String, required: no, default: -
Specifies the desired translation languages as comma separated values, eg: en,de,fr.
type: String, required: no, default: -
The identifier of a browser to combine this search with (think of this as a named filter).
type: String , required: no, default: Thumbnail
The derivation type for downloads, such as "Original", "Thumbnail", etc.
type: boolean, required: no, default: true
Whether to load and include the parent category.
type: boolean, required: no, default: true
Whether to load and include all linked categories.
type: String, required: no, default: -
Property containing the ID of the previous system (for dedupe).
type: int, required: no, default: -
Upload chunk size in bytes
type: boolean, required: no, default: false
Whether to force HTTP/1.1 as protocol
type: int, required: no, default: -
Limit the number of relations created in bulk (a value <= 0 is considered as unlimited)
type: String, required: no, default: -
Date property containing the last binary modification date of the asset. Should an asset found via previousIdProperty have an older date, a new version is uploaded.
type: boolean, required: no, default: false
Whether to set relations on the original asset when a duplicate was detected. This is only relevant for Asset entities
type: boolean, required: no, default: false
Whether to replace the version on existing assets with the presented binary (requires a previousIdProperty match)
type: String, required: no, default: -
When updating asset relations on an existing asset, remove all existing relations of the given type first
Usage: uptempo://dam.customer.com?username=x&password=y
Note that only the consumer side has been implemented so far.
type: String, required: yes, default: -
The username for REST.
type: String, required: yes, default: -
The password for REST.
type: String, required: no, default: EN
The default locale to use for translatable fields and names.
type: , required: no, default: -
An asset query, based on SearchFiltersWrapperRestDto, in JSON format, e.g. "{"filters": [{"type": "UPLOAD_DATE","value": [{"search": "1970-01-01"},{"search": null}]}]}";. See their docs for details.
type: String, required: no, default: -
Only sync these specific asset IDs (comma separated).
type: int, required: no, default: -
Which rendition (download format) to use when fetching an asset.
type: int , required: no, default: -
The maximum number of assets that a consumer generates per query.
type: , required: no, default:
System.getProperty("java.io.tmpdir")
Temporary directory to store downloaded binaries in (they don't have stable download URLs).
type: boolean, required: no, default: false
Whether the consumer should download asset binaries at all.
type: boolean, required: no, default: false
Whether the consumer should query the download statistics of an asset.
© brix Solutions AG