Skip to main content

Architecture

Presents a global overview of the software that makes up a DPU / DimML runtime.

Overview​

Architecture Overview

The DimML platform is a modular multi-tenant runtime to execute dimml files concurrently. There are many places in which the default behavior can be altered or extended. The module system is the main extension mechanism. Modules are hot-swappable isolated jar files that can have their own set of dependencies. They extend the dimml language by providing flow processors and plugins. Modules are beyond the scope of this document.

The platform source code is structured around the following sub-projects:

  • dimml-common, common utility code used throughout the other projects, e.g. URI handling, triggers, static functions, cryptography, etc. Relies on Google Guava and Bouncy Castle.
  • dimml-js, decentralized Javascript generation library based on processing actions
  • dimml-code, execution runtime for script code used in dimml files, e.g. server-side Javascript, Groovy. Relies on Nashorn, Groovy and Jython.
  • dimml-dom, Document Object Model for parsing and serializing dimml files
  • dimml-api / dimml-impl, API to access the DimML repository (DimML's main management interface) and default implementation supporting modules in dimml-impl. Relies on Aether.
  • dimml-server, Netty based TCP server, handling HTTP(S) and Websocket requests; bootstrapping the DimML repository. Relies on Netty, Hazelcast, AWS SDK, Quartz and many smaller libraries.

DimML's main entry point is the DimmlBootstrap component. When a new server is started the following simplified steps are taken by DimmlBootstrap:

  1. Load configuration.
  2. Setup repository: discovery and loading scheme for dimml files, loading scheme for modules.
  3. Start Netty TCP server with default pipeline (HTTP + Websocket).

In step 2 the default behavior of the comoponents is:

  • Discovery, discover associated dimml code based on an incoming 'subject URL'. The subject URL is usually the referrer of the HTTP request. The default discovery mechanism uses the host in the URL to build a series of host name related dimml files that should be looked up in the database.
  • Module loader, monitors a location in Amazon S3 for new or changed jar files that include module code. Changes in module code don't require a full DimML restart.
  • Dimml code loader, loads source dimml code based on a URI scheme. The default mechanism loads files from a database.
  • Event distribution, plugins are executed when specific events occur in the DimmlServer. The execution of a plugin is handled by the DimmlRepository, but it requires access to protocol details which are provided by an EventContext. Some events are: SCRIPT_STATIC, SCRIPT_DYNAMIC, FLOW, INPUT, CONCEPT_RESOLUTION, REQUEST, etc.

After a DimML server has fully started it is ready to receive requests. The rest of the components are explained by describing the steps taken to complete a request.

  1. Data enters DimML through an HTTP call handled by DimmlServer.
  2. The concept associated with the subject URL is requested from the Repository.
  3. Discovery and Dimml code loader are used by the Repository to get all dimml code.
  4. The DOM is used to parse and merge dimml code.
  5. Plugins in the concept are executed by triggering events.
  6. Data is sent to the default flow of the associated concept.
  7. Flow processors and plugins may execute script code using the Code Execution framework.
  8. Some plugins use the Javascript Builder to alter or generate Javascript code for the client.
  9. All event and flow usage is logged using HandlerUsageMetrics.

Caching occurs at various stages throughout the process described above.

Infrastructure​

Infrastructure

Communication between the browser and DimML can be divided in the following categories:

  • static Javascript code (dimml.js, domain code, concept code)
  • dynamic Javascript code (data collection, personalization)
  • API calls

These categories have different requirements on server-side processing. Static Javascript can be cached extensively to reduce latency. For that reason DimML employs a Content Delivery Network (CDN) to store static files close to the end-user:

  • cdn.dimml.io

The dynamic Javascript code requires complex processing on the server-side. It is important that failures in processing have limited effect on other partners. DimML uses different endpoints based on the partner that processes the data. There is also a separate endpoint for development code:

  • baltar.dimml.io
  • baltar-360450.dimml.io
  • baltar-w2UKFaB1.dimml.io
  • baltar-360451.dimml.io
  • baltar-dev.dimml.io
  • baltar-sandbox.dimml.io
  • alpha.dimml.io

Typical dynamic calls are generated by the static Javascript served by DimML. For these calls the integration point on the website is usually the SLOC (//cdn.dimml.io/dimml.js). DimML is in full control of the partner specific host names as used in the list above and changing them does not require updating the code of every client website.

For API calls and some other data collection requests (e.g. image requests) it is required to hardcode one of the platform-specific domains. That would make switching to a different partner or between development and production considerably more difficult. To avoid using any platform-specific domain in client HTML / Javascript, all platforms know about each other and will proxy requests to the proper platform as specified in dimml code. A special alias domain can be used to address what is known as the 'main' platform:

  • dynamic.dimml.io