Dewi - Version 2.11.0 - Release 2025-10-29¶

This is a small release with some bug fixes and preparations for future improvements on the usage of schemas and namespaces in dewi.

Features: We've created a namespace property that will replace the schema property in time. But for now it is optional.

Bug Fixes We ensured that everywhere we use utf-8 as our text encoding to prevent subtle bugs with different encoding standards. We made some small changes to the import asset fucntionality to make sure that the method get_assets() is only called once. There was an issue with importing nested assets which is now fixed.

Dewi - Version 2.10.0 - Release 2025-09-25¶

This release should have no impact (except for some whitespace changes) to your generated code when upgrading from v2.9.1

Features:

Pluggable Dewi

We've rewritten the core of dewi to support a plugin architecture. Plugins can provide configuration schemas, config preprocessors, config compilers, code generators and file assets that will be used in a dewi build command.

The sqlserver implementation has been moved to an default installed plugin (and will be removed from the main dewi repo in a later version)

To list the installed plugins run: dewi plugin ls To create a new plugin run: dewi plugin new (local-plugin, local-datamodel-plugin, standalone-plugin, standalone-datamodel-plugin)

It will guide you through a wizard to setup your first plugin. To start out always use a local-plugin or a local-datamodel-plugin. Guides and tutorials will be made available on dewi's documentation website

Asset Imports:

With the release of the plugin architecture we've added a new build phase in dewi which is the import assets phase, which you can run via dewi import or dewi build

highlevel process: Load project configuration -> Load plugins -> Import Assets -> Load configurations -> Validate configurations -> Preprocess configurations -> Compile to Datamodel -> Generate code -> Persist code to disk

The import process fetches Asset's from the installed plugins and saves them in a dewi/.dewi-plugins/{plugin-name} directory after which they will be used in the build process. This opens up the possibillity to easily share pieces of your datamodel (Yaml configurations, DQL statements) via the plugin system.

Extension Configuration:

Plugin developers are able to extend the datamodel with their own properties.

Objects with Extensions Support: Query - Stores query definitions with type, procedure_type, incremental_key, input, and content ReferenceColumn - References to columns in other tables (schema, table, column) ColumnMapping - Maps columns to their upstream sources LoadType - Defines how data is loaded with query and column mappings Flow - Named collection of load types for ETL processes Column - Table columns with framework-specific metadata, database fields, and relations Table - Complete table definitions with columns, flows, and metadata

All these objects include an extensions field with the same structure:

extensions: Optional[Dict[str, Any]] = None

For configurable properties we've added support to add extensions in a few places on all the existing configuration types. For plugins to use in their business logic

On the main level like the following:

type: SAT
...
...
extensions:
  plugin_name:
    plugin-attribute: True

and on the load type configurations, like:

type: SAT
...
...
flows:
  default:
    load_types:
      full:
        extensions:
          plugin_name:
            plugin-attribute: True
      incremental:
        extensions:
          plugin_name:
            plugin-attribute: True

Configuration types that have this extension setup: - Staging - ODS - Hub - Satellite - Link - Link Satellite - Dimension - Fact

Dewi - Version 2.9.1 - Release 2025-09-09¶

Reverts:

• Driving Key Links Data Consistency: As the current implementation only addressed this issues in CURRENT objects when the current object was virtual and not physical. We have chosen to revert to the old (but consistent) handling of the current data for LINKS. The correct implementation of this bugfix is planned for the 2.10 release

Dewi - Version 2.9.0 - Release 2025-09-08¶

Features:

Temporary Tables Support

Dewi now supports temporary table preparation steps that execute before the main data loading operations. This powerful feature enables complex data transformations using intermediate temporary tables while maintaining full data lineage tracking throughout the entire process.

How it works:

Preparation queries must use the SELECT ... INTO #temp_table syntax and be enclosed within special comment blocks: - Start block: --- PREPARE --- - End block: --- END PREPARE ---

Dewi automatically parses these preparation queries and traces data lineage through all intermediate steps, ensuring complete visibility into your data transformation pipeline.

Example usage:

--- PREPARE ---

SELECT
    customer_id,
    order_date,
    total_amount
FROM raw.orders
WHERE order_date >= '2024-01-01'
INTO #recent_orders;

SELECT
    customer_id,
    SUM(total_amount) as total_spent,
    COUNT(*) as order_count
FROM #recent_orders
GROUP BY customer_id
INTO #customer_summary;

--- END PREPARE ---

-- Main query using prepared temporary tables
SELECT
    cs.customer_id,
    cs.total_spent,
    cs.order_count,
    c.customer_name
FROM #customer_summary cs
JOIN dim.customers c ON cs.customer_id = c.customer_id

This feature is particularly useful for: - Complex data aggregations requiring multiple steps - Data quality checks before loading - Performance optimization through staged processing - Maintaining clean separation between preparation and final loading logic

Minor improvements:

• Inserts into satellite tables will use the SAT's _CURRENT table (only when stored in a physical table) when available to prevent costly recalculation of latest records

Bugfixes:

• Driving Key Links Data Consistency: Resolved a critical issue with driving key links where records could be missed when relationships appeared, changed, and then reverted over time. The fix ensures complete data consistency across all relationship state changes.

• Configuration Cleanup: Removed the unused enabled attribute from the current_view configuration in LINK and SAT objects. This attribute had no implementation and was causing confusion in configuration validation.

Dewi - Version 2.8.0 - Release 2025-07-18¶

Dewi v2.8 introduces the first Data Observabillity feature (as a preview feature). We introduced a new feature flag: dewi.generation.generate_data_profiling, which when set to True will generate a whole bunch of stored procedures to calculate and store data profiling statistics.

It generates 4 types of procedures:

• A container procedure to profile an entire database schema udp_profiling_<schema>
• A procedure for a table udp_profiling_<schema>_<table>
• A procedure for only the table stats (like row count and column count) udp_profiling_table_stats_<schema>_<table>
• A procedure for the column stats (like value distribution, amount of nulls, etc.) udp_profiling_table_stats_<schema>_<table>_<column>

Enabling you to mix an match these profiling steps in a way that suits your project. The procedures tracks the following statistics:

prf.ProfilingTable:

• Row count
• Column count

prf.ProfilingColumn:

• Record count
• Distinct value count
• Amount of NULLS
• Uniqueness of the values
• Min/Max/Average/Median values
• 5,25,50,75,95 Percentiles

With timestamps so you can analyse or show trends over time.

Generation:

• Major performance improvements in the insert into SAT procedures
• SQL Project file migration to use the new SDK syntax
• SQL Project file include the master reference table dacpac so you can use system stored procedures and tables without build errors

Dewi - Version 2.7.2 - Release 2025-07-18¶

Bugfixes:

• Fixes the issue introduced in v2.7.0 where dewi would no longer delete files

Dewi - Version 2.7.1 - Release 2024-11-06¶

Bugfixes:

• Custom current views were not respecting the feature flag for custom query parsing, now they do

Dewi - Version 2.7.0 - Release 2024-04-08¶

Compilation:

• All paths in the dewi datamodel will be formatted as POSIX paths. Improving the interoperability between operating systems (Yay Mac and Linux users!)

Generation:

• Major perfomance improvements of loading physical and virtual CURRENT objects by precalculating the MAX loaddts for all SAT hashkeys
• Adds DISTINCT to the staging query of the RST SAT insert DML. This enables adding a flow to a HUB table from a source with "duplicates" or where the hub key functions as a "foreign" key (late arriving dimensions)
• Expanded the SQL Server logging with extra parameters:
  • StartTime of the proccess
  • EndTime of the proccess
  • Duration of the proccess
  • Number of inserted records
  • Number of deleted records
  • Number of updated records
  • Status, indicates if the proccess failed or succeded
  • Comments, Object name + success or failure , can be customized
  • User that called the proccess

Bugfixes:

• Fixes parsing of "SUBSTRING" function in custom SQL queries
• Dewi does not fail on empty configs anymore (when you add --- at the end of a dewi config file)
• Again some fixes for custom queries that have a CTE
• Fixes stored procedure logging hardcoded schema name
• Fixes haskhey or hashdiff generation for tables with more than 128 columns (don't ask me why you would need this, but it's possible now)

Dewi - Version 2.6.6 - Release 2023-12-08¶

Bugfixes:

• Fixes bug with relative and absolute paths in SQL project file generation

Dewi - Version 2.6.5 - Release 2023-11-30¶

Bugfixes:

• Fixes a bug in the current_view path calculation. Input paths should always be relative to the output dir to properly support CI/CD pipelines.

Dewi - Version 2.6.4 - Release 2023-11-04¶

Bugfixes:

• Fixes bug in rewriting global query config for DIMs and FACT tables

Dewi - Version 2.6.3 - Release 2023-10-10¶

Bugfixes:

• Fixes issue with replacing hashkey postfix when deducing upstream column names in LINK configuration

Dewi - Version 2.6.2 - Release 2023-07-06¶

Bugfixes:

• Fixes bug with custom paths and content in Custom object compiler

Dewi - Version 2.6.1 - Release 2023-06-21¶

Bugfixes:

• Fixes issue with custom queries and excluded characters like ' and " in the SELECT statement

Dewi - Version 2.6.0 - Release 2023-06-12¶

General:

• Data lineage parsing is added to CUST objects (this is enabled with the dewi.compiler.custom_sql_parsing feature flag)
• custom CURRENT DQL statemens will be parsed and validated.

Bugfixes:

• Foreign key and index names will be limited to 128 characters on SQL Server
• Data lineage parsing now also supports inline subqueries, nested unions and multiple references to the same physical object

Dewi - Version 2.5.0 - Release 2023-05-10¶

Internals:

• The compilation process now exists of two steps preprocessing and compilation to start with simplifying the compiler code. In preprocessing the defaults are applied and some configuration validation is done. In compilation their should be only code to turn the config into a datamodel
• First proof of concept implementation of SQL query parsing to determine the data lineage of custom queries
• Improved overall performance by 60%

Configuration:

• There is a new table type ODS see below for more info
• You can now set your project name in the project.yaml with the attribute project_name
• It's now allowed to have any decimal type in the configuration (not only DECIMAL(19,5))
• Adds a generic way to enable and disable features in dewi, see below for more info

Bugfixes:

• Fixed bug with LINK types when referencing the same hub multiple times, this would result in using the same upstream columns for both HK calculations. It'll now prefix the source keys in the staging table (backported to v2.4.5)
• Fixed the weird behaviour where a randown query was chosen during generation of DML statements when it was impossible to properly find the parts of the configured custom query. It will now throw an exception
• Moved the SET FINISH TIME in all SQL Server stored procedures to be after the SET @RECORDCOUNT

ODS Tables:

The ODS type will make it easier to construct an Operational Datastore layer in the datawarehouse. When configuring the ODS type you will get a generated landing table for full, incremental, delete loads. The persistent staging table and when configured also a temporal table with history tracking. For every load type it'll generate load procedures which are optimized for the type of table. Non history tracking tables will perform better.

type: ODS
entity: activities
layer: STAGING # optional
schema: stg # optional
name: acitivities_CURRENT # optional
track_history: True # Current implementation will create a SQL Server system versioned temporal table for this
columns:
- name: ID
  primary_key: True # An ODS table does require a PK
- name: activity
- name: duration
  data_type: DECIMAL(19,8)  # this is allowed now :)
flow: # flows are optional. It'll default to a full and incremental load with a flow named default
  default:
    upstream:
    - schema: stg
      name: activity      # the ODS type is the first dewi type to create the staging table if it does not already exist in the config

Data lineage parsing:

In this release, data lineage parsing has been implemented for FACT and DIM tables. However, it is disabled by default and can be enabled using a feature flag (more on that below).

When enabled, the parsing functionality will extract column mapping information from custom queries and save it in the datamodel.yaml. Currently, it has the following considerations:

• The parsing logic is strict in aliasing. If a column is referenced without specifying its source, the parsing will fail even if the query runs successfully.
• It should handle subqueries, CTEs, and other common constructs well, but it may encounter difficulties with more complex or unusual query structures.

Please enable the feature in your project and test it by parsing custom queries. If any queries fail to parse correctly, add bug stories to the project and provide the query along with the expected result. This will help improve the parsing logic.

Feature flags:

Feature flags are introduced to allow early adopters to assist in the development and refinement of new features before they are rolled out to all Dewi installations. The feature flags control specific features or changes and can be configured in the project.yaml using the features attribute. Currently, there are two available features:

• dewi.compiler.custom_sql_parsing (lineage parsing): Disabled by default.
• dewi.configuration.yaml_c_loader (performance increase): Enabled by default.

If these features are not explicitly configured in project.yaml, the default settings will be as follows:

features:
- name: dewi.compiler.custom_sql_parsing  # lineage parsing
  enabled: False
- name: dewi.configuration.yaml_c_loader  # performance increase
  enabled: True

Dewi - Version 2.4.0 - Release 2022-11-28¶

Configuration:

• This release adds support for 'physical' storage of current objects for SATs and LINKs. To enable this mode set the config current_view.storage: physical on config documents of type SAT or LINK.

Example:

type: SAT
...
current_view:
  storage: physical
...

Dewi - Version 2.3.0 - Release: 2022-11-08¶

General:

• There is a new version command to checkout the version of dewi you are running. All commands will also output the dewi version

Configuration:

• You can now reference a dimension on a FACT table by using it's entity name. For this the reference now allows either a name or entity attribute configuration.

type: FACT
...
dimensions:
- name: Full Dimension Table Name DIM
- entity: Dimension
...

Compiler:

• The compiler will now complain about using columns on a Satellite that are also part of the business key of the hub. Which is not allowed

Generation:

• Generating the CURRENT objects for driving key links used the wrong HK column to check for the current row. Now it uses all the defined driving keys.

Dewi - Version 2.2.0 - Release: 2022-10-14¶

Configuration:

• Descriptive values added for almost all table and column types. Added 'description' to tables and 'description' and 'examples' to column types
• Support for defining custom current views. Add the 'current_view' attribute with a 'query' attribute with a 'sql' type attribute
• The CustomSQL directory is now configurable in the project configuration.

Compiler:

• Changed the default load types for the DIM and FACT tables into Incremental and Deletes (Be aware this will delete the default Full load flows)

Generation:

• Fixed issue with RSTSATs where 'DELETES' are inserted with NULLs for recordsource and loaddts which is wrong
• The configured CustomSQL directory is scanned recursively for *.sql files

Dewi - Version 2.1.0 - Release: 2022-09-20¶

Configuration:

• Allow defining the column type on CUSTOM tables

Compilation:

• It's no longer allowed to have a nullable primary key on a staging table
• Optimizes datamodel building by reducing the amount of data copying (drastically)

Generation:

• Enables the usage of CTE's for virtual tables
• Fixes the CURRENT VIEW templates to include the recordsource when determining if a record should exist
• Fixes tracking RSTSAT entries based on their recordsource

General:

• Adds a Changelog :)
• Adds warning to docs about the wrong dewi version
• Minor build fixes

Dewi - Version 2.0.0 - Release: 2022-06-03¶

• Initial release of the datawarehouse automation framework as an executable python package