Skip to content

comparision

Please find the difference in the changes with different version, latest three versions changes are documented

Modifications made to the version during implementation or integration

stage 0.8.0 1.0.0
rules table schema changes added additional two column
1.enable_error_drop_alert(boolean)
2.error_drop_thresholdt(int)

documentation found here
Remains same
rule table creation required yes - creation not required if you're upgrading from old version but schema changes required yes - creation not required if you're upgrading from old version but schema changes required
stats table schema changes remains same Remains Same
stats table creation required automated Remains Same
notification config setting remains same Remains Same
secret store and kafka authentication details Create a dictionary that contains your secret configuration values and register in __init__.py for multiple usage, example Remains Same. You can disable streaming if needed, in SparkExpectations class
spark expectations initialisation create spark expectations class object using SpakrExpectations by passing product_id and additional optional parameter debugger, stats_streaming_options example New arguments are added. Please follow this - example
with_expectations decorator remains same New arguments are added. Please follow this - example
WrappedDataFrameWriter Doesn't exist This is new and users need to provider the writer object to record the spark conf that need to be used while writing - example