comparision
Please find the difference in the changes with different version, latest three versions changes are documented
Modifications made to the version during implementation or integration¶
stage | 0.8.0 | 1.0.0 |
---|---|---|
rules table schema changes | added additional two column 1. enable_error_drop_alert(boolean) 2. error_drop_thresholdt(int) documentation found here |
Remains same |
rule table creation required | yes - creation not required if you're upgrading from old version but schema changes required | yes - creation not required if you're upgrading from old version but schema changes required |
stats table schema changes | remains same | Remains Same |
stats table creation required | automated | Remains Same |
notification config setting | remains same | Remains Same |
secret store and kafka authentication details | Create a dictionary that contains your secret configuration values and register in __init__.py for multiple usage, example |
Remains Same. You can disable streaming if needed, in SparkExpectations class |
spark expectations initialisation | create spark expectations class object using SpakrExpectations by passing product_id and additional optional parameter debugger , stats_streaming_options example |
New arguments are added. Please follow this - example |
with_expectations decorator | remains same | New arguments are added. Please follow this - example |
WrappedDataFrameWriter | Doesn't exist | This is new and users need to provider the writer object to record the spark conf that need to be used while writing - example |