Skip to content

Classes

spark_expectations.core.exceptions.SparkExpectOrFailException

Bases: Exception

Throw this exception if a rule fails and is expected to fail the job

spark_expectations.core.exceptions.SparkExpectationsDataframeNotReturnedException

Bases: Exception

Throw this exception if a function doesn't return a dataframe

spark_expectations.core.exceptions.SparkExpectationsEmailException

Bases: Exception

Throw this exception when spark expectations encounters exceptions while sending email notifications

spark_expectations.core.exceptions.SparkExpectationsErrorThresholdExceedsException

Bases: Exception

Throw this exception when error percentage exceeds certain configured value

spark_expectations.core.exceptions.SparkExpectationsInvalidAggDQExpectationException

Bases: Exception

Throw this exception when an invalid agg_dq expectation is encountered

spark_expectations.core.exceptions.SparkExpectationsInvalidQueryDQExpectationException

Bases: Exception

Throw this exception when an invalid query_dq expectation is encountered

spark_expectations.core.exceptions.SparkExpectationsInvalidRowDQExpectationException

Bases: Exception

Throw this exception when an invalid row_dq expectation is encountered

spark_expectations.core.exceptions.SparkExpectationsInvalidRuleTypeException

Bases: Exception

Throw this exception when an invalid rule type is encountered

spark_expectations.core.exceptions.SparkExpectationsMiscException

Bases: Exception

Throw this exception when spark expectations encounters miscellaneous exceptions

spark_expectations.core.exceptions.SparkExpectationsPagerDutyException

Bases: Exception

Throw this exception when spark expectations encounters exceptions while sending PagerDuty API notifications

spark_expectations.core.exceptions.SparkExpectationsSlackNotificationException

Bases: Exception

Throw this exception when spark expectations encounters exceptions while sending Slack notifications

spark_expectations.core.exceptions.SparkExpectationsTeamsNotificationException

Bases: Exception

Throw this exception when spark expectations encounters exceptions while sending Teams notifications

spark_expectations.core.exceptions.SparkExpectationsUserInputOrConfigInvalidException

Bases: Exception

Throw this exception when configured rule or value from the user is wrong

spark_expectations.core.exceptions.SparkExpectationsZoomNotificationException

Bases: Exception

Throw this exception when spark expectations encounters exceptions while sending Zoom notifications

Functions

spark_expectations.core.exceptions.raise_if_ansi_exception(e: Exception, rule_name: str, rule_expectation: str) -> None

Check if exception is likely cast error due to ANSI mode. Add that info to error message and raise.

Source code in spark_expectations/core/exceptions.py 
84
85
86
87
88
89
90
91
92
def raise_if_ansi_exception(e: Exception, rule_name: str, rule_expectation: str) -> None:
    """Check if exception is likely cast error due to ANSI mode. Add that info to error message and raise."""
    if "CAST_INVALID_INPUT" in str(e):
        raise SparkExpectationsMiscException(
            f"Cast error while evaluating rule '{rule_name}' with expectation/SQL '{rule_expectation}'. "
            f"This may be caused by rule expectation/SQL incompatibility with spark.sql.ansi.enabled=true "
            f"(default on Databricks Serverless). Please either update your expectation/SQL "
            f"to be compliant with ANSI mode, disable ANSI mode, or run not on serverless."
        ) from e