DGD-4258 DGD-4311 Events emission enhancements

[jrosend]

This pull request introduces a significant refactor to the event emission logic in the Spark agent, focusing on the introduction of a new NuEventEmitter class. The changes aim to improve the filtering and handling of OpenLineage events.

Introduction of NuEventEmitter:

• integration/spark/app/src/main/java/io/openlineage/spark/agent/NuEventEmitter.java: Added the NuEventEmitter class, which includes methods to filter events based on job type, event type, and job name, and to discard column lineage facets for non-complete events.

Refactoring event emission:

• integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/RddExecutionContext.java: Replaced direct calls to eventEmitter.emit with NuEventEmitter.emit in the start and end methods for SparkListenerJobStart and SparkListenerJobEnd events. [1] [2] [3]
• integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/SparkApplicationExecutionContext.java: Updated the start and end methods for SparkListenerApplicationStart and SparkListenerApplicationEnd events to use NuEventEmitter.emit. [1] [2] [3]
• integration/spark/app/src/main/java/io/openlineage/spark/agent/lifecycle/SparkSQLExecutionContext.java: Applied similar changes to use NuEventEmitter.emit for various event methods, including start and end for SparkListenerSQLExecutionStart, SparkListenerSQLExecutionEnd, SparkListenerStageSubmitted, and SparkListenerStageCompleted events. [1] [2] [3] [4] [5] [6] [7] [8] [9]

Enhancements to NuFacet:

• integration/spark/shared/src/main/java/io/openlineage/spark/agent/facets/NuFacet.java: Implemented a method to fetch the job name (jobNurn) from the Spark session configuration, adding logging for cases where the job name is not found. [1] [2]

Removal of unused fields:

• integration/spark/shared/src/main/java/io/openlineage/spark/api/OpenLineageContext.java: Removed the jobNurn field, which is now dynamically fetched in NuFacet.

Type of Change

• Bug fix
• New feature
• Non-breaking change
• Documentation update

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes