Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres trigger #9218

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Postgres trigger #9218

wants to merge 6 commits into from

Conversation

ea-rus
Copy link
Contributor

@ea-rus ea-rus commented May 16, 2024

Description

Added subscribe method to postgres handler.
How it works:

  • Creates trigger for table in postgres database
  • Postgres trigger creates event
    • if columns are defined - it checks they are not null (for insert) or changed (for update)
  • Mindsdb listen to event and executes query for every event.

Limitations:

  • Because it creates real trigger, write access to database is required

Example

create trigger tr_pg 
on my_pg.home_rentals    -- < table to track
columns rental_price, location -- < columns to track, optional
(      
  insert into my_pg.rentals_predicted (
    SELECT   t.rental_price as real_price,   m.rental_price as predicted_price
    FROM TABLE_DELTA as t
    JOIN home_rentals_model as m
  )
)

Demo
https://www.loom.com/share/8b27d9c467e6471e89ab22121daa7b85

Fixes #issue_number

Type of change

  • ⚡ New feature (non-breaking change which adds functionality)

Verification Process

To ensure the changes are working as expected:

  • Test Location: Specify the URL or path for testing.
  • Verification Steps: Outline the steps or queries needed to validate the change. Include any data, configurations, or actions required to reproduce or see the new functionality.

Additional Media:

  • I have attached a brief loom video or screenshots showcasing the new functionality or change.

Checklist:

  • My code follows the style guidelines(PEP 8) of MindsDB.
  • I have appropriately commented on my code, especially in complex areas.
  • Necessary documentation updates are either made or tracked in issues.
  • Relevant unit and integration tests are updated or added.

# Conflicts:
#	mindsdb/integrations/handlers/postgres_handler/postgres_handler.py
@ea-rus ea-rus requested a review from StpMax May 16, 2024 14:46
def subscribe(self, stop_event, callback, table_name, columns=None, **kwargs):

# psycopg2 is used
import psycopg2
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not to use psycopg?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to psycopg


while conn.notifies:
process_event(conn.notifies.pop())
conn.poll()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it required? will it clear current notifies list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was updated after switching to psycopg

conn.poll()

while conn.notifies:
process_event(conn.notifies.pop())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in this case processing order if LIFO. Do we need FIFO? If so, then need to add 0 to .pop(0)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was updated after switching to psycopg

@ea-rus ea-rus requested a review from StpMax June 4, 2024 19:50
# exit trigger
return

conn.add_notify_handler(process_event)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it required to add handler in each iteration?


conn.add_notify_handler(process_event)
# trigger getting updates
conn.execute("SELECT 1").fetchone()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spam db with query each 0.5s looks badly. Let put here comment with reminder to rewrite it to generator when psycopg 3.2.2 will be released (they added timeout arg there)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants