Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP Bridge: js-module sender input #1004

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Conversation

svix-onelson
Copy link
Contributor

@svix-onelson svix-onelson commented Jul 14, 2023

The general idea here is to let folks define a js module to use as a
data source for a "sender."

This could be general purpose but the initial use case is for polling
APIs that don't expose webhooks.

Currently, this diff contains only support code for config handling.

Some implementation details were worked out separately as a
POC in a separate repo.

By using fetch to issue HTTP requests, we can track state in the
module itself to decide if the data is worth sending a webhook for.

Example:
https://github.com/svix-onelson/poller-input-poc/blob/main/fetcher.js

There are many barriers to integrating the code in the POC linked above
which need to be cleared first.

We need:

  • a custom module loader to let us source modules from the
    bridge config instead of js files on disk.
  • Glue code to connect the op_forward deno extension calls to either a
    transformation or svix sender output.
  • existing transformation code in bridge needs to be refactored to allow
    us to use "newer deno" without also introducing a memory leak.

For the latter, https://github.com/svix/monorepo-private/issues/5670
aims to solve this.

In addition to the above, deno ops (i.e native extension code) are
supposed to be able to register state with the runtime, but I wasn't
able to get it to work for keeping track of which worker was which
(allowing us to propagate payloads to the appropriate output).

The general idea here is to let folks define a js module to use as a
data source for a "sender."

This could be general purpose but the initial use case is for polling
APIs that don't expose webhooks.

Currently this diff contains only support code for config handling.

Some implementation details were worked out separately as a
[POC in a separate repo.](https://github.com/svix-onelson/poller-input-poc/)

By using `fetch` to issue HTTP requests, we can track state in the
module itself to decide if the data is worth sending a webhook for.

Example:
<https://github.com/svix-onelson/poller-input-poc/blob/main/fetcher.js>

There are many barriers to integrating the code in the POC linked above
which need to be cleared first.

We need:
- a custom module loader to let us source modules from the
  bridge config instead of js files on disk.
- Glue code to connect the `op_forward` deno extension calls to either a
  transformation or svix sender output.
- existing transformation code in bridge needs to be refactored to allow
  us to use "newer deno" without also introducing a memory leak.

For the latter, <svix/monorepo-private#5670>
aims to solve this.

In addition to the above, deno ops (i.e native extension code) are
supposed to be able to register state with the runtime, but I wasn't
able to get it to work for keeping track of which worker was which
(allowing us to propagate payloads to the appropriate output).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant