Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added initial prototype for rathole #8632

Merged
merged 113 commits into from
Jun 30, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
11e7127
Added initial prototype for rathole
madhavajay Mar 27, 2024
91a326a
update dockerfile with additional hosts for docker internal
shubham3121 Apr 15, 2024
191193d
define intial frame for the rathole server app
shubham3121 Apr 18, 2024
02694ec
define RatholeConfig model
shubham3121 Apr 22, 2024
4d26dab
implement server toml reader/writer class
shubham3121 Apr 22, 2024
a875114
integrate rathole toml client/server manager with Rathole fastapi end…
shubham3121 Apr 22, 2024
2186a26
Merge branch 'dev' into madhava/rathole
shubham3121 Apr 22, 2024
59eff34
add a default value to Rathole service type
shubham3121 Apr 22, 2024
6ed6b10
added nginx conf builder
shubham3121 Apr 23, 2024
0084bef
Fix RatholeConfigBuilder class
shubham3121 Apr 23, 2024
bbb093f
- pass server name to nginx builder
shubham3121 Apr 24, 2024
17e9a2f
fix relative import issues
khoaguin Apr 25, 2024
3807a52
Merge branch 'dev' into madhava/rathole
khoaguin Apr 25, 2024
3f1be01
fix rathole server
shubham3121 Apr 24, 2024
f5bd6ff
add rathole service to docker compose and traefik template
shubham3121 Apr 26, 2024
284ca05
set rathole mode in hagrid cli
shubham3121 Apr 26, 2024
ce2d36e
fix rathole config in docker compose files
shubham3121 Apr 26, 2024
c179ddb
fix client.toml
shubham3121 Apr 28, 2024
895448b
fix client.toml and server.toml for testing
shubham3121 Apr 28, 2024
e04fb02
fix loadbalancer in dynamic yml for rathole forward
shubham3121 Apr 29, 2024
b22309e
fix port number for rathole forwarding
shubham3121 Apr 29, 2024
b61d70e
retry fixing traefik
shubham3121 Apr 29, 2024
6f0759e
add rathole service and statefilset yaml
shubham3121 Apr 30, 2024
bffeefa
add rathole to devspace and define a configmap to update server.toml
shubham3121 Apr 30, 2024
eee0fc1
refactor and combine client and server.sh
shubham3121 May 1, 2024
0eb42a9
fix rathole configmap
shubham3121 May 1, 2024
dc31f6b
mount volume based on rathole mode if server or client
shubham3121 May 1, 2024
00cc225
mount server.toml and client.toml based on rathole mode
shubham3121 May 1, 2024
79bcc23
update configmap to seperate path of conf loaded and used in rathole
shubham3121 May 2, 2024
c693d3f
add some comment to start script
shubham3121 May 3, 2024
3d7759e
rathole: remove fastapi and nginx ports mapping from traefik
shubham3121 May 5, 2024
a1a379b
add toml_w and tomli packages
shubham3121 May 8, 2024
81c4dfa
Merge branch 'dev' into madhava/rathole
shubham3121 May 8, 2024
619e119
add classes to handle CRUD ops for toml server and client files in ra…
shubham3121 May 9, 2024
2fda67d
Merge branch 'dev' into madhava/rathole
shubham3121 May 9, 2024
55272f8
add method to add host to client.toml in configmap
shubham3121 May 9, 2024
7b8e461
move RatholeService to seperate file
shubham3121 May 13, 2024
13e2dd0
fix add host to client method to pass remote addr
shubham3121 May 13, 2024
ad838ac
fix values.yml not being correctly propogated
shubham3121 May 15, 2024
01fe5e8
add mode to rathole in values.yml
shubham3121 May 15, 2024
f7be280
fix set remote addr method in Rathole service
shubham3121 May 15, 2024
c5c3fa6
rename RATHOLE_MODE to MODE in rathole/start.sh
shubham3121 May 15, 2024
401d974
add a retry if client.toml is invalid when no connections are setup
shubham3121 May 15, 2024
e4e55f3
remove fastapi from grid/rathole
shubham3121 May 16, 2024
dd9a037
use rathole image to build rathole
shubham3121 May 16, 2024
f9a3fcf
configure rathole toml to use websockets
shubham3121 May 17, 2024
299dbac
configure to use same port and path for both http and websocket
shubham3121 May 21, 2024
f67774c
fix proxy port set to the client toml
shubham3121 May 22, 2024
fd60a9b
Added build step for rathole for arm64 platforms
madhavajay May 22, 2024
72e7daa
start traefik in watch mode with watch on directory instead of a sing…
shubham3121 May 22, 2024
59ecc5d
update method to expose port on rathole service
shubham3121 May 23, 2024
7c54096
fix proxy rule for dynamically added router rules for rathole
shubham3121 May 23, 2024
eb284ff
Merge branch 'dev' into madhava/rathole
shubham3121 May 27, 2024
57b35b0
fix lint
shubham3121 May 27, 2024
1fbd8be
move rathole token to http routes
shubham3121 May 28, 2024
9c7983c
fix passing host name in case of rathole connection
shubham3121 May 28, 2024
2b7172e
add widget
eelcovdw May 26, 2024
667c4d4
minor fixes
eelcovdw May 27, 2024
6262647
add decision column, disable sort
eelcovdw May 27, 2024
17cc6d3
fix old resolve method + typing
eelcovdw May 27, 2024
810bd65
add comments
eelcovdw May 27, 2024
307c4b8
fix tests
eelcovdw May 27, 2024
5a931f6
Use autosplat for settings update. Add docstring to Update service
jcardonnet May 16, 2024
b1d96e4
Fix autosplat when it's called from inner services
IonesioJunior May 20, 2024
0245797
Fix unit/notebook tests
IonesioJunior May 20, 2024
ba11b4d
Update settings test
IonesioJunior May 20, 2024
552a8ed
rename new_project.send() to new_project.start()
alexnicita May 24, 2024
46a0a76
Revert "rename new_project.send() to new_project.start()"
rasswanth-s May 27, 2024
7a8368a
Fix lint
IonesioJunior May 27, 2024
3010045
Fix unit tests
IonesioJunior May 27, 2024
fc2b703
Fix lint
IonesioJunior May 27, 2024
3e92863
Fixed linting / tests adding error handling for None types
madhavajay May 29, 2024
dbf3ff3
Merge branch 'dev' into madhava/rathole
shubham3121 May 29, 2024
74ad277
Re-add changed node route
madhavajay May 29, 2024
cebd471
Merge branch 'dev' into madhava/rathole
madhavajay May 30, 2024
47d5852
Change rathole port in gateway dev mode
madhavajay May 30, 2024
6ac3253
update network service to add override rathole config is already exists
shubham3121 May 30, 2024
e27c5cd
make id optional in NodeConnection and its subclasses
shubham3121 Jun 1, 2024
b9a0e6c
add /rathole prefix to dynamic proxy config
shubham3121 Jun 3, 2024
a7de407
add rathole prefix in http connection when rathole token present
shubham3121 Jun 3, 2024
d52643b
update internal proxy url to include /rathole
shubham3121 Jun 9, 2024
db9bd4c
Merge branch 'dev' into madhava/rathole
shubham3121 Jun 9, 2024
ed6ce11
update protocol version
shubham3121 Jun 9, 2024
6ea2cff
fix parameter name reference in update peer api
shubham3121 Jun 10, 2024
4dd0b7f
added notebook
shubham3121 Jun 10, 2024
51b7209
remove hagrid and docker compose files
shubham3121 Jun 10, 2024
2df1656
lint notebook
shubham3121 Jun 10, 2024
cec992b
remove output from Network notebook
shubham3121 Jun 10, 2024
0db775f
ignore prefix for rathole in blob store path
shubham3121 Jun 11, 2024
abcc997
add a stream upload API for blob storage
shubham3121 Jun 16, 2024
43d47fc
Merge branch 'dev' into madhava/rathole
shubham3121 Jun 20, 2024
c335d53
add a test for reverse tunnel
shubham3121 Jun 24, 2024
8e88b48
modulizer rathole config builder to a seperate class
shubham3121 Jun 25, 2024
2be7e68
integrate configmap deletion
shubham3121 Jun 25, 2024
7b30c15
Merge branch 'dev' into madhava/rathole
shubham3121 Jun 25, 2024
d4074b0
fix protocol version for HttpConnection
shubham3121 Jun 26, 2024
f174537
fix url path in _make_get
shubham3121 Jun 26, 2024
70d661d
Merge branch 'dev' into madhava/rathole
shubham3121 Jun 26, 2024
2f5554c
deprecate HttpConnectionV2
shubham3121 Jun 26, 2024
5d5ac71
ignore security on hardcoded binding for rathole config
shubham3121 Jun 26, 2024
fbcfe59
mark reverse tunnel with network marker
shubham3121 Jun 27, 2024
c9be64e
remove use of route id for route deletion
shubham3121 Jun 27, 2024
7609728
refactor reverse tunnel config logic to a single method
shubham3121 Jun 27, 2024
4677e7d
Merge branch 'dev' into madhava/rathole
shubham3121 Jun 27, 2024
e6f31f0
rename rathole to reverse tunnel in syft application
shubham3121 Jun 28, 2024
b393b7a
rename reverse_tunnel to rtunnel in values and values.dev yaml
shubham3121 Jun 28, 2024
30f1132
revert backend account to have permission to patch services
shubham3121 Jun 28, 2024
ad31f5b
force enable proxy in case of gateways
shubham3121 Jun 28, 2024
a95a815
start rathole pod, config, and service if rtunnel flag is enabled
shubham3121 Jun 28, 2024
feaa0d4
add retry to test_delete_route_on_peer for flakyness
shubham3121 Jun 28, 2024
714d451
Merge branch 'dev' into madhava/rathole
yashgorana Jun 28, 2024
e95af15
fix incorrect path
yashgorana Jun 30, 2024
c2ab102
fix charts
yashgorana Jun 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
209 changes: 209 additions & 0 deletions notebooks/Experimental/Network.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,209 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"id": "bd9a2226-3e53-4f27-9213-75a8c3ff9176",
"metadata": {},
"outputs": [],
"source": [
"# syft absolute\n",
"import syft as sy"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fddf8d07-d154-4284-a27b-d74e35d3f851",
"metadata": {},
"outputs": [],
"source": [
"gateway_client = sy.login(\n",
" url=\"http://localhost\", port=9081, email=\"[email protected]\", password=\"changethis\"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8f7b106d-b784-45d8-b54d-4ce2de2da453",
"metadata": {},
"outputs": [],
"source": [
"domain_client = sy.login(\n",
" url=\"http://localhost\", port=9082, email=\"[email protected]\", password=\"changethis\"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ff504949-620d-4e26-beee-0d39e0e502eb",
"metadata": {},
"outputs": [],
"source": [
"domain_client.connect_to_gateway(gateway_client, reverse_tunnel=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ba7bc71a-4e6a-4429-9588-7b3d0ed19e27",
"metadata": {},
"outputs": [],
"source": [
"gateway_client.api.services.request"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5b4984e1-331e-4fd8-b012-768fc613f48a",
"metadata": {},
"outputs": [],
"source": [
"# gateway_client.api.services.request[0].approve()"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "90dc44bd",
"metadata": {},
"outputs": [],
"source": [
"node_peers = gateway_client.api.network.get_all_peers()\n",
"node_peers"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8c06aaa6-4157-42d1-959f-9d47722a3420",
"metadata": {},
"outputs": [],
"source": [
"node_peer = gateway_client.api.network.get_all_peers()[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "cb63a77b",
"metadata": {},
"outputs": [],
"source": [
"node_peer.node_routes"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "61882e86",
"metadata": {},
"outputs": [],
"source": [
"node_peer.node_routes[0].__dict__"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fb19dbc6-869b-46dc-92e3-5e75ee6d0b06",
"metadata": {},
"outputs": [],
"source": [
"domain_client.api.network.get_all_peers()[0].node_routes[0].__dict__"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "32d09a51",
"metadata": {},
"outputs": [],
"source": [
"# node_peer.client_with_key(sy.SyftSigningKey.generate())"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b7d9e41d",
"metadata": {},
"outputs": [],
"source": [
"# gateway_client.api.network.delete_route(node_peer.verify_key, node_peer.node_routes[1])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8fa24ec7",
"metadata": {},
"outputs": [],
"source": [
"gateway_client"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3a081250-abc3-43a3-9e06-ff0c3a362ebf",
"metadata": {},
"outputs": [],
"source": [
"gateway_client.peers"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b6fedfe4-9362-47c9-9342-5cf6eacde8ab",
"metadata": {},
"outputs": [],
"source": [
"domain_client_proxy = gateway_client.peers[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f1940e00-0337-4b56-88c2-d70f397a7016",
"metadata": {},
"outputs": [],
"source": [
"domain_client_proxy.connection"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "613125c5-6321-4238-852c-ff0cfcd9526a",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.1.-1"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
3 changes: 3 additions & 0 deletions packages/grid/backend/grid/core/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,9 @@ def get_emails_enabled(self) -> Self:
ASSOCIATION_REQUEST_AUTO_APPROVAL: bool = str_to_bool(
os.getenv("ASSOCIATION_REQUEST_AUTO_APPROVAL", "False")
)
REVERSE_TUNNEL_ENABLED: bool = str_to_bool(
os.getenv("REVERSE_TUNNEL_ENABLED", "false")
)
model_config = SettingsConfigDict(case_sensitive=True)


Expand Down
4 changes: 4 additions & 0 deletions packages/grid/default.env
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ TRAEFIK_PUBLIC_TAG=traefik-public
STACK_NAME=grid-openmined-org
DOCKER_IMAGE_BACKEND=openmined/grid-backend
DOCKER_IMAGE_FRONTEND=openmined/grid-frontend
DOCKER_IMAGE_RATHOLE=openmined/grid-rathole
DOCKER_IMAGE_TRAEFIK=traefik
TRAEFIK_VERSION=v2.11.0
REDIS_VERSION=6.2
Expand Down Expand Up @@ -109,3 +110,6 @@ ENABLE_SIGNUP=False

# Enclave Attestation
DOCKER_IMAGE_ENCLAVE_ATTESTATION=openmined/grid-enclave-attestation

# Rathole Config
RATHOLE_PORT=2333
Loading
Loading