Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(agent): Add URL whitelisting/blacklisting to config and URL validation #6848

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
+29 −2
Draft

feat(agent): Add URL whitelisting/blacklisting to config and URL validation #6848

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d67721e
falcon entering the arena
Satyam97 Feb 13, 2024
e1378d3
feat(read_webpage): added white/black listing URLs in read webpage fu…
Satyam97 Feb 14, 2024
27b1b4e
Merge branch 'master' into validation
Satyam97 Feb 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 6 additions & 0 deletions arena/falcon.json
View file
Open in desktop
Satyam97 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"github_repo_url": "https://github.com/Satyam97/AutoGPT.git",
"timestamp": "2024-02-13T18:29:44.575532",
"commit_hash_to_benchmark": "bb7f5abc6c915f4fbe7fe8331dae2ddc81a4f900",
"branch_to_benchmark": "master"
}
9 changes: 9 additions & 0 deletions autogpts/autogpt/.env.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,3 +235,12 @@ OPENAI_API_KEY=your-openai-api-key
# AP_SERVER_PORT=8000
# # AP_SERVER_DB_URL - Specifies what connection url the agent protocol database will connect to (Default: Internal SQLite)
# AP_SERVER_DB_URL=sqlite:///data/ap_server.db


################################################################################
### White/Black Listing URLS
################################################################################
##WEB_POLICY - Specifies whether the url list is black list(0) or a whilte list(1)
#WEB_POLICY=0
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a policy for users who want to continue using AutoGPT as normal. This looks like it forces them to choose between whitelisting and blacklisting, which some users might not want either. There needs to be a default option that simply disables this feature.

Comment on lines +243 to +244
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Formatting diverges from the rest of the file
  • Rather make this an enum with values blacklist and whitelist

##URL_LIST - Comma separated URLs to be black or whitelisted, sets according to WEB_POLICY value
#URL_LIST=https://www.google.com,http://www.google.com
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only thing I don't like about putting it here, is that these lists might get pretty long. That's why in my version I was attempting to parse files.

4 changes: 4 additions & 0 deletions autogpts/autogpt/autogpt/config/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,10 @@ class Config(SystemSettings, arbitrary_types_allowed=True):
# Stable Diffusion
sd_webui_auth: Optional[str] = UserConfigurable(from_env="SD_WEBUI_AUTH")

# White/Black Listing URLs
web_policy: Optional[str] = os.getenv("WEB_POLICY", 0)
url_list: Optional[list] = os.getenv("URL_LIST",[]).split(",")
Comment on lines +244 to +246
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not how the config of the application works, as you can see from how all the other config attributes are defined.


@validator("plugins", each_item=True)
def validate_plugins(cls, p: AutoGPTPluginTemplate | Any):
assert issubclass(
Expand Down
12 changes: 10 additions & 2 deletions autogpts/autogpt/autogpt/url_utils/validators.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@
import re
from typing import Any, Callable, ParamSpec, TypeVar
from urllib.parse import urljoin, urlparse
from autogpt.config import Config
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is gonna throw a linting error

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added flake8 to VS code, sorry there was some issue with flake earlier.


P = ParamSpec("P")
T = TypeVar("T")

config = Config
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this won't work

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Pwuts can you please help with why this wont work?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from autogpt.config import ConfigBuilder
config = ConfigBuilder.build_config_from_env()

web_policy = config.web_policy
url_list = config.url_list

will this work? @Pwuts


def validate_url(func: Callable[P, T]) -> Callable[P, T]:
"""
Expand All @@ -26,7 +27,9 @@ def wrapper(url: str, *args, **kwargs) -> Any:
Raises:
ValueError if the url fails any of the validation tests
"""

web_policy = config.web_policy
url_list = config.url_list

# Most basic check if the URL is valid:
if not re.match(r"^https?://", url):
raise ValueError("Invalid URL format")
Expand All @@ -38,6 +41,11 @@ def wrapper(url: str, *args, **kwargs) -> Any:
# Check URL length
if len(url) > 2000:
raise ValueError("URL is too long")
if web_policy:
if url not in url_list:
raise ValueError("URL Not Whitelisted")
elif url in url_list:
raise ValueError("URL Blacklisted.")
Comment on lines +46 to +48
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inconsistent formatting

Copy link
Author

@Satyam97 Satyam97 Feb 16, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have fixed the linting this update the PR with the above config changes @Pwuts


return func(sanitize_url(url), *args, **kwargs)

Expand Down