Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

removed check for prefix so we can fallback to empty #174

Merged
merged 3 commits into from
Jun 6, 2019
Merged

Conversation

patduin
Copy link
Contributor

@patduin patduin commented May 23, 2019

fixes #173

@patduin patduin requested a review from a team May 23, 2019 20:15
Copy link
Contributor

@massdosage massdosage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It almost looks... too.... easy!

CHANGELOG.md Outdated
### Changed
* Changed a prefixed *primary* metastore to fallback to 'empty prefix' if nothing specified. See [#173](https://github.com/HotelsDotCom/waggle-dance/issues/173).


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two new lines 😱

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(although it's not visible in the formatted version :) )

@patduin
Copy link
Contributor Author

patduin commented May 24, 2019

...to easy indeed. Found a conundrum in UDFs.
Leaving out the prefix for a udf won't work. At least not in a Hive CLI (Not sure how this works in other clients (Spark,Qubole etc..)). Hive CLI fetches all UDFs on startup (or reload function command). They are registered with prefix. So when calling show functions you get prefix_db.functionName.
In order for a select db.functionName to work we'll need to return that particular function name. We need to return both prefix_db.functionName and db.functionName when loading the functions. This means show functions will show both which is something we don't really want.
I don't think there is any way around this though when using a function (e.g. select...) there is no Metastore call to proxy it just looks up the loaded functions in the client and fails if it cannot locate a function with that name.

We need to choose either list functions prefixed and non-prefixed or force prefixed UDFs only. Thoughts/preferences anyone?

@abhimanyugupta07
Copy link
Member

abhimanyugupta07 commented May 29, 2019

...to easy indeed. Found a conundrum in UDFs.
Leaving out the prefix for a udf won't work. At least not in a Hive CLI (Not sure how this works in other clients (Spark,Qubole etc..)). Hive CLI fetches all UDFs on startup (or reload function command). They are registered with prefix. So when calling show functions you get prefix_db.functionName.
In order for a select db.functionName to work we'll need to return that particular function name. We need to return both prefix_db.functionName and db.functionName when loading the functions. This means show functions will show both which is something we don't really want.
I don't think there is any way around this though when using a function (e.g. select...) there is no Metastore call to proxy it just looks up the loaded functions in the client and fails if it cannot locate a function with that name.

We need to choose either list functions prefixed and non-prefixed or force prefixed UDFs only. Thoughts/preferences anyone?

I think I prefer showing both in show functions rather than having inconsistent behaviour when compared with rest of the queries where we are defaulting to db when no prefix is provided in the query.

@patduin
Copy link
Contributor Author

patduin commented May 29, 2019

I think I prefer showing both in show functions rather than having inconsistent behaviour when compared with rest of the queries where we are defaulting to db when no prefix is provided in the query.

The problem with showing both (same when showing both db's) is that it looks weird and user might think hey that looks wrong I'll delete on (which will delete both). :(

@abhimanyugupta07
Copy link
Member

I think I prefer showing both in show functions rather than having inconsistent behaviour when compared with rest of the queries where we are defaulting to db when no prefix is provided in the query.

The problem with showing both (same when showing both db's) is that it looks weird and user might think hey that looks wrong I'll delete on (which will delete both). :(

True. But deleting a function is not as big of a problem compared to deleting a table or database. Out of the two options, if you choose the one where you force people to use prefix, again the issue of backward compatibility will arise and also a user will be confused that for everything else my query works but for functions, its not working. I feel it's inconsistent from an end-user perspective. Also (I am guessing here) show functions might not be a commonly used query, so this might not be that big of a problem if we show both options only for this query.

@massdosage
Copy link
Contributor

Remind me what happens when one does "show databases" in this case - do we only see the prefixed ones?

@patduin
Copy link
Contributor Author

patduin commented Jun 6, 2019

Remind me what happens when one does "show databases" in this case - do we only see the prefixed ones?

yes.

@patduin patduin merged commit 7715c24 into master Jun 6, 2019
@patduin patduin deleted the issue-173 branch June 6, 2019 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fallback to primary metastore when no prefix
4 participants