You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge?
SQL (standard) was recently extended with property graph querying features (PGQ): ISO standard, theoretical foundations. I wonder if DataFusion can be extended with PGQ.
Describe the solution you'd like
All parts should be extended. The most nontrivial part is interconnection between traditional SQL and graph analysis (path-related evaluations). While it is possible to store graph in columnar storage (e.g. Apache Arrow), it may be inefficient for path-related queries (while pretty efficient for attributes-of-vertex-related analytical queries). So, specific path-indexes may be required. Even more, in some cases it may be good idea to store graph topology in separated storage in specific format (e.g. sparse adjacency matrix, similar to FalkorDB).
On the other hand, even if we store graph in columnar storage, linear-algebra-related primitives can be useful for path querying (DuckPGQ: Efficient Property Graph Queries in an analytical RDBMS).
So, logical and physical plans should provide not only specific operators, but support balancing between data representation.
Describe alternatives you've considered
No response
Additional context
Can something like this project be used for physical level of linear algebra?
I'm interested in such a system design and development, but I'm aware that such an extension of DataFusion may leads to system recreation. So, I want to discuss this direction: should we extend DataFusion or create new independent system.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem or challenge?
SQL (standard) was recently extended with property graph querying features (PGQ): ISO standard, theoretical foundations. I wonder if DataFusion can be extended with PGQ.
Describe the solution you'd like
All parts should be extended. The most nontrivial part is interconnection between traditional SQL and graph analysis (path-related evaluations). While it is possible to store graph in columnar storage (e.g. Apache Arrow), it may be inefficient for path-related queries (while pretty efficient for attributes-of-vertex-related analytical queries). So, specific path-indexes may be required. Even more, in some cases it may be good idea to store graph topology in separated storage in specific format (e.g. sparse adjacency matrix, similar to FalkorDB).
On the other hand, even if we store graph in columnar storage, linear-algebra-related primitives can be useful for path querying (DuckPGQ: Efficient Property Graph Queries in an analytical RDBMS).
So, logical and physical plans should provide not only specific operators, but support balancing between data representation.
Describe alternatives you've considered
No response
Additional context
Can something like this project be used for physical level of linear algebra?
Possible theoretical foundations.
It may be a first step to support GQL.
I'm interested in such a system design and development, but I'm aware that such an extension of DataFusion may leads to system recreation. So, I want to discuss this direction: should we extend DataFusion or create new independent system.
The text was updated successfully, but these errors were encountered: