I should be able to search across a complete list of tables, fields and descriptions in my project.

[PriPatel]

Default display:
Sketch

• By default You see a complete list of your tables , with descriptions in a column.
• The descriptions should look consistent in terms of styling for now ( eg. its possible to have variation because of the markdown)
• Display 'table groups' in a collapsible / expandable accordion.
• If there is only one table, you should not be able to collapse / expand it
• If you click a table -> You open an explore view with the table selected
• It should be clear when a table is made up of joins (by showing an icon and hover state. use a dummy icon for now)

Search bar behaviour
Sketch

• If you use the search bar, your results will show tables that match, as well as fields/descriptions within that that match (use atree structure)
• If you click on Field X -> You open an explore view with the relevant table selected and Field X must be selected by default too

If you:

• ...search for a field that lives in a group, show the group name (like a tree structure. I want to work on this a lot more, so this doesn't have to be so refined)
• ...search for a group, I should be able to see all the fields within that group too (using the tree structure)
• ...search for field that appears in two tables because of a join, display it twice.

Implementation

• New page: /catalog
• No feature flag needed for now
• Stying : Use pure mantine defaults and dont get too caught up trying to match the look of the wireframe. This is all going to change, so its all about functionality :)
• Out of scope : Filters and Fields tab (as seen in some of the sketches)

Search API

• Implement backend search for search fields across the project, and tables across the project.
• We already have the SearchModel and SearchService containing search for fields. We need to refactor this to make the field search reusable, or simply write new SearchModels
• We must refactor out the field/table search, which currently relies on loading all explores and running in-memory regex. This could be a big performance bottleneck and doesn't provide semantic search. We want to ship this in a way that makes it very easy to iterate on the search algorithm/technology without affecting the search service / API / permission checks.
• User attributes are currently slowing down search massively, can we make that better?
• Can we index tables on project compile? For example creating tables, creating search vectors, or even embeddings to make it easier to search?
• New features
  • We need syntax highlighting. Is this something we do on the backend or shall we do a frontend trick to show matches? The latter would be much easier

Open Questions

What happens when you go back? Does everything reset?

[magnew]

We are merging the main work for this ticket and I have broken out a couple of the remaining tasks into their own tickets:

• Catalog: It should be clear when a table is made up of joins #10149
• Enable linking to a field in the explores page from the catalog #10150
• Catalog: searching for a group, a table within a group, or a field within a group should show the heirarchy #10151