Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(filter): Implement wild card searching for issue #95 #131

Merged
merged 3 commits into from
Aug 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 23 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ VERSION ?= $(shell git log -1 --pretty=format:"%H")
OS := $(shell go env GOOS)
ARCH := $(shell go env GOARCH)

.PHONY: all test doc gqlgen test-all test-e2e test-app test-db
.PHONY: all test doc gqlgen mockery test-all test-e2e test-app test-db fmt compose-prepare compose-up compose-down compose-restart compose-build

# Source the .env file to use the env vars with make
-include .env
Expand Down Expand Up @@ -54,17 +54,34 @@ gqlgen:
mockery:
mockery

GINKGO := go run github.com/onsi/ginkgo/v2/ginkgo
test-all:
ginkgo -r
$(GINKGO) -r

test-e2e:
ginkgo internal/e2e
$(GINKGO) -r internal/e2e

test-app:
ginkgo internal/app
$(GINKGO) -r internal/app

test-db:
ginkgo internal/database/mariadb
$(GINKGO) -r internal/database/mariadb

fmt:
go fmt ./...
go fmt ./...

DOCKER_COMPOSE := docker-compose -f docker-compose.yaml
DOCKER_COMPOSE_SERVICES := heureka-app heureka-db
compose-prepare:
sed 's/^SEED_MODE=false/SEED_MODE=true/g' .test.env > .env

compose-up:
$(DOCKER_COMPOSE) up -d $(DOCKER_COMPOSE_SERVICES)

compose-down:
$(DOCKER_COMPOSE) down $(DOCKER_COMPOSE_SERVICES)

compose-restart: compose-down compose-up

compose-build:
$(DOCKER_COMPOSE) build $(DOCKER_COMPOSE_SERVICES)
1 change: 1 addition & 0 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ services:

heureka-app:
build: .
container_name: heureka-app
profiles:
- heureka
environment:
Expand Down
2 changes: 2 additions & 0 deletions internal/api/graphql/graph/baseResolver/issue.go
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ func IssueBaseResolver(app app.Heureka, ctx context.Context, filter *model.Issue
PrimaryName: filter.PrimaryName,
Type: lo.Map(filter.IssueType, func(item *model.IssueTypes, _ int) *string { return pointer.String(item.String()) }),

Search: filter.Search,

IssueMatchStatus: nil, //@todo Implement
IssueMatchDiscoveryDate: nil, //@todo Implement
IssueMatchTargetRemediationDate: nil, //@todo Implement
Expand Down
9 changes: 8 additions & 1 deletion internal/api/graphql/graph/generated.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions internal/api/graphql/graph/model/models_gen.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion internal/api/graphql/graph/schema/issue.graphqls
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ input IssueFilter {

componentVersionId: [String],

search: [String],

# leave away for MVP
# cveDescription: [String]
# fromAdvisory: ID,
Expand All @@ -72,4 +74,4 @@ enum IssueTypes {
Vulnerability,
PolicyViolation,
SecurityEvent
}
}
8 changes: 7 additions & 1 deletion internal/database/mariadb/database.go
Original file line number Diff line number Diff line change
Expand Up @@ -165,8 +165,14 @@ func buildFilterQuery[T any](filter []T, expr string, op string) string {
}

func buildQueryParameters[T any](params []interface{}, filter []T) []interface{} {
return buildQueryParametersCount(params, filter, 1)
}

func buildQueryParametersCount[T any](params []interface{}, filter []T, count int) []interface{} {
for _, item := range filter {
params = append(params, item)
for i := 0; i < count; i++ {
params = append(params, item)
}
Comment on lines +173 to +175
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why we need to do this?

Copy link
Collaborator Author

@michalkrzyz michalkrzyz Aug 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is used because Wild Card query has multiple logical statements with the same parameter, here is pseudo code we are using:
'WHERE ....some previous conditions... AND (primName LIKE searchParam OR secName LIKE searchParam) AND ...

search param is the same for every single element from search list of filters. It is passed currently through two SQL Query parameters passed by question marks per search element. In all other filters, we have only single question marks used.
We could add extra SET Sql keyword to set every parameter to some value but it will complicate the design a bit, but I am open for discussion.

Meanwhile I will try to find better name for this function I have added to database.go.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh thanks for explaining

}
return params
}
Expand Down
9 changes: 8 additions & 1 deletion internal/database/mariadb/issue.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ import (
"github.wdf.sap.corp/cc/heureka/internal/entity"
)

const (
wildCardFilterQuery = "IV.issuevariant_secondary_name LIKE Concat('%',?,'%') OR I.issue_primary_name LIKE Concat('%',?,'%')"
wildCardFilterParamCount = 2
)

func (s *SqlDatabase) getIssueFilterString(filter *entity.IssueFilter) string {
var fl []string
fl = append(fl, buildFilterQuery(filter.ServiceName, "S.service_name = ?", OP_OR))
Expand All @@ -23,6 +28,7 @@ func (s *SqlDatabase) getIssueFilterString(filter *entity.IssueFilter) string {
fl = append(fl, buildFilterQuery(filter.IssueVariantId, "IV.issuevariant_id = ?", OP_OR))
fl = append(fl, buildFilterQuery(filter.Type, "I.issue_type = ?", OP_OR))
fl = append(fl, buildFilterQuery(filter.PrimaryName, "I.issue_primary_name = ?", OP_OR))
fl = append(fl, buildFilterQuery(filter.Search, wildCardFilterQuery, OP_OR))
fl = append(fl, "I.issue_deleted_at IS NULL")

return combineFilterQueries(fl, OP_AND)
Expand Down Expand Up @@ -55,7 +61,7 @@ func (s *SqlDatabase) getIssueJoins(filter *entity.IssueFilter, withAggregations
`)
}

if len(filter.IssueVariantId) > 0 {
if len(filter.IssueVariantId) > 0 || len(filter.Search) > 0 {
joins = fmt.Sprintf("%s\n%s", joins, `
LEFT JOIN IssueVariant IV ON I.issue_id = IV.issuevariant_issue_id
`)
Expand Down Expand Up @@ -164,6 +170,7 @@ func (s *SqlDatabase) buildIssueStatement(baseQuery string, filter *entity.Issue
filterParameters = buildQueryParameters(filterParameters, filter.IssueVariantId)
filterParameters = buildQueryParameters(filterParameters, filter.Type)
filterParameters = buildQueryParameters(filterParameters, filter.PrimaryName)
filterParameters = buildQueryParametersCount(filterParameters, filter.Search, wildCardFilterParamCount)
if withCursor {
filterParameters = append(filterParameters, cursor.Value)
filterParameters = append(filterParameters, cursor.Limit)
Expand Down
65 changes: 65 additions & 0 deletions internal/database/mariadb/issue_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -326,6 +326,71 @@ var _ = Describe("Issue", Label("database", "Issue"), func() {
Expect(entry.Type).To(BeEquivalentTo(issueType))
}
})
It("can filter issue PrimaryName using wild card search", func() {
row := seedCollection.IssueRows[rand.Intn(len(seedCollection.IssueRows))]

const charactersToRemoveFromBeginning = 2
const charactersToRemoveFromEnd = 2
const minimalCharactersToKeep = 5

start := charactersToRemoveFromBeginning
end := len(row.PrimaryName.String) - charactersToRemoveFromEnd

Expect(start+minimalCharactersToKeep < end).To(BeTrue())

searchStr := row.PrimaryName.String[start:end]
filter := &entity.IssueFilter{Search: []*string{&searchStr}}

entries, err := db.GetIssues(filter)

issueIds := []int64{}
for _, entry := range entries {
issueIds = append(issueIds, entry.Id)
}

By("throwing no error", func() {
Expect(err).To(BeNil())
})

By("at least one element was discarded (filtered)", func() {
Expect(len(seedCollection.IssueRows) > len(issueIds)).To(BeTrue())
})

By("returning the expected elements", func() {
Expect(issueIds).To(ContainElement(row.Id.Int64))
})
})
It("can filter issue variant SecondaryName using wild card search", func() {
// select an issueVariant
issueVariantRow := seedCollection.IssueVariantRows[rand.Intn(len(seedCollection.IssueVariantRows))]

const charactersToRemoveFromBeginning = 2
const charactersToRemoveFromEnd = 2
const minimalCharactersToKeep = 5

start := charactersToRemoveFromBeginning
end := len(issueVariantRow.SecondaryName.String) - charactersToRemoveFromEnd

Expect(start+minimalCharactersToKeep < end).To(BeTrue())

searchStr := issueVariantRow.SecondaryName.String[start:end]
filter := &entity.IssueFilter{Search: []*string{&searchStr}}

entries, err := db.GetIssues(filter)

issueIds := []int64{}
for _, entry := range entries {
issueIds = append(issueIds, entry.Id)
}

By("throwing no error", func() {
Expect(err).To(BeNil())
})

By("returning the expected elements", func() {
Expect(issueIds).To(ContainElement(issueVariantRow.IssueId.Int64))
})
})
})
Context("and using pagination", func() {
DescribeTable("can correctly paginate", func(pageSize int) {
Expand Down
1 change: 1 addition & 0 deletions internal/entity/issue.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ type IssueFilter struct {
IssueMatchId []*int64 `json:"issue_match_id"`
ComponentVersionId []*int64 `json:"component_version_id"`
IssueVariantId []*int64 `json:"issue_variant_id"`
Search []*string `json:"search"`
IssueMatchStatus []*string `json:"issue_match_status"`
IssueMatchDiscoveryDate *TimeFilter `json:"issue_match_discovery_date"`
IssueMatchTargetRemediationDate *TimeFilter `json:"issue_match_target_remediation_date"`
Expand Down
2 changes: 1 addition & 1 deletion internal/mocks/mock_Database.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion internal/mocks/mock_Heureka.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading