Skip to content

Commit

Permalink
Store vector init (#24)
Browse files Browse the repository at this point in the history
* API CRUD Refactoring

* Backend API Refactoring

* Adding vectorStoreClass and embeddingModelClass to SiloEntity

* Begin of Scheduling Configuration

* Fix checkstyle fail

* Removing AsyncConfig and SchedulingConfig in order to replace them with JobRunr

* Init JobRunr

* Updating default port of JobRunr from 8000 to 1984

* refactor: Rename SecurityConfiguration to SecurityConfig

on-behalf-of: @dRAGon-Okinawa [email protected]"

* chore: Update OpenAPI mappings to use single quotes instead of double quotes

on-behalf-of: @org [email protected]

* chore: Disable JobRunr background job server, job scheduler, and dashboard for Profile("test")

This commit disables the JobRunr background job server, job scheduler, and dashboard by setting the corresponding properties to false in the application-test.yaml file. This change is necessary to prevent unnecessary resource usage during testing.

Note: This commit message follows the established convention of using the "chore" type for non-functional changes.

* refactor: Update ProviderType enum values and add fromString method

This commit updates the ProviderType enum in the `ProviderType.java` file. It adds two new enum values, `ONNX` and `OpenAI`, and modifies the existing `OPENAI` value to `OpenAI`. Additionally, a new static method `fromString` is added to the enum to retrieve the corresponding ProviderType based on a given string value.

Note: This commit message follows the established convention of using the "refactor" type for code restructuring or refactoring.

* refactor: Add unique index for name field in IngestorEntity and ProviderEntity

This commit adds a unique index for the `name` field in both the `IngestorEntity` and `ProviderEntity` classes. The unique index ensures that the `name` field values are unique within their respective collections.

Note: This commit message follows the established convention of using the "refactor" type for code restructuring or refactoring.

* feat: Add create method with beforeSaveCallback to AbstractCrudBackendApiController

This commit adds a new overloaded `create` method to the `AbstractCrudBackendApiController` class. The new method accepts an additional parameter `beforeSaveCallback`, which is a function that can be used to modify the entity before saving it to the repository. This allows for more flexibility in customizing the creation process of entities.

Note: This commit message follows the established convention of using the "feat" type for new feature additions.

* feat: Handle unique key constraint violation in AbstractCrudBackendApiController

This commit modifies the `AbstractCrudBackendApiController` class to handle unique key constraint violations when saving entities to the repository. It adds a try-catch block around the `repository.save(entity)` method and throws a `ResponseStatusException` with a `CONFLICT` status and a descriptive error message when a `UniqueConstraintException` is caught.

Note: This commit message follows the established convention of using the "feat" type for new feature additions.

* Fix checkstyleMain + checkstyleTest

* feat: Update dRAGon description to mention RAG pipeline

This commit updates the dRAGon description in the `HomepageFeatures` component to mention the RAG pipeline instead of just "your docs". This change provides more clarity and aligns with the actual functionality of the software.

* refactor: Update primary color variables in custom.css

This commit updates the primary color variables in the custom.css file. The new color values are #8b5cf6 for the primary color, #743cf4 for the dark shade, #692cf3 for the darker shade, #4d0ce0 for the darkest shade, #a27cf8 for the light shade, #ad8cf9 for the lighter shade, and #cfbcfb for the lightest shade. This change ensures consistency and aligns with the desired color scheme.

* refactor: Update ProviderEntity settings field description

* refactor: Update SiloEntity vectorStoreClass field to vectorStoreType

This commit updates the `SiloEntity` class by renaming the `vectorStoreClass` field to `vectorStoreType`. This change provides a more accurate and descriptive name for the field, improving code readability and maintainability.

* refactor: Update EmbeddingModelType enum with dimensions and maxTokens

This commit updates the `EmbeddingModelType` enum in the `EmbeddingModelType.java` file. It adds the `dimensions` and `maxTokens` fields to the `EmbeddingModelDefinition` class, allowing for more flexibility in defining the dimensions and maximum tokens for each embedding model. This change enhances the configurability of the embedding models and improves the overall functionality of the application.

* refactor: Add IngestorEntity settings field for key-value pairs

This commit adds a new field `settings` to the `IngestorEntity` class. The `settings` field is a map of key-value pairs and allows for linking additional settings to the Ingestor. This change enhances the flexibility and configurability of the IngestorEntity.

* refactor: Rename SiloEntity vectorStoreClass field to vectorStoreType

* Fix checkstyleMain

* refactor: Update AbstractCrudBackendApiController to use ObjectMapper for field updates

This commit refactors the `AbstractCrudBackendApiController` class to use the `ObjectMapper` from the Jackson library for updating entity fields. Instead of manually iterating over the fields and using reflection to set the values, the code now serializes the `fields` map to JSON using the `ObjectMapper`, and then uses the `ObjectReader` to update the entity fields. This change improves code readability and maintainability by leveraging the built-in functionality of the `ObjectMapper` and reduces the risk of errors when updating fields.

* refactor: Add ingestors field to SiloEntity

This commit adds a new field `ingestors` to the `SiloEntity` class. The `ingestors` field is a list of Ingestor UUIDs that are linked to the Silo. When linked, the Ingestors will be run in order to add documents into the Silo. This change enhances the functionality of the SiloEntity by allowing it to manage and track the associated Ingestors.

* refactor: Add showAppURLs and configureJobs methods to StartupApplicationListener

* Begin JobRunr Ingestor Job

* refactor: Remove @Profile("!test") annotation from JobRunrConfig

The @Profile("!test") annotation was removed from the JobRunrConfig class. This change ensures that the configuration is applied regardless of the active profile, improving consistency and eliminating potential issues related to test environments.
  • Loading branch information
isontheline authored May 28, 2024
1 parent e169936 commit 3e2df78
Show file tree
Hide file tree
Showing 29 changed files with 603 additions and 173 deletions.
1 change: 1 addition & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
"postStartCommand": ".devcontainer/postStart.sh",
"postCreateCommand": ".devcontainer/postCreate.sh",
"forwardPorts": [
1984,
1985,
3000,
4173
Expand Down
27 changes: 19 additions & 8 deletions backend/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -20,26 +20,37 @@ repositories {
}

dependencies {
compileOnly 'org.projectlombok:lombok:1.18.32'
annotationProcessor 'org.projectlombok:lombok:1.18.32'
annotationProcessor "org.springframework.boot:spring-boot-configuration-processor"
implementation 'org.springframework.boot:spring-boot-starter-security'
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation 'org.springframework.boot:spring-boot-starter-thymeleaf:3.2.5'
implementation 'org.springframework.boot:spring-boot-starter-validation'
implementation 'org.hibernate.validator:hibernate-validator'
implementation 'dev.langchain4j:langchain4j-open-ai:0.30.0'
developmentOnly 'org.springframework.boot:spring-boot-devtools'
implementation 'org.springdoc:springdoc-openapi-starter-webmvc-ui:2.5.0'

implementation 'dev.langchain4j:langchain4j-open-ai:0.31.0'
implementation 'dev.langchain4j:langchain4j:0.31.0'
implementation 'dev.langchain4j:langchain4j-core:0.31.0'
implementation 'dev.langchain4j:langchain4j-embeddings:0.31.0'
implementation 'dev.langchain4j:langchain4j-open-ai:0.31.0'
implementation 'dev.langchain4j:langchain4j-easy-rag:0.31.0'

implementation platform('org.dizitart:nitrite-bom:4.2.1')
implementation 'org.dizitart:nitrite'
implementation 'org.dizitart:nitrite-support'
implementation 'org.dizitart:nitrite-mvstore-adapter'
implementation 'org.dizitart:nitrite-jackson-mapper'
implementation 'org.springdoc:springdoc-openapi-starter-webmvc-ui:2.5.0'

implementation 'net.sourceforge.argparse4j:argparse4j:0.9.0'
developmentOnly("org.springframework.boot:spring-boot-devtools")
implementation 'org.jobrunr:jobrunr-spring-boot-3-starter:7.1.2'

testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
testImplementation 'org.springframework.security:spring-security-test'

compileOnly 'org.projectlombok:lombok:1.18.32'
annotationProcessor 'org.projectlombok:lombok:1.18.32'
testCompileOnly 'org.projectlombok:lombok:1.18.32'
testAnnotationProcessor 'org.projectlombok:lombok:1.18.32'
}
Expand All @@ -49,12 +60,12 @@ application {
}

openApi {
groupedApiMappings.set(["http://localhost:1985/api/docs/backendapi": "swagger-backendapi.json",
"http://localhost:1985/api/docs/publicapi": "swagger-publicapi.json"])
groupedApiMappings.set(['http://localhost:1985/api/docs/backendapi': 'swagger-backendapi.json',
'http://localhost:1985/api/docs/publicapi': 'swagger-publicapi.json'])
outputDir.set(file("$buildDir/docs"))
waitTimeInSeconds.set(10)
customBootRun {
args.set(["--spring.profiles.active=test"])
args.set(['--spring.profiles.active=test'])
}
}

Expand Down
1 change: 0 additions & 1 deletion backend/src/main/java/ai/dragon/config/AppConfig.java
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,4 @@
@Configuration
@EnableConfigurationProperties(DataProperties.class)
public class AppConfig {

}
17 changes: 17 additions & 0 deletions backend/src/main/java/ai/dragon/config/JobRunrConfig.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
package ai.dragon.config;

import org.jobrunr.jobs.mappers.JobMapper;
import org.jobrunr.storage.InMemoryStorageProvider;
import org.jobrunr.storage.StorageProvider;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class JobRunrConfig {
@Bean
public StorageProvider storageProvider(JobMapper jobMapper) {
InMemoryStorageProvider storageProvider = new InMemoryStorageProvider();
storageProvider.setJobMapper(jobMapper);
return storageProvider;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import org.springframework.security.web.SecurityFilterChain;

@Configuration(proxyBeanMethods = false)
public class SecurityConfiguration {
public class SecurityConfig {
@Bean
@Order(Ordered.HIGHEST_PRECEDENCE)
SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
package ai.dragon.controller.api.backendapi.repository;

import java.lang.reflect.Constructor;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.function.Function;

import org.dizitart.no2.exceptions.UniqueConstraintException;
import org.springframework.http.HttpStatus;
import org.springframework.web.server.ResponseStatusException;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.ObjectReader;

import ai.dragon.entity.IAbstractEntity;
import ai.dragon.repository.AbstractRepository;

abstract class AbstractCrudBackendApiController<T extends IAbstractEntity> {
public T update(String uuid, Map<String, Object> fields, AbstractRepository<T> repository) {
T entityToUpdate = repository.getByUuid(uuid);
if (entityToUpdate == null) {
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "Entity not found");
}
try {
ObjectMapper objectMapper = new ObjectMapper();
String fieldsAsJson = objectMapper.writeValueAsString(fields);
ObjectReader objectReader = objectMapper.readerForUpdating(entityToUpdate);
entityToUpdate = objectReader.readValue(fieldsAsJson);
} catch (Exception ex) {
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "Error parsing fields as JSON", ex);
}
if (!uuid.equals(entityToUpdate.getUuid().toString())) {
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "UUID must not be changed.");
}
repository.save(entityToUpdate);
return entityToUpdate;
}

public List<T> list(AbstractRepository<T> repository) {
return repository.find().toList();
}

public T create(AbstractRepository<T> repository) throws Exception {
return create(repository, null);
}

public T create(AbstractRepository<T> repository, Function<T, T> beforeSaveCallback) throws Exception {
Class<T> clazz = repository.getGenericSuperclass();
Constructor<T> cons = clazz.getConstructor();
T entity = cons.newInstance();
if (beforeSaveCallback != null) {
entity = beforeSaveCallback.apply(entity);
}
try {
repository.save(entity);
} catch (UniqueConstraintException ex) {
throw new ResponseStatusException(HttpStatus.CONFLICT, "Unique key constraint violation", ex);
}
return entity;
}

public T get(String uuid, AbstractRepository<T> repository) {
return Optional.ofNullable(repository.getByUuid(uuid))
.orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "Entity not found."));
}

public void delete(String uuid, AbstractRepository<T> repository) {
if (!repository.exists(uuid)) {
throw new ResponseStatusException(HttpStatus.NOT_FOUND, "Entity not found.");
}
repository.delete(uuid);
}
}
Original file line number Diff line number Diff line change
@@ -1,22 +1,19 @@
package ai.dragon.controller.api.backendapi.repository;

import java.time.LocalDateTime;
import java.util.List;
import java.util.Optional;
import java.util.Map;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PatchMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;

import com.fasterxml.jackson.databind.JsonMappingException;
import com.fasterxml.jackson.databind.ObjectMapper;

import ai.dragon.entity.IngestorEntity;
import ai.dragon.repository.IngestorRepository;
Expand All @@ -29,65 +26,49 @@
@RestController
@RequestMapping("/api/backendapi/repository/ingestor")
@Tag(name = "Ingestor Repository", description = "Ingestor Repository Management API Endpoints")
public class IngestorBackendApiController {
public class IngestorBackendApiController extends AbstractCrudBackendApiController<IngestorEntity> {
@Autowired
private IngestorRepository ingestorRepository;

@GetMapping("/")
@ApiResponse(responseCode = "200", description = "List has been successfully retrieved.")
@Operation(summary = "List all Ingestors", description = "Returns all Ingestor entities stored in the database.")
public List<IngestorEntity> list() {
return ingestorRepository.find().toList();
return super.list(ingestorRepository);
}

@PostMapping("/")
@ApiResponse(responseCode = "200", description = "Ingestor has been successfully created.")
@ApiResponse(responseCode = "409", description = "Constraint violation.", content = @Content)
@Operation(summary = "Create a new Ingestor", description = "Creates one Ingestor entity in the database.")
public IngestorEntity create() {
String ingestorName = String.format("Ingestor %s",
LocalDateTime.now().format(java.time.format.DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm")));

IngestorEntity ingestor = new IngestorEntity();
ingestor.setName(ingestorName);
ingestorRepository.save(ingestor);
return ingestor;
public IngestorEntity create() throws Exception {
return super.create(ingestorRepository);
}

@GetMapping("/{uuid:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}}")
@ApiResponse(responseCode = "200", description = "Ingestor has been successfully retrieved.")
@ApiResponse(responseCode = "404", description = "Ingestor not found.", content = @Content)
@Operation(summary = "Retrieve one Ingestor", description = "Returns one Ingestor entity from its UUID stored in the database.")
public IngestorEntity get(@PathVariable("uuid") @Parameter(description = "Identifier of the Ingestor") String uuid) {
return Optional.ofNullable(ingestorRepository.getByUuid(uuid))
.orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "Ingestor not found."));
public IngestorEntity get(
@PathVariable("uuid") @Parameter(description = "Identifier of the Ingestor") String uuid) {
return super.get(uuid, ingestorRepository);
}

@PatchMapping("/{uuid:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}}")
@ApiResponse(responseCode = "200", description = "Ingestor has been successfully created.")
@ApiResponse(responseCode = "404", description = "Ingestor not found.", content = @Content)
@Operation(summary = "Update a Ingestor", description = "Updates one Ingestor entity in the database.")
public IngestorEntity update(@PathVariable("uuid") @Parameter(description = "Identifier of the Ingestor", required = true) String uuid,
IngestorEntity ingestor) throws JsonMappingException {
if (!ingestorRepository.exists(uuid) || ingestor == null || !uuid.equals(ingestor.getUuid().toString())) {
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "Ingestor not found");
}

IngestorEntity ingestorToUpdate = ingestorRepository.getByUuid(uuid);
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.updateValue(ingestorToUpdate, ingestor);
ingestorRepository.save(ingestorToUpdate);

return ingestorToUpdate;
public IngestorEntity update(
@PathVariable("uuid") @Parameter(description = "Identifier of the Ingestor", required = true) String uuid,
@RequestBody Map<String, Object> fields) throws JsonMappingException {
return super.update(uuid, fields, ingestorRepository);
}

@DeleteMapping("/{uuid:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}}")
@ApiResponse(responseCode = "200", description = "Ingestor has been successfully deleted.")
@ApiResponse(responseCode = "404", description = "Ingestor not found.", content = @Content)
@Operation(summary = "Delete a Ingestor", description = "Deletes one Ingestor entity from its UUID stored in the database.")
public void delete(@PathVariable("uuid") @Parameter(description = "Identifier of the Ingestor") String uuid) {
if (!ingestorRepository.exists(uuid)) {
throw new ResponseStatusException(HttpStatus.NOT_FOUND, "Ingestor not found.");
}
ingestorRepository.delete(uuid);
super.delete(uuid, ingestorRepository);
}
}
Original file line number Diff line number Diff line change
@@ -1,23 +1,20 @@
package ai.dragon.controller.api.backendapi.repository;

import java.time.LocalDateTime;
import java.util.List;
import java.util.Optional;
import java.util.Map;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PatchMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.server.ResponseStatusException;

import com.fasterxml.jackson.databind.JsonMappingException;
import com.fasterxml.jackson.databind.ObjectMapper;

import ai.dragon.entity.ProviderEntity;
import ai.dragon.enumeration.ProviderType;
Expand All @@ -31,30 +28,28 @@
@RestController
@RequestMapping("/api/backendapi/repository/provider")
@Tag(name = "Provider Repository", description = "Provider Repository Management API Endpoints")
public class ProviderBackendApiController {
public class ProviderBackendApiController extends AbstractCrudBackendApiController<ProviderEntity> {
@Autowired
private ProviderRepository providerRepository;

@GetMapping("/")
@ApiResponse(responseCode = "200", description = "List has been successfully retrieved.")
@Operation(summary = "List all Providers", description = "Returns all Provider entities stored in the database.")
public List<ProviderEntity> list() {
return providerRepository.find().toList();
return super.list(providerRepository);
}

@PostMapping("/")
@ApiResponse(responseCode = "200", description = "Provider has been successfully created.")
@ApiResponse(responseCode = "409", description = "Constraint violation.", content = @Content)
@Operation(summary = "Create a new Provider", description = "Creates one Provider entity in the database.")
public ProviderEntity create(
@RequestParam(name = "type", required = true) @Parameter(description = "Type of the provider") String providerType) {
String providerName = String.format("Provider %s",
LocalDateTime.now().format(java.time.format.DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm")));

ProviderEntity provider = new ProviderEntity();
provider.setName(providerName);
provider.setType(ProviderType.valueOf(providerType));
providerRepository.save(provider);
return provider;
@RequestParam(name = "type", required = true) @Parameter(description = "Type of the provider") String providerType)
throws Exception {
return super.create(providerRepository, provider -> {
provider.setType(ProviderType.fromString(providerType));
return provider;
});
}

@GetMapping("/{uuid:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}}")
Expand All @@ -63,8 +58,7 @@ public ProviderEntity create(
@Operation(summary = "Retrieve one Provider", description = "Returns one Provider entity from its UUID stored in the database.")
public ProviderEntity get(
@PathVariable("uuid") @Parameter(description = "Identifier of the Provider") String uuid) {
return Optional.ofNullable(providerRepository.getByUuid(uuid))
.orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "Provider not found."));
return super.get(uuid, providerRepository);
}

@PatchMapping("/{uuid:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}}")
Expand All @@ -73,27 +67,15 @@ public ProviderEntity get(
@Operation(summary = "Update a Provider", description = "Updates one Provider entity in the database.")
public ProviderEntity update(
@PathVariable("uuid") @Parameter(description = "Identifier of the Provider", required = true) String uuid,
ProviderEntity provider) throws JsonMappingException {
if (!providerRepository.exists(uuid) || provider == null || !uuid.equals(provider.getUuid().toString())) {
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "Provider not found");
}

ProviderEntity providerToUpdate = providerRepository.getByUuid(uuid);
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.updateValue(providerToUpdate, provider);
providerRepository.save(providerToUpdate);

return providerToUpdate;
@RequestBody Map<String, Object> fields) throws JsonMappingException {
return super.update(uuid, fields, providerRepository);
}

@DeleteMapping("/{uuid:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}}")
@ApiResponse(responseCode = "200", description = "Provider has been successfully deleted.")
@ApiResponse(responseCode = "404", description = "Provider not found.", content = @Content)
@Operation(summary = "Delete a Provider", description = "Deletes one Provider entity from its UUID stored in the database.")
public void delete(@PathVariable("uuid") @Parameter(description = "Identifier of the Provider") String uuid) {
if (!providerRepository.exists(uuid)) {
throw new ResponseStatusException(HttpStatus.NOT_FOUND, "Provider not found.");
}
providerRepository.delete(uuid);
super.delete(uuid, providerRepository);
}
}
Loading

0 comments on commit 3e2df78

Please sign in to comment.