Ingestor job init (#25)

* chore: Update devcontainer.json with new VS Code extensions

* Using @PostConstruct to create mandatory directories

* Init Subscribing events for Ingestor repository

* 🚧 OnEvent

* Upgrade Gradle Plugin org.springframework.boot from 3.2.5 to 3.3.0

* feat: Update package import statements for DataProperties

The package import statements for `DataProperties` have been updated to reflect the new package structure in the `ai.dragon.config.properties` package. This change ensures that the correct class is imported and used in the `DirectoryStructureComponent` and `PropertiesConfig` classes.

* feat: Remove unused method onApplicationStartup in JobService

The method `onApplicationStartup` in the `JobService` class is no longer used and can be safely removed. This change eliminates unnecessary code and improves code readability.

* feat: Refactor IngestorJobService to use EntityChangeListener

* feat: Update package import statements for DataProperties

The package import statements for `DataProperties` have been updated to reflect the new package structure in the `ai.dragon.config.properties` package. This change ensures that the correct class is imported and used in the `DirectoryStructureComponent` and `PropertiesConfig` classes.

* feat: Rename StartupApplicationListener to DragonApplicationListener

* feat: Renaming IAbstractEntity interface

* feat: Refactor AbstractRepository to use EntityChangeListener

* feat: Refactor entity classes to use AbstractEntity instead of IAbstractEntity

The entity classes (IngestorEntity, ProviderEntity, and SiloEntity) have been refactored to use the AbstractEntity class instead of the IAbstractEntity interface. This change ensures consistency and improves code readability.

* feat: Remove unused JobScheduler dependency in IngestorJobService

* Remove unused DatabaseService dependency in DragonApplicationListener

* 🚧 Silo Job

* ⚠️ An Ingestor is now a Silo and a Silo is now a Farm (collection of Silos)

* 🚧 JobScheduler

* feat: Refactor getByUuid method in AbstractRepository

The `getByUuid` method in the `AbstractRepository` class has been refactored to return an `Optional` instead of a nullable value. This change improves code readability and ensures safer handling of null values.

* feat: Refactor JobRunrConfig

The `JobRunrConfig` class has been refactored to remove unused imports and methods. This change improves code cleanliness and reduces unnecessary dependencies.

* Refactor SiloIngestorJobHandler and SiloIngestorRequest

The `SiloIngestorJobHandler` and `SiloIngestorRequest` classes have been refactored. The previous implementation in the `ai.dragon.job` package has been deleted, and new implementations have been added in the `ai.dragon.job.silo` package.

* Refactor DragonApplicationListener to close web server application context

The `DragonApplicationListener` class has been refactored to include the closing of the web server application context. This change ensures proper cleanup and resource management.

* Refactor SiloJobService to use JobRequestScheduler

The `SiloJobService` class has been refactored to use the `JobRequestScheduler` instead of the  `JobScheduler`.

* Refactor background job server configuration for Profile(test)

Enable the background job server and set the poll interval to 5 seconds. Also, enable the job scheduler and configure the default number of retries for failing jobs.

* feat: Fix getByUuid and exists methods in AbstractRepository

* refactor: Use Optional in repository tests

Refactor the repository tests to use `Optional` when retrieving entities by UUID. This change improves code readability and ensures safer handling of null values.

* feat: Add UnusedLocalVariable check to checkstyle.xml

The `checkstyle.xml` file in the backend has been updated to include the `UnusedLocalVariable` check. This change ensures that unused local variables are flagged during code analysis.

* refactor: Add EmptyStatement check to checkstyle.xml

* refactor: Add EqualsAvoidNull check to checkstyle.xml

* refactor: Add SimplifyBooleanExpression and SimplifyBooleanReturn checks to checkstyle.xml

* refactor: Add UnnecessaryParentheses check to checkstyle.xml

* refactor: Add UnnecessarySemicolonAfterOuterTypeDeclaration check to checkstyle.xml

* refactor: Add UnnecessarySemicolonAfterTypeMemberDeclaration and UnnecessarySemicolonInEnumeration checks to checkstyle.xml

* refactor: Moving fields from FarmEntity to SiloEntity

* refactor: Rename stopRecurringJob to removeRecurringJob and stopAllRecurringJobs to removeAllRecurringJobs

The methods in the JobService class have been renamed to improve clarity and consistency. The stopRecurringJob method has been renamed to removeRecurringJob, and the stopAllRecurringJobs method has been renamed to removeAllRecurringJobs. This change aligns with the purpose of these methods and enhances code readability.

* 🚧 SiloIngestorJob progress

* refactor: Rename stopRecurringJob to removeRecurringJob and stopAllRecurringJobs to removeAllRecurringJobs

* refactor: Update EntityChangeListener to pass the entity object instead of UUID

The EntityChangeListener class has been modified to pass the entity object instead of just the UUID to the onChangeEvent method.

* refactor: Update JobService to include getRecurringJob method

The JobService class has been modified to include a new method called getRecurringJob. This method retrieves a recurring job by its ID. This change enhances the functionality of the JobService and improves code organization.

* feat: Update SiloJobService to use SiloEntity instead of UUID in job methods

The SiloJobService class has been modified to use the SiloEntity object instead of UUID in the createSiloIngestorJob, removeSiloIngestorJob, and restartSiloIngestorJob methods. This change improves code clarity and aligns with the recent modifications in the EntityChangeListener class.

* refactor: Update SiloJobService to use SiloEntity in job methods

* refactor: Remove unused import of UUID in EntityChangeListener

* refactor: ingestorType and ingestorSchedule are now mandatory

* refactor: Update SiloEntity to use List<String> for ingestorSettings

* refactor: Update EmbeddingModelType to use Lombok annotations

The EmbeddingModelType class has been refactored to use Lombok annotations for generating getters, setters, and builder methods. This change improves code readability and reduces boilerplate code.

* refactor: Renaming SiloType to IngestorType

* refactor: Update EmbeddingModelType to use Lombok annotations

* refactor: Update SiloJobService to use SiloEntity in job methods

* refactor: Add LocalSiloIngestor and LocalIngestorSettings classes

* refactor: Update SiloIngestorJobRequest to use Lombok annotations

* refactor: Update SiloIngestorJobHandler to ingest data to SiloEntity with progress tracking

This commit refactors the SiloIngestorJobHandler class to include a new method, ingestDataToSilo, which ingests data to the SiloEntity with progress tracking. The method uses a LocalSiloIngestor to list the documents to be ingested and iterates through them, updating the progress bar accordingly.

* refactor: Add ImplAbstractSiloIngestor class

The commit adds a new class, `ImplAbstractSiloIngestor`, which extends the `AbstractSiloIngestor` interface. This class provides a base implementation for ingesting data into a `SiloEntity` object. It includes methods for converting ini settings to objects and checking ingestor settings. This addition enhances the functionality of the codebase.

* refactor: Update LocalSiloIngestor to use IniSettingUtil for converting ini settings to objects

The LocalSiloIngestor class has been updated to use the IniSettingUtil class for converting ini settings to objects. This change improves code organization and promotes code reuse.

* refactor: Improve logging messages in SiloIngestorJobHandler

This commit improves the logging messages in the SiloIngestorJobHandler class. It updates the log messages to provide more descriptive information about the job being executed, such as the UUID of the Silo Ingestor Job. This change enhances the readability and clarity of the log output, making it easier to track the progress of the job.

* ```text
refactor: Update AbstractSiloIngestor to throw exception in listDocuments()

This commit updates the AbstractSiloIngestor interface to throw an exception in the listDocuments() method. This change ensures that any errors encountered during the listing of documents are properly handled and propagated.

* refactor: Add recursive document listing in LocalSiloIngestor

This commit adds the ability to recursively list documents in the LocalSiloIngestor class. It introduces a new private method, makeDocumentsList, which uses the Files.newDirectoryStream method to iterate through the directory and its subdirectories. This enhancement allows for a more comprehensive listing of documents and improves the functionality of the codebase.

* refactor: Add URI field to IngestorDocument and add new properties to LocalIngestorSettings

This commit adds a new field, `URI`, to the `IngestorDocument` class, allowing for the storage of document URIs. Additionally, it introduces new properties, `matches` and `recursive`, to the `LocalIngestorSettings` class. The `matches` property represents a string pattern for matching documents, while the `recursive` property indicates whether the document listing should be done recursively. These changes enhance the functionality and flexibility of the codebase.

.devcontainer/devcontainer.json

@@ -22,6 +22,7 @@
 		"vscode": {
 			"extensions": [
 				"redhat.java",
+				"vscjava.vscode-java-debug",
 				"vscjava.vscode-java-pack",
 				"vscjava.vscode-gradle",
 				"Vue.volar"

backend/build.gradle

@@ -3,7 +3,7 @@ plugins {
 	id 'java-library'
 	id 'checkstyle' 
 	id 'application'
-	id 'org.springframework.boot' version '3.2.5'
+	id 'org.springframework.boot' version '3.3.0'
 	id 'io.spring.dependency-management' version '1.1.4'
 	id 'org.springdoc.openapi-gradle-plugin' version '1.8.0'
 }

backend/checkstyle.xml

@@ -6,6 +6,16 @@
   <property name="charset" value="UTF-8"/>
   <module name="TreeWalker">
     <module name="UnusedImports"/>
+    <module name="UnusedLocalVariable"/>
+    <module name="EmptyStatement"/>
+    <module name="EqualsAvoidNull"/>
+    <module name="SimplifyBooleanExpression"/>
+    <module name="SimplifyBooleanReturn"/>
+    <module name="UnnecessaryParentheses"/>
+    <module name="UnnecessarySemicolonAfterOuterTypeDeclaration"/>
+    <module name="UnnecessarySemicolonAfterTypeMemberDeclaration"/>
+    <module name="UnnecessarySemicolonInEnumeration"/>
+    <module name="UnnecessarySemicolonInTryWithResources"/>
     <module name="RedundantImport"/>
     <module name="AvoidStarImport"/>
     <module name="AvoidStaticImport">

backend/src/main/java/ai/dragon/component/DirectoryStructureComponent.java

@@ -6,29 +6,28 @@
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
-import org.springframework.boot.CommandLineRunner;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.core.Ordered;
 import org.springframework.core.annotation.Order;
 import org.springframework.stereotype.Component;
 
-import ai.dragon.properties.DataProperties;
+import ai.dragon.config.properties.DataProperties;
+import jakarta.annotation.PostConstruct;
 import lombok.Getter;
 
 @Component
-@Order(1)
-public class DirectoryStructureComponent implements CommandLineRunner {
+@Order(Ordered.HIGHEST_PRECEDENCE)
+public class DirectoryStructureComponent {
     private final Logger logger = LoggerFactory.getLogger(this.getClass());
 
+    @Autowired
     private DataProperties dataProperties;
 
     @Getter
     private File dataDirectory;
 
-    public DirectoryStructureComponent(DataProperties dataProperties) {
-        this.dataProperties = dataProperties;
-    }
-
-    @Override
-    public void run(String... args) throws Exception {
+    @PostConstruct
+    private void postConstruct() throws Exception {
         dataDirectory = createDataDirectories();
         createDatabaseDirectory();
     }

backend/src/main/java/ai/dragon/config/AppConfig.java

@@ -1,11 +1,7 @@
 package ai.dragon.config;
 
-import org.springframework.boot.context.properties.EnableConfigurationProperties;
 import org.springframework.context.annotation.Configuration;
 
-import ai.dragon.properties.DataProperties;
-
 @Configuration
-@EnableConfigurationProperties(DataProperties.class)
 public class AppConfig {
 }

backend/src/main/java/ai/dragon/config/PropertiesConfig.java

@@ -0,0 +1,16 @@
+package ai.dragon.config;
+
+import org.springframework.boot.context.properties.ConfigurationProperties;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+
+import ai.dragon.config.properties.DataProperties;
+
+@Configuration
+public class PropertiesConfig {
+    @Bean
+    @ConfigurationProperties(prefix = DataProperties.PREFIX)
+    public DataProperties dataProperties() {
+        return new DataProperties();
+    }
+}

backend/src/main/java/ai/dragon/properties/DataProperties.java → backend/src/main/java/ai/dragon/config/properties/DataProperties.java

@@ -1,11 +1,8 @@
-package ai.dragon.properties;
-
-import org.springframework.boot.context.properties.ConfigurationProperties;
+package ai.dragon.config.properties;
 
 import lombok.Getter;
 import lombok.Setter;
 
-@ConfigurationProperties(prefix = DataProperties.PREFIX)
 @Getter
 @Setter
 public class DataProperties {

backend/src/main/java/ai/dragon/controller/api/backendapi/repository/AbstractCrudBackendApiController.java

@@ -3,7 +3,6 @@
 import java.lang.reflect.Constructor;
 import java.util.List;
 import java.util.Map;
-import java.util.Optional;
 import java.util.function.Function;
 
 import org.dizitart.no2.exceptions.UniqueConstraintException;
@@ -13,15 +12,13 @@
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.fasterxml.jackson.databind.ObjectReader;
 
-import ai.dragon.entity.IAbstractEntity;
+import ai.dragon.entity.AbstractEntity;
 import ai.dragon.repository.AbstractRepository;
 
-abstract class AbstractCrudBackendApiController<T extends IAbstractEntity> {
+abstract class AbstractCrudBackendApiController<T extends AbstractEntity> {
     public T update(String uuid, Map<String, Object> fields, AbstractRepository<T> repository) {
-        T entityToUpdate = repository.getByUuid(uuid);
-        if (entityToUpdate == null) {
-            throw new ResponseStatusException(HttpStatus.BAD_REQUEST, "Entity not found");
-        }
+        T entityToUpdate = repository.getByUuid(uuid)
+                .orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "Entity not found"));
         try {
             ObjectMapper objectMapper = new ObjectMapper();
             String fieldsAsJson = objectMapper.writeValueAsString(fields);
@@ -61,7 +58,7 @@ public T create(AbstractRepository<T> repository, Function<T, T> beforeSaveCallb
     }
 
     public T get(String uuid, AbstractRepository<T> repository) {
-        return Optional.ofNullable(repository.getByUuid(uuid))
+        return repository.getByUuid(uuid)
                 .orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "Entity not found."));
     }
 

backend/src/main/java/ai/dragon/controller/api/backendapi/repository/FarmBackendApiController.java

@@ -0,0 +1,73 @@
+package ai.dragon.controller.api.backendapi.repository;
+
+import java.util.List;
+import java.util.Map;
+
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.web.bind.annotation.DeleteMapping;
+import org.springframework.web.bind.annotation.GetMapping;
+import org.springframework.web.bind.annotation.PatchMapping;
+import org.springframework.web.bind.annotation.PathVariable;
+import org.springframework.web.bind.annotation.PostMapping;
+import org.springframework.web.bind.annotation.RequestBody;
+import org.springframework.web.bind.annotation.RequestMapping;
+import org.springframework.web.bind.annotation.RestController;
+
+import com.fasterxml.jackson.databind.JsonMappingException;
+
+import ai.dragon.entity.FarmEntity;
+import ai.dragon.repository.FarmRepository;
+import io.swagger.v3.oas.annotations.Operation;
+import io.swagger.v3.oas.annotations.Parameter;
+import io.swagger.v3.oas.annotations.media.Content;
+import io.swagger.v3.oas.annotations.responses.ApiResponse;
+import io.swagger.v3.oas.annotations.tags.Tag;
+
+@RestController
+@RequestMapping("/api/backendapi/repository/farm")
+@Tag(name = "Farm Repository", description = "Farm Repository Management API Endpoints")
+public class FarmBackendApiController extends AbstractCrudBackendApiController<FarmEntity> {
+    @Autowired
+    private FarmRepository farmRepository;
+
+    @GetMapping("/")
+    @ApiResponse(responseCode = "200", description = "List has been successfully retrieved.")
+    @Operation(summary = "List all Farms", description = "Returns all Farm entities stored in the database.")
+    public List<FarmEntity> list() {
+        return super.list(farmRepository);
+    }
+
+    @PostMapping("/")
+    @ApiResponse(responseCode = "200", description = "Farm has been successfully created.")
+    @ApiResponse(responseCode = "409", description = "Constraint violation.", content = @Content)
+    @Operation(summary = "Create a new Farm", description = "Creates one Farm entity in the database.")
+    public FarmEntity create() throws Exception {
+        return super.create(farmRepository);
+    }
+
+    @GetMapping("/{uuid:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}}")
+    @ApiResponse(responseCode = "200", description = "Farm has been successfully retrieved.")
+    @ApiResponse(responseCode = "404", description = "Farm not found.", content = @Content)
+    @Operation(summary = "Retrieve one Farm", description = "Returns one Farm entity from its UUID stored in the database.")
+    public FarmEntity get(@PathVariable("uuid") @Parameter(description = "Identifier of the Farm") String uuid) {
+        return super.get(uuid, farmRepository);
+    }
+
+    @PatchMapping("/{uuid:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}}")
+    @ApiResponse(responseCode = "200", description = "Farm has been successfully created.")
+    @ApiResponse(responseCode = "404", description = "Farm not found.", content = @Content)
+    @Operation(summary = "Update a Farm", description = "Updates one Farm entity in the database.")
+    public FarmEntity update(
+            @PathVariable("uuid") @Parameter(description = "Identifier of the Farm", required = true) String uuid,
+            @RequestBody Map<String, Object> fields) throws JsonMappingException {
+        return super.update(uuid, fields, farmRepository);
+    }
+
+    @DeleteMapping("/{uuid:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}}")
+    @ApiResponse(responseCode = "200", description = "Farm has been successfully deleted.")
+    @ApiResponse(responseCode = "404", description = "Farm not found.", content = @Content)
+    @Operation(summary = "Delete a Farm", description = "Deletes one Farm entity from its UUID stored in the database.")
+    public void delete(@PathVariable("uuid") @Parameter(description = "Identifier of the Farm") String uuid) {
+        super.delete(uuid, farmRepository);
+    }
+}

backend/src/main/java/ai/dragon/controller/api/backendapi/repository/SiloBackendApiController.java

@@ -49,7 +49,8 @@ public SiloEntity create() throws Exception {
     @ApiResponse(responseCode = "200", description = "Silo has been successfully retrieved.")
     @ApiResponse(responseCode = "404", description = "Silo not found.", content = @Content)
     @Operation(summary = "Retrieve one Silo", description = "Returns one Silo entity from its UUID stored in the database.")
-    public SiloEntity get(@PathVariable("uuid") @Parameter(description = "Identifier of the Silo") String uuid) {
+    public SiloEntity get(
+            @PathVariable("uuid") @Parameter(description = "Identifier of the Silo") String uuid) {
         return super.get(uuid, siloRepository);
     }
 

backend/src/main/java/ai/dragon/entity/AbstractEntity.java

@@ -0,0 +1,10 @@
+package ai.dragon.entity;
+
+import java.util.UUID;
+
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+
+@JsonIgnoreProperties(ignoreUnknown = true)
+public interface AbstractEntity {
+    public UUID getUuid();
+}

backend/src/main/java/ai/dragon/entity/FarmEntity.java

@@ -0,0 +1,43 @@
+package ai.dragon.entity;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.UUID;
+
+import org.dizitart.no2.index.IndexType;
+import org.dizitart.no2.repository.annotations.Entity;
+import org.dizitart.no2.repository.annotations.Id;
+import org.dizitart.no2.repository.annotations.Index;
+import org.dizitart.no2.repository.annotations.Indices;
+
+import io.swagger.v3.oas.annotations.media.Schema;
+import jakarta.validation.constraints.NotNull;
+import lombok.Getter;
+import lombok.Setter;
+
+@Entity(value = "farm")
+@Schema(name = "Farm", description = "Farm Entity")
+@Indices({ @Index(fields = "name", type = IndexType.UNIQUE) })
+@Getter
+@Setter
+public class FarmEntity implements AbstractEntity {
+    @Id
+    @NotNull
+    @Schema(description = "Identifier of the Farm")
+    private UUID uuid;
+
+    @NotNull
+    @Schema(description = "Name of the Farm. Must be unique.")
+    private String name;
+
+    @Schema(description = """
+            List of Silo UUIDs to be linked to the Farm.
+            A farm is a collection of Silos, each Silo is a collection of Documents.""")
+    private List<UUID> silos;
+
+    public FarmEntity() {
+        this.uuid = UUID.randomUUID();
+        this.name = String.format("Farm %s", this.uuid.toString());
+        this.silos = new ArrayList<UUID>();
+    }
+}