v1.0.3

README.md

@@ -19,12 +19,14 @@ IntelliJ plugin adding support for [Vespa AI](https://github.com/vespa-engine/v
 * Render a tree view of any trace in the YQL response.
  * Optionally render a zipkin view of your trace.
 
-* Package, prepare and activate simple vespa applications.
+* Package, prepare and activate (Java based projects).
 
-* Support for TLS/SSL connections
+* Support for TLS/SSL connections to Vespa cluster and containers.
 
 * Simple visualization of services.xml files
 
+* Show Vespa cluster logs (mapped from docker containers) 
+
 ## Dependencies
 
 ### Vespa Cluster
@@ -64,6 +66,11 @@ file [from disk](https://www.jetbrains.com/help/idea/managing-plugins.html#insta
 
 ## Change-Notes
 
+ * [1.0.3] - Support for Vespa Java Apps
+ * Package, prepare and activate will call maven before packaging and uploading the application.
+ * The Vespa tool windows (except for the Side panel) will be hidden from start until they are needed.
+ * New tool window for Vespa Cluster logs
+ * Works with log files/dirs mapped from docker containers (`vespa.log` or `logarchive` directory).
  * [1.0.2] - TLS and simple upload
  * TLS support for connections (not tested on vespa-cloud as I do not have a vespa-cloud)
  * Right click on application dir and select "Package, Prepare and Activate"
@@ -72,22 +79,39 @@ file [from disk](https://www.jetbrains.com/help/idea/managing-plugins.html#insta
  * Simple visualization of service.xml files
  * Right-click on a services.xml file and select "Show Service Overview"
  * Show connection status in the Vespa dock
+ * Bugfix for Vespa Results toolWindow
+ * Responses will now turn up on first execution
  * [1.0.1] - Fix since-build for idea-version
  * [1.0.0] - First version
 
 ## TODOs (Limitations)
 
 * Test TLS connectivity with vespa-cloud
  * We do not have access to a vespa-could to test this.
-* Application code support for prepare and activate
- * Support Vespa Java Projects
- * Trigger build before packaging the application
- * Use the generated files in the ./target dir
-* When you run a yql request the first time after starting the IDE there will be no results shown.
- * Workaround: just run the query again and results will show.
+* Add document-api support
+ * Support upload of Document PUT request from `.json` file using the `/document/v1/` api.
+ * Support for bulk uploads from `.json` files.
+* Add support for editing `services.xml` and `hosts.xml` with the help of a running cluster
+ * Discover nodes for different deployments/discovery methods
+ * docker
+ * kubernetes
+ * vespa-cloud
+ * DNS (SRV lookup)
+ * ETCD?
+ * SNMP?
+* Vespa developer cluster support (should be developed together with the docker node discovery feature)
+ * Add features to create and manage a cluster for development with docker/docker-compose.
+ * Support both single node and multi-node (secured?) clusters.
+* [bug] Logs chicken and egg problem
+ * When a multi-node vespa cluster is started the the `logarchive` directory does
+ not exist yet as that will only be created once an application has been deployed.
+ This can mess up the tailing of the logs for the plugin.
+ * We need to make the log handling more robust against file system issues like this...
+* Figure out how we can monitor the LogServer files remotely
+ * Test on k8s and vespa-cloud
 * Add more tests. The test coverage is VERY low atm.
 * Clean up and remove code that is no longer needed!
-* Improve the YQL ighlighting
+* Improve the YQL highlighting
 
 
 ## Setting up a Vespa Cluster for development
@@ -102,17 +126,21 @@ a single node vespa cluster with some books to search.
 
 1. Start vespa cluster
  ```shell
+ cd vespa-cluster/docker 
  # Start the vespa cluster/node
- docker run --detach --name vespa --hostname vespa-container \
- --publish 8080:8080 --publish 19071:19071 --publish 19050:19050 \
- vespaengine/vespa:8
+ ./vespa-cluster-start.sh
 
  # Test that the config service is running
  curl -s http://localhost:19071/state/v1/health | jq .
 
  # Stop and remove
- docker rm -f vespa
+ ./vespa-cluster-stop.sh
  ```
+
+ The above start script will start a single node vespa cluster and map the log directory
+ to the [`vespa-cluster/docker/logs`](vespa-cluster/docker/logs) directory. 
+ You can monitor the vespa cluster logs with the plugin by setting this directory as the
+ `vespa-log-dir` in the plugin setting.
 
 1. Deploy the "books" application
 
@@ -127,7 +155,7 @@ a single node vespa cluster with some books to search.
  curl -s http://localhost:19071/state/v1/health | jq .
  ```
 
- <img title="Services Overview" src="./assets/services-xml-single.png" alt="Services Overview" height="300"/>
+ <img title="Services Overview" src="./assets/services-xml-single.png" alt="Services Overview" height="200"/>
 
 1. Add some documents
  ```shell
@@ -163,8 +191,8 @@ All files needed are in the [`vespa-cluster`](vespa-cluster) directory:
 1. Start cluster nodes with docker-compose:
  ```shell
  cd vespa-cluster/docker-compose
- # Run detached
- docker-compose up -d
+ # Start the vespa cluster/node
+ ./vespa-cluster-start.sh
  
  # Try the secured config endpoint:
  curl -s \
@@ -174,10 +202,13 @@ All files needed are in the [`vespa-cluster`](vespa-cluster) directory:
  https://localhost:19071/state/v1/health | jq .
  
  # To stop just
- docker-compose stop
- # or (-v will remove the volumes and data)
- docker-compose down -v
+ ./vespa-cluster-stop.sh
+ 
  ```
+ The above start script will start a single node vespa cluster and map the log directory
+ to the [`vespa-cluster/docker-copmpose/logs`](vespa-cluster/docker/logs) directory. 
+ You can monitor the vespa cluster logs with the plugin by setting this directory as the
+ `vespa-log-dir` in the plugin setting.
 
  Docker-compose will create a cluster with the following nodes:
 
@@ -199,6 +230,13 @@ All files needed are in the [`vespa-cluster`](vespa-cluster) directory:
  --cert $(pwd)/pki/vespa/host.pem \
  --cacert $(pwd)/pki/vespa/ca-vespa.pem \
  https://localhost:19071/application/v2/tenant/default/prepareandactivate
+ 
+ # Make sure the application is up by probing the query port 
+ curl -s \
+ --key $(pwd)/tls/host.key \
+ --cert $(pwd)/tls/host.pem \
+ --cacert $(pwd)/tls/ca-vespa.pem \
+ https://localhost:8443/state/v1/health | jq .
  ```
  <img title="Services Overview" src="./assets/services-xml.png" alt="Services Overview" height="300"/>
 

build.gradle.kts

@@ -10,7 +10,7 @@ plugins {
 }
 
 group = "org.pehrs"
-version = "1.0.2"
+version = "1.0.3"
 
 repositories {
  mavenCentral()
@@ -75,7 +75,8 @@ intellij {
  System.out.println("Build for VERSION " + ideaVersion)
  type.set("IC") // Target IDE Platform
 
- plugins.set(listOf(/* Plugin Dependencies */))
+ plugins.set(listOf("com.intellij.java", "org.jetbrains.idea.maven"))
+
 }
 
 tasks {

src/main/java/com/pehrs/vespa/yql/plugin/VespaClusterChecker.java

@@ -10,26 +10,62 @@
 import java.util.HashMap;
 import java.util.List;
 import java.util.Map;
+import java.util.Optional;
 import java.util.stream.Collectors;
 import java.util.stream.Stream;
 import lombok.Getter;
+import org.apache.http.client.methods.HttpGet;
+import org.eclipse.jetty.client.ProtocolHandlers;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class VespaClusterChecker {
 
  private static final Logger log = LoggerFactory.getLogger(VespaClusterChecker.class);
 
- public static interface StatusListener {
+
+ public interface StatusListener {
  void vespaClusterStatusUpdated();
  }
 
+ public enum Status {
+ UP, DOWN, INITIALIZING, FAIL,
+ }
+
+// public final static String STATUS_UP = "up";
+// public final static String STATUS_DOWN = "down";
+// public final static String STATUS_INITIALIZING = "initializing";
+// public final static String STATUS_FAIL = "fail";
+
  static List<StatusListener> listeners = new ArrayList();
 
  @Getter
- static Map<VespaClusterConfig, String> queryEndpointStatus = new HashMap<>();
+ static Map<VespaClusterConfig, Status> queryEndpointStatus = new HashMap<>();
+ @Getter
+ static Map<VespaClusterConfig, Status> configEndpointStatus = new HashMap<>();
  @Getter
- static Map<VespaClusterConfig, String> configEndpointStatus = new HashMap<>();
+ static Map<VespaClusterConfig, String> appName = new HashMap<>();
+
+ @Getter
+ static Map<VespaClusterConfig, JsonNode> appStatus = new HashMap<>();
+
+ static Status zipkinStatus = Status.FAIL;
+
+ public static Optional<String> getAppGeneration(VespaClusterConfig config) {
+ return Optional.ofNullable(appStatus.get(config))
+ .flatMap(nodeRoot -> {
+ if(nodeRoot.has("application")) {
+ JsonNode application = nodeRoot.get("application");
+ if(application.has("meta")) {
+ JsonNode meta = application.get("meta");
+ if(meta.has("generation")) {
+ return Optional.of(meta.get("generation").asText());
+ }
+ }
+ }
+ return Optional.empty();
+ });
+ }
 
  public static void checkVespaClusters() {
  checkVespaClusters(null);
@@ -40,6 +76,7 @@ public static void checkVespaClusters(ProgressIndicator indicator) {
  if (settings != null) {
 
  List<VespaClusterConfig> clusterConfigs = settings.clusterConfigs;
+ // FIXME: Create thread pool and run all the checks in parallel
  for (int i = 0; i < clusterConfigs.size(); i++) {
  VespaClusterConfig clusterConfig = clusterConfigs.get(i);
  if (indicator != null) {
@@ -48,38 +85,93 @@ public static void checkVespaClusters(ProgressIndicator indicator) {
  indicator.setText("Vespa: " + clusterConfig.queryEndpoint);
  }
  URI configHostUri = clusterConfig.getConfigUri().resolve("/");
- String configCode = getHealthStatusCode(configHostUri.toString(), clusterConfig);
+ Status configCode = getHealthStatusCode(configHostUri.toString(), clusterConfig);
  log.trace("[" + clusterConfig.name + "]" + clusterConfig.configEndpoint + ": " + configCode);
  URI queryHostUri = clusterConfig.getQueryUri().resolve("/");
- String queryCode = getHealthStatusCode(queryHostUri.toString(), clusterConfig);
+ Status queryCode = getHealthStatusCode(queryHostUri.toString(), clusterConfig);
  log.trace("[" + clusterConfig.name + "]" + clusterConfig.queryEndpoint + ": " + queryCode);
 
+ getApplicationClusterName(configHostUri.toString(),
+ clusterConfig).ifPresentOrElse((name) -> appName.put(clusterConfig, name),
+ () -> appName.remove(clusterConfig));
+
+ getApplicationStatus(queryHostUri.toString(), clusterConfig)
+ .ifPresentOrElse(as -> appStatus.put(clusterConfig, as),
+ () -> appStatus.remove(clusterConfig));
+
+ zipkinStatus = getZipkinStatus();
+
  queryEndpointStatus.put(clusterConfig, queryCode);
  configEndpointStatus.put(clusterConfig, configCode);
  }
  notifyStatusListeners();
  }
  }
 
- public final static String STATUS_UP = "up";
- public final static String STATUS_DOWN = "down";
- public final static String STATUS_INITIALIZING = "initializing";
- public final static String STATUS_FAIL = "fail";
-
- private static String getHealthStatusCode(String endpoint, VespaClusterConfig clusterConfig) {
+ private static Status getHealthStatusCode(String endpoint, VespaClusterConfig clusterConfig) {
  String url = String.format("%s/state/v1/health", endpoint);
  try {
  JsonNode healthResponse =
  VespaClusterConnection.jsonGet(clusterConfig, url);
  if(healthResponse.has("status")) {
  if(healthResponse.get("status").has("code")) {
  String code = healthResponse.get("status").get("code").asText();
- return code;
+ try {
+ return Status.valueOf(code.toUpperCase());
+ } catch (Exception ex) {
+ return Status.FAIL;
+ }
  }
  }
- return STATUS_DOWN;
+ return Status.DOWN;
+ } catch (Exception ex) {
+ return Status.FAIL;
+ }
+ }
+
+ private static Optional<String> getApplicationClusterName(String endpoint, VespaClusterConfig clusterConfig) {
+ String url = String.format("%s/config/v2/tenant/default/application/default/cloud.config.cluster-list", endpoint);
+ try {
+ JsonNode clusterListRes =
+ VespaClusterConnection.jsonGet(clusterConfig, url);
+ if(clusterListRes.has("storage")) {
+ JsonNode storage = clusterListRes.get("storage");
+ if(storage.isArray() && storage.size() > 0) {
+ JsonNode first = storage.get(0);
+ if(first.has("name")) {
+ return Optional.of(first.get("name").asText());
+ }
+ }
+ }
+ return Optional.empty();
+ } catch (Exception ex) {
+ return Optional.empty();
+ }
+ }
+
+
+ private static Status getZipkinStatus() {
+ YqlAppSettingsState settings = YqlAppSettingsState.getInstance();
+
+ String url = String.format("%s/api/v2/traces", settings.getZipkinEndpoint());
+ try {
+ JsonNode clusterListRes = VespaClusterConnection.jsonGet(url);
+ return Status.UP;
+ } catch (Exception ex) {
+ return Status.FAIL;
+ }
+ }
+
+
+ private static Optional<JsonNode> getApplicationStatus(String endpoint, VespaClusterConfig clusterConfig) {
+ // http://localhost:8080/ApplicationStatus
+ String url = String.format("%s/ApplicationStatus", endpoint);
+ try {
+ JsonNode appStatus =
+ VespaClusterConnection.jsonGet(clusterConfig, url);
+ return Optional.of(appStatus);
  } catch (Exception ex) {
- return STATUS_FAIL;
+ return Optional.empty();
  }
  }