Add a new parseJabRefComment with unit test #12145

leaf-soba · 2024-11-04T08:27:36Z

first step in jabref-meta storage in bib file should be improved (by switching to embedded JSON) #10371

Mandatory checks

I own the copyright of the code submitted and I licence it under the MIT license
Change in CHANGELOG.md described in a way that is understandable for the average user (if change is visible to the user)
Tests created for changes (if applicable)
Manually tested changed features in running JabRef (always required)
Screenshots added in PR description (for UI changes)
Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

github-actions

Your code currently does not meet JabRef's code guidelines.
We use Checkstyle to identify issues.
Please carefully follow the setup guide for the codestyle.
Afterwards, please run checkstyle locally and fix the issues.

fix logic check string contains a valid json

koppor

Please do differently.

I highlighted the idea using the diff:

Check for jabref-meta-0.1.0string at the beginning of the comment. Then remove that line. The remainder of the comment is the complete JSON.

Do NOT read the JSON as BibEntry. It is NOT a BibEntry. It is META data for JabRef. E.g., preferences etc.

Background: JabRef displys BibEntrys in the main table - and settings in dialog. We do not render settings in the main table.

calixtus · 2024-11-04T15:33:09Z

src/main/java/org/jabref/logic/importer/fileformat/BibtexParser.java

+        } catch (
+                JsonSyntaxException e) {


Checkstyle is somewhat broken, please reduce to one line.

1. read json as metadata 2. small refactor to remove a not in use `LOGGER` in `MetaData`

github-actions

JUnit tests are failing. In the area "Some checks were not successful", locate "Tests / Unit tests (pull_request)" and click on "Details". This brings you to the test output.

You can then run these tests in IntelliJ to reproduce the failing tests locally. We offer a quick test running howto in the section Final build system checks in our setup guide.

koppor · 2024-11-11T11:42:36Z

src/main/java/org/jabref/logic/importer/fileformat/BibtexParser.java

@@ -339,7 +342,7 @@ private void parseAndAddEntry(String type) {
        }
    }

-    private void parseJabRefComment(Map<String, String> meta) {
+    void parseJabRefComment(Map<String, String> meta) {


This needs @VisibileForTesting?

koppor · 2024-11-11T11:45:45Z

src/main/java/org/jabref/logic/importer/fileformat/BibtexParser.java

+        }
+    }
+
+    private void parseCommentToJson(String comment, Map<String, String> meta) {


No - it should return the JSOn object.

The following call should be rewritten:

MetaData metaData = metaDataParser.parse( meta, importFormatPreferences.bibEntryPreferences().getKeywordSeparator());

meta should be a record of

Map<String, String> v5metaData

JsonObject v6metaData

And then be handled accordingliny in org.jabref.logic.importer.fileformat.BibtexParser#metaDataParser.

koppor · 2024-11-11T11:46:19Z

src/main/java/org/jabref/model/metadata/MetaData.java


 @AllowedToUseLogic("because it needs access to citation pattern and cleanups")
 public class MetaData {

    public static final String META_FLAG = "jabref-meta: ";
+    public static final String META_FLAG_010 = "jabref-meta-0.1.0";


Include the string Version in the constant name:

Suggested change

public static final String META_FLAG_010 = "jabref-meta-0.1.0";

public static final String META_FLAG_VERSION_010 = "jabref-meta-0.1.0";

rewrite metaDataParser.parse

github-actions

Your code currently does not meet JabRef's code guidelines.
We use Checkstyle to identify issues.
Please carefully follow the setup guide for the codestyle.
Afterwards, please run checkstyle locally and fix the issues.

fix wrong input

koppor · 2024-11-16T22:55:29Z

src/main/java/org/jabref/logic/importer/fileformat/BibtexParser.java

@@ -386,7 +390,23 @@ private void parseJabRefComment(Map<String, String> meta) {
            } catch (ParseException ex) {
                parserResult.addException(ex);
            }
+        } else if (comment.substring(0, Math.min(comment.length(), MetaData.META_FLAG_VERSION_010.length())).equals(MetaData.META_FLAG_VERSION_010)) {


Java strings support startsWith. I think, this should be used here instead of this complicated thign.

koppor · 2024-11-16T22:56:31Z

src/main/java/org/jabref/logic/importer/fileformat/BibtexParser.java

+        }
+    }
+
+    private JsonObject parseCommentToJson(String comment, Map<String, String> meta) {


Remove the map as parameter. It should not be modified.

The JsonObject should be separate - to enable handling v5.x meta data (Map) and v6.x metadata (JSON)

Moreover, return Optional<JsonObject> (which is more modern Java)

koppor · 2024-11-16T22:57:34Z

src/main/java/org/jabref/logic/importer/fileformat/BibtexParser.java

+        Pattern pattern = Pattern.compile("\\{.*}", Pattern.DOTALL);
+        Matcher matcher = pattern.matcher(comment);


I think, just doing comments.substring(MetaData.META_FLAG_VERSION_010) should to the trick. - No need for more checking here.

src/main/java/org/jabref/logic/importer/util/MetaDataParser.java

koppor · 2024-11-16T22:58:57Z

src/main/java/org/jabref/logic/importer/util/MetaDataParser.java

+        return parse(new MetaData(), data, keywordSeparator);
+    }
+
+    public MetaData parse(MetaData metaData, JsonObject data, Character keywordSeparator) throws ParseException {


Add JavaDoc - I don't get this.

I think, this should not be in this PR for now - We only focus on the JSON - and then, we can work on the parsing of the JSON.

Please, either test this method or remove it - and put it to the next PR.

This method is should be static, shouldn't it?

1. add javaDoc 2. remove necessary regex use substring 3. refactor code to `startsWith`

koppor · 2024-11-25T22:31:02Z

src/main/java/org/jabref/logic/importer/util/MetaDataParser.java

+        return parse(new MetaData(), data, keywordSeparator);
+    }
+
+    public MetaData parse(MetaData metaData, JsonObject data, Character keywordSeparator) throws ParseException {


Please, either test this method or remove it - and put it to the next PR.

koppor · 2024-11-25T22:32:38Z

src/main/java/org/jabref/logic/importer/util/MetaDataParser.java

+        return parse(new MetaData(), data, keywordSeparator);
+    }
+
+    public MetaData parse(MetaData metaData, JsonObject data, Character keywordSeparator) throws ParseException {


This method is should be static, shouldn't it?

koppor · 2024-11-25T22:33:41Z

src/main/java/org/jabref/logic/importer/fileformat/BibtexParser.java

@@ -386,9 +386,17 @@ private void parseJabRefComment(Map<String, String> meta) {
            } catch (ParseException ex) {
                parserResult.addException(ex);
            }
+        } else if (comment.startsWith(MetaData.META_FLAG_VERSION_010)) {
+            parseCommentToJson(comment);


The result is not handled. I think, it needs to be handled. It needs to be passed to the Metadata.parse method somehow.

Sorry I made this PR too bigger again, so I decided only to add the unit test for parseCommentToJson and implement of it, I'll do more in next PR.

fix the unittest to keep the PR small

koppor

I think public MetaData parse(MetaData metaData, JsonObject data, Character keywordSeparator) throws ParseException { has to be removed.

You can start with adding a parse method to org.jabref.logic.importer.util.MetaDataParser acceping a JSONObject (with the current save actions configured).

koppor · 2024-11-28T09:49:54Z

src/test/java/org/jabref/logic/importer/fileformat/BibtexParserTest.java

+        JsonObject actualJson = parser.parseCommentToJson(entries).orElse(null);
+        assertEquals(actualJson, getExpectedJson());


No. Use the power of Optionals.

Suggested change

JsonObject actualJson = parser.parseCommentToJson(entries).orElse(null);

assertEquals(actualJson, getExpectedJson());

Optional<JsonObject> actualJson = parser.parseCommentToJson(entries);

assertEquals(actualJson, Optional.of(getExpectedJson()));

koppor · 2024-11-28T11:32:16Z

src/main/java/org/jabref/logic/importer/util/MetaDataParser.java

+                String user = entry.getKey().substring(MetaData.FILE_DIRECTORY.length() + 1);
+                metaData.setUserFileDirectory(user, parseDirectory(entry.getValue()));
+            } else if (entry.getKey().startsWith(MetaData.FILE_DIRECTORY_LATEX)) {
+                // The user name starts directly after FILE_DIRECTORY_LATEX + '-'


These hacks should be removed when transitioning to JSON.

leaf-soba · 2024-11-28T13:07:19Z

Sorry I just forget to remove public MetaData parse

remove MetaData.parse

koppor

I know saw that puzzled me at getExpectedJson() {

We need to to proper engineering there, since this will be in the core of JabRef. And proper engineering takes time...

koppor · 2024-11-29T10:01:12Z

src/main/java/org/jabref/logic/importer/util/MetaDataParser.java

-
+

No additional spaces pleace

koppor · 2024-11-29T10:06:05Z

src/test/java/org/jabref/logic/importer/fileformat/BibtexParserTest.java

+        JsonObject saveActions = new JsonObject();
+        saveActions.addProperty("state", true);
+        JsonArray dateArray = new JsonArray();
+        dateArray.add("normalize_date");
+        dateArray.add("action2");
+        saveActions.add("date", dateArray);
+        JsonArray pagesArray = new JsonArray();
+        pagesArray.add("normalize_page_numbers");
+        saveActions.add("pages", pagesArray);
+        JsonArray monthArray = new JsonArray();
+        monthArray.add("normalize_month");
+        saveActions.add("month", monthArray);
+        JsonObject expectedJson = new JsonObject();
+        expectedJson.add("saveActions", saveActions);
+        return expectedJson;


This reads very ugly.

According to https://stackoverflow.com/a/4696873/873282 it can be easier

Can you try with var?

var expected = new Object() { ...

Another thought: Just simplify the contained object.

you are testing the JSON reading - then put a simple JSON inside. Test one thing in one test :-)

Reason: The searialization of saveActions will probalby changed. Especially state will be called enabled for sure (because state has a different meaning)

add a new parseJabRefComment

56c1bef

leaf-soba marked this pull request as draft November 4, 2024 08:34

github-actions bot reviewed Nov 4, 2024

View reviewed changes

fix logic check string contains a valid json

b2646ae

fix logic check string contains a valid json

leaf-soba marked this pull request as ready for review November 4, 2024 08:52

koppor requested changes Nov 4, 2024

View reviewed changes

koppor added the status: changes required Pull requests that are not yet complete label Nov 4, 2024

calixtus reviewed Nov 4, 2024

View reviewed changes

read json as metadata

e12e655

1. read json as metadata 2. small refactor to remove a not in use `LOGGER` in `MetaData`

github-actions bot reviewed Nov 5, 2024

View reviewed changes

leaf-soba requested a review from koppor November 6, 2024 00:56

Merge branch 'main' into close-issue-10371

465d6cf

leaf-soba requested a review from calixtus November 11, 2024 09:05

koppor requested changes Nov 11, 2024

View reviewed changes

rewrite metaDataParser.parse

830e7e6

rewrite metaDataParser.parse

github-actions bot reviewed Nov 14, 2024

View reviewed changes

fix wrong input

065a784

fix wrong input

koppor requested changes Nov 16, 2024

View reviewed changes

follow the review fix code

3c4ee8f

1. add javaDoc 2. remove necessary regex use substring 3. refactor code to `startsWith`

leaf-soba requested a review from koppor November 21, 2024 09:07

koppor requested changes Nov 25, 2024

View reviewed changes

fix the unittest to keep the PR small

fb161cf

fix the unittest to keep the PR small

leaf-soba requested a review from koppor November 28, 2024 07:20

koppor requested changes Nov 28, 2024

View reviewed changes

remove MetaData.parse

ae98cdc

remove MetaData.parse

koppor requested changes Nov 29, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a new parseJabRefComment with unit test #12145

Add a new parseJabRefComment with unit test #12145

leaf-soba commented Nov 4, 2024

github-actions bot left a comment

koppor left a comment

calixtus Nov 4, 2024

github-actions bot left a comment

koppor Nov 11, 2024

koppor Nov 11, 2024

koppor Nov 11, 2024

github-actions bot left a comment

koppor Nov 16, 2024

koppor Nov 16, 2024

koppor Nov 16, 2024

koppor Nov 16, 2024

koppor Nov 16, 2024

koppor Nov 25, 2024

koppor Nov 25, 2024

koppor Nov 25, 2024

koppor Nov 25, 2024

koppor Nov 25, 2024

leaf-soba Nov 28, 2024

koppor left a comment

koppor Nov 28, 2024

koppor Nov 28, 2024

leaf-soba commented Nov 28, 2024

koppor left a comment

koppor Nov 29, 2024

koppor Nov 29, 2024

	public static final String META_FLAG_010 = "jabref-meta-0.1.0";
	public static final String META_FLAG_VERSION_010 = "jabref-meta-0.1.0";

		Pattern pattern = Pattern.compile("\\{.*}", Pattern.DOTALL);
		Matcher matcher = pattern.matcher(comment);

		JsonObject actualJson = parser.parseCommentToJson(entries).orElse(null);
		assertEquals(actualJson, getExpectedJson());

Add a new parseJabRefComment with unit test #12145

Are you sure you want to change the base?

Add a new parseJabRefComment with unit test #12145

Conversation

leaf-soba commented Nov 4, 2024

Mandatory checks

github-actions bot left a comment

Choose a reason for hiding this comment

koppor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koppor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leaf-soba commented Nov 28, 2024

koppor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment