Skip to content
This repository has been archived by the owner on Feb 1, 2020. It is now read-only.

makeDictionary doesn't allow creating big dictionaries #7

Open
marmistrz opened this issue Nov 29, 2016 · 8 comments
Open

makeDictionary doesn't allow creating big dictionaries #7

marmistrz opened this issue Nov 29, 2016 · 8 comments

Comments

@marmistrz
Copy link

I generated a words.xml and then ran the gradle build task. Then makeDictionary randomly fails:

:makeDictionary
Reading words from input /home/marcin/proj/LanguagePack/dictionary/words.xml
Will store output files under /home/marcin/proj/LanguagePack/src/main/res/raw
Deleting previous versions...
Nodes = 2152096
:makeDictionary FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':makeDictionary'.
> 4194305

Ran it again:

:makeDictionary
Reading words from input /home/marcin/proj/LanguagePack/dictionary/words.xml
Will store output files under /home/marcin/proj/LanguagePack/src/main/res/raw
Deleting previous versions...
Nodes = 3466568
:makeDictionary FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':makeDictionary'.
> java.lang.ArrayIndexOutOfBoundsException (no error message)

Once again, with stacktrace:

:makeDictionary
Reading words from input /home/marcin/proj/LanguagePack.priv/dictionary/words.xml
Will store output files under /home/marcin/proj/LanguagePack.priv/src/main/res/raw
Deleting previous versions...
Nodes = 4781040
:makeDictionary FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':makeDictionary'.
> java.lang.ArrayIndexOutOfBoundsException (no error message)

* Try:
Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':makeDictionary'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:84)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:61)
        at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
        at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:88)
        at org.gradle.api.internal.tasks.execution.ResolveTaskArtifactStateTaskExecuter.execute(ResolveTaskArtifactStateTaskExecuter.java:45)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:51)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:54)
        at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:34)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:233)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:215)
        at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:74)
        at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:55)
        at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:32)
        at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:113)
        at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:37)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
        at org.gradle.execution.DefaultBuildExecuter.access$000(DefaultBuildExecuter.java:23)
        at org.gradle.execution.DefaultBuildExecuter$1.proceed(DefaultBuildExecuter.java:43)
        at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
        at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:30)
        at org.gradle.initialization.DefaultGradleLauncher$4.run(DefaultGradleLauncher.java:197)
        at org.gradle.internal.Factories$1.create(Factories.java:25)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:53)
        at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:194)
        at org.gradle.initialization.DefaultGradleLauncher.access$200(DefaultGradleLauncher.java:36)
        at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:118)
        at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:112)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:91)
        at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:63)
        at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:112)
        at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:98)
        at org.gradle.launcher.exec.GradleBuildController.run(GradleBuildController.java:66)
        at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
        at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:41)
        at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:26)
        at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:75)
        at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:49)
        at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:44)
        at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:29)
        at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:67)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:47)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:26)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:34)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:74)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:72)
        at org.gradle.util.Swapper.swap(Swapper.java:38)
        at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:72)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:60)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:72)
        at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
        at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
        at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:50)
        at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:293)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
        at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
Caused by: java.lang.ArrayIndexOutOfBoundsException

Using gradle 3.2.1, with the 2.2.0 Android gradle plugin.
@marmistrz
Copy link
Author

What's interesting, when I limit the words.xml to 49 words, no exception is thrown. My words.xml has 652988 words.
I suspect that one word causes the whole thing to crash (I'm unable to filter all the weird ones)

@marmistrz
Copy link
Author

marmistrz commented Dec 9, 2016 •
edited
Loading

Ok, I see that my frequencies are much bigger than 255. May this be the source of a problem?

I managed to get a more meaningful stack trace: http://pastebin.com/HwiYeb0e

@marmistrz
Copy link
Author

marmistrz commented Dec 9, 2016 •
edited
Loading

Ok, I see, the culprit is the following line: https://github.com/AnySoftKeyboard/AnySoftKeyboardTools/blob/master/makedictionary/src/main/groovy/com/anysoftkeyboard/tools/makedictionary/MakeBinaryDictionary.java#L244

The number of words in a dictionary seems to be a hard limit.

What's interesting, I only have 652988 words in my words.xml

@marmistrz marmistrz changed the title makeDictionary randomly crashes makeDictionary doesn't allow creating big dictionaries Dec 9, 2016
@marmistrz
Copy link
Author

Another update: if I have 1024 * 1024 / 4 words in my words.xml, makeDictionary succeeds. If I have 1024 * 1024 / 2 of them - it crashes

@kertase
Copy link

kertase commented Dec 11, 2016

I can confirm this issue and it has been existing for a while now see: #4

I think that the dictionary size limit is somehow hard coded to 4mb=4,194,304.

@kertase
Copy link

kertase commented Dec 12, 2016 •
edited
Loading

For your info, I tried changing the limit in the code as you can see here: https://github.com/kertase/AnySoftKeyboardTools
The dictionary gets build but app crashes with a dictionary of that size unfortunately so the limit is there for a reason and that is the app can’t manage a bigger dic.

@ghost
Copy link

ghost commented Dec 18, 2016

Came across the same problem, hoping for an update

@PanderMusubi
Copy link

If this is a duplicate of #4 then please close this issue. Workaround for now is to limit the number of words.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@PanderMusubi @marmistrz @kertase