Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: TaskProcessing API #45094

Merged
merged 63 commits into from
May 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
00894e2
feat: first pass at TaskProcessing API
marcelklehr Apr 29, 2024
8352b27
fix: weed out some psalm errors and run cs:fix
marcelklehr Apr 30, 2024
4b2acee
test: Add OldTasksShouldBeCleanedUp test
marcelklehr Apr 30, 2024
17486ad
fix: Add RemoveOldTasksBackgroundJob to repair step that instantiates it
marcelklehr Apr 30, 2024
ee7592f
fix: Run cs:fix
marcelklehr Apr 30, 2024
44b896f
feat: TaskProcessing OCS API
marcelklehr Apr 30, 2024
b2b93e4
feat: Add getFileContents endpoint to TaskProcessing OCS API
marcelklehr May 2, 2024
86317bb
refactor: Move validation to EShapeType
marcelklehr May 2, 2024
29cbb3c
chore: Run cs:fix
marcelklehr May 2, 2024
3b0925a
chore: Regenerate openapi.json
marcelklehr May 2, 2024
1c033ae
fix(IRootFolder): Add getAppDataDirectoryName method
marcelklehr May 2, 2024
7a94798
fix: Fix psalm issues
marcelklehr May 2, 2024
8e56626
feat: Add ExApp endpoints
marcelklehr May 3, 2024
5031a2e
fix: Typo
marcelklehr May 3, 2024
843bb62
fix: LazyRoot missing method
marcelklehr May 3, 2024
c9ea537
fix: Fix psalm issues
marcelklehr May 3, 2024
b150d77
refactor: rename getTaskType to getTaskTypeId
marcelklehr May 3, 2024
4b954d2
fix: Wire-up the new classes
marcelklehr May 3, 2024
eebeb82
fix: Small fixes
marcelklehr May 3, 2024
bd5dfd0
test: Add more tests for legacy pass-through
marcelklehr May 3, 2024
3593d9b
fix: psalm issue
marcelklehr May 3, 2024
9a2cd6b
fix: Expose task type on CoreTaskProcessingTask json
marcelklehr May 6, 2024
928d04f
fix: oc_taskProcessing_tasks.identifier: notnull = false
marcelklehr May 6, 2024
9effb55
chore: update openapi.json
marcelklehr May 6, 2024
996e507
feat: Remove stale files in AppData as well
marcelklehr May 6, 2024
6203c1c
fix: Check if user is authorized to use the files they mentioned
marcelklehr May 6, 2024
ef44af1
test: Test file authorization check
marcelklehr May 6, 2024
8ccb29a
fix: psalm issues
marcelklehr May 6, 2024
a5053d3
fix: Run cs:fix
marcelklehr May 6, 2024
b85a0ed
fix: Update autoloaders
marcelklehr May 6, 2024
5de42a5
fix: Don't use dynamic property
marcelklehr May 6, 2024
2c87809
fix: address review comments
marcelklehr May 6, 2024
ef61c50
fix: address review comments
marcelklehr May 6, 2024
ec27c53
fix: address review comments
marcelklehr May 6, 2024
fff2fb8
fix: psalm issue
marcelklehr May 7, 2024
4a3b9b8
refactor: identifier is now customId/custom_id
marcelklehr May 7, 2024
20c09d1
fix: Don't check in barmani plugin in composer autoloader
marcelklehr May 7, 2024
beeee26
fix: bump OC_Version
marcelklehr May 7, 2024
4d9a0ea
fix: update openai specs
marcelklehr May 7, 2024
f2ab6cb
fix: fix psalm issues
marcelklehr May 7, 2024
c079a61
feat: Add cancel endpoint to OCS API
marcelklehr May 7, 2024
a045e0c
fix: fix migration
marcelklehr May 7, 2024
0e06d64
Update core/Migrations/Version30000Date20240429122720.php
marcelklehr May 7, 2024
f6f4965
fix: fix tests
marcelklehr May 7, 2024
eff862b
fix: run cs:fix
marcelklehr May 8, 2024
ec94a67
fix(ocs): change /tasktypes response to combine optional and non-opti…
marcelklehr May 8, 2024
19a0aae
fix(TextToImage): Allow leaving the resources open
marcelklehr May 10, 2024
f3a88f0
fix(OCS-API): No csrf required for /tasks/taskId/file/fileId
marcelklehr May 10, 2024
a8afa7f
fix(OCS-API): Add endpoint to list user tasks
marcelklehr May 10, 2024
9cc1a01
test: Put input files in user storage
marcelklehr May 10, 2024
715245a
fix: run cs:fix
marcelklehr May 10, 2024
e4b1d23
fix(Manager#fillInputData): Load user folder if needed
marcelklehr May 10, 2024
c020490
Update lib/private/TaskProcessing/Manager.php
marcelklehr May 13, 2024
b11052f
Update lib/public/TaskProcessing/IManager.php
marcelklehr May 13, 2024
ac36c78
fix(SynchronousBackgroundJob): Only reschedule when needed
marcelklehr May 13, 2024
a9a2cbf
feat: Add some new task types
marcelklehr May 13, 2024
c1f84aa
fix(Manager#fillInputs): Try to setup user FS before access file inputs
marcelklehr May 13, 2024
4ce5aaf
chore: Check in autoloader updates
marcelklehr May 13, 2024
9864fc8
chore: fix htaccess
marcelklehr May 13, 2024
8d06338
fix: Fix pass-through stt provider
marcelklehr May 13, 2024
f5a8bda
Update core/ResponseDefinitions.php
marcelklehr May 14, 2024
cac812d
fix: address review comments
marcelklehr May 14, 2024
6c4992d
fix: expose lastUpdated in OCS API
marcelklehr May 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
430 changes: 430 additions & 0 deletions core/Controller/TaskProcessingApiController.php

Large diffs are not rendered by default.

115 changes: 115 additions & 0 deletions core/Migrations/Version30000Date20240429122720.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
<?php

declare(strict_types=1);

/**
* @copyright Copyright (c) 2024 Marcel Klehr <[email protected]>
*
* @author Marcel Klehr <[email protected]>
*
* @license AGPL-3.0-or-later
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*
*/

namespace OC\Core\Migrations;

use Closure;
use OCP\DB\ISchemaWrapper;
use OCP\DB\Types;
use OCP\Migration\IOutput;
use OCP\Migration\SimpleMigrationStep;

/**
*
*/
class Version30000Date20240429122720 extends SimpleMigrationStep {

/**
* @param IOutput $output
* @param Closure $schemaClosure The `\Closure` returns a `ISchemaWrapper`
* @param array $options
* @return null|ISchemaWrapper
*/
public function changeSchema(IOutput $output, Closure $schemaClosure, array $options): ?ISchemaWrapper {
/** @var ISchemaWrapper $schema */
$schema = $schemaClosure();

if (!$schema->hasTable('taskprocessing_tasks')) {
$table = $schema->createTable('taskprocessing_tasks');

$table->addColumn('id', Types::BIGINT, [
'notnull' => true,
'length' => 64,
'autoincrement' => true,
]);
$table->addColumn('type', Types::STRING, [
'notnull' => true,
'length' => 255,
]);
$table->addColumn('input', Types::TEXT, [
'notnull' => true,
]);
$table->addColumn('output', Types::TEXT, [
'notnull' => false,
]);
$table->addColumn('status', Types::INTEGER, [
'notnull' => false,
'length' => 6,
'default' => 0,
]);
$table->addColumn('user_id', Types::STRING, [
'notnull' => false,
'length' => 64,
]);
$table->addColumn('app_id', Types::STRING, [
'notnull' => true,
'length' => 32,
'default' => '',
]);
$table->addColumn('custom_id', Types::STRING, [
'notnull' => false,
'length' => 255,
'default' => '',
]);
$table->addColumn('last_updated', Types::INTEGER, [
'notnull' => false,
'length' => 4,
'default' => 0,
'unsigned' => true,
]);
$table->addColumn('completion_expected_at', Types::DATETIME, [
'notnull' => false,
]);
$table->addColumn('progress', Types::FLOAT, [
'notnull' => false,
'default' => 0,
]);
$table->addColumn('error_message', Types::STRING, [
'notnull' => false,
'length' => 255,
]);

$table->setPrimaryKey(['id'], 'taskp_tasks_id_index');
$table->addIndex(['status', 'type'], 'taskp_tasks_status_type');
$table->addIndex(['last_updated'], 'taskp_tasks_updated');
$table->addIndex(['user_id', 'app_id', 'custom_id'], 'taskp_tasks_uid_appid_cid');

return $schema;
}

return null;
}
}
31 changes: 31 additions & 0 deletions core/ResponseDefinitions.php
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,37 @@
* iconURL: ?string,
* iconEmoji: ?string,
* }
*
* @psalm-type CoreTaskProcessingShape = array{
* name: string,
* description: string,
* type: "Number"|"Text"|"Audio"|"Image"|"Video"|"File"|"ListOfNumbers"|"ListOfTexts"|"ListOfImages"|"ListOfAudios"|"ListOfVideos"|"ListOfFiles",
* mandatory: bool,
marcelklehr marked this conversation as resolved.
Show resolved Hide resolved
* }
*
* @psalm-type CoreTaskProcessingTaskType = array{
* name: string,
* description: string,
* inputShape: CoreTaskProcessingShape[],
* outputShape: CoreTaskProcessingShape[],
Comment on lines +190 to +191
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* inputShape: CoreTaskProcessingShape[],
* outputShape: CoreTaskProcessingShape[],
* inputShapes: CoreTaskProcessingShape[],
* outputShapes: CoreTaskProcessingShape[],

Since they are an array the names should be plural as well.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree. It's the shape of the input. It's one input (because it's input once) so one shape.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm that makes sense, how about just input and output?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well it's not an input but just the shape of the input to be :)

* }
*
* @psalm-type CoreTaskProcessingIO = array<string, numeric|list<numeric>|string|list<string>>
*
* @psalm-type CoreTaskProcessingTask = array{
* id: int,
* lastUpdated: int,
* type: string,
* status: 'STATUS_CANCELLED'|'STATUS_FAILED'|'STATUS_SUCCESSFUL'|'STATUS_RUNNING'|'STATUS_SCHEDULED'|'STATUS_UNKNOWN',
* userId: ?string,
* appId: string,
* input: CoreTaskProcessingIO,
* output: null|CoreTaskProcessingIO,
* customId: ?string,
* completionExpectedAt: ?int,
* progress: ?float
provokateurin marked this conversation as resolved.
Show resolved Hide resolved
* }
*
*/
class ResponseDefinitions {
}