Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/evi 109 option disable index #80

Open
wants to merge 68 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
b0abd23
EVI-109 disable index
FedeC87p Jul 5, 2023
8799757
EVI-129 generate thumbnail
FedeC87p Jul 9, 2023
96fb8b0
merge hotfix
tmm360 Jul 20, 2023
6a2fd19
update help and readme
tmm360 Jul 20, 2023
764fb37
merge from dev
FedeC87p Jul 21, 2023
fff3334
merge from dev
FedeC87p Jul 22, 2023
f10cf56
EVI-129 fix merge conflict
FedeC87p Jul 22, 2023
a415e09
EVI-129 refactory
FedeC87p Jul 22, 2023
42fd73b
add jetbrains ide config
tmm360 Aug 29, 2023
627e4ef
update dependencies
tmm360 Aug 29, 2023
839c714
enable deferred upload
tmm360 Aug 29, 2023
e0b4440
Merge pull request #84 from Etherna/improve/EVI-136-deferred-uploads
tmm360 Aug 29, 2023
e9e035d
add ffprobe reference in readme
tmm360 Aug 29, 2023
9b17d85
Merge pull request #85 from Etherna/improve/EVI-137-ffprove-readme
tmm360 Aug 29, 2023
780335a
catch YoutubeExplodeException
tmm360 Aug 29, 2023
93b0de7
Merge pull request #86 from Etherna/improve/EVI-139-catch-YoutubeExpl…
tmm360 Aug 29, 2023
1436d70
fix youtube get issue
tmm360 Aug 29, 2023
8ebe37b
Merge pull request #87 from Etherna/fix/EVI-140-upgrade-ytdownloader
tmm360 Aug 29, 2023
9e59b8d
handle timeoutexception with yt downloads
tmm360 Aug 30, 2023
b2fc068
Merge pull request #88 from Etherna/improve/EVI-142-handle-yt-timeout…
tmm360 Aug 30, 2023
ef65f81
EVI-119 check for ffmpeg
FedeC87p Jul 2, 2023
2a9a865
EVI-119 check FFProbe
FedeC87p Jul 2, 2023
0edc478
EVI-119 refactory
FedeC87p Jul 3, 2023
1974d8e
EVI-119 fix merge
FedeC87p Jul 20, 2023
159e004
refactory
FedeC87p Jul 20, 2023
d1c3513
fix FFmpegFolderPath
FedeC87p Jul 21, 2023
556ab2f
EVI-119 refactory
FedeC87p Jul 22, 2023
a7885eb
EVI-119 fix path returned
FedeC87p Jul 22, 2023
a44f82b
EVI-119 fix CommonCheckAndGetAsync
FedeC87p Jul 22, 2023
9c1d4c6
merge global binary support
tmm360 Aug 30, 2023
05100e8
add id migration feature
tmm360 Sep 1, 2023
7328064
minor name fix
tmm360 Sep 1, 2023
c0eca6c
Merge pull request #89 from Etherna/feature/EVI-146-add-id-migration
tmm360 Sep 1, 2023
6906fba
update license headers
tmm360 Sep 1, 2023
31afa51
update from sagl to sa
tmm360 Nov 10, 2023
cb941ae
Merge pull request #90 from Etherna/improve/EVI-152-etherna-sa
tmm360 Nov 10, 2023
72e9990
update dependencies
tmm360 Nov 10, 2023
d399331
Merge pull request #91 from Etherna/improve/EVI-154-update-YTD
tmm360 Nov 10, 2023
0808fe0
update github action build with new architectures
tmm360 Nov 10, 2023
52f27d6
Merge pull request #92 from Etherna/feature/EVI-149-add-32bit-build
tmm360 Nov 10, 2023
279636b
enable pinning by default
tmm360 Nov 10, 2023
5cb4dfd
Merge pull request #93 from Etherna/improve/EVI-153-enable-pinning-by…
tmm360 Nov 10, 2023
397c403
merge hotfix
tmm360 Nov 10, 2023
fd18694
remove unsupported linux-x86 arch
tmm360 Nov 10, 2023
2d670fc
update bee.net and etherna sdk
tmm360 Nov 15, 2023
c0ba028
minor fixes
tmm360 Jan 7, 2024
44ef580
update to dotnet8
tmm360 Jan 7, 2024
021d1ad
Merge pull request #94 from Etherna/improve/EVI-160-dotnet8
tmm360 Jan 7, 2024
4478754
update README.md
tmm360 Jan 7, 2024
40ab315
merge hotfix
tmm360 Jan 10, 2024
d528116
merge thumbnail update
tmm360 Jan 15, 2024
02c2ece
fix thumbnail creation
tmm360 Jan 16, 2024
b1ccf46
update dependencies
tmm360 Feb 26, 2024
6b4b227
fix wrong property reference
tmm360 Mar 3, 2024
1041228
Merge pull request #95 from Etherna/fix/EVI-168-print-correct-indexid
tmm360 Mar 3, 2024
760baf2
fix OriginVideoQualityLabel updating manifest
tmm360 Mar 3, 2024
0c495a8
Merge pull request #96 from Etherna/fix/EVI-167-video-quality-label-f…
tmm360 Mar 3, 2024
94890c2
update dependencies
tmm360 Mar 3, 2024
1c59e40
update dependencies
tmm360 Mar 22, 2024
8a69ec1
fix build
tmm360 Mar 22, 2024
f025ac5
update dependency
tmm360 Mar 25, 2024
57ed897
merge hotfix
tmm360 Mar 25, 2024
ab149fd
update readme and minor updates
tmm360 Mar 25, 2024
b1b8d31
update entry points and options
tmm360 Mar 25, 2024
d24f131
update domain models and services
tmm360 Mar 26, 2024
6a3f81e
update CleanerVideoService
tmm360 Mar 27, 2024
4203a20
update VideoUploaderService
tmm360 Mar 27, 2024
9682d1f
update domain models
tmm360 Mar 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 88 additions & 12 deletions .github/workflows/publish-stable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,45 +11,121 @@ jobs:

strategy:
matrix:
kind: ['linux', 'linuxselfcontained', 'macos', 'macosselfcontained', 'windows', 'windowsselfcontained']
kind: [
'linux-x64',
'linux-arm',
'linux-arm64',

'linux-selfcont-x64',
'linux-selfcont-arm',
'linux-selfcont-arm64',

'macos-x64',
'macos-arm64',

'macos-selfcont-x64',
'macos-selfcont-arm64',

'windows-x86',
'windows-x64',
'windows-arm',
'windows-arm64',

'windows-selfcont-x86',
'windows-selfcont-x64',
'windows-selfcont-arm',
'windows-selfcont-arm64']
include:
- kind: linux
- kind: linux-x64
os: ubuntu-latest
target: linux-x64
frameworktype: --no-self-contained
- kind: linuxselfcontained
- kind: linux-arm
os: ubuntu-latest
target: linux-arm
frameworktype: --no-self-contained
- kind: linux-arm64
os: ubuntu-latest
target: linux-arm64
frameworktype: --no-self-contained

- kind: linux-selfcont-x64
os: ubuntu-latest
target: linux-x64
frameworktype: --self-contained
- kind: macos
- kind: linux-selfcont-arm
os: ubuntu-latest
target: linux-arm
frameworktype: --self-contained
- kind: linux-selfcont-arm64
os: ubuntu-latest
target: linux-arm64
frameworktype: --self-contained

- kind: macos-x64
os: macOS-latest
target: osx-x64
frameworktype: --no-self-contained
- kind: macosselfcontained
- kind: macos-arm64
os: macOS-latest
target: osx-arm64
frameworktype: --no-self-contained

- kind: macos-selfcont-x64
os: macOS-latest
target: osx-x64
frameworktype: --self-contained
- kind: windows
- kind: macos-selfcont-arm64
os: macOS-latest
target: osx-arm64
frameworktype: --self-contained

- kind: windows-x86
os: windows-latest
target: win-x86
frameworktype: --no-self-contained
- kind: windows-x64
os: windows-latest
target: win-x64
frameworktype: --no-self-contained
- kind: windowsselfcontained
- kind: windows-arm
os: windows-latest
target: win-arm
frameworktype: --no-self-contained
- kind: windows-arm64
os: windows-latest
target: win-arm64
frameworktype: --no-self-contained

- kind: windows-selfcont-x86
os: windows-latest
target: win-x86
frameworktype: --self-contained
- kind: windows-selfcont-x64
os: windows-latest
target: win-x64
frameworktype: --self-contained
- kind: windows-selfcont-arm
os: windows-latest
target: win-arm
frameworktype: --self-contained
- kind: windows-selfcont-arm64
os: windows-latest
target: win-arm64
frameworktype: --self-contained

runs-on: ${{ matrix.os }}

steps:
- name: Checkout
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Setup dotnet
uses: actions/setup-dotnet@v3
uses: actions/setup-dotnet@v4
with:
dotnet-version: 7.0.x
dotnet-version: 8.0.x

- name: Build EthernaVideoImporter project
shell: bash
Expand All @@ -59,7 +135,7 @@ jobs:
# Build everything
dotnet publish src/EthernaVideoImporter/EthernaVideoImporter.csproj --runtime "${{ matrix.target }}" "${{ matrix.frameworktype }}" -c Release -o "$release_name"
# Pack files
if [ "${{ matrix.target }}" == "win-x64" ]; then
if [ "${{ matrix.kind }}" == windows* ]; then
7z a -tzip "${release_name}.zip" "./${release_name}/*"
else
tar czvf "${release_name}.tar.gz" "$release_name"
Expand All @@ -73,7 +149,7 @@ jobs:
# Build everything
dotnet publish src/EthernaVideoImporter.Devcon/EthernaVideoImporter.Devcon.csproj --runtime "${{ matrix.target }}" "${{ matrix.frameworktype }}" -c Release -o "$release_name"
# Pack files
if [ "${{ matrix.target }}" == "win-x64" ]; then
if [ "${{ matrix.kind }}" == windows* ]; then
7z a -tzip "${release_name}.zip" "./${release_name}/*"
else
tar czvf "${release_name}.tar.gz" "$release_name"
Expand Down
13 changes: 13 additions & 0 deletions .idea/.idea.EthernaVideoImporter/.idea/.gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions .idea/.idea.EthernaVideoImporter/.idea/.name

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions .idea/.idea.EthernaVideoImporter/.idea/encodings.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions .idea/.idea.EthernaVideoImporter/.idea/indexLayout.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/.idea.EthernaVideoImporter/.idea/vcs.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion EthernaVideoImporter.sln
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@ EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{6647EBF4-A8E7-4987-9A43-CF6F9DA1A3AF}"
ProjectSection(SolutionItems) = preProject
.editorconfig = .editorconfig
EthernaVideoImporter.sln.licenseheader = EthernaVideoImporter.sln.licenseheader
README.md = README.md
.gitignore = .gitignore
EthernaVideoImporter.sln.DotSettings = EthernaVideoImporter.sln.DotSettings
EndProjectSection
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "workflows", "workflows", "{1E5D1B5B-4926-4F7F-97A9-19DAE5065550}"
Expand Down
21 changes: 21 additions & 0 deletions EthernaVideoImporter.sln.DotSettings
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
<wpf:ResourceDictionary xml:space="preserve" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:s="clr-namespace:System;assembly=mscorlib" xmlns:ss="urn:shemas-jetbrains-com:settings-storage-xaml" xmlns:wpf="http://schemas.microsoft.com/winfx/2006/xaml/presentation">
<s:String x:Key="/Default/CodeStyle/FileHeader/FileHeaderText/@EntryValue">Copyright 2022-present Etherna SA

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.</s:String>
<s:Boolean x:Key="/Default/Environment/SettingsMigration/IsMigratorApplied/=JetBrains_002EReSharper_002EFeature_002EServices_002ECodeCleanup_002EFileHeader_002EFileHeaderSettingsMigrate/@EntryIndexedValue">True</s:Boolean>
<s:Boolean x:Key="/Default/UserDictionary/Words/=Dtos/@EntryIndexedValue">True</s:Boolean>
<s:Boolean x:Key="/Default/UserDictionary/Words/=etherna/@EntryIndexedValue">True</s:Boolean>
<s:Boolean x:Key="/Default/UserDictionary/Words/=Fmpeg/@EntryIndexedValue">True</s:Boolean>
<s:Boolean x:Key="/Default/UserDictionary/Words/=Fprobe/@EntryIndexedValue">True</s:Boolean>
<s:Boolean x:Key="/Default/UserDictionary/Words/=skippable/@EntryIndexedValue">True</s:Boolean>
</wpf:ResourceDictionary>
47 changes: 0 additions & 47 deletions EthernaVideoImporter.sln.licenseheader

This file was deleted.

28 changes: 18 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ Currently exists two versions:
* `EthernaVideoImporter` for a generic use
* `EthernaVideoImporter.Devcon` to import specifically Devcon Archive's videos

Etherna Video Importer requires at least [.NET 7 Runtime](https://dotnet.microsoft.com/download/dotnet/7.0) and [ASP.NET Core 7 Runtime](https://dotnet.microsoft.com/download/dotnet/7.0) installed on local machine to run, or it needs the `selfcontained` version of package, that already contains framework dependencies.
Etherna Video Importer requires at least [.NET 8 Runtime](https://dotnet.microsoft.com/download/dotnet/8.0) and [ASP.NET Core 8 Runtime](https://dotnet.microsoft.com/download/dotnet/8.0) installed on local machine to run, or it needs the `selfcontained` version of package, that already contains framework dependencies.

### Setup FFmpeg
To run the importer you need to download [FFmpeg](https://ffmpeg.org/download.html) locally, and copy the binary file into the default folder "\FFmpeg", or specify its location with arguments.
### Setup FFmpeg and FFprobe
To run the importer you need to download [FFmpeg and FFprobe](https://ffmpeg.org/download.html) locally, and copy them into the default folder "\FFmpeg", or specify its location with arguments.

### How to use

Expand All @@ -28,14 +28,17 @@ Commands:

General Options:
-k, --api-key Api Key (optional)
-f, --ffmpeg-path Path to FFmpeg folder (default: <app_dir>/FFmpeg)
-f, --ffmpeg-path Path to FFmpeg folder (default: search to <app_dir>/FFmpeg or global install)
-i, --ignore-update Ignore new version of EthernaVideoImporter
-a, --auto-purchase Accept automatically purchase of all batches

Index Options:
--no-index Disable video listing on any index

Video Management Options:
-t, --ttl TTL (days) Postage Stamp (default: 365 days)
-o, --offer Offer video downloads to everyone
-p, --pin Pin videos
--no-pin Don't pin videos (pinning by default)
--force Force upload video if they already have been uploaded
-m, --remove-missing Remove indexed videos generated with this tool but missing from source
--remove-unrecognized Remove indexed videos not generated with this tool
Expand All @@ -56,7 +59,11 @@ To import from a video list you need a metadata descriptor file. Metadata is a J
"Title": "First video title",
"Description": "My first video description",
"VideoFilePath": "path/to/your/video1.mp4",
"ThumbnailFilePath": "path/to/your/optional/thumbnail1.jpg"
"ThumbnailFilePath": "path/to/your/optional/thumbnail1.jpg",
"OldIds": [
"optionalOldId1",
"optionalOldId2"
]
},
{
"Id": "myId2",
Expand All @@ -69,8 +76,6 @@ To import from a video list you need a metadata descriptor file. Metadata is a J
]

Id field is mandatory, and is needed to trace same video through different executions. Each Id needs to be unique.
Video paths can be local or online uris. Thumbnail paths are optional, and can only be local.
Local paths can be relative or absolute, online urls can only be absolute.

Run 'evi -h' or 'evi --help' to print help.
```
Expand All @@ -81,14 +86,17 @@ Usage: evid MD_FOLDER [OPTIONS]

General Options:
-k, --api-key Api Key (optional)
-f, --ffmpeg-path Path to FFmpeg folder (default: <app_dir>/FFmpeg)
-f, --ffmpeg-path Path to FFmpeg folder (default: search to <app_dir>/FFmpeg or global install)
-i, --ignore-update Ignore new version of EthernaVideoImporter
-a, --auto-purchase Accept automatically purchase of all batches

Index Options:
--no-index Disable video listing on any index~~~~

Video Management Options:
-t, --ttl TTL (days) Postage Stamp (default: 365 days)
-o, --offer Offer video downloads to everyone
-p, --pin Pin videos
--no-pin Don't pin videos (pinning by default)
--force Force upload video if they already have been uploaded
-m, --remove-missing Remove indexed videos generated with this tool but missing from source
--remove-unrecognized Remove indexed videos not generated with this tool
Expand Down
16 changes: 4 additions & 12 deletions src/EthernaVideoImporter.Core/CommonConsts.cs
Original file line number Diff line number Diff line change
@@ -1,19 +1,17 @@
// Copyright 2022-present Etherna Sagl
//
// Copyright 2022-present Etherna SA
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//
// http://www.apache.org/licenses/LICENSE-2.0
//
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

using Etherna.BeeNet.Clients.DebugApi;
using Etherna.BeeNet.Clients.GatewayApi;
using System;
using System.IO;
using System.Runtime.InteropServices;
Expand All @@ -25,19 +23,13 @@ public sealed class CommonConsts
public const int BeeApiPort = 1633;
public const int BeeDebugPort = 1635;
public const string BeeNodeUrl = "http://localhost/";
public const GatewayApiVersion BeeNodeGatewayVersion = GatewayApiVersion.v4_0_0;
public const DebugApiVersion BeeNodeDebugVersion = DebugApiVersion.v4_0_0;
public static readonly string DefaultFFmpegFolder = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "FFmpeg");
public const int DownloadMaxRetry = 3;
public static readonly TimeSpan DownloadTimespanRetry = TimeSpan.FromMilliseconds(3500);
public const string EthereumNullAddress = "0x0000000000000000000000000000000000000000";
public const string EthernaCreditUrl = "https://credit.etherna.io/";
public const string EthernaIndexUrl = "https://index.etherna.io/";
public const string EthernaIndexContentUrlPrefix = "https://etherna.io/embed/";
public const int EthernaGatewayPort = 443;
public const string EthernaGatewayUrl = "https://gateway.etherna.io/";
public const string EthernaPermalinkContentUrlPrefix = "https://etherna.io/embed/";
public const string EthernaServicesClientName = "ethernaServicesClient";
public const string EthernaSsoUrl = "https://sso.etherna.io/";
public const string EthernaVideoImporterClientId = "ethernaVideoImporterId";
public static string FFmpegBinaryName
Expand Down
Loading