2918: Add tts on web #2985

bahaaTuffaha · 2024-11-06T12:59:17Z

Short description

Adding text-to-speech functionality for web.

Proposed changes

TtsPlayer.tsx will receive content and title from useTtsPlayer.ts hook then it will process content as following:
- Adding period to any <p> or <li> tag if there is non at the end.
- Splitting the content using a library sentencex to support most languages.
- Adding the title to the array.
TtsPlayer.tsx using EasySpeech library for web speech api .. why I used it instead of directly using the native api read here: https://github.com/leaonline/easy-speech/blob/master/FAQ.md#why-does-this-library-exists-if-i-can-use-tts-natively-in-the-browser
Web Speech API has some limitation on devices like Android I can't pause so I implemented a way to increment currentSentencesIndex at each sentence so pausing will be depend on sentence index . Another issue with Firefox about triggering onEnd function at any .cancel() method unlike chromium based browsers (added useeffect to check whether it reached the last index or not ... Note : sometimes content gets finished reading before reaching end of the array)
This is different from tts-native: voices needs to be installed from OS so a modal will show up TtsHelpModal to guide users for each platform. Note: by default for windows for example you have what the keyboard layout have of languages also it may need a restart to see the newly installed language.
Modified Modal.tsx to accept new props: icon and styling just for (backgroundColor, borderRadius)
Notes:
- boundaryGuard is created just to prevent handleBoundary from incrementing again incases like handleForward or handleBackward (I'm still not sure about this function as I did it out of frustration 😂)
- Not Tested on iOS !

Side effects

CityContentToolbar.tsx, CategoriesPage.tsx, LocalNewsPage.tsx, TuNewsDetailPage.tsx, RemoteContent.tsx

Testing

At integreat : Go to any 3rd level content in categories or news.
Click on ToolbarItem on the side called readAloud
The player will show up press play to make it start reading content.

Observations

I tested this PR on linux and I got different result at each browser:
Chrome: works as expected + nice voices.
Firefox: I can't pause (linux) + using opensource voice library from mbrola mb-en1 sounds robotic .
Brave: (linux) not working at all because there is no build in speech to text like google chrome does.
Windows on the other hand.. most browsers can run good as long you download the voices from the OS it self.
I didn't test it on mac OS...

Resolved issues

Fixes: #2918

…d unnecessary mock

…ving the volume.

…le from ttsContext

ztefanie

Nice work 🚀

Tested this on chrome and firefox.

Firefox voice is really crappy, but chrome is really nice 🚀 Maybe we add hint, that some browsers better support this feature? Maybe @hauf-toni can share some input, how to handle this best from a UI perspective.

Found a few bug:

issue 1:

Open this page: http://localhost:9000/augsburg/de/willkommen/willkommen-in-augsburg-2
Click "read aloud"
Click "play"
Click on "Weiter"
Result: The reader starts reading out the beginning of the current sentance, when clicking "weiter" for the first time. When clicking "weiter" a second time, it correctly jumps to the next sentance
Expected result: When clicking "weiter" for the first time, it should directly jump to the next sentance.

issue 2:
When opening this page: http://localhost:9000/augsburg/de/willkommen
the "read aloud" button is displayed, but does nothing when clicked.
Expected: Should either not be displayed or does something when clicked

issue 3:
Open this page: http://localhost:9000/augsburg/de/behoerden-beratung/behoerden/auslaenderbehoerde
Click "next" to jump to the last sentance
in this line "Weitere Informationen finden Sie unter [hier]" the reader switches to the "start"-mode, but there is still this whole contact blog that is read

issue 4:
When clicking pause I see the initial view of the tts-player, with the first word of the page in the heading. I think this is confusing, as clicking play does not continue reading from the beginning. I would expect that the player just changes the pause button to showing a play button.
what i get when i click pause

what i expected

issue 5:
on firefox when the tts reaches the end of the text, the player does not reset to start as it is for other browsers.

ztefanie · 2024-12-13T07:34:02Z

build-configs/common/theme/colors.ts

@@ -34,4 +36,6 @@ export const commonLightColors = {
  invalidInput: '#B3261E',
  warningColor: '#FFA726',
  linkColor: '#0b57d0',
+  ttsPlayerWarningBackground: 'rgba(255, 253, 230, 1)',
+  warning_amber: 'rgba(249, 124, 0, 1)',


Please do not use _ in names. Use camelCase instead. Also please do not put the actual name of the color in the variable nameing, because then it would need renaming if the color changes.
Please stay consistent, either prefix both variables with "ttsPlayer" or none of them

ztefanie · 2024-12-13T07:35:30Z

build-configs/integreat/index.ts

@@ -39,6 +39,7 @@ const commonIntegreatBuildConfig: CommonBuildConfigType = {
    fixedCity: null,
    cityNotCooperatingTemplate,
    chat: true,
+    tts: false,


Why is this "false". Do we only want this feature on beta for now?

ztefanie · 2024-12-13T07:37:27Z

release-notes/unreleased/2918-Add-TTS-on-web.yml

+platforms: # relevant platforms, possible values: web, android and ios
+  - web
+en: Added an exciting new Text-to-Speech feature!
+de: Eine spannende neue Text-zu-Sprache-Funktion wurde hinzugefügt!


Either remove this release_note or enable the feature on production, by also setting tts to true in the build-config for integreat.

ztefanie · 2024-12-13T07:38:34Z

translations/translations.json

@@ -6834,7 +6840,13 @@
      "openInApp": "افتح في تطبيق {{appName}}",
      "open": "افتح",
      "view": "عرض",
-      "getOnPlayStore": "احصل عليه - على متجر Google Play"
+      "getOnPlayStore": "احصل عليه - على متجر Google Play",


Nice, one more language in the team 🚀

ztefanie · 2024-12-13T09:01:43Z

translations/translations.json

+      "accessibility": "Barrierefreiheit",
+      "readAloud": "Vorlesefunktion",
+      "previous": "Zurück",
+      "next": "Weiter",


💭 We already have translations for previous and next in the translations file, but in other sections/categories. I think it is fine to keep it, but we may think about this in the future, if it is worth the effort to restructure things to lower translation costs.

ztefanie · 2024-12-13T10:41:04Z

web/src/components/TtsContainer.tsx

+        text: sentences.slice(index).join(' '),
+        voice: selectedVoice,
+        pitch: 1,
+        rate: 1,


i think pitch and rate could be deleted as these are the defaults

ztefanie · 2024-12-13T10:44:03Z

web/src/components/TtsPlayer.tsx

+  flex-direction: ${props => (props.$isPlaying ? 'column' : 'row')};
+  justify-content: center;
+  align-items: center;
+  align-self: center;


align self can be removed as it does nothing

ztefanie · 2024-12-13T10:44:46Z

web/src/components/TtsPlayer.tsx

+import Icon from './base/Icon'
+
+const StyledTtsPlayer = styled.dialog<{ $isPlaying: boolean }>`
+  background-color: #dedede;


Can you please move this color to the theme?

ztefanie · 2024-12-13T10:54:10Z

web/src/components/TtsPlayer.tsx

+  font-size: 18px;
+`
+
+const CloseButton = styled(Button)`


Please recude code duplication by moving shared styles between styled components by moving them to a shared base class like this:

const buttonBaseStyles = ` display: flex; justify-content: center; align-items: center; transition: box-shadow 0.2s ease, transform 0.1s ease; &:active { box-shadow: none; transform: translateY(2px); } ` const CloseButton = styled(Button)` ${buttonBaseStyles}; // other stylings that are not shared

ztefanie · 2024-12-13T10:57:03Z

web/src/components/TtsPlayer.tsx

+`
+
+const StyledPlayIcon = styled(Button)`
+  background-color: #232323;


please move this color to the theme

bahaaTuffaha · 2024-12-13T12:18:43Z

issue 1:

Open this page: http://localhost:9000/augsburg/de/willkommen/willkommen-in-augsburg-2

Click "read aloud"

Click "play"

Click on "Weiter"
Result: The reader starts reading out the beginning of the current sentance, when clicking "weiter" for the first time. When clicking "weiter" a second time, it correctly jumps to the next sentance
Expected result: When clicking "weiter" for the first time, it should directly jump to the next sentance.

issue 2: When opening this page: http://localhost:9000/augsburg/de/willkommen the "read aloud" button is displayed, but does nothing when clicked. Expected: Should either not be displayed or does something when clicked

issue 3: Open this page: http://localhost:9000/augsburg/de/behoerden-beratung/behoerden/auslaenderbehoerde Click "next" to jump to the last sentance in this line "Weitere Informationen finden Sie unter [hier]" the reader switches to the "start"-mode, but there is still this whole contact blog that is read

Can you please mention which browser you used for each issue?

ztefanie · 2024-12-16T08:17:10Z

Can you please mention which browser you used for each issue?

I mostly used Chrome, sometimes I double checked with Firefox

bahaaTuffaha and others added 12 commits October 7, 2024 21:10

TTS for web part1 testing SpeechSynthesis

9530674

2918: More improvements to tts

9eb0b45

2918: more enhancement to speechSynthesis

53a59cc

2918: Refactoring TtsPlayer

a0bc6a1

2918: used easy-speech cross platform support

d838533

more improvements to ttsWeb

a3a4dbf

2918: Adding TtsContextProvider and more improvements

c0741da

2918 :Adding Mocks and test also adjusting the TtsHelpModal

8592cec

2918: Fixing TtsPlayer test file and more improvements

8acb476

2918: Fixing volume and changes for TtsHelpModal

5ec63a2

2918: Trying to solve onEnd issue

e4fecf3

Merge branch 'main' into 2918-Add-TTS-on-web

ac7f3d8

steffenkleinle changed the title ~~2918 add tts on web~~ 2918: add tts on web Nov 11, 2024

steffenkleinle changed the title ~~2918: add tts on web~~ 2918: Add tts on web Nov 11, 2024

bahaaTuffaha and others added 10 commits November 11, 2024 19:06

Merge remote-tracking branch 'origin' into 2918-Add-TTS-on-web

5d8a435

2918: Fixing a bug where I cant redirect to the categories and remove…

a36dee7

…d unnecessary mock

2918: Small Refactoring for codeSense

af707de

2918: Adding tts to FeatureFlagsType

8805341

Fixing the codeSense error for CategoriesPage at navigateToCategories

2ee6fdb

2918: Splitting Tts for web into TtsPlayer and TtsContainer plus remo…

b720631

…ving the volume.

Merge remote-tracking branch 'origin' into 2918-Add-TTS-on-web

366397f

2918: changes similar to tts-Android replacing react Dispatch and tit…

b24583a

…le from ttsContext

2918: Modifying to be similar to TtsOnAndroid

1ff36c5

Merge branch 'main' into 2918-Add-TTS-on-web

2631024

ztefanie reviewed Dec 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2918: Add tts on web #2985

2918: Add tts on web #2985

bahaaTuffaha commented Nov 6, 2024 •

edited

Loading

ztefanie left a comment

ztefanie Dec 13, 2024

ztefanie Dec 13, 2024

ztefanie Dec 13, 2024

ztefanie Dec 13, 2024

ztefanie Dec 13, 2024

ztefanie Dec 13, 2024

ztefanie Dec 13, 2024

ztefanie Dec 13, 2024

ztefanie Dec 13, 2024

ztefanie Dec 13, 2024

bahaaTuffaha commented Dec 13, 2024 •

edited

Loading

ztefanie commented Dec 16, 2024

2918: Add tts on web #2985

Are you sure you want to change the base?

2918: Add tts on web #2985

Conversation

bahaaTuffaha commented Nov 6, 2024 • edited Loading

Short description

Proposed changes

Side effects

Testing

Observations

Resolved issues

ztefanie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bahaaTuffaha commented Dec 13, 2024 • edited Loading

ztefanie commented Dec 16, 2024

bahaaTuffaha commented Nov 6, 2024 •

edited

Loading

bahaaTuffaha commented Dec 13, 2024 •

edited

Loading