Lots of enhancements (#26)

* Migrated to jinja2, use pipfile and supervisord

* Add kubernetes manifests, fix nginx logging, streamline the dockerfile

* Use the upstream image name, fix typos

* Switch to single quotes

* Fix typos in template, update readme

README.md

@@ -22,31 +22,47 @@ docker run -it -p 80:80 -p 1935:1935 \
 
 If this is a host than where you built the docker image, you will need to push the docker image to that host (or build it there). Alternatively, you could use the DockerHub build of this image by pulling and using the `kamprath/multistreaming-server:latest` [Docker image](https://hub.docker.com/repository/docker/kamprath/multistreaming-server). Also, if you plan on doing any transcoding when rebroadcasting a stream, you need to ensure that your docker host's CPU is sufficient for the job. It is recommend that the host CPU has at least four cores for each distinct transcoding the multi-streaming server will do. If the host CPU is not sufficient, bit rates on the transcoded streams will suffer.
 
-Note that an environment variable is set when running the Docker image:
+Note that some environment variables should be set when running the Docker image:
 
 * `MULTISTREAMING_PASSWORD` _(REQUIRED)_ - This is a password you define and will be used by your steaming software. This is a marginally secure way to prevent other people from pushing to your stream.
+* `CONFIG_DISABLE_RECORD` _(OPTIONAL)_ - If this env var is set to a non-empty value, the stream recording feature will be disabled. 
+* `CONFIG_NGINX_DEBUG` _(OPTINAL)_ - If this env var is set to a non-empty value, it will enable the nginx debug logging to stderr. This is useful for debugging.
+* `CONFIG_FFMPEG_LOG` _(OPTIONAL)_ - If this env var is set to a non-empty value, each ffmpeg process will start logging their output to `/tmp/ffmpeg-$APP.log` in the container. This is useful for debugging issues related to encoding.
+* `CONFIG_FFMPEG_MAX_MUXING_QUEUE_SIZE` _(OPTIONAL)_ - This env var must be a numerical value, it will be used as the global default for `-max_muxing_queue_size` argument for the ffmpeg processes. You may need to set this to a higher value (i.e: `4096`) if you run in to this [bug.](https://trac.ffmpeg.org/ticket/6375)
 
 You must also create and JSON file with the RTMP rebroadcasting configuration you desire. This file should get mapped from your local file system to the `/rtmp-configuration.json` file path within the Docker container. The JSON file has the following elements:
 
 * `endpoint`- This is the name of the RTMP ingest endpoint that the source stream will be pushed to. Defaults to `live` if not specified.
+* `transcodeProfiles` - This is a JSON object with keys indicating a unique identifier for a transcode profile, and the values specifying the supported transcode options.
+    * `pixels` - The pixel dimension that the stream should be transcoded to. Formatted like "1920x1080". If not specified, defaults to "1280x720".
+    * `videoBitRate` - The video bit rate that should be used when sending the stream to this destination. Should be a number followed by "k" or "m" for kilo- and mega- bits-per-second. If not specified defaults to "4500k".
+    * `videoKeyFrameSecs` - The number of seconds between key frames in the transcoded stream. If not specified, defaults to 2.
+    * `videoFrameRate` - The frames per second which the video stream will be re-encoded with, defaults to "30".
+    * `audioBitRate` - The bit rate that should be used for the transcoded audio signal. Should be a number followed by "k" or "m" for kilo- and mega- bits-per-second. If not specified, defaults to "160k". If neither `audioBitRate` or `audioSampleRate` are specified, then the audio signal is simply copied from source with no alteration.
+    * `audioSampleRate` - The sampling rate to be used for the transcoded audio signal. Should be an integer indicating the sampling Hertz. If not specified, defaults to `48000`. If neither `audioBitRate` or `audioSampleRate` are specified, then the audio signal is simply copied from source with no alteration.
+    * `maxMuxingQueueSize` - Individual override for the `CONFIG_FFMPEG_MAX_MUXING_QUEUE_SIZE` env var. See the env var section for the usage.
 * `rebroadcastList`- _Required_ Contains a list of JSON objects that each configure a distinct RTMP destination that the stream pushed to the ingest endpoint will be rebroadcasted to. At least one destination should be configured. There is no specific limit on the number of destinations except for the hardware limitations of your host. Each destination is configured with the following JSON elements:
   * `name` - _Required_ A distinct label for this destination. Can be any alphanumeric string with no white space. Must be distinct from all the other destination names in this list.
   * `platform` - _Required_ The platform that this specific rebroadcast stream should be pushed to. The default RTMP destinations will be used for each platform. Supported platforms values are: `youtube`, `facebook`, `twitch`, `instagram`, `periscope`, `microsoft-stream`, `mixcloud`, `dlive` and `custom`. Note that specifying `microsoft-stream` or `custom` will cause the `streamKey` element to be ignored if present and instead use the `fullRTMPURL`
   * `regionCode` - If `periscope` is specified as the platform for this destination, this is the two letter region code that is part of the Periscope server URL. If undefined, it will default to `ca` (the "US West" region)
   * `streamKey` - This is the stream key that identifies the unique stream on the specified platform. This value is provided by the platform. This element must be provided for all `platform` types except for `custom` and `microsoft-stream`.
   * `fullRTMPURL` - If `custom` or `microsoft-stream` is specified in the `platform`, the URL specified in this element is used for the forming destination URL. This should include the `rtmp://` prefix. For the `microsoft-stream` platform, this is the full URL provided in their stream set up. This element is ignored for all other platform types.
-  * `transcode` - If present, the stream will be transcoded before rebroadcasting it to this list item's destination. Note that when using this transcoding, the stream will be trancoded to 30 FPS and CBR bit rate. The value is a JSON object that contains the following configuration elements:
-    * `pixels` - The pixel dimension that the stream should be transcoded to. Formatted like "1920x1080". If not specified, defaults to "1280x720".
-    * `videoBitRate` - The video bit rate that should be used when sending the stream to this destination. Should be a number followed by "k" or "m" for kilo- and mega- bits-per-second. If not specified defaults to "4500k".
-    * `videoKeyFrameSecs` - The number of seconds between key frames in the transcoded stream. If not specified, defaults to 2.
-    * `audioBitRate` - The bit rate that should be used for the transcoded audio signal. Should be a number followed by "k" or "m" for kilo- and mega- bits-per-second. If not specified, defaults to "160k". If neither `audioBitRate` or `audioSampleRate` are specified, then the audio signal is simply copied from source with no alteration.
-    * `audioSampleRate` - The sampling rate to be used for the transcoded audio signal. Should be an integer indicating the sampling Hertz. If not specified, defaults to `48000`. If neither `audioBitRate` or `audioSampleRate` are specified, then the audio signal is simply copied from source with no alteration.
+  * `transcode` - If present, the stream will be transcoded before rebroadcasting it to this list item's destination. Note that when using this transcoding, the stream will be trancoded to 30 FPS and CBR bit rate. The value is a JSON object with the support for same optoins as an entry in `transcodeProfiles`. Prefer to use `transcodeProfile` key if you have multiple outputs using the same encoding configuration.
+  * `transcodeProfile` - If present, this stream will use the transcode settings provided by the `transcodeProfile` referenced by the string identifier set here. If multiple streams refer to the same `transcodeProfile`, the transcoding will only be done once. This option is mutually exclusive with the `transcode` key.
+
 
 Here is an example of the JSON configuration file:
 ```
 {
   "endpoint": "live",
-  "rebroacastList": [
+  "transcodeProfiles": {
+    "720_60fps": {
+      "pixels": "1280x720",
+      "videoBitRate": "4500k",
+      "videoFrameRate": 60
+    }
+  },
+  "rebroadcastList": [
     {
       "name": "youtube",
       "platform": "youtube",
@@ -56,19 +72,13 @@ Here is an example of the JSON configuration file:
       "name": "facebook-1",
       "platform": "facebook",
       "streamKey": "def456",
-      "transcode": {
-        "pixels": "1280x720",
-        "videoBitRate": "4500k"
-      }
+      "transcodeProfile": "720_60fps"
     },
     {
       "name": "facebook-2",
       "platform": "facebook",
       "streamKey": "ghi789",
-      "transcode": {
-        "pixels": "1280x720",
-        "videoBitRate": "4500k"
-      }
+      "transcodeProfile": "720_60fps"
     },
     {
       "name": "periscope",
@@ -88,7 +98,7 @@ Here is an example of the JSON configuration file:
 ```
 Note that as long as the `name` elements are different, you can have more than one destination pushing to the same `platform`, each with different `streamKey` values.
 
-If you would like to capture a recording of the stream sent to the ingest endpoint, bind a local directory on your host to the `/var/www/html/recordings/` file path within the Docker image when launching the Docker container. You can also see statistics about the various streams this server is pushing by visiting this web page: `http://__docker_host_IP_address__/stat`.
+If you would like to capture a recording of the stream sent to the ingest endpoint, bind a local directory on your host to the `/var/www/html/recordings/` file path within the Docker image when launching the Docker container. You can also see statistics about the various streams this server is pushing by visiting this web page: `http://__docker_host_IP_address__/stat`. You can optionally disable recording by setting the env var `CONFIG_DISABLE_RECORD` to a non-empty value.
 
 Once the Docker image is running, set up your stream software with the following parameters:
 
@@ -99,6 +109,15 @@ In OBS, you would set the above parameters for a "Custom..." Service in the Stre
 
 The next thing to do is start your stream!
 
+## Development
+
+In order to develop on this repository, you must have a python 3.7 environment and [`Pipenv`](https://pypi.org/project/pipenv/) installed.
+
+- Clone the repository
+- Navigate to the `multistreaming-server/` directory
+- Install dependencies and create the virtual env by running `pipenv install`
+- Activate the virtual env by `pipenv shell`
+
 ## Future work
 
 Goals for future improvements to this project include:

deployment-scripts/README.md

@@ -71,3 +71,20 @@ When running this script, it takes the following arguments:
 * `-h` - This will display more detailed information on how to use the script, including environment variables that are supported.
 
 If the script successfully completes, you will be running a Docker container locally with the Multi-Service RTMP Broadcaster software running. The script also prints some useful information and commands to use at the end of its run. The IP address you should use to configure your streaming software's destination is `127.0.0.1` or `localhost`.
+
+## Kubernetes
+
+You will find the required manifests to deploy the container to a Kubernetes cluster under the `./kubernetes-kustomize` directory. 
+
+* Exposed RTMP/stats port: The example manifest in this repo will create a Service of type `NodePort` which will expose the RTMP endpoint on port 33195, and the stats endpoint on port 30080 on all the nodes. If you want to change this behavior, update `service.yaml` with the `nodePort` values you want.
+* Stream password: Stream password will be created a Kubernetes Secret in the cluster, to set the password open the file `kustomization.yaml` and find the entry with the name `multistreaming` in the list under the key `secretGenerator`. 
+* RTMP Configuration JSON: The configuration file must be placed in the same path as the `kustomization.yaml` and should be named `rtmp-configuration.json`. The deployment will automatically read this file and create it in the cluster as a configmap.
+* Kubernetes namespace: By default the manifests will be deployed to a namespace called `multistreaming`. You may change this in `kustomization.yaml`. This namespace should be created manually before it can be used for deployments.
+
+Deployment command:
+```
+# Make sure you have your KUBECONFIG set.
+$ cd kubernetes-kustomize/ # or the path where you are storing the manifests
+$ kubectl create namespace multistreaming # the target namespace, should only be run once 
+$ kubectl apply -k . # can be run many times to refresh the configuration
+```

deployment-scripts/kubernetes-kustomize/deployment.yaml

@@ -0,0 +1,45 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: multistreaming
+  labels:
+    app: multistreaming
+spec:
+  selector:
+    matchLabels:
+      app: multistreaming
+  template:
+    metadata:
+      labels:
+        app: multistreaming
+    spec:
+      containers:
+      - name: multistreaming
+        image: kamprath/multistreaming-server:latest
+        imagePullPolicy: Always
+        volumeMounts:
+        - name: config-volume
+          mountPath: /rtmp-configuration.json
+          subPath: rtmp-configuration.json
+        env:
+        - name: MULTISTREAMING_PASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: multistreaming
+              key: password
+        - name: CONFIG_NGINX_DEBUG
+          value: "true"
+        - name: CONFIG_FFMPEG_LOG
+          value: "true" 
+        - name: CONFIG_FFMPEG_MAX_MUXING_QUEUE_SIZE
+          value: "4096"
+        - name: CONFIG_DISABLE_RECORD
+          value: "true"
+        ports:
+        - containerPort: 80
+        - containerPort: 1935
+      volumes:
+      - name: config-volume
+        configMap:
+          name: multistreaming-config
+

deployment-scripts/kubernetes-kustomize/ingress.yaml

@@ -0,0 +1,13 @@
+apiVersion: networking.k8s.io/v1beta1
+kind: Ingress
+metadata:
+  name: multistreaming
+spec:
+  rules:
+  - host: multistreaming.ls90.co
+    http:
+      paths:
+      - path: /
+        backend:
+          serviceName: multistreaming
+          servicePort: 80

deployment-scripts/kubernetes-kustomize/kustomization.yaml

@@ -0,0 +1,19 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+nameSuffix: -v1
+namespace: multistreaming
+commonLabels:
+  app: multistreaming
+  version: v1
+resources:
+- deployment.yaml
+- service.yaml
+# - ingress.yaml # Optional, must configure the hostname if you need to expose the stat page as an ingress
+configMapGenerator:
+- name: multistreaming-config
+  files:
+  - rtmp-configuration.json 
+secretGenerator:
+- name: multistreaming
+  literals:
+  - password=sekritPassword

deployment-scripts/kubernetes-kustomize/rtmp-configuration.json

@@ -0,0 +1,23 @@
+{
+  "endpoint": "live",
+  "transcodeProfiles": {
+    "720_60fps": {
+      "pixels": "1280x720",
+      "videoBitRate": "4500k",
+      "videoFrameRate": 60
+    }
+  },
+  "rebroadcastList": [
+	  {
+      "name": "twitch",
+      "platform": "twitch",
+      "streamKey": "foobar"
+    },
+    {
+      "name": "fbpage",
+      "platform": "facebook",
+      "streamKey": "secretKey",
+      "transcodeProfile": "720_60fps"
+    }
+  ]
+}

deployment-scripts/kubernetes-kustomize/service.yaml

@@ -0,0 +1,19 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: multistreaming
+spec:
+  selector:
+    app: multistreaming
+  type: NodePort
+  ports:
+    - name: web
+      protocol: TCP
+      port: 80
+      targetPort: 80
+      nodePort: 30080
+    - name: rtmp
+      protocol: TCP
+      port: 1935
+      targetPort: 1935
+      nodePort: 31935

multistreaming-server/Dockerfile

@@ -4,6 +4,9 @@ MAINTAINER Michael Kamprath "https://github.com/michaelkamprath"
 ARG NGINX_VERSION=1.19.2
 ARG RTMP_REPO=uizaio
 ARG RTMP_MODULE_VERSION=1.4.0.4
+ARG TINI_VERSION=v0.19.0
+ARG SUPERVISORD_VERSION=4.2.1
+ARG PIPENV_PACKAGE_VERSION=2020.8.13
 
 RUN set -x \
  && addgroup -S stunnel \
@@ -22,34 +25,33 @@ RUN set -x \
  && ./configure --with-http_ssl_module --add-module=../nginx-rtmp-module-${RTMP_MODULE_VERSION} \
  && make \
  && make install \
+ && cp /nginx-rtmp-module-${RTMP_MODULE_VERSION}/stat.xsl /usr/local/nginx/html/ \
  && apk del build-deps \
  && mkdir -p /var/www/html/recordings \
  && mkdir -p /var/run/stunnel/ \
  && chown nobody:nobody -R /var/www/html \
- && chown stunnel:stunnel /var/run/stunnel/
+ && chown stunnel:stunnel /var/run/stunnel/ \
+ && wget -O /tini https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-static \
+ && chmod +x /tini \
+ && pip3 install supervisor==${SUPERVISORD_VERSION} pipenv==${PIPENV_PACKAGE_VERSION}
 
-COPY index.html /usr/local/nginx/html/
-COPY nginx-conf/nginx.conf /base-nginx.conf
-COPY launch-nginx-server.sh launch-nginx-server.sh
-COPY rtmp-conf-generator.py rtmp-conf-generator.py
+COPY Pipfile Pipfile.lock /
+RUN  pipenv install --system --deploy
 
+COPY supervisord.conf /etc/supervisor/supervisord.conf
 COPY stunnel-conf/etc-default-stunnel /etc/default/stunnel
 COPY stunnel-conf/etc-stunnel-conf.d-fb.conf /etc/stunnel/conf.d/fb.conf
 COPY stunnel-conf/etc-stunnel-conf.d-ig.conf /etc/stunnel/conf.d/ig.conf
 COPY stunnel-conf/etc-stunnel-stunnel.conf /etc/stunnel/stunnel.conf
 
-# forward request and error logs to docker log collector
-RUN ln -sf /dev/stdout /usr/local/nginx/logs/access.log
-RUN ln -sf /dev/stderr /usr/local/nginx/logs/error.log
-
-RUN cp /nginx-rtmp-module-${RTMP_MODULE_VERSION}/stat.xsl /usr/local/nginx/html/
+COPY index.html /usr/local/nginx/html/
+COPY nginx-conf/nginx.conf /base-nginx.conf
+COPY launch-nginx-server.sh launch-nginx-server.sh
+COPY rtmp-conf-generator.py nginx-template.conf.j2 /
 
 EXPOSE 1935
 EXPOSE 80
 
 STOPSIGNAL SIGTERM
-
-# Remove the entry point from jrottenberg/ffmpeg
-ENTRYPOINT []
-
-CMD ["/bin/sh", "/launch-nginx-server.sh"]
+ENTRYPOINT ["/tini", "--"]
+CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/supervisord.conf"]

multistreaming-server/Pipfile

@@ -0,0 +1,17 @@
+[[source]]
+name = "pypi"
+url = "https://pypi.org/simple"
+verify_ssl = true
+
+[dev-packages]
+pylint = "*"
+black = "*"
+
+[packages]
+jinja2 = "*"
+
+[requires]
+python_version = "3.7"
+
+[pipenv]
+allow_prereleases = true