Remote sign language interpretation in live video productions
This article is about adding sign interpretation to a livecast produced at a remote location.
The result may look like this video clip from the Swedish Riksdag (parliament) building, recorded live in September 2011: The media player is loading...
Contents |
Introduction
This article provides a guideline on how to build your own studio that supports live sign interpretation. The studio is designed so that it will facilitate live signing of videos produced at remote locations. While the focus here is on superimposing sign interpreters with the help of chromakey technology, the same studio technology is useful in other situations. These include all kinds of video-over commenting on political speeches, sports events, business conferences, product launches, etc.
Quick start
You may do this offline with a pre-recorded video, or online by grabbing a live video stream from a remote location. The result can then be delivered in real time over the internet as a separate live video stream.
For this kind of production all you need is a decent Windows PC, at least VidBlaster Professional, a camera, something that works as a green screen, a few lamps, a small sound mixer, and of course an internet connection if you want to do it online.
Here is a short video clip from the studio used to develop the solution decribed in this guideline: The media player is loading...
As you can see there is no need to spend thousands of Euros, unless you have high quality ambitions and believe that HD is the way to go. In other words, the total cost can be really low or rather high.
From idea to practice
The reader is assumed to initially have at least some vague understanding of the basic technologies discussed. These include MS-Windows, video production, chromakey, live streaming, and audio mixing. Necessary in depth knowledge may easily be obtained later by searching the internet and participating in relevant discussion forums, such as the VidBlaster forum.
Designing the VidBlaster profile
Screen dump 1 on the right illustrates how it looks like on the PC screen when the production setup described below is used. The left part of the screen, discussed in more detail below, shows the VidBlaster application window. The right part is a browser window displaying the video being sign interpreted.
Five steps from original to signed
In theory there are five steps that take you from any original video to a nice sign interpreted live video. The steps are outlined here, together with an example that illustrates how it might look like in a real VidBlaster profile. In practice you may need fewer or more steps, depending on your previous experiences and current ambition.
A: Decide on how to get the original video
We begin by looking at three different types of sources for the original video. The first one is video from a cable attached to a local camera. The second alternative is any video played on the productions PC. This type include videos displayed in a web browser, such as those from YouTube or TED/talks. Finally, our third type is the so called RTSP/RTMP stream broadcasted from a live streaming service.
VidBlaster offers modules that support all the sources mentioned above. In our example “Camera 1” is prepared for a local camera, but not used here. Instead “Camera 2” is selected. That module is using its screen capture device to grab the video played in the PC:s browser window. In this case the video is played on demand. Of course you may easily capture a live video instead.
If you are able to find out the URL to the source live stream, you could potentially take the live feed directly as it is coming from the streaming server. For this you need a stream supported by the "RTSP/RTMP In" module (available only in the high end Broadcast edition of VidBlaster). In the illustration “RTSP/RTMP Input 1” is prepared for such streams, but not used.
The video below, from a live presentation of this platform as well as capblaster (for live subtitling), was actually produced live using the RTSP/RTMP-input method. English subtitles for this video are available if you watch the clip How to add subtitles and sign language to webcasts on YouTube (i.e. not embedded here in the wiki).
The media player is loading...
B: Make room for the sign interpreter
To prevent the sign interpreter obscuring too much of the image in the original video, we need to create some empty space. How much, and where, is a balance between the need of the viewers and your artistic ambitions.
In the example we have shrunk the original video by about 20 percent, and repositioned it as a virtual screen on a neutral blue background. “Video Effect 1” is the module that takes the background in “Player 1” (jpg image of black box on a blue background) and positions “Camera 2” (our original screen grabbed video) as a PiP aligned in the black rectangle which is part of the background image.
C: Position the sign interpreter in the screen
The chromakey effect works by combining two full screen images. So, either you have to put the sign interpreter in front of a huge green screen, properly scaled and positioned as they will appear in the final composition. Or, as in our example, you can use another video effect to obtain the same result.
Hence, “Video Effect 2” is used here to scale and position the sign interpreter on the video screen. In order for the chromakey effect to work, everything surrounding the sign interpreter must be green. The background image from “Player 2” takes care of this. To ensure an exact match of the green color, the image was created by capturing a still image from the sign interpreter’s camera. As you might notice we are also using the PiP effect’s crop option to remove unwanted edges from the “Camera 3” source
D: Superimpose the sign interpreter with chromakey
The next step is pretty straightforward. “Video Effect 3” is set to use Chroma Key and combine the output from “Video Effect 1” (shrunken original) with the output from “Video Effect 2” (positioned interpreter on green background). Hopefully the module’s “auto-key” option will be sufficient to give you a nice result with few artifacts. If not, you might be forced to adjust the chromakey settings manually. Or more likely, adjust the lighting around the sign interpreter and perhaps try a different green image in “Player 2”.
Now is the time to make some final adjustments regarding the size and position of the sign interpreter. This might involve the camera’s pan, tilt, and zoom, as well as the sign interpreter’s own feet. And of course, trimming the PiP window in the “Video Effect 2” module.
E: Stream and record
As of writing this instruction, VidBlaster is only capable of streaming and recording the final composited shot. While this is really easily done, and typically the main point, you might also want to stream/record the sign interpreter separately.
The camera’s built in recorder is of course the most obvious way to get a separate recording of the sign interpreter. Alternatively you can split the video signal from the camera, and feed the video into any suitable type of recording equipment.
Setting up the studio
- a production PC
- a video camera
- a properly lit green screen
- an audio mixer
- loudspeakers
- at least one PC monitor
- an internet connection
Configure a production PC
VidBlaster is designed for Windows, running on everything from XP to 7, including the 64 bit versions. As a general indication of the hardware requirements VidBlaster’s help section includes the following recommendations. For 640*480: CPU Core 2 Quad/Intel i7, 4 GB RAM, accelerated video card, Windows XP+. For HD: CPU Intel Core i7 Extreme, 6 GB RAM, accelerated video card, Windows 7.
A suitable configuration is always dependent on what you want to do, and the video quality required. The example discussed in this article is based on successful productions of live streamed and recorded productions with a resolution of 864x480 pixels at 25 fps. The PC, which is overpowered to support future expansions, includes the following hardware and software:
- CPU: Intel i7 2600K
- RAM: Corsair 8 Gb, DDR3 1600 MHz Vengeance
- MOBO: Gigabyte GA-P67AUD4
- GPU: Gigabyte GTX560
- HD: Corsair SSD 120 Gb Force series
- OS: Windows 7/64bit
- Video production software: VidBlaster Broadcast, versions 1.36 and 2.08
While some may prefer to build their own PC to optimize and save money, other prefer off the shelf systems from well known brand names. Either way, both types of system work, as will probably a powerful laptop, as long as the video resolution is not too high (perhaps 640x360 @ 25 fps).
Camera, lights and sound
The video camera can be anything from a decent web cam to a professional HD camera. We have successfully used the Logitech Quickcam Pro 9000 web camera. Regular consumer and professional cameras will need some kind of interface to the production PC.
You may use a special green screen, or simply paint a wall in a green color. Most likely you will need more than one lamp in order to offer evenly distributed light with minimal shadows. For help on how to succeed with chroma key in video production, google around, read the Working with Chroma Key article here on the Wiki, or check out the “Chromakey green screen lighting tutorial” video at www.smick.co.uk. With some luck, as in our example case, good office light from the ceiling just needed extra support from a desk top lamp. The media player is loading...
Of course the signers must be able to listen to the sound from the video they are interpreting. Unfortunately it is not possible to just attach speakers to the production PC when it plays the original video stream (unless you are able to enable the Stereo Mix input in Windows 7). Normally you need to feed the audio back into the PC in order for it to be included in the video you are producing. The most convenient way to do this is with the help from an analog audio mixer, for example the Behringer Xenyx 502.
The PC audio out should be connected to one of the mixer’s input channels. Main out from the mixer goes to the PC audio in. Finally, connect the mixer’s headphone/monitor output to the loudspeaker’s input. This way the signers can conveniently control the sound level of what they hear, as well as the level fed back to the signed video. In case there is a need to record sound from the studio, such as oral comments from the signers, just connect a microphone to the mixer.
At least one big PC monitor is the last piece in the studio jigsaw puzzle. In terms of resolution the screen must be able to simultaneously display both the VidBlaster application with all its modules, as well as provide room for a video player set to the desired source resolution. Sign interpreters typically work in pairs. Therefore, a second monitor comes in handy, allowing both interpreters to get a good look of what’s happening on the screen. This monitor can either replicate the main monitor, or just the “Program” out from VidBlaster (by using a Video Out module).
Of course an internet connection is required too. You will need around 2 Mb/s downstream and 600 Kb/s upstream in order to assure a decent and smooth video. The exact bandwith requirement has to be determined by your quality needs, combined with some testing.
Physical layout and training
When the studio is built, and of course tested, it is almost ready to be used in live situations. Exactly how you go ahead will of course be determined by your particular studio design, as well as production requirements.
Here are some general hints.
Beware of mirroring
When you look in a mirror, left becomes right and the other way around. This effect needs to be considered when you work in the studio. If, as in our example, the shrunken original video is positioned left on the resulting video image, then the sign interpreter should have her/his monitor to the right. This way the sign interpreter will “look” and “point” in the “right” direction at the viewer’s end.
In a similar fashion walking in and out of the camera is mirrored. Depending on the direction the interpreter takes, it will look like she or he is walking over the video screen or out to its side.
The best way to sort this confusion out is by experimenting, and training.
Train your staff
Besides the production manager, who should be able to set up everything and is confident on how to use VidBlaster, you need sign interpreters that understand how the studio works. It is really easy to simulate a live situation, and thus allow the interpreters to test and try.
Rehearse with the source
In the case you are live interpreting an event, try to rehearse with the producer at the original location. If possible, do this live in order to make sure everything runs smoothly and that everyone knows what to do in different situations.
Additional information
VidBlaster versions
This guideline was originally written for, and tested with, VidBlaster versions 1.36 and 2.08.
Limitiations
Live streaming implies slight delays. Due to the way live video is transmitted over the internet, there will always be a slight delay, typically between 2 and 30 seconds from source to viewer. This means that in practice it is impossible to show the source at the same time as the sign interpreted version. Thus, for example, killing the idea of showing the interpreted video at the original location.
More or less one way. Another way the delay limits the functionality is reduced interaction. If the interpreter would like to ask the source for some clarification, this question might come too late. Likewise, if the source wants to direct the interpreters, the delay has to be considered.
Credits and copyright
The first version of this guideline was written and published by Richard Gatarski. The studio solution was invented and developed by Richard together with staff at the Interpreter Centre with Örebro County Council, Sweden. Funding for the hardware, software and additional studio equipment was provided by The Swedish Post and Telecom Authority (PTS).
The original guideline by Westreamu AB (westreamu.se) is licensed under a Creative Commons Attribution 3.0 Unported License.
External links
<a rel="license" href="http://creativecommons.org/licenses/by/3.0/"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by/3.0/88x31.png" /></a>
This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/3.0/">Creative Commons Attribution 3.0 Unported License</a>.
