Implementing custom captions renderer for Android VideoView


From Android KitKat (API level 16), programmers can use ‘addSubtitleSource()’ on VideoView to add a WebVTT format subtitle track. That’s it. Nothing else to do. The native subtitle handler even automatically listens and changes drawing styles according to system accessibility caption settings. (FYI – System captions settings are under Settings->Accessibility->Captions) Awesome!

But only that it isn’t. Most of the application workflows have a captions button for the user to turn captions on/off. Unfortunately, there are no public member functions on VideoView control to allow the developer, in proxy of app user, to control captions visibility from within the app. All the developer can do is redirect the user to system accessibility page using ‘Intent’. Even then, the captions settings are global. Sure, if the user has permanent hearing disability, this is no problem. He/she would want to turn on caption for all the apps with one switch. Unfortunately, this is not always the case. Even a normal user may want to mute sound and watch the video with captions in a quiet environment. Then, the app will need a button within it for the user to toggle captions. Another, though minor, quirk is that ‘addSubtitleSource()’ works with WebVTT format. The ubiquitous captions format is SRT. This will require you to wire an intermediate converter (A simple SRT to WebVTT converter that I wrote for testing can be downloaded here). Hence, there is atleast two strong cases to implement a custom captions handler.

Android Custom Captions Renderer

1. Get an instance of internal MediaPlayer

The VideoView control has a private member object instance of MediaPlayer control and implements ‘addSubtitleSource()’ on top of it. So, the first plan of attack is getting an instance of this MediaPlayer object instance. One way of doing this is using Java’s reflection capabilities to call hidden member functions in VideoView class – i.e. ‘VideoView.class.getDeclaredMethod(“<Method name>”).invoke(<Params>)’. This, of course, is very ‘hacky’ and we want to stay away from this as much as we can in production code. Fortunately, there’s another way. If we create a new derived class from ‘VideoView’, we can override the ‘onPrepared()’ event and get an instance of its internal MediaPlayer object in the event listener’s function parameter. When this event fires, the video player has loaded the video and is ready to accept captions track. Using MediaPlayer’s ‘addTimedTextSource()’, we can add an external captions track. This can be SRT format.

2. Handle special condition where caption file is on the network

If your captions file is on the network though, we have another hurdle to jump. The function accepts only local file path or a local file descriptor so we have to download the captions to a local file in a temp directory and then add code to handle file clean up when it is no longer needed. If your code fails to properly cleanup, you will be cluttering user’s precious storage space. Alternatively, we could go another path by using ‘MemoryFile’. MemoryFile is a file mapped into memory used for inter-process communication. Nice thing about this is that the system takes care of the file maintenance part while we just read/write to memory. Using Java’s reflection capabilities we can call hidden member ‘getFileDescriptor()’ on MemoryFile to get a ‘FileDescriptor’ to use with ‘addTimedTextSource()’. I would argue that though ‘getFileDescriptor()’ is hidden, it is safe to call it. It ties directly with underlying Linux API and there isn’t any reason for Android  framework to change or remove this function in the future.

3. Rendering caption on specific time

When you set a listener using ‘MediaPlayer.setOnTimedTextListener()’, we have only asked MediaPlayer to notify us using ‘onTimedText’ event when caption text start/end timing is reached on playback. We still have to put a TextView on top of VideoView to show the captions. This can be done easily in the video activity. When the listener handler is invoked, it will have a ‘TimedText’ object on its function parameter. The ‘getText()’ method of this parameter will return the caption text to be rendered at this playback time. If the end time for a caption is reached, this will return a null string. Then, we just hide the TextView captions text control. There is also a ‘getBounds()’ method in the ‘TimedText’ object. Optionally, this may be a non-null ‘Rect’ object. This is a valid Rect object when the input captions file contains positioning information. A rigorous implementation will use this bounds by moving TextView accordingly. One case where this may be used is when an object/person important to the video’s subject matter is on the bottom part of the video which is usually occluded by the captions text.

Pitfalls

The native SRT parser is confused by subtitle information blocks where there is sequence and timing information but the captions text is empty. For instance, as in caption sequence 2 below:

1
00:00:00,000 --> 00:00:00,100
Caption text 1

2
00:00:01,200 --> 00:00:02,000


3
00:00:04,000 --> 00:00:06,000
Caption text 3

One way to fix this is by replacing the ‘\r\n\r\n\r\n’ (assuming Windows style line break) after the end time in sequence 2 with ‘\r\n{Any non-white space character here}\n\r\n’.

Advertisements

5 thoughts on “Implementing custom captions renderer for Android VideoView

    1. The project I was working on is a commercial project so I cannot distribute it and neither do I have a standalone sample project. But I have put relevant part of the code in PasteBin here – http://pastebin.com/NMjP4ZQc. Here are some points you need to know (besides that has already been mentioned in the article):
      1. Create a new subclass of VideoView. Override its ‘OnPrepared()’ event, copy MediaPlayer object from function parameter to class member variable called ‘mMediaPlayer’ and call ‘loadCaptionsIfRequired()’ in the provided code.
      2. Have a TextView in your video player page over the VideoView control to write subtitle text. Have a image control called ‘closedcaptionsButton’, to function as button to toggle captions on/off.
      3. Override ‘OnTimedText()’ event using ‘mMediaPlayer.setOnTimedTextListener()’ and set text of the TextView to show the subtitle at a given time.
      4. The code contains support for fetching captions in compressed GZip format and converting from Windows-1252 to UTF-8. Might want to change this according to your needs.

      I am sure you can find your way from the provided snippet.

      1. Thank you. I have videos and WebVTT files that reside on the device, so my implementation will be much easier.

  1. I got everything working with one exception: onTimeText is not called when the subtitle text should disappear. Can MediaPlayer be configured to call onTimeText when the subtitle text should be displayed AND should disappear? Or is there a way to get the current subtitle’s duration so that I can hide subtitle when appropriate?

    1. The Android document promises that a call to this function with TimedText.GetText() as null is made (ref: https://developer.android.com/reference/android/media/TimedText.html#getText()) when you have to hide the captions TextBlock but this unfortunately doesn’t seem to happen. Nonetheless, it would be wise to assume this behavior can occur (perhaps in future Android?) and code to prepare TimedText.GetText() returning null.

      As to how to fix this problem now, the simplest way would be to reset a timer every time your onTimedText event gets called. When the timer expires, it should hide captions TextBlock and cancel itself. Given that captions are usually very close to each other, this solution produces acceptable results to hide captions properly which have large amount of silence after them.

      Another way is to parse, the subtitle file yourselves (I am not sure whether the start stop time dictionary is available when the native Android subtitle file parser finishes) and make a list of start and stop time. But unless you really don’t like the first solution, this is way too much work.

      Lastly, if none of the above works probably moving to a custom MediaPlayer implementation might be a better option.

      In a different note, be sure to handle empty captions text condition as mentioned around the end of the article. That might give you unexpected failure in the future when a totally valid subtitle file is passed.

Leave a reply here, thanks!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s