Using MythTV as a Video Aggregator

From MythTV Book

This chapter is a published version, which means it can only be changed by editors. If you find an error, or want to expand this chapter, then please do that by editing Using MythTV as a Video AggregatorBeta. If you have questions, please contact the editors.

Introduction

A few years ago if you wanted to check a set of web pages daily, you'd open each one of them in a browser window and check for updates. As blogs became more common, this became quite painful, as there started to be many sites to check. Clearly, some automated way of checking for updates was required. Dave Winer proposed one solution, which was an XML representation of the contents of the blog called Really Simple Syndication (RSS). Now, instead of a user checking sites for updates, they could have a piece of client software, known as an aggregator, download these XML files, and inform the user of updates to the site when they occur. RSS' contribution is largely that it is easier to parse XML than HTML, and that the XML format includes information which might not be included in a standard web page -- such as the date that a particular portion of the page was last updated.

Once blogging became popular, bloggers branched out into other formats. Some users started adding audio files to their blog posts, and called themselves podcasters -- a name which was a play on broadcasting intended for MP3 players such as Apple's iPod. Other bloggers started shooting video content, and adding this to their posts. These video posts are either called video-blogs or vlogs. At this point users still needed to download the media content of a blog post when their aggregator informed them of a new post. The next step in the evolution was therefore to add to the XML syntax of RSS to allow for automated download of media content. This was done by implementing an extension which termed these media files enclosures.

Video blogging is now a common method of distributing video content. Individuals do it, popular group blogs such as Boing Boing have a video version, and even major media outlets like television networks have video blog content. An example is Australia's ABC, which provides many of these television programs as in video blog format, as well as broadcasting them over the air.

MythTV is capable of acting as a video aggregator via a plugin called MythNetTV. MythNetTV was originally written as a test harness to import content for an earlier version of this book, and is now used by various MythTV users to import video blogs into MythTV. MythNetTV's main strengths are that it downloads a variety of video aggregation formats, handles converting the video files to a format that MythTV can play and imports the recordings into MythTV's program database so that they are treated as regular television programs. In fact, unless you look at the name of the channel the program came from, its not possible to tell any difference between an imported video and one with was recorded by a tuner under the control of MythTV.

MythNetTV is a little different from other MythTV plugins, in that it isn't available for download from the main MythTV web site, and doesn't have a visible presence in MythTV's user interface. This, however, may change in the future.

This chapter will guide you through downloading and installing MythNetTV, as well as configuring it.

Introducing MythNetTV

As mentioned earlier, MythNetTV is a video aggregator for MythTV. Video files for syndication feeds such as RSS and ATOM are downloaded, and the programs from those feeds which haven't yet been downloaded are fetched, converted into the right format for MythTV, and then imported into MythTV. MythNetTV can fetch programs using HTTP, HTTPS, RTSP and BitTorrent protocols. MythNetTV is a python program which runs separately from MythTV, although there is a simple user interface available as well.

Installing MythNetTV

Installing MythNetTV from source is relatively simple. First off you need to ensure that you have the dependencies installed. For Ubuntu, these are installed like this:

$ sudo apt-get install python python-feedparser python-mysqldb mplayer mencoder bittornado

This will fetch the needed packages and install them in the usual manner.

Next you need to fetch the MythNetTV source. If you want to use the latest stable version, then check the MythNetTV site for a recent release. The most recent release is release 7 at the time of writing this chapter. What you will download is a tar file, which is extracted like this (although you may need to update some version numbers):

$ tar xvzf mythnettv-release-7.tgz
release-7/
release-7/ChangeLog
release-7/database.py
release-7/gflags.py
release-7/mythnettvcore.py
release-7/mythnettv_test.py
release-7/program.py
release-7/proxyhandler.py
... snip ...

It doesn't really matter where you put the extracted directory, as MythTV doesn't call MythNetTV. If you need the latest bug fixes, or just want to test the unstable version, then you need to fetch the source code from the source repository. The source repository uses subversion (SVN), so you'll need to ensure that you have subversion installed first:

$ sudo apt-get install subversion

Now you're ready to fetch the latest version of the MythNetTV code. You do that like this:

$ svn co http://www.stillhq.com/mythtv/mythnettv/svn/trunk mythnettv
A    mythnettv/submitchange.sh
A    mythnettv/recordings_tool.py
A    mythnettv/testsetup.py
A    mythnettv/README.BITTORRENT
A    mythnettv/utility_test.py
A    mythnettv/release.sh
A    mythnettv/gflags.py
A    mythnettv/database.py
A    mythnettv/video_test.py
A    mythnettv/video.py
A    mythnettv/program.py
A    mythnettv/runtests.sh
... snip ...
Checked out revision 225.

This checks out the trunk of the MythNetTV source code repository. The trunk is the place where current development occurs. There are other things in the source repository -- such as old releases and the tar files from the web site, but they're only of interest to developers.

Its a good idea to run the unit tests before you use the MythNetTV code. This will ensure that MythNetTV is working correctly on your system, and that you have all the dependancies installed. Generally new unit tests are added when users report problems, so if the unit tests pass then your install should be ok.

The unit tests use a MythTV database, so you need to create that database first. MythNetTV includes a simple SQL script to create a test user and database, which should be ok in most circumstances -- it creates a database called mythnettv_test, and a user named test who can only log on from the local machine. The command to do that will look like this:

$ mysql -u root -p < setuptestdb.sql
Enter password:
$ mysql -u test -p mythnettv_test
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 186832
Server version: 5.0.67-0ubuntu6-log (Ubuntu)
 
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
 
mysql> quit
Bye

Here we run the script as root, and then test the "test" user to make sure it is working. You run the unit tests with a simple shell script. Change directory into the source you downloaded, and then run the tests like this:

$ ./runtests.sh 
 
Running mythnettv_test.py
 
..Dropping old testing table: channel
Dropping old testing table: mythnettv_archive
Dropping old testing table: mythnettv_category
Dropping old testing table: mythnettv_group
Dropping old testing table: mythnettv_log
Dropping old testing table: mythnettv_programs
Dropping old testing table: mythnettv_proxies
Dropping old testing table: mythnettv_proxy_usage
Dropping old testing table: mythnettv_settings
Dropping old testing table: mythnettv_subscriptions
Dropping old testing table: recorded
Dropping old testing table: settings
Info: Creating log table
Info: Creating settings table
Info: Creating programs table
Info: Creating subscriptions table
Updating tables
Updated the data directory to /tmp/testdata
Updated the data directory to /tmp/testdata
Updated the data directory to /tmp/testdata
Updated the data directory to /tmp/testdata
Updated the data directory to /tmp/testdata
Updated the data directory to /tmp/testdata
Downloading http://www.stillhq.com/mythtv/mythnettv/testdata//gruen_2008_ep10.mp4
... snip ...

You'll know if a unit test failed because there will be error text at the end of the output. If there is a line saying "all tests passed" at the end of the output, then all is well.

Configuring MythNetTV

Configuring MythNetTV is reasonably straightforward. These instructions assume that you already have MythTV setup on the machine you're using for MythNetTV though, as the database configuration is read from the mysql.txt file created by the MythTV frontend. First off, MythNetTV needs a directory to store temporary files in which downloads are occurring. This is also where partial downloads are stored between attempts, so it can get quite large. In this example, we'll put these files in /tmp, which means it will get cleaned out on boot. Let's run the list command, which wont return anything, but lets us set the data directory:

$ mythnettv --datadir=/tmp/ list
Updated the data directory to /tmp/
You are not currently subscribed to the MythNetTV
announcements video feed. This feed is very low volume
and is used to announce important things like new releases
and major bug fixes.
 
Would you like to be subscribed? You will only be asked
this once.
 
Type yes to subscribe: yes
Subscribed to MythNetTV announcements
MythNetTV announcements
 - http://www.stillhq.com/mythtv/mythnettv/announce/feed.xml

Notice that we were also asked if we wanted to subscribe to the MythNetTV announcements feed, which is an extremely low volume video blog used for announcing new MythNetTV releases.

MythNetTV Functionality

MythNetTV has many command line options. This section serves as a reference for those commands, but is written in the form of a tutorial -- in other words options are introduced in the order you are likely to need them.

Processing Syndication Files

The first step with using MythNetTV is to tell it about programs you want to download. There are several ways to do this. First off, you can ask MythNetTV to process a local syndication file with the file command, like this:

$ mythnettv file <filename> <title>

In this case we're processing the file named filename, and telling MythNetTV that the name of the program is title. For example, let's say that we have an RSS file containing a list of videos from a conference, and we want to add those videos to MythTV for later viewing. We would use a command line like this:

$ mythnettv file lca2008.xml "linux.conf.au 2008"

In the above example, you would need to obtain the XML file containing the videos from somewhere else. Its more likely though that you've found a syndication file for the videos online somewhere. In that case, you would use the url command to download the syndication file, and then process it:

$ mythnettv url http://mirror.linux.org.au/linux.conf.au/2008/videos.rss "linux.conf.au 2008"

This version of the command works well if you only want to process the syndication file once -- for example when a conference's videos have been put online. However, if you want to check regularly for updates, as would be the case with something like a blog or online television show, then you're better off subscribing to the syndication file:

$ mythnettv subscribe http://mirror.linux.org.au/linux.conf.au/2008/videos.rss "linux.conf.au 2008"

The difference between url and subscribe is that with subscribe, MythNetTV makes a note of the URL, and you can check the URL for updates without having to retype the URL.

Subscription Management

The most common way of downloading programs with MythNetTV is to subscribe to syndication URLs as described in the previous section. Once you've created subscriptions you will of course want to manage them. First off, you create a subscription with the subscribe command discussed above. You can list existing subscriptions with the list command. For example, here are some of my subscriptions at the moment:

$ mythnettv list
ABC 7:30 report
 - http://www.abc.net.au/7.30/xml/730_vodcast_m4v.xml
 (inactive)
  Category is Australia
  Group is AustraliaGroup
 
Ask a ninja
 - http://feeds.feedburner.com/AskANinja-WMW
 
blip.tv
 - http://make.blip.tv/rss (proxy: proxyhost.stillhq.com:3128 budget 1 gb (1073741824 bytes))
 (inactive)
 
Boing Boing
 - http://feeds.boingboing.net/boingboing/tv
 
...

There is quite a lot of information in this listing. For example, you can see that two of my subscriptions are inactive (which means that no more programs will be downloaded from those syndication feeds). You can also see that one of my downloads has a category and group set, and that another of my subscriptions uses a HTTP proxy when fetched. All of these features are discussed below.

To unsubscribe to a syndication URL, you use the unsubscribe command. This command has caused quite a lot of discussion, as it doesn't actually remove the subscription -- it simply marks it as inactive. This was implemented in this way because if you removed a syndication URL and then re-add it later, all the programs currently listed in the syndication file will be marked as new unless they already appear in the database. This way, re-enabling a subscription will ignore programs from before you run the subscribe command again.

To unsubscribe:

  unsubscribe http://mirror.linux.org.au/linux.conf.au/2008/videos.rss "linux.conf.au 2008"

Once you've created subscriptions, you'll want to check if there are new programs listed in those syndication URLs. This doesn't happen automatically -- you need to ask MythNetTV to check for updates. You do this with the update command, which can be run like this:

$ mythnettv update

If you only want to update one program, perhaps because you have many subscriptions and it takes a while to download them all, then you can specify the program title to update. This will download all of the syndication URLs for that title.

$ mythnettv update "linux.conf.au 2008"

Downloading Programs

Now that we have some programs in the database, either from processing a syndication file as a one-off, or by subscribing to some syndication URLs, and then running the update command, it's time to download some programs. The simplest invocation of the download command is quite simple:

$ mythnettv download 10

This will simply download the "next 10" programs. I use quotes around "next 10", because the algorithm for selecting programs to download isn't simply a case of downloading the ten newest programs. Instead, the default is to download the five newest programs, and the five oldest. This is done in case you want to watch the first episodes of a program before moving onto the more recent ones. You can use flags to change this behavior. Specifically, using the --newestfirst flag will download only the newest programs, whereas oldestfirst will download the oldest. An example use of one of these flags:

$ mythnettv --oldestfirst download 10

Will download the ten oldest programs. Its possible that you only want to download episodes of a given program. You can do that by specifying the title of the program. For example:

$ mythnettv download 10 "linux.conf.au 2008"

You can be more specific than this though. Perhaps you only want to download talks about MythTV from the conference. You can do this by specifying a title, and then a regexp to use against the subtitle (episode title) of the program. Only episodes matching both criteria will be downloaded. So, to download only talks about MythTV from the conference:

$ mythnettv download 100 "linux.conf.au 2008" ".*MythTV.*"

There is one final download option I want to discuss in this section. Imagine that the conference has two versions of one talk in their syndication file, and you only want one copy of the program. Perhaps you don't really care which version you get either -- so long as you get a copy. In that case, construct a filter which uniquely identifies all the duplicates, and then append justone to the end of the command line, like this:

$ mythnettv download 1 "linux.conf.au 2008" "MythTV Tutorial.*" justone

Once one version has been successfully downloaded, all the others will be marked as downloaded as well, even though they haven't actually been downloaded. If you just want to find out what episodes will be downloaded next, then you can use the nextdownload command to find out. It takes the same flags as the download command, and either just a number, or a number and program title.

Skipping Downloads

Sometimes you want to skip downloading some episodes of a program. This is implemented using the markread command, which mirrors the download command. In fact, the same logic is used to select the programs to download, but instead of downloading them they are simply marked as having already been downloaded and imported into MythTV. Some examples:

$ mythnettv markread 100
$ mythnettv markread 100 "linux.conf.au 2008"
$ mythnettv markread 100 "linux.conf.au 2008" ".*MythTV.*"

The markread command is interactive, which means you'll be prompted for each show. For example:

$ mythnettv markread 1
1 matches
  linux.conf.au 2008: An Introduction to ANTLR A parser toolkit for problems large and small
  (Wed, 31 Dec 2008 08:00:04)
 
Are you sure you want to mark this show as downloaded?
 
Type yes to do this: yes
Done

If you want to re-watch an episode, or regret marking a show as downloaded, then you can also markunread. markunread is currently limited to the most recent programs, which can make it harder to mark old programs as unread. This may change in the future. Finally, if you want to remove everything from the list of episodes to download, you can use the cleartodo command, which takes no arguments. This is the "nuke from orbit" option, with no way of undoing, so use it with care.

Attempts Are Rate Limited

MythNetTV's download command is intended to be run in an unattended manner -- for example via cron or a screen session. This means that MythNetTV needs to rate limit attempts to download an episode, as well as limiting the number of overall attempts for a single episode as well. This stops a single failed episode from blocking other episodes from downloading. The rate limiting is implemented by only attempting to download a given episode once per hour, as well as only trying to download a given episode three times before giving up.

You can instruct MythNetTV to ignore the time at which the last attempt to download programs was with the --force flag. You can also reset the counter of attempts for a program with the resetattempts command. This is useful when it is something other than the remote server which causes the download to fail -- for example if your home DSL is down, then downloads will fail. Once you fix your DSL, you'll want to reset the attempt counters. The resetattempts command will interactive reset the attempts counter for episodes which have not yet completed downloading. You can also restrict the command to a given program title by including that on the command line. Some examples:

$ mythnettv resetattempts
$ mythnettv resetattempts "linux.conf.au 2008"

One-off Downloads

Sometimes you don't want to subscribe to a program -- you just want to quickly download a single episode to watch. You do that with the todoremote command. If you run the command with no arguments, it will prompt for needed details of the program -- things like the title, episode title, and description of the show. For example:

$ mythnettv todoremote http://mirror.linux.org.au/linux.conf.au/2008/Thu/mel8-114.ogg
title >> linux.conf.au 2008
subtitle >> Memory-Efficient and Fast Websites -- Pick Two!   
description >>

This will create an entry in the list of episodes to download for this program. You can also enter all the needed information on the command line if you don't want to be prompted:

mythnettv todoremote http://mirror.linux.org.au/linux.conf.au/2008/Thu/mel8-178.ogg
"linux.conf.au 2008" "Clustered Samba - not just a hack any more" " "

Here I passed a space for the program description because you can't pass an empty string on the command line. If you don't want to make an entry in the TODO list, but would rather download and import the program immediately, then use the importremote command, which takes the same arguments as todoremote, but does all the work immediately.

Bittorrent downloads need some special handling for this style of import. This is because there are two downloads for a bittorrent download -- first the torrent file needs to be fetched, and then the actual content may be fetched. There is therefore the importtorrent command, which is the same as importremote, but works for bittorrent downloads.

Finally for one-off imports, you can of course also import local files. This is useful because these files will be transcoded to a format MythTV can play (if they're not already in a playable format), and then appear as a TV program inside the MythTV user interface in the same manner that any other MythNetTV download does. This can be useful if you have transcoded videos on disk that you want to reimport to watch again, or if you have videos from another source such as a handycam. To do this, use the importlocal command, which again takes the same arguments as the importremote and importtorrent commands. Note that the original source file will not be removed by this command.

If you want to import a directory full of videos, then you can also use the importmanylocal command to speed the process up. This command takes a path to a directory, a regexp to match, and a program title as command line arguments. For example, if we already had some conference videos downloaded, but wanted to watch them with MythTV:

$ mythnettv importmanylocal /data/la-mirror/linux.conf.au/2008/Thu/
".*.ogg" "linux.conf.au 2008"
Considering mel8-114.ogg
Destination will be /data/src/mythnettv/data/mel8-114.ogg
Importing /data/src/mythnettv/data/mel8-114.ogg
Files in format theo do need transcoding
Transcoding
Importing video /data/la-mirror/linux.conf.au/2008/Thu//mel8-114.ogg...
Rebuilding seek table
Adding commercial flag job to backend queue
Done
 
 
Considering mel8-120.spx
 
Considering mel8-099a.spx
 
Considering mel8-092.spx
 
Considering mel8-117.ogg
Destination will be /data/src/mythnettv/data/mel8-117.ogg
Importing /data/src/mythnettv/data/mel8-117.ogg
Files in format theo do need transcoding
Transcoding
Importing video /data/la-mirror/linux.conf.au/2008/Thu//mel8-117.ogg...
Rebuilding seek table
Adding commercial flag job to backend queue
Done
 
 
Considering mel8-041a.spx
 
Considering mel8-102.spx
 
Considering mel8-178.spx
 
Considering mel8-122.ogg
Destination will be /data/src/mythnettv/data/mel8-122.ogg
Importing /data/src/mythnettv/data/mel8-122.ogg
Files in format theo do need transcoding
Transcoding
...

You can see in this output that the videos needed to be transcoded, as MythTV can't play OGG videos natively, and that only the files matching the regular expression were imported.

Archiving Your Downloads

If you're downloading a program from the Internet, then sometimes you want to keep an archive of its episodes on disk as well. I don't like doing this in MythTV itself, as I've found that MythTV's GUI becomes sluggish when I have more than a few hundred recordings. Instead, I archive my downloads to a directory on disk. This is supported by the archive command, which takes a program title and a path to archive to. In my case:

$ mythnettv archive "linux.conf.au 2008" /data/video/linux.conf.au/

Categories and Groups

MythTV records have extra attributes associated with them. Obviously the program title, episode title, description and channel the program was recorded from are a few examples. Additionally, recordings have a category, which is used for parental controls, and a group, which is used for playback. A typical category would be something like "childrens" or "drama", whereas playback groups are user defined and used to help you select programs to watch.

MythNetTV supports setting both of these attributes with the category and group commands. Some examples:

$ mythnettv category "linux.conf.au 2008" "conference"
$ mythnettv group "linux.conf.au 2008" "AustraliaGroup"

Using a HTTP Proxy

Sometimes you want your HTTP and HTTPS downloads to go through a proxy. Perhaps this is because of requirements at your ISP, or the proxy tunnels through a VPN connection. MythNetTV can use proxies if required, although the configuration is unusually flexible. You specify a proxy with the http_proxy command, but you also need to specify a regexp for URLs that should use that proxy. To have all traffic use the proxy, use the regexp ".*". For example, let's send all Australian domain names to a proxy:

$ mythnettv http_proxy ".*\.au" aussieproxy.stillhq.com:3128

You can additionally specify a maximum amount of traffic to send through a proxy in a given day. This is useful if you want to limit the amount of traffic you are being billed for. Just append a number of megabytes to the end of the command line. The proxyusage command will give you a summary of traffic through any configured HTTP proxies.

Reporting

MythNetTV provides a bunch of reporting as well. The most interesting is probably the statistics command, which will tell you how many episodes of your various subscriptions it has downloaded, as well as how many are waiting to still be downloaded. It will also show you how much traffic each of your various subscriptions has cost you, although the accounting for that isn't 100% accurate for bittorrent. The log command is helpful for debugging problems, as it displays MythNetTV's internal log of stuff which has happened.

Debugging Video Directories

MythNetTV needs to know where MythTV is storing its recordings so it can move imported programs to the right place. This is probably the most fragile part of MythNetTV because sometimes it is hard to determine the correct location. There are therefore two commands which help with this: videodir will show you where MythTV thinks it should be putting recordings, and explainvideodir will explain why it picked that directory.

Flags

I've already explained some of the flags that you can use to tweak MythNetTV behavior. There are a few others that I would like to mention as well.

• --help: Show a help message

• --datadir: The location of the data directory. Will change the previous value. This is useful the first time you run MythNetTV.
• --noprompt: Suppress prompting for user input when required
• --nopromptforannounce: Should the user be prompted about subscribing to the announce video feed? This is only used the first time MythNetTV is run.

• --defaultuploadrate: The default bittorrent upload rate. If set will change the previous value.
• --uploadrate: Override the default upload rate for bittorrent for just this one download.

• --db_host: The name of the host the MySQL database is on, don't define if you want to parse mysql.txt instead.
• --db_name: The name of the database which MythNetTV uses, don't define if you want to parse mysql.txt instead.
• --db_password: The password for the database user, don't define if you want to parse mysql.txt instead.
• --db_user: The name of the user to connect to the database with, don't define if you want to parse mysql.txt instead

• --nocommflag: Run the mythcommflag command on new videos.
• --force: Force downloads to run, even if they failed recently.

Other Command Line Tools

MythNetTV is implemented as a set of python modules, and some of these modules export helpful functionality of their own that I want to briefly mention.

Video Tools

The MythNetTV video subsystem may be queried directly -- it is in the file named video.py. This can be useful if you want to perform simple operations such as asking if a given video file needs to be transcoded before being imported into MythTV, or actually performing the transcode.

The length command takes a path and returns the length of a video. The info command also takes a path and returns all available information for a given video, such as the codec used to encode it. The needstranscode takes a path and determines if a given file needs to be transcoded before being imported into MythTV. The transcode command takes a path and an output path and transcodes the file into a format suitable for MythTV.

Recordings Tools

In the process of developing MythNetTV, I ended up creating a lot of test recordings which were painful to clean up in the user interface. I therefore wrote recordings_tool to help me with those tasks. To print a summary of your MythTV recordings, use the summary command. You can get a list of all of all titles with a recording currently available with the list command, which takes no arguments. You can limit the output of list to a specific title by including that on the command line as well -- in which case you'll get a summary of all the recordings with that title.

To get a list of previous recordings for a given title, use the previously' command, with the title as an argument. To allow re-recording of a previously recorded program, use the rerecord command with a program title. If you just want to re-record one episode, then pass the title and the subtitle of the episode.

You can delete all the recordings with a given title by using the delete command, passing it a title as an argument.

MythNetTV Community

There is a mailing list if you're interested in discussion and announcements of new releases (archives of the list are here). There is a public SVN repository as well. If you want to keep up with MythNetTV developments, your best bet is to keep in touch with the MythNetTV web page.

Conclusion

In this chapter we've discussed MythNetTV, a video aggregator for MythTV. We've discussed how to install it, how to configure it, as well as its functionality. MythNetTV allows you to import downloads from the Internet into MythTV so they appear like any other recording, and with many television networks now offering their programming as free downloads, this will become increasingly common in the future.