Record and archive video from IP cameras

I’ve had a couple of Foscam FI9805W H264 IP cameras for almost a year, and I’ve been very happy with them: the 1280×960 image is sharp and clear both during the day and during the night, and their firmware has been very reliable.

One thing I wanted, though, was having the footage from the last 1-7 days available at any time. The onboard firmware allows to record to an FTP server, but this was suboptimal, there was no easy way to define the clip length and was pretty clunky to set up.

I started digging around, and I found that ffmpeg could easily record the RTSP stream from the cameras. In the cameras’ settings you can choose the video bitrate, up to 4 Mbit. I found that the optimal bitrate is 2 Mbit: going to 4 only meant that the files were twice the size without any noticeable improvement of the quality.
This results in approximately 15 Gb per day per camera of video files. This is way lower than 2 Mbit average, and that’s due to the fact that during the night, with the IR lights on, the image turns to black and white and the bitrate lowers to about half of the usual value.

I came up with a complete solution which is made of the following parts:

• Two Foscam FI9805W IP cameras, but any reasonable number of cams can be used
• My home server, running Debian, which is on 24/7
• A cronjob that fires up ffmpeg every 15 minutes to record 15 minutes clips. This makes searching much easier than having to deal with giant multi-hour recordings.
• A cronjob that triggers everyday at midnight, which converts recordings older than 24 hours to a much smaller low-framerate and low-quality files to reduce disk usage
• A cronjob that triggers everyday at 4 am to purge older recordings.

Recording video

This is the easy part. I just use this script, which I named recordCam.sh:

You’ll have to edit the IP address, port number and login credentials to suit your needs and add/remove additional lines at the bottom to match the number of cameras you need to record. Also, set your own paths.

You need to add a cronjob to your system, to fire this script every 15 minutes:

*/15 * * * * /path/to/recordCam.sh

Quick tip: in the settings of each of your cameras, add a “Visitor” type user for esclusive use in this script, so that if somebody finds its password (which as you can see is saved in the clear) he cannot mess up with your cameras’ settings.

Converting to low-quality for archival

I decided I don’t need to save full-quality recordings of every single second, so my compromise was to heavily re-compress videos older than 24 hours (1440 minutes).

After lots of tests, I chose to reduce the framerate from 30 to 5 fps and set the bitrate to 100 kbits. That’s a really low bitrate for 960p videos, but since the footage is mostly static the quality is still half decent. The space usage is about 1 GB per day per camera.

The script I use, convertVideo.sh, is this:

It takes the file you pass to it, creates the appropriate folder structure and encodes it, then it deletes the original file.

This is the cronjob that launches the script:

0 0 * * * find /path/to/surveillance/folder/video/ -mmin +1440 -size +10000k -exec /path/to/convertVideo.sh {} \;

I use the find command to get the videos that need to be converted, and it looks for files over 10 megabytes that were last modified more than 1440 minutes ago. Of course, you are free to change these parameters as you wish.

Pruning old videos

Even with this heavy compression, the files add up quickly, so I decided it’s not worth keeping videos older than a week.

So, here is the cronjob to do the job (pun intended):

0 4 * * * find /path/to/surveillance/folder/archive -mindepth 3 -type d -mtime +5 -exec rm -r {} \;

It looks into the archive folders, looking for directories, as each day has its own, older than 5 days (that’s a weird side effect of confusing date math: you’ll end up having 7 days worth of recordings, plus the high quality last day).

The -mindepth 3 parameter was required due to the folder structure I chose, which is: archive/camXX/YYYY-MM/DD/*.mp4

• At the first depth level there are the folders of each camera. Their last-modified date changes every time you add or remove a file/foder inside it, so this actually happens the 1st of evert month, when the month’s folder is created.
• At the second level, there are the YYYY-MM folders, so we shouldn’t touch them
• Finally, at the third level there are our “day” folders, which we want to delete when they get too old.

Then a final cronjob that removes old and empty month directories

1 4 * * * find /path/to/surveillance/folder/archive -mindepth 2 -maxdepth 2 -type d -empty -exec rmdir {} \;

You’re done

Yes. That’s it. I admit it’s not very straightforward, but it does work once all the pieces are in place. The nice thing is that all the mp4 files, both those saved directly from the cameras and the re-encoded ones play nicely on my iOS devices (I presume Android as well, but I don’t have a device handy to test), so I can just VPN back home to retrive a recording, should I need to.

If you have any questions feel free to leave a comment below, I’ll try to reply to everyone.