After doing a bit of digging I was able to get PS3 Media Server running on Ubuntu 9.10 64bit, with the ability to play hd x264 mkv files. I thought I'd post what steps (I remember) I took, and hopefully this helps others in a similar situation. I think these steps should work on a 32bit install, but I can't verify that.

Note that this worked for me, but I can't guarantee that it won't completely destroy your computer, so use at your own risk.


At a high level, here are the steps that I took:


1. Prerequisites
2. Install mplayer, mencoder, ffmpeg
3. Compile the latest mplayer with the fix mentioned in this post: viewtopic.php?f=3&t=4224
4. Compile the latest SVN of ps3 media server.
5. Optionally, make PS3 Media Server start every time I log in


And now for the nitty gritty details:

1. Prerequisites
Make sure that the ubuntu restricted extras package is installed. From a terminal window, run


sudo apt-get install ubuntu-restricted-extras



Next, install checkinstall


sudo apt-get install checkinstall


This package is required to add our compiled version of mplayer to our ubuntu package repository after we compile it.

Next, we need to install ant, so that we can compile PS3 Media Server.


sudo apt-get install ant



2. Install mplayer, mencoder, ffmpeg

If you had a look at the readme for PS3 Media Server, you'll see that it doesn't include these in the Linux version, and they need to be installed seperately.

Open up a terminal window and run the following:


sudo apt-get install mplayer mencoder ffmpeg



Note that while we are compiling mplayer in the next step, we are still installing mplayer here first. We are doing this because this will install various other mplayer packages, like mplayer-fonts, that the compiler may require in the next step.


3. Compile the latest mplayer with the fix mentioned in this post: viewtopic.php?f=3&t=4224

This code is taken directly from this forum posting. In the same terminal window, run these seperate lines of code one at a time:


$ sudo apt-get install subversion
$ cd $HOME
$ svn checkout svn://svn.mplayerhq.hu/mplayer/trunk mplayer
$ cd $HOME/mplayer
$ CFLAGS="-O2 -fomit-frame-pointer" ./configure --confdir=/etc/mplayer
$ make
$ sudo checkinstall -D --install=yes --fstrans=no --pakdir "$HOME/Desktop"
--pkgname mplayer --backup=no --deldoc=yes --deldesc=yes --delspec=yes --default
--pkgversion "3:1.0~svn-`grep "#define VERSION" version.h | cut -d"-" -f2`"
$ make distclean




4. Compile the latest SVN of ps3 media server

I tried with the current stable release and MKV files were still very choppy. After following these steps, it seemed to resolve the issue.

First download JDK 6 from Sun. We need this to compile PS3 Media Server. Go to Google JDK 6, and grab the latest JDK from Sun. As of this writing it is JDK 6 Update 16, and the filename I downloaded for 64bit was jdk-6u16-linux-x64.bin
Once the file has finished downloading, in your terminal window navigate to your Downloads


cd ~/Downloads



and run


sh ./jdk-6u16-linux-x64.bin



Read the EULA and accept it. Once it finishes the install, we need to set our Java HOME to the jre directory within the JDK we just installed. To do that run:

(replace this directory with the valid directory you just installed JDK to)


JAVA_HOME=/home/YOUR_UBUNTU_USERNAME/Downloads/jdk1.6.0_16/jre


then export the JAVA_HOME variable so it is available to other applications in this terminal session.


export JAVA_HOME





Next, change to your home directory and download the SVN of PS3 Media server:


cd ~


svn checkout http://ps3mediaserver.googlecode.com/svn/trunk/ ps3mediaserver-read-only



Once that completes, we can go into the directory where the PS3 Media Server code is downloaded and compile it with ant


cd ps3mediaserver-read-only/ps3mediaserver
ant



Done!

You have to make sure that PMS.sh has execute permissions on it.


chmod a+x PMS.sh



The other option is to run


sh ./PMS.sh



You also might have to do the same thing to tsMuxeR, located in the linux directory.


cd linux
chmod a+x tsMuxeR



Now you can run ps3 media server by typing


./PMS.sh

from within the ps3mediaserver directory. If all goes well, you should have no problem running HD Movies across your blazing fast gigabit network connection :lol:


5. Optionally, make PS3 Media Server start every time I log in

PMS.sh HAS to be ran from the directory it is in... if you run it by typing /home/username/ps3ms/PMS.sh, you will get errors when playing your files, and if you click on the README tab or FAQ tab, they will be blank. This is because it uses relative paths in it's code. The only correct way to run PMS.sh is to first cd into the directory, then run ./PMS.sh.

It is because of this, that when you try to run PMS.sh from your Startup programs, you get the above issues. To get around this, I create a script file that will first cd into the directory, and then run PMS.sh.

Create a file in your ps3 media server directory called start-pms.sh, with a text editor add the following to it:


#!/bin/sh
cd /home/murray/ps3mediaserver
./PMS.sh



you'll want to give that file execute permissions


chmod a+x start-pms.sh



Next, click System > Preferences > Startup Applications

Click Add..

Name: PS3 Media Server
Command: /path/to/start-pms.sh

Click Add.

Click close. Now the next time you log in, you should see PS3 Media Server start no problemo. I check "start minimized" under general configuration so that I don't get the main window each time I log in.