I can't speak for an Octane with EMXI, but on a lowly O2 with CRM graphics, I've found MPEG-4 part 2 video with AAC audio plays nicely on mplayer.
300 MHz R5K O2
Full screen video: 320x240 at 300 kbps and audio at 64 kbps.
Widescreen video: 320x180 at 300 kbps and audio at 64 kbps.
CPU usage is at around 40-55%
400 MHz R12K O2
Full screen video: 480x360 at 800 kbps and audio at 128 kbps.
Widescreen video: 480x270 at 800 kbps and audio at 128 kbps.
CPU usage is at around 50%
* 640x480 (full) and 640x360 (wide) resolutions are also playable but CPU usage goes up to 90-95%
So if these are passable on an O2, then a faster system with better graphics will obviously have no problem. I think the key is to use the old MPEG-4 part 2 codec when converting videos for playback on an SGI with mplayer. And keep the resolution to the strict 320/480/640 widths. IRIX seems to love those for video. Can anyone with a VPro Fuel or Octane let me know if 1280x720 MPEG-4 (1200 kbps) is playable on theirs?