Have attached below the steps I had to take to compile xmame 034.2 on Irix 5.3. This is probably a good starting point to get things going, hopefully may be of use if you would like to try this out yourself...
Steps taken to compile xmame 034.2:
------------------------------------
1. Before you start:
Install the IDO for Irix 5.3
Extract the xmame source files where you intend to do your work: tar -xvf xmame.tar
Install GNU Make "make-3.81-2.tgc-irix5.3-mips1-tgcware.tardist", and depenencies:
expat-2.0.1-2.tgc-irix5.3-mips1-tgcware.tardist
gettext-0.16.1-2.tgc-irix5.3-mips1-tgcware.tardist
libiconv-1.11-2.tgc-irix5.3-mips1-tgcware.tardist
libgcc_s1-3.4.6-2.tgc-irix5.3-mips1-tgcware.tardist
libstdcxx7-7-6.tgc-irix5.3-mips1-tgcware.tardist
2. Make the following changes in makefile.unix
> MAKE = gmake ## Irix make requires more effort, need to to use gmake. Line 40
> cc in lieu of gcc ## line 44, use the IDO compiler
> remove -n32 -mips4 ## we are compiling on mips3, causes error below for lib64/cmplrs/cc
> ARCH = IRIX ## line 330
> Enabled sound line 330 (used ARCH = irix # automatically includes sound)
> Did not need to change rm -fR to rm -fr # Line 433
> Changed OPTFLAGS = -O -Wall to " -O -fullwarn ## Line 85
## -Wall is gnu all warnings
## ERROR on flag -Wall in "OPTFLAGS = -O -Wall"
3. 1st Attempt to compile: "gmake -f makefile.unix"
Error on src/unix/video.c: 181:Unterminated string or character constant
FIX: Added backslash '\' at the end of lines 253, 254, 255 (of file video.c)
4. 2nd Attempt to compile: "gmake -f makefile.unix"
cc Error: cc is not installed in /usr/lib64/cmplrs/cc
FIX: remove -n32 -mips4 from makefile ## causing error for lib64/cmplrs/cc
5. 3rd Attempt to compile: "gmake -f makefile.unix"
Ignore Warning on optimisation, suggest to use -Olimit option on Dasm68000
Errors:
ranlib ../../obj/unix.x11/osdepend.a
gmake[1] ranlib: Command not found
gmake[1] *** [../../obj/unix.x11/osdepend.a] Error 127
gmake[1] Leaving directory '/usr/people/guest/xmame34.2/src/unix'
gmake[1]: *** [osdepend] Error 2
FIX: Edit makefile.unix and comment out use of ranlib on line 70.
# RANLIB = ranlib
Need to be careful of line 499 and 508 as these have references to RANLIB.
6. 4th Attempt to compile: "gmake -f makefile.unix"
gmake: execvp: obj/pacman.a: Permission denied
gmake: *** [obj/pacman.a] Error 127
Caused by line 507 which has the line $(RANLIB) $@ # and hence executes nothing.
Fix: Removing RANLIB fixes the problem
6. 5th Attempt to compile: "gmake -f makefile.unix"
Compiled and linked successfully...