Octane video options are either low end (composite analog: Personal Video), or high end (digital: SDI and HD-SDI).
So you're probably going to need an RBG-to-(HD)SDI converter, maybe an AJA HD10A or similar? You didn't tell us the source of your RGB, if it's not broadcast video but VGA or RGB workstation output you may need something else.
An Octane1 Digital Video board has SDI input/output,
An Octane2 DM2 with VBOB has HD-SDI input/output (this can be rather pricey).
Don't forget that all video input is stored uncompressed so especially HD formats need ample disk I/O bandwidth (up to ~ 240MB/s for 4:4:4 I think). Fibre Channel is your friend.
Finally, it's possible to use an Onyx2 DIVO card in an Octane (if you replace the XIO latch with an Octane style latch). It provides SDI input/output. I have one of those left for sale and can replace the latch.