BakaSub ATI Genlock Hack

BakaSubDocs | Some devious ATI docs

This page may be very out of date or no longer relevant.

BakaSub/Linux

Version 0.1.1

Copyright (C)1998 Dan Potter

This file contains supplementary info on using an ATI All-in-Wonder or All-in-Wonder PRO under Linux for subtitling. You can't do a whole lot else useful with it but it does do some neat tricks. I am, e.g., watching The Wall in the background of the Pico session while typing this. Pretty trippy...

I recommend you read this all the way through and understand what I'm talking about BEFORE STARTING. You may notice something I've left out that you'll need. No, I didn't leave anything out on purpose (as that kinda sounds =), but I may have left something out on accident. Better to know now than when you've got a trippy half-working display of The Wall running through your screen.. hehe =)

This file is valid for the ATI All-in-Wonder and PRO *PCI* cards, although it may work with AGP. I dunno.

Background

So... you seek the spell of GENLOCK?? winking smiley

Well, TOO BAD cause ATI DOESN'T LOVE YOU like their WINDOWS USERS!!

But that's alright. That's what hackers are for.

I am currently working on an evil work-around to get a basic 640x480 genlock working in Linux using the ATI All-in-Wonder and All-in-Wonder Pro cards. I have the former.. quality ain't too good but it's a lot better than nothing =). The genlock itself is actually about as good as you can expect to get in Windows, and there are a few caveats that go with it.

There are two major components to this little hack:

  • video input
  • video output

The first is, paradoxically, much easier to obtain. The current workaround I have involves using DOSEMU to run their M64DIAG program, and then switching to X (which is already running in 640x480). The result is a nice chromakeyed video input taking up the whole screen. Definitely not perfect but not too bad either.

Their tech support told me that setting up the video input is VERY HARD (as if we couldn't do it) but the above suggests that the video overlay almost sets itself up. Because you can switch modes in X and the TV input window stays there, same size. Which ought to tell you once again how much they love us.

The second problem, again pardoxically, is much harder. The reason is that the TV output is _very_ picky about its mode timings. I say paradoxically because it's very easy to get the TV output to turn on in Linux. You can just hook up your TV and then boot Linux... works great. You can also go into DOSEMU and use "install tvon" from the M64 diags, and that works fine. HOWEVER, switching video modes manually (without using the ATI BIOS) causes the TV output and the standard video output to both go crazy. I'm not sure if there's an easy workaround for this one unfortunately. I tried capturing the SVGA regs during the M64DIAG program and constructing X mode timings from it, but something's still not right.

I did have a thought that may solve both of these problems but would require a GREAT DEAL of effort. Well, if you're a die-hard, you'll do it, right? winking smiley

You need a Linux 2.1.12x kernel with "vesa framebuffer console" turned on, and the XFree86 framebuffer server. What you're gonna do is basically the same as the above, except you'll set Linux to use 640x480x24 (or whatever) on Linux startup, then let ATI do the same for you under DOSEMU. Now you can switch consoles, no problem, and you're in the same video mode.

Maybe ATI will get smart and release their specs eventually but I kinda doubt it. My guess is that this will go like all other people reluctant to release hardware specs eventually -- people will get sick of not having the features they paid for, will hack it out WITHOUT ATI's help, and their card will look bad... at that point they have nothing to lose so they help. I hope so anyway =)

Down and dirty

Ok, no matter how you get things rolling, you're gonna need one thing for sure -- a working DOSEMU setup that can access your video card's BIOS and PCI address space.

I'm using RedHat 5.2, which comes (handily) with DOSEMU 0.98.1. I highly recommend that you obtain this version to work with, as it reduces the number of variables and makes sure everything is up to date enough to work with your card.

Ok, to start off, let DOSEMU and friends get at your video card. Look in the file /proc/pci for your video card. You should see "VGA compatible controller" and then a chip designation. A few lines down will be a line that says something like "I/O at 0xe400 [0xe401]". The first number (0xe400) is the one to note. That is what mine says, so I will use it as an example from here out. Note that if you change motherboards or move your PCI cards around, this number may change.

Now edit your /etc/dosemu.conf file and turn on the video stuff. You can probably figure out the equivalent lines in your version of DOSEMU (if not 0.98.1) but here is what it is for that version. Change these lines: 

 $_video = "vga"
 $_console = (1)
 $_graphics = (1)
 $_videoportaccess = (1)
 $_vbios_seg = (0xc000)
 $_vbios_size = (0x10000)
 $_vmemsize = (1024)
 $_chipset = "plainvga"

Ok, that's a good start. Now you need to give DOSEMU a bunch of port access on your machine. Yeah, this may make you a little nervous, and it should. But this is actually very harmless beyond maybe losing your console and having to reboot... Note that you should replace the 0xe4xx lines with the PCI base you found above. 

 $_ports = "range 0xe400,0xe4ff"

Alright. Cross your fingers, do a chicken dance, and type "dos" at the prompt. if all goes well, you should get some screen flickers / multisync monitor switches, and see a DOS prompt in 80x25 text mode, maybe with "ATI GTB BIOS" at the top somewhere. No luck? Sigh.. Try disabling ALL shadow segments in your BIOS and see if that helps. (You don't need them in Linux anyway..)

Next step is to get a good DOSEMU hard disk setup. If you haven't done this already, I recommend setting up an image that has the smallest (memory-wise) version of DOS you can find. FreeDos will not work! It's too big! You should also make a copy of the installed "ATI Utility Disk" onto that DOSEMU image. You can get this utility disk off the ATI FTP site. Look for "Mach64 Diagnostics and Utilities". Get the newest version! The one I have is: ftp://ftp.atitech.ca/pub/Install/64utl303.exe

Ok, once you've got a DOSEMU setup that has a good amount of free memory and the M64 utils, we'll start to pull the first bit of black magic. Start up DOSEMU and then run "M64DIAG". You should get a screen change and then their goofy test display. Play around a bit, whatever, and once you're done, make sure to exit the util. Then exit DOSEMU.

If you've gotten that far, you're well on your way to getting this work. If that didn't work, but the DOSEMU startup with graphics BIOS did, then I don't know what to tell you.. =(

The Wrong Way(tm)

Well, I didn't want to wait to download a new kernel and X server, so I did things the supa-hack way the first time around. Here's what I did. It DOES work but it SUCKS! =) And you also can't get TV output.. =(

Basically, start Xwindows; switch to a Linux console; start the M64DIAG program under DOSEMU; select a video mode matching your X11 video mode; select "Extensions" and then "VT Functions"; use the "Functional Test" menu to select an input source; select "Demos" and "Video Capture". Voila. You've now got a big full-screen input window under Linux. Now's the tricky part... hit CTRL-ALT-F7 to switch back to Xwindows.

If smoke doesn't rise from your computer or something similar, I really want to hear from you. ^_^ For me the image remained, but it chose black as the key color and then dumped a stream of garbage on the upper part of my display. One way to get around this is to setup a virtual desktop of, say, 1280x1024, and then use a 640x480 resolution under X. You can then scroll down to the bottom part of the screen w/no corruption and watch TV while using an Xterm (trippy experience =).

You can get TV output working in a similar way, but not both at the same time! (as far as I know..) You just go into DOSEMU and run "INSTALL TVON". Poof! Problem is, if you try to change video modes without the help of the ATI BIOS, you're screwed. TV output goes crazy and often the monitor output doesn't come back at all. Note that you can also get TV output by attaching a TV to the output when you turn your computer on.

Recovery

If you are deft at typing without looking at the result, you can switch to an unused console, log in as root, and kill your current DOSEMU session; then start a new one. If you are using text mode (like most normal people in Linux) you can now exit DOSEMU (don't forget to turn off TV output if going into X) and be fine. This will also work with vesafb (described below), however. You should just change to the video mode matching your console in M64DIAG and then switch over to a console. That'll leave an ugly hardware mouse pointer on your screen, which is going to plague you all the way through this, but it's better than nothing winking smiley

The Right Way(tm)

Well, short of those hosers giving us Linux support anyway.

I'm going to start off by saying that this is really involved and unless you are crazy about seeing that genlock throughput through your Linux box and being able to subtitle with, e.g., BakaSub, don't bother. If you're a hacker type and you just can't stand the suspense then go right ahead. I'm pretty new to this myself but I can also give you some limited assistance over email if you want to try.

Supply list:

  • Redhat 5.2 or similarly modern distribution. Make sure it can run Linux 2.1.12x kernels.
  • Relatively modern compiler (can compile new kernels).
  • Newest linux 2.1.xxx kernel. I am using 2.1.129 presently, it seems to work Ok, though no extensive tests.
  • Ok, you've got a little choice on this one. Either obtain the X333servonly.tgz package from ftp.xfree86.org (X server sources) or find a pre-compiled binary for the FBDev server. I have one, so if you can't find it lemme know. I may post it on my web site. If you get the sources, you'll need to compile it yourself. You may also need to patch fbdev.c -- it got a SEGV on me without some changes.

Ignoring the "modern distribution" requirement, this is about 18 megs of stuff to download. So if you've only got a modem, start it going and go to bed for the evening or something =)

Ok, first thing to do is compile and install a new kernel. This is mostly covered elsewhere; the pertinent info is that under "code maturity" you need to enable "prompt for development and/or incomplete drivers" (duhh, it's a 2.1 kernel =), and under "Console drivers" you need to enable "video mode selection support", "support for frame buffer devices", and "VESA VGA graphics console". Compile and install your kernel as per usual. You may want to take a gander at Documentation/fb/vesafb.txt (in the Linux sources) so you know what's going on. Pick a nice video mode (I highly recommend 640x480x16b for subtitling, something bigger for normal usage, which I don't recommend because it's slow =) and reboot.

When the kernel starts up, you'll need to give it the command line parameter "vga=ask". Hit <ENTER> at the prompt to make it ask you, and then type the mode number you figured out. E.G., 640x480x16b would be "311" (what I use). Hit ENTER again and your computer will start to boot with a beautiful penguin logo at the top (aren't you glad you got into this?? ^_-; Now you know what the LinuxPPC people have been holding out on us, because they use this same code for their consoles.)

Now get ready to Be Impressed Version 2.0. Run DOSEMU from the console (as before). If you have a TV output hooked up, it'll be following along with you hopefully. Run M64DIAG and select the video mode matching what you told your Vesa frame buffer startup (in my example, 640x480, 16 True color 565). Exit that menu and go do the "video capture" thing in VT extensions. Now switch consoles! Bang. You should be watching your video input in the background of the console. DON'T SWITCH BACK TO DOSEMU! If you do, you'll lose it all. Instead, do a "killall dos" (fun, wasn't it? =) and switch consoles a few times to clear everything up. Of course, now you're stuck with TV input. But hey, if you were following my example, you are also looking at TV input, going through your Linux box, and out to the TV! Yes, you are now genlocking your Linux console onto the video stream! (whoopiee... =)

The key to this whole thing, as you have probably figured out, is making the ATI BIOS take care of video mode switching and TV inputs, without it realizing it's being hijacked for a much cooler purpose =). And now we come to what was basically the impossible part before, but which is now feasable... the X server!

The FBDev server was designed specifically for usage with the frame buffer devices. XFree86 3.3.3 and above have proper support for this server and the vesafb device in Linux (which we are using). Thus you really need this version (3.3.3). If you have the precompiled binary from somewhere, skip down past the part describing configuration and compilation (next few paragraphs).

Remember how I said you should have RedHat 5.2 or something similar? Well, here's another place it comes in handy. You should have the "Vesafb" mini-howto for reference. In RedHat 5.2, that's located at /usr/doc/HOWTO/mini/Vesafb. This file actually has the info you need to obtain and compile a shiny new XF86_FBDev server. You'll want to get this thing compiled and then move it into /usr/X11R6/bin (where your XF86_Mach64 is currently sitting). Although it looks pretty arcane, it is really very straightforward -- edit the config file the howto mentions (xc/config/cf/xf86site.def) and change the options you want to change. The most important is to enable building of the FBDev server. You can also build a new Mach64 server if you want a real one with internationalization support.

On the kernel I have, the kernel call to do panning around a virtual screen with a vesafb is apparently not implemented, so the server will segfault on startup. To fix this (I recommend it since you don't need this feature), edit the file xc/programs/Xserver/hw/xfree86/fbdev/fbdev.c -- go to the function "fbdevAdjustFrame" and comment out the "ioctl" line (around line 1324). Save that and exit. Note that my pre-compiled binary has these changes in it (and some debug info you can ignore...)

Now go to the xc/ directory (root of the X source tree) and type:

make World >& world.py &

That'll pipe the output into a file you can tail -f if you want, or leave it in the background. Once that's done, you can "make install" but I recommend installing the binary itself manually. You'll find it as xc/programs/Xserver/XF86_FBDev.

Ok, everyone should come back to this point (compile or pre-compiled). I'm assuming that if you got the pre-compiled binary, it went into /usr/X11R6/bin. Look in /etc/X11 and find the link "X". It should point to your current X server (XF86_Mach64 if you're set up right). rename that link to "X.old" and then make a new link like so: 

 cd /etc/X11
 mv X X.old
 ln -s ../../usr/X11R6/bin/XF86_FBDev X

You'll need to do one more thing, which is edit your /etc/XF86Config file and add a section for the fbdev device. This doesn't seem to be documented very well (even as well as compiling your own FBDev server), so here's how you do it.

Add this section next to your existing "Device" section: 

 Section "Device"
    Identifier   "Linux Frame Buffer Device"
 EndSection

And add this at the bottom: 

 # Framebuffer device
 Section "Screen"
     Driver      "fbdev"
     Device      "Linux Frame Buffer Device"
     Monitor     "<MONITOR NAME>"
     DefaultColorDepth 16
     Subsection "Display"
         Depth       8 
         Modes       "default"
     EndSubsection
     Subsection "Display"
         Depth       16
         Modes       "default"
     EndSubsection
     Subsection "Display"
         Depth       32
         Modes       "default"
     EndSubsection
 EndSection

Replace "<MONITOR NAME>" in the above with whatever your real XF86Config option for your regular X server says (it's installation dependant!). You should also use a different number for "16" in the above if you use a different color depth.

Hold your breath and type 'startx'. You should be presented with an X display. Note that it is VERY important that *NO MODE CHANGES* happen when you do this. If they do, something is wrong (maybe your Mach64 server is still active somehow...)

Now what we're gonna do is in principle the same as before, except this time we'll switch to the console containing the vesafb X server instead of just another Linux console.

Start X windows, and for good measure, get to a shell and type "xsetroot -solid black". That will chromakey your background. Sorry, you can't change the key color... =( Use off-black for your fonts. =)

Now switch to another console (e.g. CTRL-ALT-F1). Start up DOSEMU as before, and run M64DIAG, select your video mode, start video capture. Now switch to the X11 console (CTRL-ALT-F7). If all went well, you should be looking at genlocked video through your X setup.

Caveats

Since the first method is one big caveat, I am ignoring it and proceeding to the second one. =)

The first and foremost is the major pain you have to go through to get this working, even once you get past the initial setup. It's going to involve a reboot (unless you're crafty) and some config swapping each time you want to switch between subtitling mode and normal mode.

The second caveat is that the vesafb server is VERY slow compared to the Mach64 server. If you try this you'll find that out *really* fast. Remember back when your windows used to jerk around while redrawing? Hellloooo again =). PRO might have a better time at it, my regular AiW didn't.

I mentioned craftiness above... well here's another possibility. The Mach64 X server doesn't disagree with the vesafb driver in the kernel. So what you can do (in theory) is start up with the vesafb, but run your normal Mach64 server (which will clobber the vesafb console, so you lose your system console... oh whoop.. use xdm =P =). Now you have a relatively normal setup, and when you want to do subtitling, you switch to your now defunct console (maybe! I have seen DOSEMU at least restore the vesafb console successfully!) and kill X. Swap out X servers (change the symlink in /etc/X11). Start up DOSEMU and get your subtitling video mode going. Now when you switch consoles from DOSEMU the vesafb console will be back like normal and you can start up the vesafb X server. When finished, kill the X server, start up DOSEMU again, and turn off the TV stuff. Then exit DOSEMU and start up your normal X config again.

Geez, that's all a pain, but unless you want to try and interpret the several thousand line port traces I got from DOSEMU... ^_^;

Conclusion

ATI, you bum-heads, get us some Linux support for this card!! Until then I guess we'll keep on using stupid workarounds like this one.

If you had troubles or have comments, please email me!

Also, check out the BakaSub web page (if you haven't done so) at: