BackupExec 12.5 does support agents for backing up ESXi, the kicker is you need to have additional software from VMware in order for the agent to work.
As things sit today, you use agents to backup Windows regardless of whether or not the Windows in question is on physical or virtual hardware. In the case of your VMware Server guest VM’s, you can back up their constituent VMware Server files IF the guest VM is shutdown, the files are then backed up at the Windows level on the host machine.
ESX and ESXi are bare-metal hypervisors so Backup Exec needs a way to get at the VMDK files (and other constituent files) if you wish to back up at the ESX level; regular Windows backups happen just as they do now with the Windows agents. VMware supplies an API for backups at the host machine level through the VCB framework supplied with their Virtual Machine Infrastructure. This infrastructure is “paid for” software — ESXi is “free” but you have to purchase the infrastructure if you want additional management features or the VCB framework. The BackupExec agent for ESX talks to the VCB framework and is supposed to allow you to then take backups of your guests at the VMDK level. I say “supposed to” because, frankly, we have not had much success in getting it to work reliably. VCB works as we have used the framework at one client to script backups manually and we have successfully used another company’s backup software for ESX that relies on VCB.