Is there any reason that what you're doing can't be accomplished using a Matrix field?
If the answer is "no, a Matrix field isn't sufficient", then you'll need to build a custom plugin to handle it. Assuming that this is the path that you are taking, you'll want to get familiar with this file:
/app/templates/_includes/forms.html
These macros will help you to generate just about every field type that Craft has natively available.
I built a plugin awhile back which puts a select dropdown menu inside of a table. The table macro is pre-rendered via PHP, turning it into a string. I then output that string directly into my actual page template.
To see exactly how I did it, check out this example code...
This may not answer your question 100%, but hopefully it points you in the right direction.