Polyalphabetic substitution is, as it name says, a substitution cipher
(letters are replaces by other letters) where multiple alphabets are used,
unlike the simple shift cipher, where we use only one alphabet.
I'll show you an example of the Vigenère cipher, since it's the most well-known
among all polyalphabetic substitution ciphers.
The Vigenère cipher is harder to crack than a standard shift cipher, since we
can't break it using standard letter frequency analysis.
Check out Wikipedia, for the Charless Babbage and Friedrich Kasiski cryptoanalysis.
Let's see an example: The plaintext "APPLE IS RED", encrypted with the key "MANGO"
will become "MPCRS US EKR".
As you can see, the letters "PP" from "APPLE" will become "PC" when encrypted using
the key "MANGO". Why? Here's a small scheme:

Plaintext: APPLEISRED
Key: MANGOMANGO
Ciphertext: MPCRSUSEKR

So, the key decides how many places will the letters be shifted.
The letter "A" has the value 0, "B" is 1, and so on. When we add 0 to "P" it will be
"P" in the ciphertext, because nothing has changed.
But, if we would add "B" to "P", it would result in "Q", the next letter to "P" in
the alphabet. I think you get the idea of the Vigenère cipher now.
If the key is not long as the plaintext itself (like in this example), it is repeated.
However, if you want better security, your key should be as long as the plaintext.
Now, enough of "blablabla", let's see the code:

#include <stdio.h>
#include <string.h>

int main(int argc, char * argv[]){

int pos, i=0, key, r;
char key_buff[255];

if(argc != 3){
printf("Usage: %s [text] [key]\n", argv[0]);
return 1;
}
while(strlen(argv[2]) < strlen(argv[1])){ /* Extend the key */
strcpy(key_buff, argv[2]);
strcat(argv[2], key_buff);
}

while(argv[1][i] != '\0'){
pos = argv[1][i];
key = argv[2][i] - 65; /* Make the key */
if(isupper(pos)){
r = pos + key;
if(r > 'Z')
r-=26;
printf("%c", r); /* Main stuff */
}
i++;
}
return 0;
}

The int "pos" stores the ASCII code of the plaintext array's current character.
"i" stores the position in the plaintext string and in the key.
The value of "key" decides how many places will the characters will be shifted.
"r" is the resulting character's ASCII code.
"key_buff" is needed to extend the key if it's shorter than plaintext.
Two parameters are required, the plaintext and the key. After we get them,
we need to check if the key is shorter than the plaintext. If it is, we need to extend it by
doubling it. This means, that "MANGO" will become "MANGOMANGO".
If the key is longer than the plaintext, we simply won't use the uneeded characters from the
end of the key.
Now we enter a while loop and sum "pos" with "key" to get the ciphertext "r".
65 is substracted from the ASCII code of the key character, because in this code we work
only with upper case letters - the lowest upper case letter is "A", with the ASCII value of 65.
And since we want our ciphertext in upper case letters, we can't add more than 25 to "pos".
Note that in this example I only showed you how to encrypt. Feel free to figure out the
decryption routine (it's simple).