Pete Finnigan's Oracle Security Weblog

OK, what are encoded or decoded shell codes? - First I had better say what shell code is. Shell code is the instructions (machine instructions) that a hacker sends to a server via a buffer overflow or similar to gain control of the server. This is a common practice in hacking computers. A hacker finds an incorrectly written piece of code in an application that allows you to send incorrect input that in turn ends up with the server running your own machine code rather than the application processing some business function.

Oracle seems to be very susceptible to this kind of attack in recent times. A lot of the alert 68 issues seem to be cases of PL/SQL built in procedures being exploited by sending long strings to them. This kind of attack can be used to send shell code via a PL/SQL function to the database server. So how does a hacker do this? Well part of the long string that the hacker would pass to a vulnerable PL/SQL function or procedure parameter would be shell code. A buffer overflow works because the string passed overflows the end of the buffer assigned to handle it and may go on to overflow a return address of a function on the machines stack (There are also other ways that this kind of overwrite attack can work but let's stick to this one for now). The idea is to get an executable instruction that is supplied by the hacker into the CPU. When the function returns the server will execute the hackers code instead. This kind of attack works because a hacker is able to send machine code to the server, either through a formal program parameter or as other supplied input or possibly via PL/SQL functions or procedures.

Applications are becoming more clever and various filters between the hacker and the server aim to filter out anything that could be machine code. This is where an encoding loop comes in, as it allows the hacker to use valid character sets such as A-Z0-9a-z and then the decoder takes care of making it executable.

http://www.edup.tudelft.nl/~bjwever/whitepaper_shellcode.html - (broken link) I found this paper by Berend-Jan Wever (skylined) that is excellent. It gives some good ideas on how to write a decoder loop and gives some example source code at the end. This paper build on two previous papers, the first by Rix called "Writing ia32 alphanumeric shellcodes" and the second by obscou called http://www.phrack.org/show.php?p=61&a=11 - (broken link) Building IA32 'Unicode-Proof' Shellcodes. It is worth reading these two papers first before skylineds paper.

If you want to understand how a hacker thinks and to understand the lengths that they will go to exploit applications including getting past filters then you need to read these papers. The two by Rix and obscou are superb and well worth reading. The paper by skylined that I found first is also excellent.