Sybase Adaptive Server Enterprise SHA256-based password hashing algorithm
Authors: Laszlo Toth, Marcell Major
Sybase ASE versions before version 15.0.2 used a proprietary hashing algorithm for storing database user passwords in the table syslogins. From version ASE 15.0.2, user passwords are stored in a new SHA256-based format. The old password hashes stored in the proprietary format can be used in newer versions if the "allow password downgrade" parameter is not set to zero.
Sybase ASE System Administration Guide  contains a high level description of the new SHA256-based format. Using that description, the actually stored hashes cannot be reproduced, the information seems to be inaccurate. This article presents the algorithm in detail.
Steps of the hash generation (the sample password "test12" is used):
1. Convert the password to UTF-16 format (Big Endian):
"test12" -> "\x00t\x00e\x00s\x00t\x001\x002"
2. Append zero bytes as padding, so that the length of the result is 510 bytes:
buffer = "\x00t\x00e\x00s\x00t\x001\x002\x00\x00\x00\x00\x00\x00\x00..."
3. Append the 8-byte salt, thus the length of the result is 518 bytes:
buffer = "\x00t\x00e\x00s\x00t\x001\x002\x00\x00\x00\x00\x00\x00\x00...\x00\x4f\x9c\xc8\xc0\xd5\x5d\x98\x03"
4. Generate the SHA256 hash of the data (518 bytes):
result = 0xb0c0816e127f2a56ee080230af5b4ce3da1f3d9fcc5449fcfcf3fb9595eb8ea6
5. Store the salt and the hash in the format described below.
Format of the data stored in syslogins table:
The sample data: 0xc0074f9cc8c0d55d9803b0c0816e127f2a56ee080230af5b4ce3da1f3d9fcc5449fcfcf3fb9595eb8ea6
Composed of the following:
-2 constant bytes: 0xc007
-8 bytes long salt: 4f9cc8c0d55d9803
-32 bytes long hash: b0c0816e127f2a56ee080230af5b4ce3da1f3d9fcc5449fcfcf3fb9595eb8ea6
 Adaptive Server© Enterprise 15.0.2, System Administration Guide: Volume 1
Sybase password cracker
You can download the proof of concept password cracker with source code for Sybase SHA256 based password hashes using the link bellow. This version is beta and not yet optimized for speed. The code is based on the woraauthbf Oracle password cracker developed by Laszlo Toth.
Sybcrack windows binary (requires Microsoft Visual C++ 9 runtime library)
Sybcrack source code