SYSCALL / SYSENTER / INT 0x80

SYSCALL is the fast kernel-entry instruction for x86-64 (Long Mode). It replaces the older SYSENTER (32-bit fast call) and INT 0x80 (legacy 32-bit software interrupt) mechanisms.

x86-64 Linux syscall ABI

Register	Role
`rax`	Syscall number
`rdi`	Argument 1
`rsi`	Argument 2
`rdx`	Argument 3
`r10`	Argument 4 (not `rcx` — that's trashed by SYSCALL itself)
`r8`	Argument 5
`r9`	Argument 6
`rax` (return)	Return value (negative errno on error)

rcx and r11 are clobbered by SYSCALL — the CPU saves RIP in rcx and RFLAGS in r11.

Example: write(1, buf, len)

asm

mov  rax, 1          ; SYS_write = 1
mov  rdi, 1          ; fd = stdout
lea  rsi, [rip + buf]; pointer to data
mov  rdx, 13         ; length
syscall

32-bit Linux INT 0x80 ABI (legacy)

asm

; 32-bit ABI — different register layout AND different syscall numbers
mov eax, 4           ; SYS_write = 4  (not 1!)
mov ebx, 1
mov ecx, buf
mov edx, len
int 0x80

Key difference: INT 0x80 uses ebx, ecx, edx, esi, edi, ebp; SYSCALL uses rdi, rsi, rdx, r10, r8, r9. The syscall numbers also differ entirely between the two ABIs.

Reverse-engineering notes

When analysing shellcode, the calling convention (rdi/rsi vs ebx/ecx) immediately reveals whether it targets the 64-bit or 32-bit ABI.
int 0x80 in a 64-bit binary is valid but unusual — often seen in shellcode that was ported from 32-bit or in intentionally compact stagers.
The vsyscall / vDSO page maps common syscalls (like gettimeofday) as regular functions to avoid the kernel-transition overhead; calls to 0xFFFFFFFFFF600000 are vsyscall stubs, not direct SYSCALL instructions.
On Windows, the syscall number (SSN) varies per OS version and patch level — making direct syscall in user-mode shellcode (a "direct syscall") a technique for bypassing security tools that hook ntdll.dll.