reverse::engineering

reverse static · dynamic x86 · ARM
First StepsStatic Analysis GhidraGDBltrace/strace x86 AssemblyAnti-debug Other FormatsTools
01First Steps
Identify the binary
# File type
file challenge
# ELF 64-bit, x86-64 → Linux binary
# PE32+                → Windows binary
# Mach-O               → macOS
# Java class           → javap / jadx
# Python bytecode .pyc → uncompyle6

# Strings — flag or hints often visible
strings challenge | grep -i "flag\|picoCTF\|pass\|key\|correct\|wrong"
strings -el challenge  # wide strings (Unicode)

# Quick run (sandbox if suspicious)
chmod +x challenge && ./challenge
# ltrace to see library calls
ltrace ./challenge
strace ./challenge
ELF headers & symbols
# ELF header info
readelf -h challenge      # header
readelf -S challenge      # sections
readelf -s challenge      # symbol table
readelf --all challenge

# Dynamic symbols / imports
objdump -d challenge      # disassemble
objdump -T challenge      # dynamic symbols
nm challenge              # symbol table
ldd challenge             # linked libraries

# Check security mitigations
checksec challenge
# NX, PIE, RELRO, stack canary, FORTIFY
02Static Analysis
objdump
# Disassemble all
objdump -d challenge
objdump -d -M intel challenge  # Intel syntax

# Specific function
objdump -d challenge | grep -A 30 "<main>"

# Data section (look for encoded strings)
objdump -s -j .data challenge
objdump -s -j .rodata challenge

# Hex dump
xxd challenge | head -40
hexdump -C challenge | head
Decompilers
ToolUse
GhidraFree NSA decompiler, excellent C output
IDA FreeIndustry standard, free tier for x86-64
Binary NinjaClean UI, MLIL IR, good for CTF
CutterGUI for radare2, free
radare2CLI powerhouse, steep learning curve
jadxAndroid APK / Java class decompiler
dotPeek / dnSpy.NET / C# decompiler
uncompyle6Python .pyc → .py source
RetDecOnline decompiler (retdec.com)
03Ghidra Workflow
Key workflow steps
# 1. Import binary → Auto-analyze (accept defaults)

# 2. Find main / interesting functions
#    Window → Functions → search "main", "check", "verify"

# 3. Rename variables for readability
#    Right-click → Rename Variable (L key)

# 4. Re-type variables for better decompilation
#    Right-click → Retype Variable

# 5. Find strings and cross-reference
#    Search → For Strings
#    Right-click string → References → Show References

# 6. Patch bytes (disable checks)
#    Right-click instruction → Patch Instruction

# Headless Ghidra (CLI)
analyzeHeadless /tmp/proj myproj \
  -import challenge -postScript PrintAST.java
Common patterns to spot
PatternWhat it means
strcmp / strncmpPassword/flag comparison
Loop + XORSimple decode routine — extract key
if (result != 0) failCondition to patch: JNZ → JZ
Global array + indexLookup table / encoded data
scanf / gets / fgetsUser input point
puts("Correct!")Success branch — trace backwards
MD5/SHA init constsHash being computed on input
ptrace(0,0,0,0)Anti-debug check
04GDB / GDB-pwndbg
Essential commands
# Start
gdb ./challenge
gdb -q ./challenge        # quiet
r < input.txt             # run with input
r arg1 arg2               # with args

# Breakpoints
b main                    # break at main
b *0x401234               # break at address
b strcmp                  # break at function
info break
delete 1

# Step / continue
c                         # continue
n                         # next (step over)
s                         # step (step into)
ni                        # next instruction
si                        # step instruction
finish                    # run to end of function
Inspect state
# Registers
info registers
p $rax                    # print register
p/x $rsp                  # hex format

# Memory
x/20x $rsp                # 20 hex words at rsp
x/s 0x404060              # string at address
x/20i $rip                # disassemble 20 instr

# pwndbg-specific (if installed)
context                   # full context view
telescope $rsp            # smart stack view
search -s "flag"          # search memory
vmmap                     # memory map

# Modify values
set $rax = 0
set *(int*)0x404060 = 1
patch byte 0x401234 0x90  # NOP an instruction
CTF tricks
# Break on strcmp to see what's being compared
b strcmp
r
# When hit: rdi = string1, rsi = string2
x/s $rdi
x/s $rsi

# Patch a conditional jump → always true
# Find: jne 0x40xxxx
set *(unsigned char*)0x401234 = 0x74  # jne → je
# Or NOP it (0x90)

# Trace all calls
set pagination off
catch syscall
commands
  backtrace
  continue
end

# Record & replay (rr)
rr record ./challenge
rr replay
05ltrace / strace
ltrace — library calls
# Trace all library calls (strcmp, malloc, etc.)
ltrace ./challenge
ltrace -s 100 ./challenge    # longer strings
ltrace -e strcmp ./challenge # only strcmp
ltrace -f ./challenge        # follow forks

# When binary checks your input vs flag:
# strcmp("your_input", "picoCTF{...}") = ...
# The flag appears in plaintext!

# With arguments
ltrace ./challenge myinput
echo "myinput" | ltrace ./challenge
strace — syscalls
# Trace all syscalls
strace ./challenge
strace -s 200 ./challenge    # longer strings
strace -e read,write ./challenge
strace -e openat ./challenge # files opened

# Useful: see what files are read/written
# openat(AT_FDCWD, "/flag", O_RDONLY) ← path!

# Attach to running process
strace -p <pid>
06x86-64 Assembly Quick Ref
Registers & calling convention
RegisterPurpose
raxReturn value, accumulator
rdiArg 1
rsiArg 2
rdxArg 3
rcxArg 4
r8, r9Arg 5, 6
rspStack pointer
rbpBase (frame) pointer
ripInstruction pointer
Key instructions
InstructionMeaning
mov dst, srcdst = src
lea dst, [addr]dst = address (not value)
cmp a, bsets flags: a-b (doesn't store)
test a, bsets flags: a&b
je / jzjump if equal / zero flag set
jne / jnzjump if not equal
jg / jljump greater / less (signed)
xor a, azero a register
push / popstack operations
call / retfunction call / return
07Anti-Debug / Obfuscation
Anti-debug techniques & bypass
TechniqueBypass
ptrace(0,0,0,0)Patch ret val to 0, or LD_PRELOAD fake ptrace
/proc/self/status TracerPidPatch strcmp or LD_PRELOAD fake open
Timing check (rdtsc)Patch jle/jge after timing compare
SIGTRAP handlerNOP the int3 or handle signal
IsDebuggerPresent (Windows)Patch return value to 0
Checksum on codeRecalculate after patching
Packing & obfuscation
# Detect packer
strings packed | grep -i "UPX\|packed\|compress"
detect-it-easy challenge     # die / diec
exeinfo challenge

# UPX unpack
upx -d challenge -o unpacked

# Run packed binary, dump from memory
# Set breakpoint at OEP (original entry point)
# Then dump with: dump binary memory out.bin start end

# Python: deobfuscate XOR loop
python3 -c "
data = [0x41, 0x62, 0x23, ...]  # from .data section
key = 0x42
print(bytes(b ^ key for b in data))
"
08Other File Formats
Python .pyc
# Decompile back to source
uncompyle6 challenge.pyc
pycdc challenge.pyc       # alternative
decompile3 challenge.pyc  # Python 3

# If uncompyle6 fails:
dis  # Python disassembler (built-in)
python3 -c "
import dis, marshal, struct
f = open('challenge.pyc', 'rb')
f.read(16)  # skip header
code = marshal.loads(f.read())
dis.dis(code)
"
Java / Android
# Java .class → source
javap -c Challenge.class    # bytecode
jadx Challenge.class        # full decompile
cfr Challenge.class         # alternative

# Android .apk
jadx -d output/ app.apk
# Then browse Java source in output/sources/

# APK as zip
unzip app.apk -d apk_contents/
strings apk_contents/classes.dex | grep "flag"
.NET / Scripting
# .NET → C# source
dnSpy challenge.exe        # GUI decompiler
ilspy challenge.exe        # or ILSpy

# Bash script obfuscation
# Look for eval $(base64 -d <<< ...) patterns
# Replace eval with echo to see decoded command

# JavaScript (Node)
node --inspect-brk challenge.js  # debug
# Use Chrome DevTools for debugging

# WebAssembly .wasm
wasm2wat challenge.wasm    # binary → text
wabt tools                 # wabt toolkit
RE CHECKLIST →  ① file + strings | grep flag  ② ltrace ./chal — see strcmp arguments  ③ strace ./chal — file access, syscalls  ④ checksec — mitigations  ⑤ Ghidra: find main → trace to "Correct!" output  ⑥ GDB: break strcmp → x/s $rdi, x/s $rsi  ⑦ Patch: change jneje to bypass check  ⑧ XOR loop in decompiler → extract key + data → decode