# File type file challenge # ELF 64-bit, x86-64 → Linux binary # PE32+ → Windows binary # Mach-O → macOS # Java class → javap / jadx # Python bytecode .pyc → uncompyle6 # Strings — flag or hints often visible strings challenge | grep -i "flag\|picoCTF\|pass\|key\|correct\|wrong" strings -el challenge # wide strings (Unicode) # Quick run (sandbox if suspicious) chmod +x challenge && ./challenge # ltrace to see library calls ltrace ./challenge strace ./challenge
# ELF header info readelf -h challenge # header readelf -S challenge # sections readelf -s challenge # symbol table readelf --all challenge # Dynamic symbols / imports objdump -d challenge # disassemble objdump -T challenge # dynamic symbols nm challenge # symbol table ldd challenge # linked libraries # Check security mitigations checksec challenge # NX, PIE, RELRO, stack canary, FORTIFY
# Disassemble all objdump -d challenge objdump -d -M intel challenge # Intel syntax # Specific function objdump -d challenge | grep -A 30 "<main>" # Data section (look for encoded strings) objdump -s -j .data challenge objdump -s -j .rodata challenge # Hex dump xxd challenge | head -40 hexdump -C challenge | head
| Tool | Use |
|---|---|
| Ghidra | Free NSA decompiler, excellent C output |
| IDA Free | Industry standard, free tier for x86-64 |
| Binary Ninja | Clean UI, MLIL IR, good for CTF |
| Cutter | GUI for radare2, free |
| radare2 | CLI powerhouse, steep learning curve |
| jadx | Android APK / Java class decompiler |
| dotPeek / dnSpy | .NET / C# decompiler |
| uncompyle6 | Python .pyc → .py source |
| RetDec | Online decompiler (retdec.com) |
# 1. Import binary → Auto-analyze (accept defaults) # 2. Find main / interesting functions # Window → Functions → search "main", "check", "verify" # 3. Rename variables for readability # Right-click → Rename Variable (L key) # 4. Re-type variables for better decompilation # Right-click → Retype Variable # 5. Find strings and cross-reference # Search → For Strings # Right-click string → References → Show References # 6. Patch bytes (disable checks) # Right-click instruction → Patch Instruction # Headless Ghidra (CLI) analyzeHeadless /tmp/proj myproj \ -import challenge -postScript PrintAST.java
| Pattern | What it means |
|---|---|
| strcmp / strncmp | Password/flag comparison |
| Loop + XOR | Simple decode routine — extract key |
| if (result != 0) fail | Condition to patch: JNZ → JZ |
| Global array + index | Lookup table / encoded data |
| scanf / gets / fgets | User input point |
| puts("Correct!") | Success branch — trace backwards |
| MD5/SHA init consts | Hash being computed on input |
| ptrace(0,0,0,0) | Anti-debug check |
# Start gdb ./challenge gdb -q ./challenge # quiet r < input.txt # run with input r arg1 arg2 # with args # Breakpoints b main # break at main b *0x401234 # break at address b strcmp # break at function info break delete 1 # Step / continue c # continue n # next (step over) s # step (step into) ni # next instruction si # step instruction finish # run to end of function
# Registers info registers p $rax # print register p/x $rsp # hex format # Memory x/20x $rsp # 20 hex words at rsp x/s 0x404060 # string at address x/20i $rip # disassemble 20 instr # pwndbg-specific (if installed) context # full context view telescope $rsp # smart stack view search -s "flag" # search memory vmmap # memory map # Modify values set $rax = 0 set *(int*)0x404060 = 1 patch byte 0x401234 0x90 # NOP an instruction
# Break on strcmp to see what's being compared b strcmp r # When hit: rdi = string1, rsi = string2 x/s $rdi x/s $rsi # Patch a conditional jump → always true # Find: jne 0x40xxxx set *(unsigned char*)0x401234 = 0x74 # jne → je # Or NOP it (0x90) # Trace all calls set pagination off catch syscall commands backtrace continue end # Record & replay (rr) rr record ./challenge rr replay
# Trace all library calls (strcmp, malloc, etc.) ltrace ./challenge ltrace -s 100 ./challenge # longer strings ltrace -e strcmp ./challenge # only strcmp ltrace -f ./challenge # follow forks # When binary checks your input vs flag: # strcmp("your_input", "picoCTF{...}") = ... # The flag appears in plaintext! # With arguments ltrace ./challenge myinput echo "myinput" | ltrace ./challenge
# Trace all syscalls strace ./challenge strace -s 200 ./challenge # longer strings strace -e read,write ./challenge strace -e openat ./challenge # files opened # Useful: see what files are read/written # openat(AT_FDCWD, "/flag", O_RDONLY) ← path! # Attach to running process strace -p <pid>
| Register | Purpose |
|---|---|
| rax | Return value, accumulator |
| rdi | Arg 1 |
| rsi | Arg 2 |
| rdx | Arg 3 |
| rcx | Arg 4 |
| r8, r9 | Arg 5, 6 |
| rsp | Stack pointer |
| rbp | Base (frame) pointer |
| rip | Instruction pointer |
| Instruction | Meaning |
|---|---|
| mov dst, src | dst = src |
| lea dst, [addr] | dst = address (not value) |
| cmp a, b | sets flags: a-b (doesn't store) |
| test a, b | sets flags: a&b |
| je / jz | jump if equal / zero flag set |
| jne / jnz | jump if not equal |
| jg / jl | jump greater / less (signed) |
| xor a, a | zero a register |
| push / pop | stack operations |
| call / ret | function call / return |
| Technique | Bypass |
|---|---|
| ptrace(0,0,0,0) | Patch ret val to 0, or LD_PRELOAD fake ptrace |
| /proc/self/status TracerPid | Patch strcmp or LD_PRELOAD fake open |
| Timing check (rdtsc) | Patch jle/jge after timing compare |
| SIGTRAP handler | NOP the int3 or handle signal |
| IsDebuggerPresent (Windows) | Patch return value to 0 |
| Checksum on code | Recalculate after patching |
# Detect packer strings packed | grep -i "UPX\|packed\|compress" detect-it-easy challenge # die / diec exeinfo challenge # UPX unpack upx -d challenge -o unpacked # Run packed binary, dump from memory # Set breakpoint at OEP (original entry point) # Then dump with: dump binary memory out.bin start end # Python: deobfuscate XOR loop python3 -c " data = [0x41, 0x62, 0x23, ...] # from .data section key = 0x42 print(bytes(b ^ key for b in data)) "
# Decompile back to source uncompyle6 challenge.pyc pycdc challenge.pyc # alternative decompile3 challenge.pyc # Python 3 # If uncompyle6 fails: dis # Python disassembler (built-in) python3 -c " import dis, marshal, struct f = open('challenge.pyc', 'rb') f.read(16) # skip header code = marshal.loads(f.read()) dis.dis(code) "
# Java .class → source javap -c Challenge.class # bytecode jadx Challenge.class # full decompile cfr Challenge.class # alternative # Android .apk jadx -d output/ app.apk # Then browse Java source in output/sources/ # APK as zip unzip app.apk -d apk_contents/ strings apk_contents/classes.dex | grep "flag"
# .NET → C# source dnSpy challenge.exe # GUI decompiler ilspy challenge.exe # or ILSpy # Bash script obfuscation # Look for eval $(base64 -d <<< ...) patterns # Replace eval with echo to see decoded command # JavaScript (Node) node --inspect-brk challenge.js # debug # Use Chrome DevTools for debugging # WebAssembly .wasm wasm2wat challenge.wasm # binary → text wabt tools # wabt toolkit
file + strings | grep flag
② ltrace ./chal — see strcmp arguments
③ strace ./chal — file access, syscalls
④ checksec — mitigations
⑤ Ghidra: find main → trace to "Correct!" output
⑥ GDB: break strcmp → x/s $rdi, x/s $rsi
⑦ Patch: change jne→je to bypass check
⑧ XOR loop in decompiler → extract key + data → decode