Advanced Malware Analysis at Georgia Tech
07 Dec 2025
This course was my introduction to malware analysis and all of the labs were done with a real malware sample. I had the opportunity to work with x86 Assembly, Ghidra, Pin, and was introduced to the concepts of data/control dependence, symbolic execution, malware sandboxes, and anti-analysis techniques. Unfortunately, I am unable to explain any of the course details, but I can outline some of the valuable experience I've gained.
What I Learned:
- x86 Assembly/Static Analysis
- Windows Calling Conventions
- Writing Ghidra Plugins
- Binary Instrumentation
- Other Topics and Future Work
x86 Assembly/Static Analysis
Static analysis required using Ghidra to manually inspect all the Assembly instructions of the malware sample, which gave me enough experience to feel very comfortable reading low-level code and recognizing what constructs from higher-level languages look like in Assembly. This also gave me a solid understanding of the x86 architecture and how various instructions affect the environment. I paid careful attention to how every instruction affected the stack, pointers, and registers, and this knowledge helped me when writing Ghidra plugins where I had to implement a shadow stack. Static analysis also helped to become familiar with the structure of a PE file and even gave me some experience reverse engineering data obfuscation techniques.
Windows Calling Conventions
Windows 32-bit calling conventions were new to me in this course and they're now committed to memory. I found it interesting how different conventions might use different registers for parameters or returns, even depending on the number of arguments like in thiscall and fastcall, or how some conventions clean up the stack and others do not. Understanding these calling conventions in detail familiarized me even more with the x86 architecture and made me think about how a malware author might use some of these details to confuse an analyst.
Writing Ghidra Plugins
I had to write two Ghidra plugins for this course. The first was a def/use plugin that iterated over all instructions to track where registers and memory locations were defined and used. The second plugin took the logic from the def/use plugin and used it to calculate data dependencies of instructions. Of course these types of plugins are not anything new, but really solidified the concepts of data flow and dependencies in my mind.
Binary Instrumentation
I also gained experience using Intel's Pin to instrument the malware sample, which was also a new concept to me. I wrote a Pin tool that output a control flow graph and although this was a fairly simple assignment, it made me understand the power of being able to intercept every instruction of a binary and write my own functions around them. My imagination ran wild with the possibilities of this type of analysis and I look forward to using it more in future analysis.
Other Topics and Future Work
Aside from the projects, I read over 30 studies concerning binary analysis and familiarized myself with the techniques used by both defenders (symbolic/concolic execution, forced execution, machine learning) and attackers (anti-analysis techniques, packing/obfuscation, virtualization). Now that this course is complete I will set up my own personal malware analysis environment to begin working on the labs in the book Practical Malware Analysis and other malware samples. I also plan to use this knowledge for my work on bug bounty programs, where I will concentrate on binary analysis in the future. I look forward to continuing to educate myself in this field and pursue it as a career.