Malware API Logs Dataset

The problem with malware detection or classification research is that it does not have STANDARD dataset.

So I try to accumulate several datasets that being used by researchers so that you will be able to replicate and present the method that you proposed for your research.

This is important so that the examiners will not refuting the credibility of your methods or algorithms during your viva-voce

From NEXGINRC

This list will be updated from time to time based on your inputs ;)

"Becoming a malware analyst"

You want to become a malware analyst?

Read this

---------------------------------------------------------------------------------------------------------------------------------

There are few jobs in this industry that seem as appealing and interesting to me as that of a malware analyst. In my mind, these professionals were waking up each day to continue a complex game not unlike the Glass Bead Game from the eponymous novel by Herman Hesse - a pure pursuit of the mind that makes connections where there are seemingly none, all for the sake of solving intricate puzzles in order to satisfy their curiosity and cravings for intellectual challenges. But I was wrong!

To satisfy my own personal craving to know what it was all about, I decided to contact a number of malware analysts working for some of the most high-profile security companies out there and ask them a few questions.

The traits and skills of good malware analysts

Some malware researchers, like McAfee Lab’s Principal Research Architect Igor Muttik, entered the field in the '80s, when the anti-virus programs were only appearing and there was no multi-billion AV industry yet. Others, like Jana Barborikova, a Junior Virus Analyst at Avast, have been in it for less than a year.

But the one thing they all have in common - beside insatiable curiosity - is the satisfaction of knowing that they are keeping users safe. In fact, the willingness to help people is one of the main qualities of a good malware researcher according to Muttik. "In this regard what we do is very similar to the work of the doctors, police and firefighters," he muses.

"What are the others?" I asked. A high IQ, he says. "Anyone can be a good programmer but to successful in computer security one has to be smarter than the best of the attackers. This requires dedication and the more brain cells you can contribute - the better!"

"Crucial for malware analysts is the ability to get a full overview of what modern malware does, how it does it and why it’s doing it,"Bogdan Botezatu, Senior E-Threat Analyst with Bitdefender, tells me. 

"Patience is also mandatory. Decrypting a piece of malware with server-side polymorphism or tracking down its behavior in a virtualized environment can get extremely frustrating. Last but not least is a strong sense of ethics. The lack of affiliation with black-hat or cybercriminal groups is just a start. Since we’re trusted with lots of confidential information and access to zero-day samples or still unpatched exploit code, we need to know that no employee would use the code for malicious purposes."

He sees the job more as a vocation. "I know quite a few antivirus researchers who are designated economists, MDs or, as in my case, historians or journalists, but are experts on cybercrime. Of course, IT-related educational backgrounds make it easier to learn how computers, operating systems, network communication and applications work, but it is not mandatory."

Kaspersky Lab Senior Malware Analyst Denis Maslennikov agrees. "The most important thing is to be interested in this field, because if you are, this interest will drive you and guide you while you search for new knowledge and experience. It’s more about the knowledge you have than about the diploma. If you have some basic background and are able to learn new stuff you can become malware analyst."

He also reiterates Botezatu's opinion on ethics. "Stay out of the black / grey area. No antivirus company will hire you or trust you with zero-day code if you have worked for or have been affiliated with exploit writers, black hat hackers or unauthorized pen-testers. Most disclosure about ongoing operations follows a strict vouching process in which the candidate receives approval or denial from peers in the industry."

The ability of not letting failure to put you off is another crucial trait according to Barborikova. "An analyst cannot be afraid to try new approaches and think outside of the box."

Finally, you need to be ready and able to communicate. "The best reverse engineer in the world is useless if she cannot report her findings in a clear and concise way," points out Guillaume Lovet, Senior Manager FortiGuard's Labs in EMEA at Fortinet.

But what kind of base knowledge is a must-have? Or, at least, is highly recommended?

"Most malware nowadays requires analysts to understand assembly languages. Learning and understanding this will unlock many doors in the field of malware analysis," says Liam O’Murchu, Manager of Operations, Symantec Security Response.

Barborikova concurs, and that is why she's currently focusing on learning them. "The analyst does nothing without some programming skills, fundamentals of networking and a basic knowledge of operating systems," she adds.

"Reverse engineering – although the focus of antivirus research – is not everything a candidate needs to understand. Most of the time, you will need to build your own tools and extend them to suit your new purposes," Botezatu weighs in.

"If you already understand assembly language, you should start learning a programming language (such as C++ and Python), as you’re going to use it to automate day-to-day tasks, write custom scripts to help you with your work or develop state-of-the-art disinfection routines that will reach millions of customers on the next update."

Lovet agrees, and considers some developer skills in scripting languages almost mandatory. He also points out that - unlike him - not all analysts have been professional C++ developers before becoming analysts, and that he finds this a significant advantage when it comes down to reverse-engineering malware pieces, which are usually coded in C++.

What none of them (or the companies they work for) consider important is having certifications.

"We do not require any certifications for new malware engineers joining our team. The most important thing is to have hands on experience analyzing malware or performing security investigations," says O’Murchu.

How does one become a malware researcher?

The roads that lead to this are many and various. Maslennikov and O'Murchu studied, respectively, information security and computer engineering at college. For the former, the road was very straight - while still at university, he got a call from Kaspersky Lab and was offered a malware analyst position. 

The latter went through several jobs such as a security tester for an internet kiosk company and working at an anti-spam company that was ultimately bought by Symantec. "We were given a tour of the new Symantec offices and as soon as I entered the malware analysis lab I knew that was the job I wanted. I was fortunate to have the opportunity to transfer into that department and short time later and have been here since," he says.

Muttik and Barborikova have an education in natural and formal sciences. Both were interested in a career in information security, and Muttik practiced reverse-engineering viruses as a hobby.

A previously mentioned, Botezatu studied history and journalism, but was also interested in reverse-engineering malware since he was a teenager. Following a stint as a network administrator for his university, he applied for a job at Bitdefender two times. After having overslept and missed the interview the first time, he worked half a year as a tech journalist before trying his luck again. This time, he was recruited by the company's communication team.

"Since joining Bitdefender, I’ve worked in a multitude of fields, from technical communication to anti-malware research and new product development. I grew to understand security from tracking down malicious activity to actually developing solutions to mitigate it, and speaking about developments in the industry at international conferences. As part of a cross-disciplinary team, we’re exposed to everything that happens in the anti-malware field, so we have a full perspective on the industry," he shared.

Lovet became a malware analyst after a 2-year-long developer experience. "Being a developer satisfied my analytic and synthetic mind, as well as my creativity, yet it lacked the 'passion' component," he says, adding that he began working as a Malware Analyst at Fortinet in 2004.

"At the beginning, we'd manually process loads of legacy DOS viruses - because we needed to have detection for those to earn some certifications. These were fun times: studying 20+ viruses per day is the equivalent of playing poker online, at 5 different tables at once: you play a LOT of hands, and gain experience faster," he shares. 

After becoming the AV Team leader, he turned more toward researcher and presenting at international conferences such as AVAR, EICAR, and Virus Bulletin.

"Eventually, I got promoted to AV and IPS team manager, then senior manager. Today, I still do my share of research (last year I presented 2 papers at BlackHat in Amsterdam), and some management of people. This was my choice, in order to diversify my skills. It is perfectly possible to stick to purely technical tasks and progress in the company aside of the management ladder, up to the rank of Fellow, which equates a VP rank in the management career," he points out.

Malware researchers' typical working day and the tools they use

"During my work I deal mainly with web malware," shares Barborikova. "I go through a list of potentially dangerous URLs and select domains which are actually malicious. Then I analyze samples, especially HTML and PHP files. Apart from handy internal tools developed in our virus lab department I use freeware tools like VirtualBox, Process Monitor or Wireshark and online deobfuscators and decoders."

"We use IDA Pro and OllyDbg for reverse engineering. And our own tools for intelligence and monitoring (probes and honeypots), says Lovet. "On the secluded replication machines, where we safely run viruses to study their behavior, we don't use virtual machines, as some malware spots those. On the mobile malware side, we have our own in-lab, secluded GSM network.

We built a base-station with a modified USRP board. The software part is OpenBTS, an open source system. When we register infected phones to that network, we can therefore trace what they do on the network: send SMS, place calls, etc."

IDA and OllyDbg are O’Murchu's "weapons of choice," as well, since they are standards for the two primary tools any malware analyst needs: a disassembler and a debugger.

"We have separate machines that we use for malware analysis. In addition to having all the tools needed for analysis installed, these machines are also isolated with no Internet connection. This prevents any malware from escaping when we are testing it," he says. 

"We generally run the threat to look at observable behaviors first, then dig deeper as needed. Using hex editors and file format parsers and learning about different file formats is also a big part of a malware analyst’s role. For example if a PDF file is being used to distribute a piece of malware, then analyst will need to become familiar with how PDF files are created and how to break them apart."

Botezatu says that he is not sure where his typical day ends and where it starts. "Antivirus research is a 24-hour mission - if your phone rings or the SMS alert beeps in the middle of the night, you take off to work, or at least VPN into the company immediately," he notes. "If an outbreak has been detected, we start developing a removal tool for computer users who are not running a Bitdefender solution.

If everything is running normally, we proceed to solving support tickets, clustering new malware and improving heuristics, while keeping an eye on security (highly private) mailing lists for new samples and developments. You know - the save-the-world-while-having-coffee activities."

"As far as tools are concerned, we’re using a lot of readily available tools such as Far Manager, IDA, Process Explorer, Process Monitor, Malzilla, and Wireshark. But the heavy lifting is done with proprietary tools built in-house, tools that don’t even have names. In the fight against malware, it’s every man for himself, we’re mostly using tools that we develop ad-hoc, ranging from unpackers to utilities for clustering files, rebooting remote machines or controlling operations off-site," he concludes.

Maslennikov says that there is really no typical working day, as there is always something new and / or urgent going on. He does his testing on two desktops with Windows and Linux and a lot of smartphones with different OS, and can't do without the Far file manager, IDA, Hiew and a number of various internal tools. Oh, and coffee - plenty of it, and often.

Finally, I asked them:

What surprised them the most during their current career?

"I have been in the security industry for almost 10 years, and I am continuously surprised by the new attacks the malware cyber criminals dream up," says O’Murchu. "Although the vast majority of attacks are predictable and nothing out of the ordinary, there is always a small number of threats that push the boundaries of what is possible."

The threat that has amazed him the most was Stuxnet. "We had never seen a piece of malware capable of changing how physical machinery works. That was a threat that really pushed the boundaries of what malware can do," he added.

Botezatu has been most surprised by the success of the Slammer worm, the virality of Conficker, the way cyber-crooks made easy money with the Rogue AV campaigns and the complexity of the TDSS family.

"But the piece of malware that went through our hands and surprised the entire world was Flamer, a piece so elegantly designed that it tricked the user into acting as a mule for the stolen data," he shares. "It took espionage to a whole new level: the ability to prioritize importance of stolen data, the way it carried the data to a gateway and the fact that it lacked compression and obfuscation, hiding its code in plain sight. This was clearly not the result of a single man, but rather the work of a team of specialists."

And while Barborikova, who has only been doing this for a year, says that she naturally often encounter things that are new for her, Lovet says that the thing that fascinates him the most is that cybercriminals have not begun exploiting mobile phones earlier. 

"Back in 2006, I predicted that they would, since a smartphone was basically a computer with something more: an integrated payment system (i.e. premium numbers). It'd simplify a lot the business model you need to set up to turn infected machines into cash," he pointed out. 

Nevertheless, it didn't really happen before 2011, and even now, he says, the scale on which it's happening is still moderate as compared to the PC world.

Conclusion

I am very grateful for the peek that these experts gave me into their profession, and I hope that you have enjoyed this as well, especially if you're contemplating a (new) career in malware research. 

Common sense says that cyber attack will never stop, but just become different, so put in you "pros" column the fact that you'll rarely be bored at your job and that, if you become good at it, will probably never experience a lack of job offers.

Computers infected with DNSChanger malware will be blocked from Internet beginning 9th July 2012

What is DNSChanger:
http://download.cnet.com/8301-2007_4-57467164-12/what-the-dnschanger-malware-is-and-why-you-should-care-faq/

From McAffee's FB:

Today is your last chance to check your computer to see if it is infected with the DNSChanger Trojan. If your computer is infected you are at risk of losing Internet access tomorrow. Check to see if you’re infected with the DNSChanger Trojan here: http://www.mcafee.com/dnscheck. If you’re infected you can download McAfee’s free tool, to restore your Internet settings.

http://blogs.mcafee.com/consumer/family-safety/on-july-9th-millions-may-lose-access-to-internet

Check if you're affected:

http://www.mcafee.com/dnscheck

FBI also issued a whitepaper on this:
http://www.fbi.gov/news/stories/2011/november/malware_110911/DNS-changer-malware.pdf

Flamer malware

Mila already shared in her malware repositories several samples of the infamous "Flame" malware

Get it here and start analyzing!

Ulasan Buku "Practical Malware Analysis"

Saya sedang membaca buku "Practical Malware Analysis" . Buku ini diterbitkan oleh penerbit buku "geek" yang terkenal, "NoStarchPress" . Penyampaian buku ini adalah teknikal dan bersahaja. Kedua-dua pengarang bekerja di Mandiant, sebuah syarikat keselamatan komputer.

Sekiranya anda berminat dengan bidang analisis malware, saya menyarankan anda miliki buku ini sebagai pelaburan.

Buku ini bagi saya memerlukan latar belakang "intermediate" memandangkan anda perlu tahu asas sistem operasi, network dan Windows. Ia memfokuskan kepada "windows binaries" dan sekiranya anda berminat dengan mobile malware, ada buku lain yang lebih sesuai.  Fokus analisis dimulakan dengan asas analisis statik dan dinamik, kemudian ke peringkat statik dan dinamik lanjutan.

Rating saya, 4.5/5

Kurang 0.5 sebab tiada DVD disertakan untuk hands-on lab pada setiap akhir bab.

"Data Mining Tools for Malware Detection" book

I recently bought this fresh, new book on the application of Data Mining for malware detection.

I bought it on-line from UK as it is not available yet in Malaysia. Perhaps you can try the following method to get the book:

Amazon
CRC Press

My talks at the recent HITB 2011 Kuala Lumpur

Abstract:

Malware infects a host with exploiting known and unknown vulnerabilities. Among the list of current detections methods, one of them is by using malware signatures in which it is considered fast and works with acceptable computing overhead. The signature generation begins with analysis done by the malware analysts. Since most antiviruses are commercial, least known to the public the method being done for large scale malware signature detection.

One method of detecting malware is by using dynamic analysis, where the behaviour of malware being monitored. The other one is by using static analysis, where it involves the process of diassembling the binary. Here I will share how we can use Python, a powerful interpreted language to do a malicious call analysis.

Drop page: here

Slides: here

Nepenthes on FreeBSD

I just installed FreeBSD 8.2 on my old laptop, since I can't find a buyer yet (it's cheap by the way, only RM500!, COD anywhere if you wanna buy from me). Basically I know that FreeBSD sockets work differently unlike Linux, so some of the works done on Dionaea honeypot can't be ported directly to FreeBSD. 

So now I use Nepenthes port on FreeBSD, the installation wasn't difficult if you're using prepackage pkg, just 

pkg_add -rv nepenthes

or go to /usr/ports and make search name=nepenthes for you to later run make install clean

In FreeBSD however, the Nepenthes will not work directly, since it cannot find /var directory within the /usr/local

So basically, you can simply

mkdir -p /usr/local/var/binaries

mkdir -p /usr/local/var/hexdumps

since in the etc/nepenthes.conf the configuration was stated as such.

For me, since I want to send my malware collection info to mwcollect Alliance, what I have to is to add the following line:

  "submitmwserv.so",               "submit-mwserv.conf",       ""

And your nepenthes should work. 

However this tutorial isn't complete by itself, unlucky for you since I am lazy.

Metasploit for shellcode generation & Libemu for offset locator

Hi,

I found Soritov's blog and decided to follow his write up

Corpus

Membicarakan tentang corpus, anda mungkin akan terbayang istilah perundang-undangan, habeas corpus. Dalam tulisan saya kali ini bukan itu yang dimaksudkan, tetapi adalah suatu kerangka ataupun framework untuk sesuatu yang kita ingin kaji.

Contohnya, bila disebut phishing corpus, ia lebih kepada kerangka bagaimana emel phishing itu secara umum. Kebiasaannya kita akan menggunakan istilah ini untuk mengkaji bagaimanakah emel phishing itu, dan bagaimana membezakan:

1. Emel Phishing dengan sah
2. Emel Phishing daripada emel spam

Merujuk kepada isu malware, kita perlu bina satu corpus untuk mengenali malware itu. Apakah yang membezakan malware dengan bukan malware. 

Sesetengah kertas kajian menggunakan istilah taksonomi(taxonomy). Bagi saya taksonomi ialah satu langkah ke arah membina corpus.

Atau mungkin sebaliknya, vice versa.

Malware Stuffs

Today I read a paper(thesis) on WildCat, malware stuffs, a thesis which contributes on the dynamic analysis of malware.

Also, in my hand, personally I contacted the author of another thesis which was recently defended, entitled Reform. This thesis contributes on the packer technology for Win 32 binaries. Thorough analysis done, among them is by using statistical analysis.