Online Malware & URL Analysis
malsub is a Python 3.6.x framework that wraps several web services of online malware and URL analysis sites through their RESTful Application Programming Interfaces (APIs). It supports submitting files or URLs for analysis, retrieving reports by hash values, domains, IPv4 addresses or URLs, downloading samples and other files, making generic searches and getting API quota values. The framework is designed in a modular way so that new services can be added with ease by following the provided template module and functions to make HTTP GET and POST requests and to pretty print results. This approach avoids having to write individual and specialized wrappers for each and every API by leveraging what they have in common in their calls and responses. The framework is also multi-threaded and dispatches service API functions across a thread pool for each input argument, meaning that it spawns a pool of threads per each file provided for submission or per each hash value provided for report retrieval, for example.
The following publicly available services are currently included in malsub:
AVCaesar;
Hybrid Analysis;
MalShare;
maltracker;
malwr;
Metadefender;
OpenPhish;
PDF Examiner;
PhishTank;
QuickSand;
Safe Browsing;
Threat Crowd;
URLVoid;
VirusTotal.
Most of these services require API keys that are generated after registering an account in their respective websites, which need to be specified in the apikey.yaml file according to the given structure. Note that some of the already bundled services are limited in supported operations due to the fact that they were developed with free API keys. API keys associated with paid subscriptions are allowed to make additional calls not open to the public and may not be restricted by a given quota. Yet, malsub can process multiple input arguments and pause between requests as a workaround for cooldown periods.
The main goal of malsub is to serve as a one-stop-shop for querying multiple online services of malware analysis and for aiding investigators. It is thus suitable for incident response, forensic and malware analysts, as well as for security practitioners alike.
Dependencies and Usage
malsub requires a few moduels that are specified in requirements.txt. The framework is structured into a package and sub packages. Its folder structure and some key files to be taken into consideration when using it or developing additional service models are described as follows:
malsub/malsub.py: application entry point;
malsub/data: miscellaneous data folder;
apikey.yaml: YAML data file of the API key and username pairs;
malsub/downl: downloads folder of files and samples;
malsub/malsub/: malsub package;
malsub/malsub/common/: modules that have a common use all throughout;
out.py: module with output displaying functions according to specific formats and log level (debug, verbose, informational or error);
frmt.py: module with pretty display functions like dictionary to JSON and tabular formats;
rw.py: module with read and write functions;
malsub/malsub/core/: core modules of the application;
web.py: module responsible for handling HTTP requests;
malsub/malsub/service/: services developed as modules parsed during runtime;
base.py: base template module for service construction.
The supported options are the following:
Usage: malsub [-h] [-a <service>] (-d | -f | -q | -r | -s | -t) [-i | -o | -l | -u] [-p <seconds>] [-v ...] [<input> ...]
Interact with online malware and phishing analysis services for malware samples, domain names, IP addresses or URLs.
Options:
-h, --help show this help message and exit
-a, --analysis <service> character-separated list of services (class or short names) [default: all]
-p, --pause <seconds> wait an interval between service requests (rate limit) [default: 0]
-v, --verbose display verbose and debug messages
API functions:
-d, --download download files or malware samples
-f, --find search for arbitrary terms (input format irrelevant)
-q, --quota retrieve API user quota
-r, --report retrieve submission reports for domains, hash values, IP addresses or URLs
-s, --submit submit malware samples or URLs for analysis
-t, --test test API calls by calling each service function as defined with some default values
Input formats (hash values or files are given as default depending on options):
-i, --ipaddr input are IPv4 addresses (applies to '-r' only)
-o, --domain input are domain names (applies to '-r' only)
-l, --appl input are hash values for application lookups (applies to '-r' only)
-u, --url input are URLs (applies to '-r' and '-s' only)
Supported hash values: MD5, SHA1, SHA-256 and SHA-512.
Service Modules
Modules of services are developed as subclasses of the Service class in malsub/service/base.py. Service is an abstract class that lays out the attributes and functions that must be implemented by subclasses to ensure that service modules have all that is necessary for the main application. The full list of supported API functions is the following:
download_file: download a file or a sample matching a given hash value;
report_file: retrieve an analysis report for a file submission identified by its hash value;
submit_file: submit a file for analysis;
report_app: retrieve a report for a known application given a hash value;
report_dom: retrieve a report for a domain name;
report_ip: retrieve a report for an IPv4 address;
report_url: retrieve a report for a URL;
submit_url: submit a URL for analysis;
search: perform searches of arbitrary terms;
quota: query user quota data.
All the above listed API functions have specific signatures in respect to arguments that must be respected. The framework works with simple custom File and Hash classes to respectively represent files and hash values by defining a few file attributes like name and size and hash type. Some of the API functions receive such File and Hash objects as arguments, which must be accessed by their attributes in order to build request parameters.
Download
