Release Notes¶

0.6.2 (2016-05-11)¶

Bugfixes:

The n6sdk_api_test script now performs a cleanup after using the pkg_resources stuff (when executed with the --generate-config option).
A minor template fix in the n6sdk Pyramid scaffold.

Documentation-related changes:

A lot of documentation fixes and improvements.

0.6.1 (2015-10-21)¶

Documentation-related changes:

The 0.6.0 release made the contents of the examples/BasicExample directory out-of-date. Now, the directory has been completely removed and relevant documentation parts have been adjusted.

Note: to generate an n6sdk-based project template (richly commented) you can use the pcreate -s n6sdk YourProjectName command (within the virtualenv in which n6sdk has been installed). See also: the Tutorial.
Several documentation fixes and improvements (including fixes and rearrangements in these release notes).

0.6.0 (2015-10-13)¶

Significant or backward incompatible changes:

Documentation: some security-related fixes in the Installation for production... and Gluing it together sections of the Tutorial – especially, related to necessary access restrictions on production.ini file.
A new utility script added: n6sdk_api_test – a tool to perform basic validation of your n6sdk-based API. (The script is automatically installed in the appropriate place when you install n6sdk.)

See: the n6sdk_api_test: API testing tool chapter of the documentation.
A new Pyramid scaffold added that makes it easy to create a skeleton of a new n6sdk-based REST API project – just run the shell command: pcreate -s n6sdk YourProjectName (within the virtualenv in which n6sdk has been installed).

See: the updated Tutorial.
The n6sdk.data_spec.DataSpec data specification class has been deeply changed as well as a new data specification class has been added: n6sdk.data_spec.AllSearchableDataSpec (being a subclass of DataSpec).

Now, all field specifications defined within DataSpec have the flag in_params set to None, that is, are not marked as query parameters.

It means that you need to create a subclass of DataSpec and, in that subclass, extend desired field specifications to enable them explicitly as query parameters (e.g., by using Ext(in_params='optional')).

Alternatively, you can create a subclass of AllSearchableDataSpec (which is a DataSpec subclass similar to the previous version of DataSpec, i.e., with most of fields marked as possible query parameters, using in_params='optional') – then you may want to extend some field specifications (by using Ext(in_params=None)) to disable them as query parameters.

See: the updated Tutorial.
Now, the field specification DataSpec.url_pattern requires that url_pattern values (concerning query parameter values as well as result values), if present, are not empty.

To implement this restriction, a new boolean flag has been added to n6sdk.data_spec.fields.UnicodeField (and all its subclasses): disallow_empty, settable as a constructor argument or a subclass attribute. If the flag is true, values are not allowed to be empty. The default value of the flag is False.
A fix that affects n6sdk.data_spec.DataSpec.fqdn, n6sdk.data_spec.fields.DomainNameField as well as n6sdk.regexes.DOMAIN_ASCII_LOWERCASE_REGEX and n6sdk.regexes.DOMAIN_ASCII_LOWERCASE_STRICT_REGEX: now, the top-level part of a domain name (TLD) is no longer allowed to consist of digits only (thanks to that, the possibility to erroneously accept an IPv4 address as a domain name has been suppressed).
A new incident category added to DataSpec.category.enum_values: 'malware-action'.
A new field specification added to DataSpec: action, intended to be used for incidents whose category is 'malware-action' (mentioned above).
Exception handling has been revamped. Among others, now content-type of HTTP error pages is set to text/plain (not to text/html).

A few significant changes related to that novelties have been applied to the following n6sdk.pyramid_commons classes: DefaultStreamViewBase, HttpResource and ConfigHelper (please analyze their code if you need detailed information); most notably:
- the DefaultStreamViewBase.concrete_view_class() class method takes the fifth obligatory argument (and obligatory subclass attribute): adjust_exc (see the documentation of the method);
- the HttpResource.configure_views() method (which calls DefaultStreamViewBase.concrete_view_class()) takes the second obligatory argument: adjust_exc;
- the ConfigHelper.complete() method (called, if needed, by ConfigHelper.make_wsgi_app()) registers appropriate Pyramid exception views as well as specifies appropriate callable as the adjust_exc argument for HttpResource.configure_views() (if you are interested in details, please, see the ConfigHelper source code, especially the code of complete() and of two new class methods: exception_view() and exc_to_http_exc()).
Now, all ConfigHelper constructor arguments are officially required to be specified as keyword arguments (not as positional ones).
The pyramid library (an existing external dependency) is now restricted to be not newer than version 1.5.7.

Other changes:

New external dependencies added: python-cjson and requests (used by the n6sdk_api_test tool mentioned above).
A bugfix: now, log messages from the n6sdk.pyramid_commons module are emitted using the 'n6sdk.pyramid_commons' logger rather than the root logger.
A new public helper function added: n6sdk.pyramid_commons.renderers.data_dict_to_json(); it defines how the standard renderers json and sjson serialize each data record (for details, see the documentation of the function);
Various minor code cleanups, refactorizations and improvements.
New and improved unit tests and doctests.
A lot of documentation improvements and fixes.

0.5.0 (2015-04-18)¶

Significant or backward incompatible changes:

Now, multiple values for a client query parameter can be specified in URL query strings in two alternative ways:
- separated with commas, within one query string item (as in past n6sdk versions), e.g.: category=bots,dos-attacker,phish;
- as individual query string items (the way introduced in this n6sdk release), e.g.: category=bots&category=dos-attacker&category=phish.
Implementation of the extension caused the following changes in the n6sdk programming interfaces:
- now, the argument for <data specification>.clean_param_dict() is a dictionary that maps query parameter names to lists of individual uncleaned parameter values (in past n6sdk versions it mapped to strings consisting of comma-separated uncleaned parameter values);
- extraction of individual query parameter values from the URL’s query string – including splitting comma-separated sequences of values – is now entirely outside of the data specification machinery and field classes; the n6sdk.data_spec.fields.Field._split_raw_param_value() non-public method has been removed.
- the interface of the n6sdk.exceptions.ParamValueCleaningError constructor has been extended a bit: now the second item of a 3-tuple being an item of an error_info_seq argument can be either a single value (as previously) or a list of values.
The Tutorial and other parts of the documentation have been adjusted appropriately.
A lot of changes related to data specification fields:
- New field classes in the n6sdk.data_spec.fields module:
  - IPv6Field (for IPv6 addresses),
  - IPv6NetField (for IPv6 network specifications),
  - EmailSimplifiedField (for e-mail addresses),
  - IBANSimplifiedField (for International Bank Account Numbers),
  - ListOfDictsField (for lists of dictionaries containing arbitrary data),
  - DirField (two-value enumeration: 'src' or 'dst'),
  - ExtendedAddressField (for lists of address data items – see the change in the address field specification, mentioned below).
- Modified field classes in the n6sdk.data_spec.fields module:
  - DictResultField:
    - the key_to_subfield_factory attribute is no longer obligatory;
    - the required_keys attribute is gone;
    - the clean_param_value() method now raises TypeError instead of NotImplementedError;
  - AddressField:
    - now inherits from ListOfDictsField, not directly from ResultListFieldMixin and DictResultField;
    - the required_keys attribute is gone; ip subfield is still obligatory – but now this requirement is implemented internally;
    - the clean_param_value() method now raises TypeError instead of NotImplementedError.
- New field specifications added to the n6sdk.data_spec.DataSpec class:
  - time.until (DateTimeField, params-only),
  - active.until (DateTimeField, params-only),
  - modified (DateTimeField, results-only),
  - modified.min (DateTimeField, params-only),
  - modified.max (DateTimeField, params-only),
  - modified.until (DateTimeField, params-only),
  - ipv6 (IPv6Field, params-only),
  - ipv6.net (IPv6NetField, params-only),
  - injects (ListOfDictsField, results-only),
  - registrar (UnicodeLimitedField),
  - url_pattern (UnicodeLimitedField),
  - username (UnicodeLimitedField),
  - x509fp_sha1 (SHA1Field),
  - email (EmailSimplifiedField),
  - iban (IBANSimplifiedField),
  - phone (UnicodeLimitedField).
- The address field specification (at n6sdk.data_spec.DataSpec) has been changed: now it is an ExtendedAddressField instance – its subfields include:
  - ip/ipv6 (IPv4Field/IPv6Field, obligatory – which means that either 'ip' or `ipv6', but not both, must be present in each member dictionary),
  - cc (CCField),
  - asn (ASNField),
  - dir (DirField),
  - rdns (DomainNameField).
- New categories added to DataSpec.category.enum_values:
  - 'amplifier',
  - 'backdoor',
  - 'dns-query',
  - 'flow',
  - 'flow-anomaly',
  - 'fraud',
  - 'leak',
  - 'vulnerable',
  - 'webinject'.
The Tutorial has been adjusted appropriately.
Both standard renderers (json and sjson) now add the "Z" suffix (indicating the UTC time) to all date+time values.
The sjson renderer now generates an additional empty line to indicate the end of data stream.

Other changes:

A new external dependency: the ipaddr library.
New and improved unit tests and doctests.
Several documentation improvements and fixes.

0.4.0 (2014-12-23)¶

This is the first public, free/open-source-licensed release of n6sdk.

Backward incompatible (though rather minor) changes:

Changed behaviour of the standard json and sjson renderers (defined in n6sdk.pyramid_commons.renderers as the StreamRenderer_json and StreamRenderer_sjson classes): now they make use of a new helper function, dict_with_nulls_removed(), that replaces the old mechanism of recursive removing of None-or-empty values from result dictionaries: previously, values equal to zero (such as 0, 0.0 or False) were also removed; now they are kept (note that values being None, empty containers and empty strings are still removed).
Now, in the n6sdk.pyramid_commons.DefaultStreamViewBase.call_api() method, an n6sdk.exceptions.TooMuchDataError exception from call_api_method() or from data specification’s clean_result_dict() causes pyramid.httpexceptions.HTTPForbidden and not pyramid.httpexceptions.HTTPServerError.
The n6sdk.class_helpers.singleton() class decorator is now more lenient: instantiation does not count if __init__() of a decorated class raised (or propagated) an exception.

Other changes:

Bugfix in the n6sdk.pyramid_commons.DefaultStreamViewBase.concrete_view_class() class method: now the check of the given renderer labels against the set of registered renderers works properly; previously it behaved nonsensically: accepted unregistered labels (causing further KeyError exceptions) and at the same time demanded that all registeted labels had to be used.
Furthermore, n6sdk.pyramid_commons.DefaultStreamViewBase has a new class attribute: break_on_result_cleaning_error, by default set to True. In custom subclasses it can be set to False – then result dictionaries that cannot be cleaned will be skipped (and a proper warning will be recorded to the logs) instead of causing pyramid.httpexceptions.HTTPServerError.
The n6sdk.pyramid_commons.renderers.dict_with_nulls_removed() function (mentioned above) is exposed as a public helper (it may be useful when implementing custom renderers).
The n6sdk.data_spec.fields.Field class (and its subclasses) as well as n6sdk.datetime_helpers.FixedOffsetTimezone – have custom implementations of the __repr__() method (producing more readable results).
Various minor code cleanups, refactorizations and improvements.
New and improved unit tests and doctests.

Documentation-related news (including big ones!):

Now the documentation is generated with Sphinx.
A new, long Tutorial has been added.
A bunch of docstrings have been added.
Contents of many docstrings have been improved.
All docstrings are now reStructuredText-formatted and used as a part of the Sphinx-generated documentation.
The former CHANGES.txt file has been reStructuredText-formatted, renamed to NEWS.rst and used as a part of the Sphinx-generated documentation. There is also a new README.rst file, also included in the generated documentation.
The former README.txt file has been moved to examples/BasicExample and sligthly improved.
Furthermore, some other BasicExample improvements have been made (cleanups, refactorizations and minor fixes; among others, the version field in the BasicExample‘s setup.py file no longer follows the n6sdk version; from now it is just "0.0.1").

0.3.0 (2014-08-12)¶

Significant or backward incompatible changes:

Network incident category "ddos" has been replaced with two separate categories: "dos-attacker" and "dos-victim" (see: n6sdk.data_spec.CATEGORY_ENUMS).
n6sdk.data_spec.fields.ResultListFieldMixin.clean_result_value() no longer accepts collections.Set instances (now it accepts only collection.Sequence instances that are not str/unicode instances).

0.2.0 (2014-08-08)¶

Significant or backward incompatible changes:

Changes in the base data specification class (n6sdk.data_spec.DataSpec) and/or in the classes defined in the n6sdk.data_spec.fields module:
- the source field is now an instance of a new class: n6sdk.data_spec.fields.SourceField – which implements more restricted validation of values; now each value not only needs to be at most 32-characters long, but also it must consist of two non-empty parts, separated with exactly one dot character ('.'), containing only lowercase ASCII letters, digits and hyphens ('-').
- a change in n6sdk.data_spec.fields.DateTimeField that affects the time, expires and until fields of DataSpec: the clean_result_value() method now accepts also ISO-formatted date-and-time strings (not only datetime.datetime instances);
- a change in n6sdk.data_spec.fields.IntegerField that affects the sport, dport and count fields of DataSpec: in clean_result_value(), the former strict is-instance check (int/long) has been replaced with a duck-typed coercion, accepting anything that can be converted using int() without information loss (e.g. a float being an integer number, such as 42.0, or a string being a decimal representation of an integer number, such as '42' – but not '42.0');
- a change in n6sdk.data_spec.fields.ASNField that affects the address (namely: asn of its subitems) and asn fields of DataSpec: the clean_*_value() methods now accept strings (str/unicode):
  - either being a decimal representation of an integer number in range 0..``2**32-1``, e.g. '98765432' (formely only clean_param_value() accepted such strings),
  - or consisting of two dot-separated decimal representations of integer numbers in range 0..``2**16-1``, e.g. '34567.65432' (formely such a notation was not accepted at all);
  note: clean_result_value() still accepts also int and long values in range 0..``2**32-1`` (and still does not accept instances of float and other types).
- a change in n6sdk.data_spec.fields.CCField that affects the address (namely: cc of its subitems) and cc fields of DataSpec: the clean_*_value() methods now accept also lowercase letters (which are automatically uppercased);
- a change in n6sdk.data_spec.fields.DomainNameSubstringField that affects the fqdn (note: DomainNameField is a subclass of DomainNameSubstringField) and fqdn.sub fields of DataSpec: the value of max_length has been changed from 253 to 255;
- a change in n6sdk.data_spec.fields.DomainNameField that affects the fqdn field of DataSpec: the regular expression the values are matched against is now more liberal (especially, underscores are now allowed; rationale: real-life domain names – especially those maliciously constructed – are not necessarily RFC-compliant; see: n6sdk.regexes.DOMAIN_ASCII_LOWERCASE_REGEX for details);
- a change in n6sdk.data_spec.fields.AnonymizedIPv4Field that affects the adip field of DataSpec: the clean_*_value() methods now accept also 'X' (uppercased 'x') segments which are automatically lowercased;
- the adip field is no longer enabled as a query parameter (field’s in_params is now set to None);
- a change in n6sdk.data_spec.fields.HexDigestField that affects the md5 and sha1 fields of DataSpec: the clean_*_value() methods now accept also non-lowercase hexadecimal digit letters (which are automatically lowercased);
- the former hash_algo attribute of UnicodeField class/subclasses/instances has been renamed to hash_algo_descr;
- n6sdk.data_spec.fields.URLField is now a subclass of n6sdk.data_spec.fields.URLSubstringField;
- n6sdk.data_spec.fields.ListField has been removed (use ResultListFieldMixin instead);
- the former n6sdk.data_spec.fields.AddressField implementation has been replaced with a new one, especially the implementation of the methods has been factored out to new generic base classes: ResultListFieldMixin and DictResultField; some details have changed in a backwards-incompatible way – notably: key_to_subfield_class has been renamed to key_to_subfield_factory.
Changes in signatures of the n6sdk.data_spec.BaseDataSpec methods: clean_param_dict(), clean_param_keys(), clean_result_dict(), clean_result_keys():
- replaced the optional argument keys_to_ignore with the ignored_keys keyword-only argument (still optional),
- added other optional arguments: forbidden_keys, extra_required_keys, discarded_keys.
Changes in n6sdk.pyramid_commons:
- functions init_pyramid_config() and complete_pyramid_config() have been removed; use the new ConfigHelper class instead (for details – see its documentation, its code and the examples in examples/BasicExample...);
- a new function added: register_stream_renderer() (see below);
- the signature of the StreamResponse class constructor changed: renderer has been renamed to renderer_name; also, now the value of that argument can be any name registered with the new function register_stream_renderer() (see its documentation for details); 'json' and 'sjson' are registered out-of-the-box;
- the DefaultStreamViewBase class has been revamped in a backward-incompatibile way (please analyze its code if you need detailed information); most notably:
  - now the concrete_view_class() class method has completely different signature (see its documentation for details; note that data_spec now must be an instance, not a class); now each concrete subclass must have the resource_id, renderers, data_spec and data_backend_api_method attributes specified (for more information, also see the documentation of the concrete_view_class() class method mentioned above);
  - formely, the data specification’s clean_param_dict() call performed in prepare_params() was guarded only against ParamCleaningError (transformed into pyramid.httpexceptions.HTTPBadRequest, when caught); now, also other exceptions are handled: n6sdk.exceptions.AuthorizationError (transformed into pyramid.httpexceptions.HTTPForbidden) and generic n6sdk.exceptions.DataAPIError (logged as an error and transformed into pyramid.httpexceptions.HTTPServerError) [note the symmetry between the prepare_params() and call_api() methods];
  - the possibility of specifying keyword arguments for data specification’s clean_*_dict() calls as well as for data backend API’s method call has been added (see the get_clean_param_dict_kwargs(), get_clean_result_dict_kwargs() and get_extra_api_kwargs() hook methods; the default implementation of each of them returns just an empty dict);
- backward-incompatibile chages in the signature of the constructor of the HttpResource class:
  - now all arguments should be specified as keyword ones (never positional, i.e. you cannot rely on argument order any more);
  - now data_spec must be an instance, not a class;
  note: see the documentation of this class for details.
The module n6sdk.data_backend_api (together with the decorator n6sdk.data_backend_api.data_backend_api_method) has been removed. It is no longer required to decorate or mark your custom data backend API class or its methods in any special way.
Unused n6sdk.exceptions.InvalidCallError has been removed.
n6sdk.exceptions.FieldValueTooLongError has been added (see below).

Other changes:

Appropriate adjustments in examples/BasicExample.
Some non-essential changes related to n6sdk.data_spec.fields:
- if the given value is too long, the clean_*_value() methods of n6sdk.data_spec.fields.UnicodeLimitedField (and of its subclasses) now raise a new exception n6sdk.exceptions.FieldValueTooLongError (which is a subclass of n6sdk.exceptions.FieldValueError that was formely raised) – see its documentation for details about attributes of its instances (that attributes can be useful, for example, when implementing external trimming of too long values...);
- it is now explicitly required for n6sdk.data_spec.fields.HexDigestField instances (and for instances of its subclasses) that num_of_characters and hash_algo_descr are specified (as subclass attributes or constructor arguments);
- it is now explicitly required for n6sdk.data_spec.fields.UnicodeLimitedField instances (and for instances of its subclasses) that max_length is not less than 1.
Module n6sdk.addr_helpers added.
Major refactorings and several minor additions, improvements, fixes and cleanups.
Improvements in the documentation (a lot of improved/added docstrings, improved README.txt, added CHANGES.txt...) and code comments.
MANIFEST.in and other package setup improvements and cleanups.
New and improved unit tests and doctests.

0.0.1 (2014-04-25)¶

Initial release.