Release Notes

0.6.2 (2016-05-11)

Bugfixes:

• The n6sdk_api_test script now performs a cleanup after using the pkg_resources stuff (when executed with the --generate-config option).
• A minor template fix in the n6sdk Pyramid scaffold.

Documentation-related changes:

• A lot of documentation fixes and improvements.

0.6.1 (2015-10-21)

Documentation-related changes:

• The 0.6.0 release made the contents of the examples/BasicExample directory out-of-date. Now, the directory has been completely removed and relevant documentation parts have been adjusted.

  Note: to generate an n6sdk-based project template (richly commented) you can use the pcreate -s n6sdk YourProjectName command (within the virtualenv in which n6sdk has been installed). See also: the Tutorial.

• Several documentation fixes and improvements (including fixes and rearrangements in these release notes).

0.6.0 (2015-10-13)

Significant or backward incompatible changes:

• Documentation: some security-related fixes in the Installation for production... and Gluing it together sections of the Tutorial – especially, related to necessary access restrictions on production.ini file.

• A new utility script added: n6sdk_api_test – a tool to perform basic validation of your n6sdk-based API. (The script is automatically installed in the appropriate place when you install n6sdk.)

  See: the n6sdk_api_test: API testing tool chapter of the documentation.

• A new Pyramid scaffold added that makes it easy to create a skeleton of a new n6sdk-based REST API project – just run the shell command: pcreate -s n6sdk YourProjectName (within the virtualenv in which n6sdk has been installed).

  See: the updated Tutorial.

• The n6sdk.data_spec.DataSpec data specification class has been deeply changed as well as a new data specification class has been added: n6sdk.data_spec.AllSearchableDataSpec (being a subclass of DataSpec).

  Now, all field specifications defined within DataSpec have the flag in_params set to None, that is, are not marked as query parameters.

  It means that you need to create a subclass of DataSpec and, in that subclass, extend desired field specifications to enable them explicitly as query parameters (e.g., by using Ext(in_params='optional')).

  Alternatively, you can create a subclass of AllSearchableDataSpec (which is a DataSpec subclass similar to the previous version of DataSpec, i.e., with most of fields marked as possible query parameters, using in_params='optional') – then you may want to extend some field specifications (by using Ext(in_params=None)) to disable them as query parameters.

  See: the updated Tutorial.

• Now, the field specification DataSpec.url_pattern requires that url_pattern values (concerning query parameter values as well as result values), if present, are not empty.

  To implement this restriction, a new boolean flag has been added to n6sdk.data_spec.fields.UnicodeField (and all its subclasses): disallow_empty, settable as a constructor argument or a subclass attribute. If the flag is true, values are not allowed to be empty. The default value of the flag is False.

• A fix that affects n6sdk.data_spec.DataSpec.fqdn, n6sdk.data_spec.fields.DomainNameField as well as n6sdk.regexes.DOMAIN_ASCII_LOWERCASE_REGEX and n6sdk.regexes.DOMAIN_ASCII_LOWERCASE_STRICT_REGEX: now, the top-level part of a domain name (TLD) is no longer allowed to consist of digits only (thanks to that, the possibility to erroneously accept an IPv4 address as a domain name has been suppressed).

• A new incident category added to DataSpec.category.enum_values: 'malware-action'.

• A new field specification added to DataSpec: action, intended to be used for incidents whose category is 'malware-action' (mentioned above).

• Exception handling has been revamped. Among others, now content-type of HTTP error pages is set to text/plain (not to text/html).

  A few significant changes related to that novelties have been applied to the following n6sdk.pyramid_commons classes: DefaultStreamViewBase, HttpResource and ConfigHelper (please analyze their code if you need detailed information); most notably:

  • the DefaultStreamViewBase.concrete_view_class() class method takes the fifth obligatory argument (and obligatory subclass attribute): adjust_exc (see the documentation of the method);
  • the HttpResource.configure_views() method (which calls DefaultStreamViewBase.concrete_view_class()) takes the second obligatory argument: adjust_exc;
  • the ConfigHelper.complete() method (called, if needed, by ConfigHelper.make_wsgi_app()) registers appropriate Pyramid exception views as well as specifies appropriate callable as the adjust_exc argument for HttpResource.configure_views() (if you are interested in details, please, see the ConfigHelper source code, especially the code of complete() and of two new class methods: exception_view() and exc_to_http_exc()).

• Now, all ConfigHelper constructor arguments are officially required to be specified as keyword arguments (not as positional ones).

• The pyramid library (an existing external dependency) is now restricted to be not newer than version 1.5.7.

Other changes:

• New external dependencies added: python-cjson and requests (used by the n6sdk_api_test tool mentioned above).
• A bugfix: now, log messages from the n6sdk.pyramid_commons module are emitted using the 'n6sdk.pyramid_commons' logger rather than the root logger.
• A new public helper function added: n6sdk.pyramid_commons.renderers.data_dict_to_json(); it defines how the standard renderers json and sjson serialize each data record (for details, see the documentation of the function);
• Various minor code cleanups, refactorizations and improvements.
• New and improved unit tests and doctests.
• A lot of documentation improvements and fixes.

0.5.0 (2015-04-18)

Significant or backward incompatible changes:

• Now, multiple values for a client query parameter can be specified in URL query strings in two alternative ways:

  • separated with commas, within one query string item (as in past n6sdk versions), e.g.: category=bots,dos-attacker,phish;
  • as individual query string items (the way introduced in this n6sdk release), e.g.: category=bots&category=dos-attacker&category=phish.

  Implementation of the extension caused the following changes in the n6sdk programming interfaces:

  • now, the argument for <data specification>.clean_param_dict() is a dictionary that maps query parameter names to lists of individual uncleaned parameter values (in past n6sdk versions it mapped to strings consisting of comma-separated uncleaned parameter values);
  • extraction of individual query parameter values from the URL’s query string – including splitting comma-separated sequences of values – is now entirely outside of the data specification machinery and field classes; the n6sdk.data_spec.fields.Field._split_raw_param_value() non-public method has been removed.
  • the interface of the n6sdk.exceptions.ParamValueCleaningError constructor has been extended a bit: now the second item of a 3-tuple being an item of an error_info_seq argument can be either a single value (as previously) or a list of values.

  The Tutorial and other parts of the documentation have been adjusted appropriately.

• A lot of changes related to data specification fields:

  • New field classes in the n6sdk.data_spec.fields module:
    • IPv6Field (for IPv6 addresses),
    • IPv6NetField (for IPv6 network specifications),
    • EmailSimplifiedField (for e-mail addresses),
    • IBANSimplifiedField (for International Bank Account Numbers),
    • ListOfDictsField (for lists of dictionaries containing arbitrary data),
    • DirField (two-value enumeration: 'src' or 'dst'),
    • ExtendedAddressField (for lists of address data items – see the change in the address field specification, mentioned below).
  • Modified field classes in the n6sdk.data_spec.fields module:
    • DictResultField:
      • the key_to_subfield_factory attribute is no longer obligatory;
      • the required_keys attribute is gone;
      • the clean_param_value() method now raises TypeError instead of NotImplementedError;
    • AddressField:
      • now inherits from ListOfDictsField, not directly from ResultListFieldMixin and DictResultField;
      • the required_keys attribute is gone; ip subfield is still obligatory – but now this requirement is implemented internally;
      • the clean_param_value() method now raises TypeError instead of NotImplementedError.
  • New field specifications added to the n6sdk.data_spec.DataSpec class:
    • time.until (DateTimeField, params-only),
    • active.until (DateTimeField, params-only),
    • modified (DateTimeField, results-only),
    • modified.min (DateTimeField, params-only),
    • modified.max (DateTimeField, params-only),
    • modified.until (DateTimeField, params-only),
    • ipv6 (IPv6Field, params-only),
    • ipv6.net (IPv6NetField, params-only),
    • injects (ListOfDictsField, results-only),
    • registrar (UnicodeLimitedField),
    • url_pattern (UnicodeLimitedField),
    • username (UnicodeLimitedField),
    • x509fp_sha1 (SHA1Field),
    • email (EmailSimplifiedField),
    • iban (IBANSimplifiedField),
    • phone (UnicodeLimitedField).
  • The address field specification (at n6sdk.data_spec.DataSpec) has been changed: now it is an ExtendedAddressField instance – its subfields include:
    • ip/ipv6 (IPv4Field/IPv6Field, obligatory – which means that either 'ip' or `ipv6', but not both, must be present in each member dictionary),
    • cc (CCField),
    • asn (ASNField),
    • dir (DirField),
    • rdns (DomainNameField).
  • New categories added to DataSpec.category.enum_values:
    • 'amplifier',
    • 'backdoor',
    • 'dns-query',
    • 'flow',
    • 'flow-anomaly',
    • 'fraud',
    • 'leak',
    • 'vulnerable',
    • 'webinject'.

  The Tutorial has been adjusted appropriately.

• Both standard renderers (json and sjson) now add the "Z" suffix (indicating the UTC time) to all date+time values.

• The sjson renderer now generates an additional empty line to indicate the end of data stream.

Other changes:

• A new external dependency: the ipaddr library.
• New and improved unit tests and doctests.
• Several documentation improvements and fixes.

0.4.0 (2014-12-23)

This is the first public, free/open-source-licensed release of n6sdk.

Backward incompatible (though rather minor) changes:

• Changed behaviour of the standard json and sjson renderers (defined in n6sdk.pyramid_commons.renderers as the StreamRenderer_json and StreamRenderer_sjson classes): now they make use of a new helper function, dict_with_nulls_removed(), that replaces the old mechanism of recursive removing of None-or-empty values from result dictionaries: previously, values equal to zero (such as 0, 0.0 or False) were also removed; now they are kept (note that values being None, empty containers and empty strings are still removed).
• Now, in the n6sdk.pyramid_commons.DefaultStreamViewBase.call_api() method, an n6sdk.exceptions.TooMuchDataError exception from call_api_method() or from data specification’s clean_result_dict() causes pyramid.httpexceptions.HTTPForbidden and not pyramid.httpexceptions.HTTPServerError.
• The n6sdk.class_helpers.singleton() class decorator is now more lenient: instantiation does not count if __init__() of a decorated class raised (or propagated) an exception.

Other changes:

• Bugfix in the n6sdk.pyramid_commons.DefaultStreamViewBase.concrete_view_class() class method: now the check of the given renderer labels against the set of registered renderers works properly; previously it behaved nonsensically: accepted unregistered labels (causing further KeyError exceptions) and at the same time demanded that all registeted labels had to be used.
• Furthermore, n6sdk.pyramid_commons.DefaultStreamViewBase has a new class attribute: break_on_result_cleaning_error, by default set to True. In custom subclasses it can be set to False – then result dictionaries that cannot be cleaned will be skipped (and a proper warning will be recorded to the logs) instead of causing pyramid.httpexceptions.HTTPServerError.
• The n6sdk.pyramid_commons.renderers.dict_with_nulls_removed() function (mentioned above) is exposed as a public helper (it may be useful when implementing custom renderers).
• The n6sdk.data_spec.fields.Field class (and its subclasses) as well as n6sdk.datetime_helpers.FixedOffsetTimezone – have custom implementations of the __repr__() method (producing more readable results).
• Various minor code cleanups, refactorizations and improvements.
• New and improved unit tests and doctests.

Documentation-related news (including big ones!):

• Now the documentation is generated with Sphinx.
• A new, long Tutorial has been added.
• A bunch of docstrings have been added.
• Contents of many docstrings have been improved.
• All docstrings are now reStructuredText-formatted and used as a part of the Sphinx-generated documentation.
• The former CHANGES.txt file has been reStructuredText-formatted, renamed to NEWS.rst and used as a part of the Sphinx-generated documentation. There is also a new README.rst file, also included in the generated documentation.
• The former README.txt file has been moved to examples/BasicExample and sligthly improved.
• Furthermore, some other BasicExample improvements have been made (cleanups, refactorizations and minor fixes; among others, the version field in the BasicExample‘s setup.py file no longer follows the n6sdk version; from now it is just "0.0.1").

0.3.0 (2014-08-12)

Significant or backward incompatible changes:

• Network incident category "ddos" has been replaced with two separate categories: "dos-attacker" and "dos-victim" (see: n6sdk.data_spec.CATEGORY_ENUMS).
• n6sdk.data_spec.fields.ResultListFieldMixin.clean_result_value() no longer accepts collections.Set instances (now it accepts only collection.Sequence instances that are not str/unicode instances).

0.2.0 (2014-08-08)

Significant or backward incompatible changes:

• Changes in the base data specification class (n6sdk.data_spec.DataSpec) and/or in the classes defined in the n6sdk.data_spec.fields module:

  • the source field is now an instance of a new class: n6sdk.data_spec.fields.SourceField – which implements more restricted validation of values; now each value not only needs to be at most 32-characters long, but also it must consist of two non-empty parts, separated with exactly one dot character ('.'), containing only lowercase ASCII letters, digits and hyphens ('-').

  • a change in n6sdk.data_spec.fields.DateTimeField that affects the time, expires and until fields of DataSpec: the clean_result_value() method now accepts also ISO-formatted date-and-time strings (not only datetime.datetime instances);

  • a change in n6sdk.data_spec.fields.IntegerField that affects the sport, dport and count fields of DataSpec: in clean_result_value(), the former strict is-instance check (int/long) has been replaced with a duck-typed coercion, accepting anything that can be converted using int() without information loss (e.g. a float being an integer number, such as 42.0, or a string being a decimal representation of an integer number, such as '42' – but not '42.0');

  • a change in n6sdk.data_spec.fields.ASNField that affects the address (namely: asn of its subitems) and asn fields of DataSpec: the clean_*_value() methods now accept strings (str/unicode):

    • either being a decimal representation of an integer number in range 0..``2**32-1``, e.g. '98765432' (formely only clean_param_value() accepted such strings),
    • or consisting of two dot-separated decimal representations of integer numbers in range 0..``2**16-1``, e.g. '34567.65432' (formely such a notation was not accepted at all);

    note: clean_result_value() still accepts also int and long values in range 0..``2**32-1`` (and still does not accept instances of float and other types).

  • a change in n6sdk.data_spec.fields.CCField that affects the address (namely: cc of its subitems) and cc fields of DataSpec: the clean_*_value() methods now accept also lowercase letters (which are automatically uppercased);

  • a change in n6sdk.data_spec.fields.DomainNameSubstringField that affects the fqdn (note: DomainNameField is a subclass of DomainNameSubstringField) and fqdn.sub fields of DataSpec: the value of max_length has been changed from 253 to 255;

  • a change in n6sdk.data_spec.fields.DomainNameField that affects the fqdn field of DataSpec: the regular expression the values are matched against is now more liberal (especially, underscores are now allowed; rationale: real-life domain names – especially those maliciously constructed – are not necessarily RFC-compliant; see: n6sdk.regexes.DOMAIN_ASCII_LOWERCASE_REGEX for details);

  • a change in n6sdk.data_spec.fields.AnonymizedIPv4Field that affects the adip field of DataSpec: the clean_*_value() methods now accept also 'X' (uppercased 'x') segments which are automatically lowercased;

  • the adip field is no longer enabled as a query parameter (field’s in_params is now set to None);

  • a change in n6sdk.data_spec.fields.HexDigestField that affects the md5 and sha1 fields of DataSpec: the clean_*_value() methods now accept also non-lowercase hexadecimal digit letters (which are automatically lowercased);

  • the former hash_algo attribute of UnicodeField class/subclasses/instances has been renamed to hash_algo_descr;

  • n6sdk.data_spec.fields.URLField is now a subclass of n6sdk.data_spec.fields.URLSubstringField;

  • n6sdk.data_spec.fields.ListField has been removed (use ResultListFieldMixin instead);

  • the former n6sdk.data_spec.fields.AddressField implementation has been replaced with a new one, especially the implementation of the methods has been factored out to new generic base classes: ResultListFieldMixin and DictResultField; some details have changed in a backwards-incompatible way – notably: key_to_subfield_class has been renamed to key_to_subfield_factory.

• Changes in signatures of the n6sdk.data_spec.BaseDataSpec methods: clean_param_dict(), clean_param_keys(), clean_result_dict(), clean_result_keys():

  • replaced the optional argument keys_to_ignore with the ignored_keys keyword-only argument (still optional),
  • added other optional arguments: forbidden_keys, extra_required_keys, discarded_keys.

• Changes in n6sdk.pyramid_commons:

  • functions init_pyramid_config() and complete_pyramid_config() have been removed; use the new ConfigHelper class instead (for details – see its documentation, its code and the examples in examples/BasicExample...);

  • a new function added: register_stream_renderer() (see below);

  • the signature of the StreamResponse class constructor changed: renderer has been renamed to renderer_name; also, now the value of that argument can be any name registered with the new function register_stream_renderer() (see its documentation for details); 'json' and 'sjson' are registered out-of-the-box;

  • the DefaultStreamViewBase class has been revamped in a backward-incompatibile way (please analyze its code if you need detailed information); most notably:

    • now the concrete_view_class() class method has completely different signature (see its documentation for details; note that data_spec now must be an instance, not a class); now each concrete subclass must have the resource_id, renderers, data_spec and data_backend_api_method attributes specified (for more information, also see the documentation of the concrete_view_class() class method mentioned above);
    • formely, the data specification’s clean_param_dict() call performed in prepare_params() was guarded only against ParamCleaningError (transformed into pyramid.httpexceptions.HTTPBadRequest, when caught); now, also other exceptions are handled: n6sdk.exceptions.AuthorizationError (transformed into pyramid.httpexceptions.HTTPForbidden) and generic n6sdk.exceptions.DataAPIError (logged as an error and transformed into pyramid.httpexceptions.HTTPServerError) [note the symmetry between the prepare_params() and call_api() methods];
    • the possibility of specifying keyword arguments for data specification’s clean_*_dict() calls as well as for data backend API’s method call has been added (see the get_clean_param_dict_kwargs(), get_clean_result_dict_kwargs() and get_extra_api_kwargs() hook methods; the default implementation of each of them returns just an empty dict);

  • backward-incompatibile chages in the signature of the constructor of the HttpResource class:

    • now all arguments should be specified as keyword ones (never positional, i.e. you cannot rely on argument order any more);
    • now data_spec must be an instance, not a class;

    note: see the documentation of this class for details.

• The module n6sdk.data_backend_api (together with the decorator n6sdk.data_backend_api.data_backend_api_method) has been removed. It is no longer required to decorate or mark your custom data backend API class or its methods in any special way.

• Unused n6sdk.exceptions.InvalidCallError has been removed.

• n6sdk.exceptions.FieldValueTooLongError has been added (see below).

Other changes:

• Appropriate adjustments in examples/BasicExample.
• Some non-essential changes related to n6sdk.data_spec.fields:
  • if the given value is too long, the clean_*_value() methods of n6sdk.data_spec.fields.UnicodeLimitedField (and of its subclasses) now raise a new exception n6sdk.exceptions.FieldValueTooLongError (which is a subclass of n6sdk.exceptions.FieldValueError that was formely raised) – see its documentation for details about attributes of its instances (that attributes can be useful, for example, when implementing external trimming of too long values...);
  • it is now explicitly required for n6sdk.data_spec.fields.HexDigestField instances (and for instances of its subclasses) that num_of_characters and hash_algo_descr are specified (as subclass attributes or constructor arguments);
  • it is now explicitly required for n6sdk.data_spec.fields.UnicodeLimitedField instances (and for instances of its subclasses) that max_length is not less than 1.
• Module n6sdk.addr_helpers added.
• Major refactorings and several minor additions, improvements, fixes and cleanups.
• Improvements in the documentation (a lot of improved/added docstrings, improved README.txt, added CHANGES.txt...) and code comments.
• MANIFEST.in and other package setup improvements and cleanups.
• New and improved unit tests and doctests.

0.0.1 (2014-04-25)

Initial release.